API: Add endpoints to trigger data shredders (#2731)

This commit is contained in:
Raphael Michel
2022-07-25 18:34:40 +02:00
committed by GitHub
parent 59edb355f8
commit b06e98ace4
8 changed files with 611 additions and 9 deletions

View File

@@ -182,7 +182,7 @@ endpoints:
Content-Type: application/json
{
"download": "https://pretix.eu/api/v1/organizers/bigevents/events/sampleconf/orderlist/download/29891ede-196f-4942-9e26-d055a36e98b8/3f279f13-c198-4137-b49b-9b360ce9fcce/"
"download": "https://pretix.eu/api/v1/organizers/bigevents/events/sampleconf/exporters/orderlist/download/29891ede-196f-4942-9e26-d055a36e98b8/3f279f13-c198-4137-b49b-9b360ce9fcce/"
}
:param organizer: The ``slug`` field of the organizer to fetch

View File

@@ -38,6 +38,7 @@ at :ref:`plugin-docs`.
webhooks
seatingplans
exporters
shredders
sendmail_rules
billing_invoices
billing_var

View File

@@ -0,0 +1,177 @@
.. spelling:: checkin
Data shredders
==============
pretix and it's plugins include a number of data shredders that allow you to clear personal information from the system.
This page shows you how to use these shredders through the API.
.. versionchanged:: 4.12
This feature has been added to the API.
.. warning::
Unlike the user interface, the API will not force you to download tax-relevant data before you delete it.
Listing available shredders
---------------------------
.. http:get:: /api/v1/organizers/(organizer)/events/(event)/shredders/
Returns a list of all exporters shredders for a given event.
**Example request**:
.. sourcecode:: http
GET /api/v1/organizers/bigevents/events/sampleconf/shredders/ HTTP/1.1
Host: pretix.eu
Accept: application/json, text/javascript
**Example response**:
.. sourcecode:: http
HTTP/1.1 200 OK
Vary: Accept
Content-Type: application/json
{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"identifier": "question_answers",
"verbose_name": "Answers to questions"
}
]
}
:query integer page: The page number in case of a multi-page result set, default is 1
:param organizer: The ``slug`` field of the organizer to fetch
:param event: The ``slug`` field of the event to fetch
:statuscode 200: no error
:statuscode 401: Authentication failure
:statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
Running an export
-----------------
Before you can delete data, you need to start a data export.
Since exports often include large data sets, they might take longer than the duration of an HTTP request. Therefore,
creating an export is a two-step process. First you need to start an export task with one of the following to API
endpoints:
.. http:post:: /api/v1/organizers/(organizer)/events/(event)/shredders/export/
Starts an export task. If your input parameters validate correctly, a ``202 Accepted`` status code is returned.
The body points you to the download URL of the result as well as the URL for the next step.
**Example request**:
.. sourcecode:: http
POST /api/v1/organizers/bigevents/events/sampleconf/shredders/export/ HTTP/1.1
Host: pretix.eu
Accept: application/json, text/javascript
Content-Type: application/json
{
"shredders": ["question_answers"]
}
**Example response**:
.. sourcecode:: http
HTTP/1.1 202 Accepted
Vary: Accept
Content-Type: application/json
{
"download": "https://pretix.eu/api/v1/organizers/bigevents/events/sampleconf/shredders/download/29891ede-196f-4942-9e26-d055a36e98b8/3f279f13-c198-4137-b49b-9b360ce9fcce/",
"shred": "https://pretix.eu/api/v1/organizers/bigevents/events/sampleconf/shredders/shred/29891ede-196f-4942-9e26-d055a36e98b8/3f279f13-c198-4137-b49b-9b360ce9fcce/"
}
:param organizer: The ``slug`` field of the organizer to fetch
:param event: The ``slug`` field of the event to fetch
:param identifier: The ``identifier`` field of the exporter to run
:statuscode 202: no error
:statuscode 400: Invalid input options or event data is not ready to be deleted
:statuscode 401: Authentication failure
:statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
Downloading the result
----------------------
When starting an export, you receive a ``download`` URL for downloading the result. Running a ``GET`` request on that result will
yield one of the following status codes:
* ``200 OK`` The export succeeded. The body will be your resulting file. Might be large!
* ``409 Conflict`` Your export is still running. The body will be JSON with the structure ``{"status": "running"}``. ``status`` can be ``waiting`` before the task is actually being processed. Please retry, but wait at least one second before you do.
* ``410 Gone`` Running the export has failed permanently. The body will be JSON with the structure ``{"status": "failed", "message": "Error message"}``
* ``404 Not Found`` The export does not exist / is expired / belongs to a different API key.
Shredding the data
------------------
When starting an export, you receive a ``shred`` URL for actually shredding the data.
You can only start the actual shredding process after the export file was generated, however you are not forced to download
the file (we'd recommend it in most cases, though).
The download will no longer be possible after the shredding.
Since shredding often requires deleting large data sets, it might take longer than the duration of an HTTP request.
Therefore, shredding again is a two-step process. First you need to start a shredder task with one of the following to API
endpoints:
.. http:post:: /api/v1/organizers/(organizer)/events/(event)/shredders/shred/(id1)/(id2)/
Starts an export task. If your input parameters validate correctly, a ``202 Accepted`` status code is returned.
The body points you to an URL you can use to check the status.
**Example request**:
.. sourcecode:: http
POST /api/v1/organizers/bigevents/events/sampleconf/shredders/shred/29891ede-196f-4942-9e26-d055a36e98b8/3f279f13-c198-4137-b49b-9b360ce9fcce/ HTTP/1.1
Host: pretix.eu
Accept: application/json, text/javascript
**Example response**:
.. sourcecode:: http
HTTP/1.1 202 Accepted
Vary: Accept
Content-Type: application/json
{
"status": "https://pretix.eu/api/v1/organizers/bigevents/events/sampleconf/shredders/status/29891ede-196f-4942-9e26-d055a36e98b8/3f279f13-c198-4137-b49b-9b360ce9fcce/"
}
:param organizer: The ``slug`` field of the organizer to fetch
:param event: The ``slug`` field of the event to fetch
:param id1: Opaque value given to you in the previous response
:param id2: Opaque value given to you in the previous response
:statuscode 202: no error
:statuscode 400: Invalid input options
:statuscode 401: Authentication failure
:statuscode 404: The export does not exist / is expired / belongs to a different API key.
:statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
:statuscode 409: Your export is still running. The body will be JSON with the structure ``{"status": "running"}``. ``status`` can be ``waiting`` before the task is actually being processed. Please retry, but wait at least one second before you do.
:statuscode 410: Either the job has timed out or running the export has failed permanently. The body will be JSON with the structure ``{"status": "failed", "message": "Error message"}``
Checking the result
-------------------
When starting to shred, you receive a ``status`` URL for checking for success.
Running a ``GET`` request on that result will yield one of the following status codes:
* ``200 OK`` The shredding succeeded.
* ``409 Conflict`` Shredding is still running. The body will be JSON with the structure ``{"status": "running"}``. ``status`` can be ``waiting`` before the task is actually being processed. Please retry, but wait at least one second before you do.
* ``410 Gone`` We no longer know about this process, probably the process was started more than an hour ago. Might also occur after successful operations on small pretix installations without asynchronous task handling.
* ``417 Expectation Failed`` Running the export has failed permanently. The body will be JSON with the structure ``{"status": "failed", "message": "Error message"}``

View File

@@ -0,0 +1,36 @@
#
# This file is part of pretix (Community Edition).
#
# Copyright (C) 2014-2020 Raphael Michel and contributors
# Copyright (C) 2020-2021 rami.io GmbH and contributors
#
# This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation in version 3 of the License.
#
# ADDITIONAL TERMS APPLY: Pursuant to Section 7 of the GNU Affero General Public License, additional terms are
# applicable granting you additional permissions and placing additional restrictions on your usage of this software.
# Please refer to the pretix LICENSE file to obtain the full terms applicable to this work. If you did not receive
# this file, see <https://pretix.eu/about/en/license>.
#
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
#
from rest_framework import serializers
class ShredderSerializer(serializers.Serializer):
identifier = serializers.CharField()
verbose_name = serializers.CharField()
class JobRunSerializer(serializers.Serializer):
shredders = serializers.MultipleChoiceField(choices=[])
def __init__(self, *args, **kwargs):
shredders = kwargs.pop('shredders')
super().__init__(*args, **kwargs)
self.fields['shredders'].choices = ((k.identifier, k.identifier) for k in shredders)

View File

@@ -42,7 +42,8 @@ from pretix.api.views import cart
from .views import (
checkin, device, discount, event, exporters, idempotency, item, oauth,
order, organizer, upload, user, version, voucher, waitinglist, webhooks,
order, organizer, shredders, upload, user, version, voucher, waitinglist,
webhooks,
)
router = routers.DefaultRouter()
@@ -84,6 +85,7 @@ event_router.register(r'waitinglistentries', waitinglist.WaitingListViewSet)
event_router.register(r'checkinlists', checkin.CheckinListViewSet)
event_router.register(r'cartpositions', cart.CartPositionViewSet)
event_router.register(r'exporters', exporters.EventExportersViewSet, basename='exporters')
event_router.register(r'shredders', shredders.EventShreddersViewSet, basename='shredders')
checkinlist_router = routers.DefaultRouter()
checkinlist_router.register(r'positions', checkin.CheckinListPositionViewSet, basename='checkinlistpos')

View File

@@ -0,0 +1,240 @@
#
# This file is part of pretix (Community Edition).
#
# Copyright (C) 2014-2020 Raphael Michel and contributors
# Copyright (C) 2020-2021 rami.io GmbH and contributors
#
# This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation in version 3 of the License.
#
# ADDITIONAL TERMS APPLY: Pursuant to Section 7 of the GNU Affero General Public License, additional terms are
# applicable granting you additional permissions and placing additional restrictions on your usage of this software.
# Please refer to the pretix LICENSE file to obtain the full terms applicable to this work. If you did not receive
# this file, see <https://pretix.eu/about/en/license>.
#
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
#
from datetime import timedelta
from celery.result import AsyncResult
from django.conf import settings
from django.http import Http404
from django.shortcuts import get_object_or_404
from django.utils.functional import cached_property
from django.utils.timezone import now
from rest_framework import status, viewsets
from rest_framework.decorators import action
from rest_framework.exceptions import ValidationError
from rest_framework.response import Response
from rest_framework.reverse import reverse
from pretix.api.serializers.shredders import (
JobRunSerializer, ShredderSerializer,
)
from pretix.base.models import CachedFile
from pretix.base.services.shredder import export, shred
from pretix.base.shredder import shred_constraints
from pretix.helpers.http import ChunkBasedFileResponse
class ShreddersMixin:
def list(self, request, *args, **kwargs):
res = ShredderSerializer(self.shredders, many=True)
return Response({
"count": len(self.shredders),
"next": None,
"previous": None,
"results": res.data
})
def get_object(self):
instances = [e for e in self.shredders if e.identifier == self.kwargs.get('pk')]
if not instances:
raise Http404()
return instances[0]
def retrieve(self, request, *args, **kwargs):
instance = self.get_object()
serializer = ShredderSerializer(instance)
return Response(serializer.data)
@action(detail=False, methods=['GET'], url_name='download', url_path='download/(?P<asyncid>[^/]+)/(?P<cfid>[^/]+)')
def download(self, *args, **kwargs):
cf = get_object_or_404(
CachedFile,
id=kwargs['cfid'],
session_key=f'api-shredder-{str(type(self.request.user or self.request.auth))}-{(self.request.user or self.request.auth).pk}'
)
if cf.file:
resp = ChunkBasedFileResponse(cf.file.file, content_type=cf.type)
resp['Content-Disposition'] = 'attachment; filename="{}"'.format(cf.filename).encode("ascii", "ignore")
return resp
elif not settings.HAS_CELERY:
return Response(
{'status': 'failed', 'message': 'Unknown file ID or export failed'},
status=status.HTTP_410_GONE
)
res = AsyncResult(kwargs['asyncid'])
if res.failed():
if isinstance(res.info, dict) and res.info['exc_type'] in ('ShredError', 'ExportError'):
msg = res.info['exc_message']
else:
msg = 'Internal error'
return Response(
{'status': 'failed', 'message': msg},
status=status.HTTP_410_GONE
)
return Response(
{
'status': 'running' if res.state in ('PROGRESS', 'STARTED', 'SUCCESS') else 'waiting',
},
status=status.HTTP_409_CONFLICT
)
@action(detail=False, methods=['GET'], url_name='status', url_path='status/(?P<asyncid>[^/]+)/(?P<cfid>[^/]+)')
def status(self, *args, **kwargs):
if settings.HAS_CELERY:
res = AsyncResult(kwargs['asyncid'])
if res.failed():
if isinstance(res.info, dict) and res.info['exc_type'] in ('ShredError', 'ExportError'):
msg = res.info['exc_message']
else:
msg = 'Internal error'
return Response(
{'status': 'failed', 'message': msg},
status=status.HTTP_417_EXPECTATION_FAILED
)
elif res.successful():
return Response(
{'status': 'ok', 'message': 'OK'},
status=status.HTTP_200_OK
)
try:
CachedFile.objects.get(
id=kwargs['cfid'],
session_key=f'api-shredder-{str(type(self.request.user or self.request.auth))}-{(self.request.user or self.request.auth).pk}'
)
except CachedFile.DoesNotExist:
return Response(
{'status': 'gone', 'message': 'May have succeeded or timed out'},
status=status.HTTP_410_GONE
)
return Response(
{
'status': 'running' if res.state in ('PROGRESS', 'STARTED', 'SUCCESS') else 'waiting',
},
status=status.HTTP_409_CONFLICT
)
@action(detail=False, methods=['POST'], url_name='shred', url_path='shred/(?P<asyncid>[^/]+)/(?P<cfid>[^/]+)')
def shred(self, *args, **kwargs):
cf = get_object_or_404(
CachedFile,
id=kwargs['cfid'],
session_key=f'api-shredder-{str(type(self.request.user or self.request.auth))}-{(self.request.user or self.request.auth).pk}'
)
if cf.file:
async_result = self.do_shred(cf)
url_kwargs = {
'asyncid': str(async_result.id),
'cfid': str(cf.id),
}
url_kwargs.update(self.kwargs)
return Response({
'status': reverse('api-v1:shredders-status', kwargs=url_kwargs, request=self.request),
}, status=status.HTTP_202_ACCEPTED)
elif not settings.HAS_CELERY:
return Response(
{'status': 'failed', 'message': 'Unknown file ID or export failed'},
status=status.HTTP_410_GONE
)
res = AsyncResult(kwargs['asyncid'])
if res.failed():
if isinstance(res.info, dict) and res.info['exc_type'] in ('ShredError', 'ExportError'):
msg = res.info['exc_message']
else:
msg = 'Internal error'
return Response(
{'status': 'failed', 'message': msg},
status=status.HTTP_410_GONE
)
return Response(
{
'status': 'running' if res.state in ('PROGRESS', 'STARTED', 'SUCCESS') else 'waiting',
},
status=status.HTTP_409_CONFLICT
)
@action(detail=False, methods=['POST'])
def export(self, *args, **kwargs):
serializer = JobRunSerializer(shredders=self.shredders, data=self.request.data, **self.get_serializer_kwargs())
serializer.is_valid(raise_exception=True)
cf = CachedFile(web_download=False)
cf.date = now()
cf.expires = now() + timedelta(hours=2)
cf.session_key = f'api-shredder-{str(type(self.request.user or self.request.auth))}-{(self.request.user or self.request.auth).pk}'
cf.save()
d = serializer.data
for k, v in d.items():
if isinstance(v, set):
d[k] = list(v)
async_result = self.do_export(cf, serializer.validated_data['shredders'])
url_kwargs = {
'asyncid': str(async_result.id),
'cfid': str(cf.id),
}
url_kwargs.update(self.kwargs)
return Response({
'download': reverse('api-v1:shredders-download', kwargs=url_kwargs, request=self.request),
'shred': reverse('api-v1:shredders-shred', kwargs=url_kwargs, request=self.request),
}, status=status.HTTP_202_ACCEPTED)
class EventShreddersViewSet(ShreddersMixin, viewsets.ViewSet):
permission = 'can_change_orders'
def get_serializer_kwargs(self):
return {}
@cached_property
def shredders(self):
shredders = []
for k, v in sorted(self.request.event.get_data_shredders().items(), key=lambda s: s[1].verbose_name):
shredders.append(v)
return shredders
def do_export(self, cf, shredders):
constr = shred_constraints(self.request.event)
if constr:
raise ValidationError(constr)
return export.apply_async(args=(
self.request.event.id,
list(shredders),
f'api-shredder-{str(type(self.request.user or self.request.auth))}-{(self.request.user or self.request.auth).pk}',
str(cf.pk)
))
def do_shred(self, cf):
constr = shred_constraints(self.request.event)
if constr:
raise ValidationError(constr)
return shred.apply_async(args=(
self.request.event.id,
str(cf.pk),
True,
))

View File

@@ -51,7 +51,7 @@ from pretix.celery_app import app
@app.task(base=ProfiledEventTask)
def export(event: Event, shredders: List[str], session_key=None) -> None:
def export(event: Event, shredders: List[str], session_key=None, cfid=None) -> None:
known_shredders = event.get_data_shredders()
with NamedTemporaryFile() as rawfile:
@@ -85,13 +85,16 @@ def export(event: Event, shredders: List[str], session_key=None) -> None:
rawfile.seek(0)
cf = CachedFile()
cf.date = now()
if cfid:
cf = CachedFile.objects.get(pk=cfid)
else:
cf = CachedFile()
cf.date = now()
cf.session_key = session_key
cf.web_download = True
cf.expires = now() + timedelta(hours=1)
cf.filename = event.slug + '.zip'
cf.type = 'application/zip'
cf.session_key = session_key
cf.web_download = True
cf.expires = now() + timedelta(hours=1)
cf.save()
cf.file.save(cachedfile_name(cf, cf.filename), rawfile)
@@ -115,7 +118,7 @@ def shred(event: Event, fileid: str, confirm_code: str) -> None:
if not shredder:
continue
shredders.append(shredder)
if any(shredder.require_download_confirmation for shredder in shredders):
if confirm_code is not True and any(shredder.require_download_confirmation for shredder in shredders):
if indexdata['confirm_code'] != confirm_code:
raise ShredError(_("The confirm code you entered was incorrect."))
if event.logentry_set.filter(datetime__gte=parse(indexdata['time'])):

View File

@@ -0,0 +1,143 @@
#
# This file is part of pretix (Community Edition).
#
# Copyright (C) 2014-2020 Raphael Michel and contributors
# Copyright (C) 2020-2021 rami.io GmbH and contributors
#
# This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
# Public License as published by the Free Software Foundation in version 3 of the License.
#
# ADDITIONAL TERMS APPLY: Pursuant to Section 7 of the GNU Affero General Public License, additional terms are
# applicable granting you additional permissions and placing additional restrictions on your usage of this software.
# Please refer to the pretix LICENSE file to obtain the full terms applicable to this work. If you did not receive
# this file, see <https://pretix.eu/about/en/license>.
#
# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
# details.
#
# You should have received a copy of the GNU Affero General Public License along with this program. If not, see
# <https://www.gnu.org/licenses/>.
#
# This file is based on an earlier version of pretix which was released under the Apache License 2.0. The full text of
# the Apache License 2.0 can be obtained at <http://www.apache.org/licenses/LICENSE-2.0>.
#
# This file may have since been changed and any changes are released under the terms of AGPLv3 as described above. A
# full history of changes and contributors is available at <https://github.com/pretix/pretix>.
#
# This file contains Apache-licensed contributions copyrighted by: Benjamin Hättasch
#
# Unless required by applicable law or agreed to in writing, software distributed under the Apache License 2.0 is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under the License.
import copy
import uuid
from datetime import datetime, timedelta
from decimal import Decimal
import pytest
from django.utils.timezone import now
from django_scopes import scopes_disabled
from pytz import UTC
from pretix.base.models import Order, OrderPosition
SAMPLE_SHREDDER_CONFIG = {
"identifier": "question_answers",
"verbose_name": "Question answers",
}
@pytest.fixture
@scopes_disabled()
def order(event, item):
o = Order.objects.create(
code='FOO', event=event, email='dummy@dummy.test',
status=Order.STATUS_PENDING, secret="k24fiuwvu8kxz3y1",
datetime=datetime(2017, 12, 1, 10, 0, 0, tzinfo=UTC),
expires=datetime(2017, 12, 10, 10, 0, 0, tzinfo=UTC),
total=23, locale='en'
)
OrderPosition.objects.create(
order=o,
item=item,
variation=None,
price=Decimal("23"),
attendee_name_parts={'_legacy': "Peter"},
secret="z3fsn8jyufm5kpk768q69gkbyr5f4h6w",
pseudonymization_id="ABCDEFGHKL",
)
return o
@pytest.mark.django_db
def test_event_list(token_client, organizer, event):
event.has_subevents = True
event.save()
c = copy.deepcopy(SAMPLE_SHREDDER_CONFIG)
resp = token_client.get('/api/v1/organizers/{}/events/{}/shredders/'.format(organizer.slug, event.slug))
assert resp.status_code == 200
assert c in resp.data['results']
resp = token_client.get(
'/api/v1/organizers/{}/events/{}/shredders/question_answers/'.format(organizer.slug, event.slug))
assert resp.status_code == 200
assert c == resp.data
@pytest.mark.django_db
def test_event_validate(token_client, organizer, team, event):
event.date_from = now() + timedelta(days=3)
event.date_to = now() + timedelta(days=4)
event.save()
resp = token_client.post(
'/api/v1/organizers/{}/events/{}/shredders/export/'.format(organizer.slug, event.slug),
data={
'shredders': ['question_answers']
}, format='json'
)
assert resp.status_code == 400
assert resp.data == ["Your event needs to be over to use this feature."]
@pytest.mark.django_db
def test_run_success(token_client, order, organizer, team, event):
event.date_from = now() - timedelta(days=91)
event.date_to = now() - timedelta(days=90)
event.save()
resp = token_client.post(
'/api/v1/organizers/{}/events/{}/shredders/export/'.format(organizer.slug, event.slug),
data={
'shredders': ['order_emails']
}, format='json'
)
assert resp.status_code == 202
assert "download" in resp.data
assert "shred" in resp.data
resp2 = token_client.get("/" + resp.data["download"].split("/", 3)[3])
assert resp2.status_code == 200
assert resp2["Content-Type"] == "application/zip"
resp3 = token_client.post("/" + resp.data["shred"].split("/", 3)[3])
assert resp3.status_code == 202
assert "status" in resp3.data
resp4 = token_client.get("/" + resp3.data["status"].split("/", 3)[3])
assert resp4.status_code == 410 # because we have no celery
resp2 = token_client.get("/" + resp.data["download"].split("/", 3)[3])
assert resp2.status_code == 404 # shredded now
order.refresh_from_db()
assert not order.email
@pytest.mark.django_db
def test_download_nonexisting(token_client, organizer, team, event):
resp = token_client.get('/api/v1/organizers/{}/events/{}/shredders/download/{}/{}/'.format(
organizer.slug, event.slug, uuid.uuid4(), uuid.uuid4()
))
assert resp.status_code == 404