Fix sqlparse version

* Fix #1025 -- Python 3.7 support

* Upgrade redis-py

* Travis: xenial

* Fix version specifier

.travis.yml

@@ -1,4 +1,5 @@
 language: python
+dist: xenial
 sudo: false
 install:
   - pip install -U pip wheel setuptools
@@ -12,23 +13,23 @@ services:
   - postgresql
 matrix:
   include:
-    - python: 3.6
+    - python: 3.7
       env: JOB=tests PRETIX_CONFIG_FILE=tests/travis_sqlite.cfg
-    - python: 3.6
+    - python: 3.7
       env: JOB=tests-cov PRETIX_CONFIG_FILE=tests/travis_postgres.cfg
-    - python: 3.6
+    - python: 3.7
       env: JOB=style
-    - python: 3.6
+    - python: 3.7
       env: JOB=tests PRETIX_CONFIG_FILE=tests/travis_mysql.cfg
-    - python: 3.6
+    - python: 3.7
       env: JOB=tests PRETIX_CONFIG_FILE=tests/travis_postgres.cfg
     - python: 3.5
       env: JOB=tests PRETIX_CONFIG_FILE=tests/travis_postgres.cfg
-    - python: 3.6
+    - python: 3.7
       env: JOB=plugins
-    - python: 3.6
+    - python: 3.7
       env: JOB=doc-spelling
-    - python: 3.6
+    - python: 3.7
       env: JOB=translation-spelling
 addons:
   postgresql: "9.4"

doc/admin/installation/manual_smallscale.rst

@@ -68,10 +68,6 @@ To build and run pretix, you will need the following debian packages::
                       python3-dev libxml2-dev libxslt1-dev libffi-dev zlib1g-dev libssl-dev \
                       gettext libpq-dev libmariadbclient-dev libjpeg-dev libopenjp2-7-dev
 
-.. note:: Python 3.7 is not yet supported, so if you run a very recent OS, make sure to get
-          Python 3.6 from somewhere. You can check the current state of things in our
-          `Python 3.7 issue`_.
-
 Config file
 -----------
 
@@ -314,4 +310,3 @@ example::
 .. _redis: https://blog.programster.org/debian-8-install-redis-server/
 .. _ufw: https://en.wikipedia.org/wiki/Uncomplicated_Firewall
 .. _strong encryption settings: https://mozilla.github.io/server-side-tls/ssl-config-generator/
-.. _Python 3.7 issue: https://github.com/pretix/pretix/issues/1025

doc/api/fundamentals.rst

@@ -181,4 +181,37 @@ as the string values ``true`` and ``false``.
 If the ``ordering`` parameter is documented for a resource, you can use it to sort the result set by one of the allowed
 fields. Prepend a ``-`` to the field name to reverse the sort order.
 
+
+Idempotency
+-----------
+
+Our API supports an idempotency mechanism to make sure you can safely retry operations without accidentally performing
+them twice. This is useful if an API call experiences interruptions in transit, e.g. due to a network failure, and you
+do not know if it completed successfully.
+
+To perform an idempotent request, add a ``X-Idempotency-Key`` header with a random string value (we recommend a version
+4 UUID) to your request. If we see a second request with the same ``X-Idempotency-Key`` and the same ``Authorization``
+and ``Cookie`` headers, we will not perform the action for a second time but return the exact same response instead.
+
+Please note that this also goes for most error responses. For example, if we returned you a ``403 Permission Denied``
+error and you retry with the same ``X-Idempotency-Key``, you will get the same error again, even if you were granted
+permission in the meantime! This includes internal server errors on our side that might have been fixed in the meantime.
+
+There are only three exceptions to the rule:
+
+* Responses with status code ``409 Conflict`` are not cached. If you send the request again, it will be executed as a
+  new request, since these responses are intended to be retried.
+
+* Rate-limited responses with status code ``429 Too Many Requests`` are not cached and you can safely retry them.
+
+* Responses with status code ``503 Service Unavailable`` are not cached and you can safely retry them.
+
+If you send a request with an ``X-Idempotency-Key`` header that we have seen before but that has not yet received a
+response, you will receive a response with status code ``409 Conflict`` and are asked to retry after five seconds.
+
+We store idempotency keys for 24 hours, so you should never retry a request after a longer time period.
+
+All ``POST``, ``PUT``, ``PATCH``, or ``DELETE`` api calls support idempotency keys. Adding an idempotency key to a
+``GET``, ``HEAD``, or ``OPTIONS`` request has no effect.
+
 .. _CSRF policies: https://docs.djangoproject.com/en/1.11/ref/csrf/#ajax

doc/api/resources/orders.rst

@@ -750,6 +750,7 @@ Creating orders
      should only use this if you know the specific payment provider in detail. Please keep in mind that the payment
      provider will not be called to do anything about this (i.e. if you pass a bank account to a debit provider, *no*
      charge will be created), this is just informative in case you *handled the payment already*.
+   * ``payment_date`` (optional) – Date and time of the completion of the payment.
    * ``comment`` (optional)
    * ``checkin_attention`` (optional)
    * ``invoice_address`` (optional)

doc/development/api/general.rst

@@ -20,7 +20,7 @@ Order events
 There are multiple signals that will be sent out in the ordering cycle:
 
 .. automodule:: pretix.base.signals
-   :members: validate_cart, order_fee_calculation, order_paid, order_placed, order_fee_type_name, allow_ticket_download
+   :members: validate_cart, order_fee_calculation, order_paid, order_placed, order_canceled, order_expired, order_modified, order_changed, order_approved, order_denied, order_fee_type_name, allow_ticket_download
 
 Frontend
 --------

src/pretix/api/middleware.py

@@ -0,0 +1,91 @@
+import json
+from hashlib import sha1
+
+from django.conf import settings
+from django.db import transaction
+from django.http import HttpRequest, HttpResponse, JsonResponse
+from django.utils.timezone import now
+from rest_framework import status
+
+from pretix.api.models import ApiCall
+
+
+class IdempotencyMiddleware:
+    def __init__(self, get_response):
+        self.get_response = get_response
+
+    def __call__(self, request: HttpRequest):
+        if request.method in ('GET', 'HEAD', 'OPTIONS'):
+            return self.get_response(request)
+
+        if not request.path.startswith('/api/'):
+            return self.get_response(request)
+
+        if not request.headers.get('X-Idempotency-Key'):
+            return self.get_response(request)
+
+        auth_hash_parts = '{}:{}'.format(
+            request.headers.get('Authorization', ''),
+            request.COOKIES.get(settings.SESSION_COOKIE_NAME, '')
+        )
+        auth_hash = sha1(auth_hash_parts.encode()).hexdigest()
+        idempotency_key = request.headers.get('X-Idempotency-Key', '')
+
+        with transaction.atomic():
+            call, created = ApiCall.objects.select_for_update().get_or_create(
+                auth_hash=auth_hash,
+                idempotency_key=idempotency_key,
+                defaults={
+                    'locked': now(),
+                    'request_method': request.method,
+                    'request_path': request.path,
+                    'response_code': 0,
+                    'response_headers': '{}',
+                    'response_body': b''
+                }
+            )
+
+        if created:
+            resp = self.get_response(request)
+            with transaction.atomic():
+                if resp.status_code in (409, 429, 503):
+                    # This is the exception: These calls are *meant* to be retried!
+                    call.delete()
+                else:
+                    call.response_code = resp.status_code
+                    if isinstance(resp.content, str):
+                        call.response_body = resp.content.encode()
+                    elif isinstance(resp.content, memoryview):
+                        call.response_body = resp.content.tobytes()
+                    elif isinstance(resp.content, bytes):
+                        call.response_body = resp.content
+                    elif hasattr(resp.content, 'read'):
+                        call.response_body = resp.read()
+                    elif hasattr(resp, 'data'):
+                        call.response_body = json.dumps(resp.data)
+                    else:
+                        call.response_body = repr(resp).encode()
+                    call.response_headers = json.dumps(resp._headers)
+                    call.locked = None
+                    call.save(update_fields=['locked', 'response_code', 'response_headers',
+                                             'response_body'])
+            return resp
+        else:
+            if call.locked:
+                r = JsonResponse(
+                    {'detail': 'Concurrent request with idempotency key.'},
+                    status=status.HTTP_409_CONFLICT,
+                )
+                r['Retry-After'] = 5
+                return r
+
+            content = call.response_body
+            if isinstance(content, memoryview):
+                content = content.tobytes()
+            r = HttpResponse(
+                content=content,
+                status=call.response_code,
+            )
+            for k, v in json.loads(call.response_headers).values():
+                r[k] = v
+            return r

src/pretix/api/migrations/0004_auto_20190405_1048.py

@@ -0,0 +1,44 @@
+# Generated by Django 2.1.5 on 2019-04-05 10:48
+
+import django.db.models.deletion
+from django.conf import settings
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+        ('pretixbase', '0116_auto_20190402_0722'),
+        ('pretixapi', '0003_webhook_webhookcall_webhookeventlistener'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='ApiCall',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('idempotency_key', models.CharField(db_index=True, max_length=190)),
+                ('auth_hash', models.CharField(db_index=True, max_length=190)),
+                ('created', models.DateTimeField(auto_now_add=True)),
+                ('locked', models.DateTimeField(null=True)),
+                ('request_method', models.CharField(max_length=20)),
+                ('request_path', models.CharField(max_length=255)),
+                ('response_code', models.PositiveIntegerField()),
+                ('response_headers', models.TextField()),
+                ('response_body', models.BinaryField()),
+            ],
+        ),
+        migrations.AlterModelOptions(
+            name='webhookcall',
+            options={'ordering': ('-datetime',)},
+        ),
+        migrations.AlterModelOptions(
+            name='webhookeventlistener',
+            options={'ordering': ('action_type',)},
+        ),
+        migrations.AlterUniqueTogether(
+            name='apicall',
+            unique_together={('idempotency_key', 'auth_hash')},
+        ),
+    ]

src/pretix/api/models.py

@@ -77,6 +77,9 @@ class WebHook(models.Model):
     all_events = models.BooleanField(default=True, verbose_name=_("All events (including newly created ones)"))
     limit_events = models.ManyToManyField('pretixbase.Event', verbose_name=_("Limit to events"), blank=True)
 
+    class Meta:
+        ordering = ('id',)
+
     @property
     def action_types(self):
         return [
@@ -106,3 +109,20 @@ class WebHookCall(models.Model):
 
     class Meta:
         ordering = ("-datetime",)
+
+
+class ApiCall(models.Model):
+    idempotency_key = models.CharField(max_length=190, db_index=True)
+    auth_hash = models.CharField(max_length=190, db_index=True)
+    created = models.DateTimeField(auto_now_add=True)
+    locked = models.DateTimeField(null=True)
+
+    request_method = models.CharField(max_length=20)
+    request_path = models.CharField(max_length=255)
+
+    response_code = models.PositiveIntegerField()
+    response_headers = models.TextField()
+    response_body = models.BinaryField()
+
+    class Meta:
+        unique_together = (('idempotency_key', 'auth_hash'),)

src/pretix/api/serializers/event.py

@@ -31,10 +31,10 @@ class PluginsField(Field):
     def to_representation(self, obj):
         from pretix.base.plugins import get_all_plugins
 
-        return {
+        return sorted([
             p.module for p in get_all_plugins()
             if not p.name.startswith('.') and getattr(p, 'visible', True) and p.module in obj.get_plugins()
-        }
+        ])
 
     def to_internal_value(self, data):
         return {

src/pretix/api/serializers/order.py

@@ -458,11 +458,12 @@ class OrderCreateSerializer(I18nAwareModelSerializer):
     payment_info = CompatibleJSONField(required=False)
     consume_carts = serializers.ListField(child=serializers.CharField(), required=False)
     force = serializers.BooleanField(default=False, required=False)
+    payment_date = serializers.DateTimeField(required=False, allow_null=True)
 
     class Meta:
         model = Order
         fields = ('code', 'status', 'testmode', 'email', 'locale', 'payment_provider', 'fees', 'comment', 'sales_channel',
-                  'invoice_address', 'positions', 'checkin_attention', 'payment_info', 'consume_carts', 'force')
+                  'invoice_address', 'positions', 'checkin_attention', 'payment_info', 'payment_date', 'consume_carts', 'force')
 
     def validate_payment_provider(self, pp):
         if pp not in self.context['event'].get_payment_providers():
@@ -533,6 +534,7 @@ class OrderCreateSerializer(I18nAwareModelSerializer):
         positions_data = validated_data.pop('positions') if 'positions' in validated_data else []
         payment_provider = validated_data.pop('payment_provider')
         payment_info = validated_data.pop('payment_info', '{}')
+        payment_date = validated_data.pop('payment_date', now())
         force = validated_data.pop('force', False)
 
         if 'invoice_address' in validated_data:
@@ -617,7 +619,7 @@ class OrderCreateSerializer(I18nAwareModelSerializer):
                     amount=order.total,
                     provider=payment_provider,
                     info=payment_info,
-                    payment_date=now(),
+                    payment_date=payment_date,
                     state=OrderPayment.PAYMENT_STATE_CONFIRMED
                 )
             elif payment_provider:

src/pretix/api/signals.py

@@ -3,7 +3,7 @@ from datetime import timedelta
 from django.dispatch import Signal, receiver
 from django.utils.timezone import now
 
-from pretix.api.models import WebHookCall
+from pretix.api.models import ApiCall, WebHookCall
 from pretix.base.signals import periodic_task
 
 register_webhook_events = Signal(
@@ -19,3 +19,8 @@ instances.
 @receiver(periodic_task)
 def cleanup_webhook_logs(sender, **kwargs):
     WebHookCall.objects.filter(datetime__lte=now() - timedelta(days=30)).delete()
+
+
+@receiver(periodic_task)
+def cleanup_api_logs(sender, **kwargs):
+    ApiCall.objects.filter(datetime__lte=now() - timedelta(hours=24)).delete()

src/pretix/api/views/__init__.py

@@ -31,10 +31,10 @@ class RichOrderingFilter(OrderingFilter):
 class ConditionalListView:
 
     def list(self, request, **kwargs):
-        if_modified_since = request.META.get('HTTP_IF_MODIFIED_SINCE')
+        if_modified_since = request.headers.get('If-Modified-Since')
         if if_modified_since:
             if_modified_since = parse_http_date_safe(if_modified_since)
-        if_unmodified_since = request.META.get('HTTP_IF_UNMODIFIED_SINCE')
+        if_unmodified_since = request.headers.get('If-Unmodified-Since')
         if if_unmodified_since:
             if_unmodified_since = parse_http_date_safe(if_unmodified_since)
         if not hasattr(request, 'event'):

src/pretix/api/views/checkin.py

@@ -7,7 +7,7 @@ from django.utils.functional import cached_property
 from django.utils.timezone import now
 from django_filters.rest_framework import DjangoFilterBackend, FilterSet
 from rest_framework import viewsets
-from rest_framework.decorators import detail_route
+from rest_framework.decorators import action
 from rest_framework.fields import DateTimeField
 from rest_framework.response import Response
 
@@ -77,7 +77,7 @@ class CheckinListViewSet(viewsets.ModelViewSet):
         )
         super().perform_destroy(instance)
 
-    @detail_route(methods=['GET'])
+    @action(detail=True, methods=['GET'])
     def status(self, *args, **kwargs):
         clist = self.get_object()
         cqs = Checkin.objects.filter(
@@ -242,7 +242,7 @@ class CheckinListPositionViewSet(viewsets.ReadOnlyModelViewSet):
 
         return qs
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def redeem(self, *args, **kwargs):
         force = bool(self.request.data.get('force', False))
         ignore_unpaid = bool(self.request.data.get('ignore_unpaid', False))

src/pretix/api/views/event.py

@@ -13,7 +13,7 @@ from pretix.api.serializers.event import (
 )
 from pretix.api.views import ConditionalListView
 from pretix.base.models import (
-    Device, Event, ItemCategory, TaxRule, TeamAPIToken,
+    CartPosition, Device, Event, ItemCategory, TaxRule, TeamAPIToken,
 )
 from pretix.base.models.event import SubEvent
 from pretix.helpers.dicts import merge_dicts
@@ -272,6 +272,8 @@ class SubEventViewSet(ConditionalListView, viewsets.ModelViewSet):
                     auth=self.request.auth,
                     data=self.request.data
                 )
+                CartPosition.objects.filter(addon_to__subevent=instance).delete()
+                instance.cartposition_set.all().delete()
                 super().perform_destroy(instance)
         except ProtectedError:
             raise PermissionDenied('The sub-event could not be deleted as some constraints (e.g. data created by '

src/pretix/api/views/item.py

@@ -4,7 +4,7 @@ from django.shortcuts import get_object_or_404
 from django.utils.functional import cached_property
 from django_filters.rest_framework import DjangoFilterBackend, FilterSet
 from rest_framework import viewsets
-from rest_framework.decorators import detail_route
+from rest_framework.decorators import action
 from rest_framework.exceptions import PermissionDenied
 from rest_framework.filters import OrderingFilter
 from rest_framework.response import Response
@@ -16,8 +16,8 @@ from pretix.api.serializers.item import (
 )
 from pretix.api.views import ConditionalListView
 from pretix.base.models import (
-    Item, ItemAddOn, ItemBundle, ItemCategory, ItemVariation, Question,
-    QuestionOption, Quota,
+    CartPosition, Item, ItemAddOn, ItemBundle, ItemCategory, ItemVariation,
+    Question, QuestionOption, Quota,
 )
 from pretix.helpers.dicts import merge_dicts
 
@@ -84,7 +84,8 @@ class ItemViewSet(ConditionalListView, viewsets.ModelViewSet):
             user=self.request.user,
             auth=self.request.auth,
         )
-        self.get_object().cartposition_set.all().delete()
+        CartPosition.objects.filter(addon_to__item=instance).delete()
+        instance.cartposition_set.all().delete()
         super().perform_destroy(instance)
 
 
@@ -498,7 +499,7 @@ class QuotaViewSet(ConditionalListView, viewsets.ModelViewSet):
             )
         super().perform_destroy(instance)
 
-    @detail_route(methods=['get'])
+    @action(detail=True, methods=['get'])
     def availability(self, request, *args, **kwargs):
         quota = self.get_object()
 

src/pretix/api/views/order.py

@@ -12,7 +12,7 @@ from django.utils.timezone import make_aware, now
 from django.utils.translation import ugettext as _
 from django_filters.rest_framework import DjangoFilterBackend, FilterSet
 from rest_framework import mixins, serializers, status, viewsets
-from rest_framework.decorators import detail_route
+from rest_framework.decorators import action
 from rest_framework.exceptions import (
     APIException, NotFound, PermissionDenied, ValidationError,
 )
@@ -42,7 +42,9 @@ from pretix.base.services.orders import (
     extend_order, mark_order_expired, mark_order_refunded,
 )
 from pretix.base.services.tickets import generate
-from pretix.base.signals import order_placed, register_ticket_outputs
+from pretix.base.signals import (
+    order_modified, order_placed, register_ticket_outputs,
+)
 
 
 class OrderFilter(FilterSet):
@@ -125,7 +127,7 @@ class OrderViewSet(viewsets.ModelViewSet):
         serializer = self.get_serializer(queryset, many=True)
         return Response(serializer.data, headers={'X-Page-Generated': date})
 
-    @detail_route(url_name='download', url_path='download/(?P<output>[^/]+)')
+    @action(detail=True, url_name='download', url_path='download/(?P<output>[^/]+)')
     def download(self, request, output, **kwargs):
         provider = self._get_output_provider(output)
         order = self.get_object()
@@ -147,7 +149,7 @@ class OrderViewSet(viewsets.ModelViewSet):
             )
             return resp
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def mark_paid(self, request, **kwargs):
         order = self.get_object()
 
@@ -188,7 +190,7 @@ class OrderViewSet(viewsets.ModelViewSet):
             status=status.HTTP_400_BAD_REQUEST
         )
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def mark_canceled(self, request, **kwargs):
         send_mail = request.data.get('send_email', True)
         cancellation_fee = request.data.get('cancellation_fee', None)
@@ -222,7 +224,7 @@ class OrderViewSet(viewsets.ModelViewSet):
             )
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def approve(self, request, **kwargs):
         send_mail = request.data.get('send_email', True)
 
@@ -240,7 +242,7 @@ class OrderViewSet(viewsets.ModelViewSet):
             return Response({'detail': str(e)}, status=status.HTTP_400_BAD_REQUEST)
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def deny(self, request, **kwargs):
         send_mail = request.data.get('send_email', True)
         comment = request.data.get('comment', '')
@@ -258,7 +260,7 @@ class OrderViewSet(viewsets.ModelViewSet):
             return Response({'detail': str(e)}, status=status.HTTP_400_BAD_REQUEST)
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def mark_pending(self, request, **kwargs):
         order = self.get_object()
 
@@ -277,7 +279,7 @@ class OrderViewSet(viewsets.ModelViewSet):
         )
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def mark_expired(self, request, **kwargs):
         order = self.get_object()
 
@@ -294,7 +296,7 @@ class OrderViewSet(viewsets.ModelViewSet):
         )
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def mark_refunded(self, request, **kwargs):
         order = self.get_object()
 
@@ -311,7 +313,7 @@ class OrderViewSet(viewsets.ModelViewSet):
         )
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def create_invoice(self, request, **kwargs):
         order = self.get_object()
         has_inv = order.invoices.exists() and not (
@@ -343,7 +345,7 @@ class OrderViewSet(viewsets.ModelViewSet):
             status=status.HTTP_201_CREATED
         )
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def resend_link(self, request, **kwargs):
         order = self.get_object()
         if not order.email:
@@ -357,7 +359,7 @@ class OrderViewSet(viewsets.ModelViewSet):
             status=status.HTTP_204_NO_CONTENT
         )
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     @transaction.atomic
     def regenerate_secrets(self, request, **kwargs):
         order = self.get_object()
@@ -375,7 +377,7 @@ class OrderViewSet(viewsets.ModelViewSet):
         )
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def extend(self, request, **kwargs):
         new_date = request.data.get('expires', None)
         force = request.data.get('force', False)
@@ -451,61 +453,64 @@ class OrderViewSet(viewsets.ModelViewSet):
             )
         return super().update(request, *args, **kwargs)
 
-    @transaction.atomic
     def perform_update(self, serializer):
-        if 'comment' in self.request.data and serializer.instance.comment != self.request.data.get('comment'):
-            serializer.instance.log_action(
-                'pretix.event.order.comment',
-                user=self.request.user,
-                auth=self.request.auth,
-                data={
-                    'new_comment': self.request.data.get('comment')
-                }
-            )
+        with transaction.atomic():
+            if 'comment' in self.request.data and serializer.instance.comment != self.request.data.get('comment'):
+                serializer.instance.log_action(
+                    'pretix.event.order.comment',
+                    user=self.request.user,
+                    auth=self.request.auth,
+                    data={
+                        'new_comment': self.request.data.get('comment')
+                    }
+                )
 
-        if 'checkin_attention' in self.request.data and serializer.instance.checkin_attention != self.request.data.get('checkin_attention'):
-            serializer.instance.log_action(
-                'pretix.event.order.checkin_attention',
-                user=self.request.user,
-                auth=self.request.auth,
-                data={
-                    'new_value': self.request.data.get('checkin_attention')
-                }
-            )
+            if 'checkin_attention' in self.request.data and serializer.instance.checkin_attention != self.request.data.get('checkin_attention'):
+                serializer.instance.log_action(
+                    'pretix.event.order.checkin_attention',
+                    user=self.request.user,
+                    auth=self.request.auth,
+                    data={
+                        'new_value': self.request.data.get('checkin_attention')
+                    }
+                )
 
-        if 'email' in self.request.data and serializer.instance.email != self.request.data.get('email'):
-            serializer.instance.log_action(
-                'pretix.event.order.contact.changed',
-                user=self.request.user,
-                auth=self.request.auth,
-                data={
-                    'old_email': serializer.instance.email,
-                    'new_email': self.request.data.get('email'),
-                }
-            )
+            if 'email' in self.request.data and serializer.instance.email != self.request.data.get('email'):
+                serializer.instance.log_action(
+                    'pretix.event.order.contact.changed',
+                    user=self.request.user,
+                    auth=self.request.auth,
+                    data={
+                        'old_email': serializer.instance.email,
+                        'new_email': self.request.data.get('email'),
+                    }
+                )
 
-        if 'locale' in self.request.data and serializer.instance.locale != self.request.data.get('locale'):
-            serializer.instance.log_action(
-                'pretix.event.order.locale.changed',
-                user=self.request.user,
-                auth=self.request.auth,
-                data={
-                    'old_locale': serializer.instance.locale,
-                    'new_locale': self.request.data.get('locale'),
-                }
-            )
+            if 'locale' in self.request.data and serializer.instance.locale != self.request.data.get('locale'):
+                serializer.instance.log_action(
+                    'pretix.event.order.locale.changed',
+                    user=self.request.user,
+                    auth=self.request.auth,
+                    data={
+                        'old_locale': serializer.instance.locale,
+                        'new_locale': self.request.data.get('locale'),
+                    }
+                )
+
+            if 'invoice_address' in self.request.data:
+                serializer.instance.log_action(
+                    'pretix.event.order.modified',
+                    user=self.request.user,
+                    auth=self.request.auth,
+                    data={
+                        'invoice_data': self.request.data.get('invoice_address'),
+                    }
+                )
+
+            serializer.save()
 
         if 'invoice_address' in self.request.data:
-            serializer.instance.log_action(
-                'pretix.event.order.modified',
-                user=self.request.user,
-                auth=self.request.auth,
-                data={
-                    'invoice_data': self.request.data.get('invoice_address'),
-                }
-            )
-
-        serializer.save()
+            order_modified.send(sender=serializer.instance.event, order=serializer.instance)
 
     def perform_create(self, serializer):
         serializer.save()
@@ -614,7 +619,7 @@ class OrderPositionViewSet(mixins.DestroyModelMixin, viewsets.ReadOnlyModelViewS
                 return prov
         raise NotFound('Unknown output provider.')
 
-    @detail_route(url_name='download', url_path='download/(?P<output>[^/]+)')
+    @action(detail=True, url_name='download', url_path='download/(?P<output>[^/]+)')
     def download(self, request, output, **kwargs):
         provider = self._get_output_provider(output)
         pos = self.get_object()
@@ -665,7 +670,7 @@ class PaymentViewSet(viewsets.ReadOnlyModelViewSet):
         order = get_object_or_404(Order, code=self.kwargs['order'], event=self.request.event)
         return order.payments.all()
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def confirm(self, request, **kwargs):
         payment = self.get_object()
         force = request.data.get('force', False)
@@ -686,7 +691,7 @@ class PaymentViewSet(viewsets.ReadOnlyModelViewSet):
             pass
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def refund(self, request, **kwargs):
         payment = self.get_object()
         amount = serializers.DecimalField(max_digits=10, decimal_places=2).to_internal_value(
@@ -751,7 +756,7 @@ class PaymentViewSet(viewsets.ReadOnlyModelViewSet):
                     payment.order.save(update_fields=['status', 'expires'])
             return Response(OrderRefundSerializer(r).data, status=status.HTTP_200_OK)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def cancel(self, request, **kwargs):
         payment = self.get_object()
 
@@ -779,7 +784,7 @@ class RefundViewSet(CreateModelMixin, viewsets.ReadOnlyModelViewSet):
         order = get_object_or_404(Order, code=self.kwargs['order'], event=self.request.event)
         return order.refunds.all()
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def cancel(self, request, **kwargs):
         refund = self.get_object()
 
@@ -796,7 +801,7 @@ class RefundViewSet(CreateModelMixin, viewsets.ReadOnlyModelViewSet):
             }, user=self.request.user if self.request.user.is_authenticated else None, auth=self.request.auth)
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def process(self, request, **kwargs):
         refund = self.get_object()
 
@@ -821,7 +826,7 @@ class RefundViewSet(CreateModelMixin, viewsets.ReadOnlyModelViewSet):
             refund.order.save(update_fields=['status', 'expires'])
         return self.retrieve(request, [], **kwargs)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def done(self, request, **kwargs):
         refund = self.get_object()
 
@@ -911,7 +916,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
             nr=Concat('prefix', 'invoice_no')
         )
 
-    @detail_route()
+    @action(detail=True, )
     def download(self, request, **kwargs):
         invoice = self.get_object()
 
@@ -929,7 +934,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
         resp['Content-Disposition'] = 'attachment; filename="{}.pdf"'.format(invoice.number)
         return resp
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def regenerate(self, request, **kwarts):
         inv = self.get_object()
         if inv.canceled:
@@ -948,7 +953,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
             )
             return Response(status=204)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def reissue(self, request, **kwarts):
         inv = self.get_object()
         if inv.canceled:

src/pretix/api/views/voucher.py

@@ -7,7 +7,7 @@ from django_filters.rest_framework import (
     BooleanFilter, DjangoFilterBackend, FilterSet,
 )
 from rest_framework import status, viewsets
-from rest_framework.decorators import list_route
+from rest_framework.decorators import action
 from rest_framework.exceptions import PermissionDenied
 from rest_framework.filters import OrderingFilter
 from rest_framework.response import Response
@@ -111,9 +111,12 @@ class VoucherViewSet(viewsets.ModelViewSet):
             user=self.request.user,
             auth=self.request.auth,
         )
-        super().perform_destroy(instance)
+        with transaction.atomic():
+            instance.cartposition_set.filter(addon_to__isnull=False).delete()
+            instance.cartposition_set.all().delete()
+            super().perform_destroy(instance)
 
-    @list_route(methods=['POST'])
+    @action(detail=False, methods=['POST'])
     def batch_create(self, request, *args, **kwargs):
         if any(self._predict_quota_check(d, None) for d in request.data):
             lockfn = request.event.lock

src/pretix/api/views/waitinglist.py

@@ -1,7 +1,7 @@
 import django_filters
 from django_filters.rest_framework import DjangoFilterBackend, FilterSet
 from rest_framework import viewsets
-from rest_framework.decorators import detail_route
+from rest_framework.decorators import action
 from rest_framework.exceptions import PermissionDenied, ValidationError
 from rest_framework.filters import OrderingFilter
 from rest_framework.response import Response
@@ -69,7 +69,7 @@ class WaitingListViewSet(viewsets.ModelViewSet):
         )
         super().perform_destroy(instance)
 
-    @detail_route(methods=['POST'])
+    @action(detail=True, methods=['POST'])
     def send_voucher(self, *args, **kwargs):
         try:
             self.get_object().send_voucher(

src/pretix/base/middleware.py

@@ -97,7 +97,7 @@ def get_language_from_event(request: HttpRequest) -> str:
 
 
 def get_language_from_browser(request: HttpRequest) -> str:
-    accept = request.META.get('HTTP_ACCEPT_LANGUAGE', '')
+    accept = request.headers.get('Accept-Language', '')
     for accept_lang, unused in parse_accept_lang_header(accept):
         if accept_lang == '*':
             break

src/pretix/base/models/auth.py

@@ -111,6 +111,7 @@ class User(AbstractBaseUser, PermissionsMixin, LoggingMixin):
     class Meta:
         verbose_name = _("User")
         verbose_name_plural = _("Users")
+        ordering = ('email',)
 
     def save(self, *args, **kwargs):
         self.email = self.email.lower()

src/pretix/base/models/event.py

@@ -759,6 +759,7 @@ class Event(EventMixin, LoggedModel):
         return not self.orders.exists() and not self.invoices.exists()
 
     def delete_sub_objects(self):
+        self.cartposition_set.filter(addon_to__isnull=False).delete()
         self.cartposition_set.all().delete()
         self.items.all().delete()
         self.subevents.all().delete()

src/pretix/base/models/fields.py

@@ -37,7 +37,7 @@ class MultiStringField(TextField):
     def get_prep_lookup(self, lookup_type, value):  # NOQA
         raise TypeError('Lookups on multi strings are currently not supported.')
 
-    def from_db_value(self, value, expression, connection, context):
+    def from_db_value(self, value, expression, connection):
         if value:
             return [v for v in value.split(DELIMITER) if v]
         else:

src/pretix/base/models/tax.py

@@ -118,6 +118,9 @@ class TaxRule(LoggedModel):
     )
     custom_rules = models.TextField(blank=True, null=True)
 
+    class Meta:
+        ordering = ('event', 'rate', 'id')
+
     def allow_delete(self):
         from pretix.base.models.orders import OrderFee, OrderPosition
 

src/pretix/base/payment.py

@@ -721,13 +721,10 @@ class BoxOfficeProvider(BasePaymentProvider):
         return False
 
     def payment_control_render(self, request, payment) -> str:
-        template = None
-        payment_info = None
-
-        if payment.info:
-            payment_info = json.loads(payment.info)
-            if payment_info['payment_type'] == "sumup":
-                template = get_template('pretixcontrol/boxoffice/payment_sumup.html')
+        if not payment.info:
+            return
+        payment_info = json.loads(payment.info)
+        template = get_template('pretixcontrol/boxoffice/payment.html')
 
         ctx = {
             'request': request,
@@ -737,11 +734,7 @@ class BoxOfficeProvider(BasePaymentProvider):
             'payment': payment,
             'provider': self,
         }
-
-        if template:
-            return template.render(ctx)
-        else:
-            return
+        return template.render(ctx)
 
 
 class ManualPayment(BasePaymentProvider):

src/pretix/base/services/cart.py

@@ -276,6 +276,10 @@ class CartManager:
         err = None
         changed_prices = {}
         for cp in expired:
+            removed_positions = {op.position.pk for op in self._operations if isinstance(op, self.RemoveOperation)}
+            if cp.pk in removed_positions or (cp.addon_to_id and cp.addon_to_id in removed_positions):
+                continue
+
             if cp.is_bundled:
                 try:
                     bundle = cp.addon_to.item.bundles.get(bundled_item=cp.item, bundled_variation=cp.variation)

src/pretix/base/services/invoices.py

@@ -120,7 +120,7 @@ def build_invoice(invoice: Invoice) -> Invoice:
         positions = list(
             invoice.order.positions.select_related('addon_to', 'item', 'tax_rule', 'subevent', 'variation').annotate(
                 addon_c=Count('addons')
-            )
+            ).order_by('positionid', 'id')
         )
 
         reverse_charge = False

src/pretix/base/services/mail.py

@@ -212,15 +212,21 @@ def mail_send_task(self, *args, to: List[str], subject: str, body: str, html: st
                 order = None
             else:
                 if attach_tickets:
+                    args = []
+                    attach_size = 0
                     for name, ct in get_tickets_for_order(order):
-                        try:
-                            email.attach(
-                                name,
-                                ct.file.read(),
-                                ct.type
-                            )
-                        except:
-                            pass
+                        content = ct.file.read()
+                        args.append((name, content, ct.type))
+                        attach_size += len(content)
+
+                    if attach_tickets < 4 * 1024 * 1024:
+                        # Do not attach more than 4MB, it will bounce way to often.
+
+                        for a in args:
+                            try:
+                                email.attach(*a)
+                            except:
+                                pass
 
         email = email_filter.send_chained(event, 'message', message=email, order=order)
 

src/pretix/base/services/orders.py

@@ -42,7 +42,9 @@ from pretix.base.services.mail import SendMailException
 from pretix.base.services.pricing import get_price
 from pretix.base.services.tasks import ProfiledTask
 from pretix.base.signals import (
-    allow_ticket_download, order_fee_calculation, order_placed, periodic_task,
+    allow_ticket_download, order_approved, order_canceled, order_changed,
+    order_denied, order_expired, order_fee_calculation, order_placed,
+    periodic_task,
 )
 from pretix.celery_app import app
 from pretix.helpers.models import modelcopy
@@ -134,55 +136,58 @@ def mark_order_refunded(order, user=None, auth=None, api_token=None):
     )
 
 
-@transaction.atomic
 def mark_order_expired(order, user=None, auth=None):
     """
     Mark this order as expired. This sets the payment status and returns the order object.
     :param order: The order to change
     :param user: The user that performed the change
     """
-    if isinstance(order, int):
-        order = Order.objects.get(pk=order)
-    if isinstance(user, int):
-        user = User.objects.get(pk=user)
-    with order.event.lock():
-        order.status = Order.STATUS_EXPIRED
-        order.save(update_fields=['status'])
+    with transaction.atomic():
+        if isinstance(order, int):
+            order = Order.objects.get(pk=order)
+        if isinstance(user, int):
+            user = User.objects.get(pk=user)
+        with order.event.lock():
+            order.status = Order.STATUS_EXPIRED
+            order.save(update_fields=['status'])
 
-    order.log_action('pretix.event.order.expired', user=user, auth=auth)
-    i = order.invoices.filter(is_cancellation=False).last()
-    if i:
-        generate_cancellation(i)
+        order.log_action('pretix.event.order.expired', user=user, auth=auth)
+        i = order.invoices.filter(is_cancellation=False).last()
+        if i:
+            generate_cancellation(i)
 
+    order_expired.send(order.event, order=order)
     return order
 
 
-@transaction.atomic
 def approve_order(order, user=None, send_mail: bool=True, auth=None):
     """
     Mark this order as approved
     :param order: The order to change
     :param user: The user that performed the change
     """
-    if not order.require_approval or not order.status == Order.STATUS_PENDING:
-        raise OrderError(_('This order is not pending approval.'))
+    with transaction.atomic():
+        if not order.require_approval or not order.status == Order.STATUS_PENDING:
+            raise OrderError(_('This order is not pending approval.'))
 
-    order.require_approval = False
-    order.set_expires(now(), order.event.subevents.filter(id__in=[p.subevent_id for p in order.positions.all()]))
-    order.save(update_fields=['require_approval', 'expires'])
+        order.require_approval = False
+        order.set_expires(now(), order.event.subevents.filter(id__in=[p.subevent_id for p in order.positions.all()]))
+        order.save(update_fields=['require_approval', 'expires'])
 
-    order.log_action('pretix.event.order.approved', user=user, auth=auth)
-    if order.total == Decimal('0.00'):
-        p = order.payments.create(
-            state=OrderPayment.PAYMENT_STATE_CREATED,
-            provider='free',
-            amount=0,
-            fee=None
-        )
-        try:
-            p.confirm(send_mail=False, count_waitinglist=False, user=user, auth=auth)
-        except Quota.QuotaExceededException:
-            raise OrderError(error_messages['unavailable'])
+        order.log_action('pretix.event.order.approved', user=user, auth=auth)
+        if order.total == Decimal('0.00'):
+            p = order.payments.create(
+                state=OrderPayment.PAYMENT_STATE_CREATED,
+                provider='free',
+                amount=0,
+                fee=None
+            )
+            try:
+                p.confirm(send_mail=False, count_waitinglist=False, user=user, auth=auth)
+            except Quota.QuotaExceededException:
+                raise OrderError(error_messages['unavailable'])
+
+    order_approved.send(order.event, order=order)
 
     invoice = order.invoices.last()  # Might be generated by plugin already
     if order.event.settings.get('invoice_generate') == 'True' and invoice_qualified(order):
@@ -234,30 +239,32 @@ def approve_order(order, user=None, send_mail: bool=True, auth=None):
     return order.pk
 
 
-@transaction.atomic
 def deny_order(order, comment='', user=None, send_mail: bool=True, auth=None):
     """
     Mark this order as canceled
     :param order: The order to change
     :param user: The user that performed the change
     """
-    if not order.require_approval or not order.status == Order.STATUS_PENDING:
-        raise OrderError(_('This order is not pending approval.'))
+    with transaction.atomic():
+        if not order.require_approval or not order.status == Order.STATUS_PENDING:
+            raise OrderError(_('This order is not pending approval.'))
 
-    with order.event.lock():
-        order.status = Order.STATUS_CANCELED
-        order.save(update_fields=['status'])
+        with order.event.lock():
+            order.status = Order.STATUS_CANCELED
+            order.save(update_fields=['status'])
 
-    order.log_action('pretix.event.order.denied', user=user, auth=auth, data={
-        'comment': comment
-    })
-    i = order.invoices.filter(is_cancellation=False).last()
-    if i:
-        generate_cancellation(i)
+        order.log_action('pretix.event.order.denied', user=user, auth=auth, data={
+            'comment': comment
+        })
+        i = order.invoices.filter(is_cancellation=False).last()
+        if i:
+            generate_cancellation(i)
 
-    for position in order.positions.all():
-        if position.voucher:
-            Voucher.objects.filter(pk=position.voucher.pk).update(redeemed=Greatest(0, F('redeemed') - 1))
+        for position in order.positions.all():
+            if position.voucher:
+                Voucher.objects.filter(pk=position.voucher.pk).update(redeemed=Greatest(0, F('redeemed') - 1))
+
+    order_denied.send(order.event, order=order)
 
     if send_mail:
         try:
@@ -294,7 +301,6 @@ def deny_order(order, comment='', user=None, send_mail: bool=True, auth=None):
     return order.pk
 
 
-@transaction.atomic
 def _cancel_order(order, user=None, send_mail: bool=True, api_token=None, device=None, oauth_application=None,
                   cancellation_fee=None):
     """
@@ -302,85 +308,87 @@ def _cancel_order(order, user=None, send_mail: bool=True, api_token=None, device
     :param order: The order to change
     :param user: The user that performed the change
     """
-    if isinstance(order, int):
-        order = Order.objects.get(pk=order)
-    if isinstance(user, int):
-        user = User.objects.get(pk=user)
-    if isinstance(api_token, int):
-        api_token = TeamAPIToken.objects.get(pk=api_token)
-    if isinstance(device, int):
-        device = Device.objects.get(pk=device)
-    if isinstance(oauth_application, int):
-        oauth_application = OAuthApplication.objects.get(pk=oauth_application)
-    if isinstance(cancellation_fee, str):
-        cancellation_fee = Decimal(cancellation_fee)
+    with transaction.atomic():
+        if isinstance(order, int):
+            order = Order.objects.get(pk=order)
+        if isinstance(user, int):
+            user = User.objects.get(pk=user)
+        if isinstance(api_token, int):
+            api_token = TeamAPIToken.objects.get(pk=api_token)
+        if isinstance(device, int):
+            device = Device.objects.get(pk=device)
+        if isinstance(oauth_application, int):
+            oauth_application = OAuthApplication.objects.get(pk=oauth_application)
+        if isinstance(cancellation_fee, str):
+            cancellation_fee = Decimal(cancellation_fee)
 
-    if not order.cancel_allowed():
-        raise OrderError(_('You cannot cancel this order.'))
-    i = order.invoices.filter(is_cancellation=False).last()
-    if i:
-        generate_cancellation(i)
+        if not order.cancel_allowed():
+            raise OrderError(_('You cannot cancel this order.'))
+        i = order.invoices.filter(is_cancellation=False).last()
+        if i:
+            generate_cancellation(i)
+
+        if cancellation_fee:
+            with order.event.lock():
+                for position in order.positions.all():
+                    if position.voucher:
+                        Voucher.objects.filter(pk=position.voucher.pk).update(redeemed=Greatest(0, F('redeemed') - 1))
+                    position.canceled = True
+                    position.save(update_fields=['canceled'])
+                for fee in order.fees.all():
+                    fee.canceled = True
+                    fee.save(update_fields=['canceled'])
+
+                f = OrderFee(
+                    fee_type=OrderFee.FEE_TYPE_CANCELLATION,
+                    value=cancellation_fee,
+                    tax_rule=order.event.settings.tax_rate_default,
+                    order=order,
+                )
+                f._calculate_tax()
+                f.save()
+
+                if order.payment_refund_sum < cancellation_fee:
+                    raise OrderError(_('The cancellation fee cannot be higher than the payment credit of this order.'))
+                order.status = Order.STATUS_PAID
+                order.total = f.value
+                order.save(update_fields=['status', 'total'])
+
+            if i:
+                generate_invoice(order)
+        else:
+            with order.event.lock():
+                order.status = Order.STATUS_CANCELED
+                order.save(update_fields=['status'])
 
-    if cancellation_fee:
-        with order.event.lock():
             for position in order.positions.all():
                 if position.voucher:
                     Voucher.objects.filter(pk=position.voucher.pk).update(redeemed=Greatest(0, F('redeemed') - 1))
-                position.canceled = True
-                position.save(update_fields=['canceled'])
-            for fee in order.fees.all():
-                fee.canceled = True
-                fee.save(update_fields=['canceled'])
 
-            f = OrderFee(
-                fee_type=OrderFee.FEE_TYPE_CANCELLATION,
-                value=cancellation_fee,
-                tax_rule=order.event.settings.tax_rate_default,
-                order=order,
-            )
-            f._calculate_tax()
-            f.save()
+        order.log_action('pretix.event.order.canceled', user=user, auth=api_token or oauth_application or device,
+                         data={'cancellation_fee': cancellation_fee})
 
-            if order.payment_refund_sum < cancellation_fee:
-                raise OrderError(_('The cancellation fee cannot be higher than the payment credit of this order.'))
-            order.status = Order.STATUS_PAID
-            order.total = f.value
-            order.save(update_fields=['status', 'total'])
-
-        if i:
-            generate_invoice(order)
-    else:
-        with order.event.lock():
-            order.status = Order.STATUS_CANCELED
-            order.save(update_fields=['status'])
-
-        for position in order.positions.all():
-            if position.voucher:
-                Voucher.objects.filter(pk=position.voucher.pk).update(redeemed=Greatest(0, F('redeemed') - 1))
-
-    order.log_action('pretix.event.order.canceled', user=user, auth=api_token or oauth_application or device,
-                     data={'cancellation_fee': cancellation_fee})
-
-    if send_mail:
-        email_template = order.event.settings.mail_text_order_canceled
-        email_context = {
-            'event': order.event.name,
-            'code': order.code,
-            'url': build_absolute_uri(order.event, 'presale:event.order', kwargs={
-                'order': order.code,
-                'secret': order.secret
-            })
-        }
-        with language(order.locale):
-            email_subject = _('Order canceled: %(code)s') % {'code': order.code}
-            try:
-                order.send_mail(
-                    email_subject, email_template, email_context,
-                    'pretix.event.order.email.order_canceled', user
-                )
-            except SendMailException:
-                logger.exception('Order canceled email could not be sent')
+        if send_mail:
+            email_template = order.event.settings.mail_text_order_canceled
+            email_context = {
+                'event': order.event.name,
+                'code': order.code,
+                'url': build_absolute_uri(order.event, 'presale:event.order', kwargs={
+                    'order': order.code,
+                    'secret': order.secret
+                })
+            }
+            with language(order.locale):
+                email_subject = _('Order canceled: %(code)s') % {'code': order.code}
+                try:
+                    order.send_mail(
+                        email_subject, email_template, email_context,
+                        'pretix.event.order.email.order_canceled', user
+                    )
+                except SendMailException:
+                    logger.exception('Order canceled email could not be sent')
 
+    order_canceled.send(order.event, order=order)
     return order.pk
 
 
@@ -1377,6 +1385,8 @@ class OrderChangeManager:
             if self.split_order:
                 self._notify_user(self.split_order)
 
+        order_changed.send(self.order.event, order=self.order)
+
     def _clear_tickets_cache(self):
         CachedTicket.objects.filter(order_position__order=self.order).delete()
         CachedCombinedTicket.objects.filter(order=self.order).delete()

src/pretix/base/settings.py

@@ -49,6 +49,10 @@ DEFAULTS = {
         'default': 'True',
         'type': bool,
     },
+    'invoice_address_not_asked_free': {
+        'default': 'False',
+        'type': bool,
+    },
     'invoice_name_required': {
         'default': 'False',
         'type': bool,

src/pretix/base/signals.py

@@ -275,6 +275,66 @@ because an already-paid order has been split.
 As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
 """
 
+order_canceled = EventPluginSignal(
+    providing_args=["order"]
+)
+"""
+This signal is sent out every time an order is canceled. The order object is given
+as the first argument.
+
+As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
+"""
+
+order_expired = EventPluginSignal(
+    providing_args=["order"]
+)
+"""
+This signal is sent out every time an order is marked as expired. The order object is given
+as the first argument.
+
+As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
+"""
+
+order_modified = EventPluginSignal(
+    providing_args=["order"]
+)
+"""
+This signal is sent out every time an order's information is modified. The order object is given
+as the first argument.
+
+As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
+"""
+
+order_changed = EventPluginSignal(
+    providing_args=["order"]
+)
+"""
+This signal is sent out every time an order's content is changed. The order object is given
+as the first argument.
+
+As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
+"""
+
+order_approved = EventPluginSignal(
+    providing_args=["order"]
+)
+"""
+This signal is sent out every time an order is being approved. The order object is given
+as the first argument.
+
+As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
+"""
+
+order_denied = EventPluginSignal(
+    providing_args=["order"]
+)
+"""
+This signal is sent out every time an order is being denied. The order object is given
+as the first argument.
+
+As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
+"""
+
 logentry_display = EventPluginSignal(
     providing_args=["logentry"]
 )

src/pretix/base/templates/400.html

@@ -1,6 +1,6 @@
 {% extends "error.html" %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 {% block title %}{% trans "Bad Request" %}{% endblock %}
 {% block content %}
     <i class="fa fa-frown-o fa-fw big-icon"></i>

src/pretix/base/templates/403.html

@@ -1,6 +1,6 @@
 {% extends "error.html" %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 {% block title %}{% trans "Permission denied" %}{% endblock %}
 {% block content %}
     <i class="fa fa-fw fa-lock big-icon"></i>

src/pretix/base/templates/404.html

@@ -1,6 +1,6 @@
 {% extends "error.html" %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 {% block title %}{% trans "Not found" %}{% endblock %}
 {% block content %}
     <i class="fa fa-meh-o fa-fw big-icon"></i>

src/pretix/base/templates/500.html

@@ -1,6 +1,6 @@
 {% extends "error.html" %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 {% block title %}{% trans "Internal Server Error" %}{% endblock %}
 {% block content %}
     <i class="fa fa-bolt big-icon fa-fw"></i>

src/pretix/base/templates/csrffail.html

@@ -1,6 +1,6 @@
 {% extends "error.html" %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 {% block title %}{% trans "Verification failed" %}{% endblock %}
 {% block content %}
     <i class="fa fa-frown-o big-icon fa-fw"></i>

src/pretix/base/templates/error.html

@@ -1,6 +1,6 @@
 {% load compress %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 <!DOCTYPE html>
 <html>
 <head>

src/pretix/base/views/js_catalog.py

@@ -2,7 +2,7 @@ from django.utils import timezone
 from django.utils.translation.trans_real import DjangoTranslation
 from django.views.decorators.cache import cache_page
 from django.views.decorators.http import etag
-from django.views.i18n import JavaScriptCatalog, render_javascript_catalog
+from django.views.i18n import JavaScriptCatalog
 
 # Yes, we want to regenerate this every time the module has been imported to
 # refresh the cache at least at every code deployment
@@ -21,4 +21,5 @@ js_info_dict = {
 def js_catalog(request, lang):
     c = JavaScriptCatalog()
     c.translation = DjangoTranslation(lang, domain='djangojs')
-    return render_javascript_catalog(c.get_catalog(), c.get_plural())
+    context = c.get_context_data()
+    return c.render_to_response(context)

src/pretix/base/views/metrics.py

@@ -20,10 +20,10 @@ def serve_metrics(request):
         return unauthed_response()
 
     # check if the user is properly authorized:
-    if "HTTP_AUTHORIZATION" not in request.META:
+    if "Authorization" not in request.headers:
         return unauthed_response()
 
-    method, credentials = request.META["HTTP_AUTHORIZATION"].split(" ", 1)
+    method, credentials = request.headers["Authorization"].split(" ", 1)
     if method.lower() != "basic":
         return unauthed_response()
 

src/pretix/base/views/mixins.py

@@ -1,5 +1,6 @@
 import json
 from collections import OrderedDict
+from decimal import Decimal
 
 from django import forms
 from django.core.files.uploadedfile import UploadedFile
@@ -186,25 +187,35 @@ class OrderQuestionsViewMixin(BaseQuestionsViewMixin):
         except InvoiceAddress.DoesNotExist:
             return InvoiceAddress(order=self.order)
 
+    @cached_property
+    def address_asked(self):
+        return self.request.event.settings.invoice_address_asked and (
+            self.order.total != Decimal('0.00') or not self.request.event.settings.invoice_address_not_asked_free
+        )
+
     @cached_property
     def invoice_form(self):
-        if not self.request.event.settings.invoice_address_asked and self.request.event.settings.invoice_name_required:
+        if not self.address_asked and self.request.event.settings.invoice_name_required:
             return self.invoice_name_form_class(
                 data=self.request.POST if self.request.method == "POST" else None,
                 event=self.request.event,
                 instance=self.invoice_address, validate_vat_id=False,
                 all_optional=self.all_optional
             )
-        return self.invoice_form_class(
-            data=self.request.POST if self.request.method == "POST" else None,
-            event=self.request.event,
-            instance=self.invoice_address, validate_vat_id=False,
-            all_optional=self.all_optional,
-        )
+        if self.address_asked:
+            return self.invoice_form_class(
+                data=self.request.POST if self.request.method == "POST" else None,
+                event=self.request.event,
+                instance=self.invoice_address, validate_vat_id=False,
+                all_optional=self.all_optional,
+            )
+        else:
+            return forms.Form(data=self.request.POST if self.request.method == "POST" else None)
 
     def get_context_data(self, **kwargs):
         ctx = super().get_context_data(**kwargs)
         ctx['order'] = self.order
         ctx['formgroups'] = self.formdict.items()
         ctx['invoice_form'] = self.invoice_form
+        ctx['invoice_address_asked'] = self.address_asked
         return ctx

src/pretix/control/forms/event.py

@@ -621,6 +621,10 @@ class InvoiceSettingsForm(SettingsForm):
         widget=forms.CheckboxInput(attrs={'data-checkbox-dependency': '#id_invoice_address_asked'}),
         required=False
     )
+    invoice_address_not_asked_free = forms.BooleanField(
+        label=_('Do not ask for invoice address if an order is free'),
+        required=False
+    )
     invoice_include_free = forms.BooleanField(
         label=_("Show free products on invoices"),
         help_text=_("Note that invoices will never be generated for orders that contain only free "

src/pretix/control/forms/item.py

@@ -43,6 +43,7 @@ class QuestionForm(I18nModelForm):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.fields['items'].queryset = self.instance.event.items.all()
+        self.fields['items'].required = True
         self.fields['dependency_question'].queryset = self.instance.event.questions.filter(
             type__in=(Question.TYPE_BOOLEAN, Question.TYPE_CHOICE, Question.TYPE_CHOICE_MULTIPLE)
         )

src/pretix/control/templates/pretixcontrol/auth/base.html

@@ -1,6 +1,6 @@
 {% load compress %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 <!DOCTYPE html>
 <html>
 <head>

src/pretix/control/templates/pretixcontrol/boxoffice/payment.html

@@ -1,11 +1,15 @@
 {% load i18n %}
 
-{% if payment_info %}
-    <dl class="dl-horizontal">
+<dl class="dl-horizontal">
+    <dt>{% trans "Device ID" %}</dt>
+    <dd>{{ payment_info.pos_id }}</dd>
+    <dt>{% trans "Receipt ID" %}</dt>
+    <dd>{{ payment_info.receipt_id }}</dd>
+    {% if payment_info.payment_type == "sumup" %}
         <dt>{% trans "Payment provider" %}</dt>
         <dd>SumUp</dd>
         <dt>{% trans "Transaction Code" %}</dt>
-        <dd>{{ payment_info.payment_data.tx_code}}</dd>
+        <dd>{{ payment_info.payment_data.tx_code }}</dd>
         <dt>{% trans "Merchant Code" %}</dt>
         <dd>{{ payment_info.payment_data.merchant_code }}</dd>
         <dt>{% trans "Currency" %}</dt>
@@ -17,6 +21,8 @@
         <dt>{% trans "Card Entry Mode" %}</dt>
         <dd>{{ payment_info.payment_data.entry_mode }}</dd>
         <dt>{% trans "Card number" %}</dt>
-        <dd><i class="fa fa-cc-{{ payment_info.payment_data.card_type|lower }}"></i> **** **** **** {{ payment_info.payment_data.last4 }}</dd>
-    </dl>
-{% endif %}
+        <dd>
+            <i class="fa fa-cc-{{ payment_info.payment_data.card_type|lower }}"></i> **** **** **** {{ payment_info.payment_data.last4 }}
+        </dd>
+    {% endif %}
+</dl>

src/pretix/control/templates/pretixcontrol/event/invoicing.html

@@ -24,6 +24,7 @@
             {% bootstrap_field form.invoice_address_company_required layout="control" %}
             {% bootstrap_field form.invoice_address_vatid layout="control" %}
             {% bootstrap_field form.invoice_address_beneficiary layout="control" %}
+            {% bootstrap_field form.invoice_address_not_asked_free layout="control" %}
         </fieldset>
         <fieldset>
             <legend>{% trans "Your invoice details" %}</legend>

src/pretix/control/templates/pretixcontrol/order/change_questions.html

@@ -18,7 +18,7 @@
     <form method="post" class="form-horizontal" href="" enctype="multipart/form-data">
         {% csrf_token %}
         <div class="panel-group" id="questions_accordion">
-            {% if request.event.settings.invoice_address_asked or order.invoice_address or request.event.settings.invoice_name_required %}
+            {% if invoice_address_asked or order.invoice_address or request.event.settings.invoice_name_required %}
                 <details class="panel panel-default" open>
                     <summary class="panel-heading">
                         <h4 class="panel-title">

src/pretix/control/templates/pretixcontrol/organizers/device_connect.html

@@ -1,6 +1,6 @@
 {% extends "pretixcontrol/organizers/base.html" %}
 {% load i18n %}
-{% load staticfiles %}
+{% load static %}
 {% load bootstrap3 %}
 {% block inner %}
     <h1>{% trans "Connect to device:" %} {{ device.name }}</h1>

src/pretix/control/views/item.py

@@ -18,8 +18,8 @@ from django.views.generic.edit import DeleteView
 
 from pretix.base.forms import I18nFormSet
 from pretix.base.models import (
-    CachedTicket, Item, ItemCategory, ItemVariation, Order, Question,
-    QuestionAnswer, QuestionOption, Quota, Voucher,
+    CachedTicket, CartPosition, Item, ItemCategory, ItemVariation, Order,
+    Question, QuestionAnswer, QuestionOption, Quota, Voucher,
 )
 from pretix.base.models.event import SubEvent
 from pretix.base.models.items import ItemAddOn, ItemBundle
@@ -49,7 +49,9 @@ class ItemList(ListView):
             event=self.request.event
         ).annotate(
             var_count=Count('variations')
-        ).prefetch_related("category")
+        ).prefetch_related("category").order_by(
+            'category__position', 'category', 'position'
+        )
 
 
 def item_move(request, item, up=True):
@@ -1158,6 +1160,7 @@ class ItemDelete(EventPermissionRequiredMixin, DeleteView):
         success_url = self.get_success_url()
         o = self.get_object()
         if o.allow_delete():
+            CartPosition.objects.filter(addon_to__item=self.get_object()).delete()
             self.get_object().cartposition_set.all().delete()
             self.get_object().log_action('pretix.event.item.deleted', user=self.request.user)
             self.get_object().delete()

src/pretix/control/views/orders.py

@@ -57,7 +57,7 @@ from pretix.base.services.orders import (
 from pretix.base.services.stats import order_overview
 from pretix.base.services.tickets import generate
 from pretix.base.signals import (
-    register_data_exporters, register_ticket_outputs,
+    order_modified, register_data_exporters, register_ticket_outputs,
 )
 from pretix.base.templatetags.money import money_filter
 from pretix.base.templatetags.rich_text import markdown_compile_email
@@ -1305,7 +1305,8 @@ class OrderModifyInformation(OrderQuestionsViewMixin, OrderView):
             messages.error(self.request,
                            _("We had difficulties processing your input. Please review the errors below."))
             return self.get(request, *args, **kwargs)
-        self.invoice_form.save()
+        if hasattr(self.invoice_form, 'save'):
+            self.invoice_form.save()
         self.order.log_action('pretix.event.order.modified', {
             'invoice_data': self.invoice_form.cleaned_data,
             'data': [{
@@ -1320,6 +1321,8 @@ class OrderModifyInformation(OrderQuestionsViewMixin, OrderView):
 
         CachedTicket.objects.filter(order_position__order=self.order).delete()
         CachedCombinedTicket.objects.filter(order=self.order).delete()
+
+        order_modified.send(sender=self.request.event, order=self.order)
         return redirect(self.get_order_url())
 
 

src/pretix/control/views/subevents.py

@@ -16,6 +16,7 @@ from django.utils.translation import pgettext_lazy, ugettext_lazy as _
 from django.views import View
 from django.views.generic import CreateView, DeleteView, ListView, UpdateView
 
+from pretix.base.models import CartPosition
 from pretix.base.models.checkin import CheckinList
 from pretix.base.models.event import SubEvent, SubEventMetaValue
 from pretix.base.models.items import (
@@ -117,6 +118,7 @@ class SubEventDelete(EventPermissionRequiredMixin, DeleteView):
             return HttpResponseRedirect(self.get_success_url())
         else:
             self.object.log_action('pretix.subevent.deleted', user=self.request.user)
+            CartPosition.objects.filter(addon_to__subevent=self.object).delete()
             self.object.cartposition_set.all().delete()
             self.object.delete()
             messages.success(request, pgettext_lazy('subevent', 'The selected date has been deleted.'))
@@ -512,6 +514,7 @@ class SubEventBulkAction(EventPermissionRequiredMixin, View):
         elif request.POST.get('action') == 'delete_confirm':
             for obj in self.objects:
                 if obj.allow_delete():
+                    CartPosition.objects.filter(addon_to__subevent=obj).delete()
                     obj.cartposition_set.all().delete()
                     obj.log_action('pretix.subevent.deleted', user=self.request.user)
                     obj.delete()

src/pretix/control/views/vouchers.py

@@ -17,7 +17,7 @@ from django.views.generic import (
     CreateView, DeleteView, ListView, TemplateView, UpdateView, View,
 )
 
-from pretix.base.models import LogEntry, Voucher
+from pretix.base.models import CartPosition, LogEntry, OrderPosition, Voucher
 from pretix.base.models.vouchers import _generate_random_code
 from pretix.control.forms.filter import VoucherFilterForm
 from pretix.control.forms.vouchers import VoucherBulkForm, VoucherForm
@@ -143,6 +143,7 @@ class VoucherDelete(EventPermissionRequiredMixin, DeleteView):
             messages.error(request, _('A voucher can not be deleted if it already has been redeemed.'))
         else:
             self.object.log_action('pretix.voucher.deleted', user=self.request.user)
+            CartPosition.objects.filter(addon_to__voucher=False).delete()
             self.object.cartposition_set.all().delete()
             self.object.delete()
             messages.success(request, _('The selected voucher has been deleted.'))
@@ -348,6 +349,7 @@ class VoucherBulkAction(EventPermissionRequiredMixin, View):
             for obj in self.objects:
                 if obj.allow_delete():
                     obj.log_action('pretix.voucher.deleted', user=self.request.user)
+                    OrderPosition.objects.filter(addon_to__voucher=obj).delete()
                     obj.cartposition_set.all().delete()
                     obj.delete()
                 else:

src/pretix/helpers/security.py

@@ -13,7 +13,7 @@ class SessionReauthRequired(Exception):
 
 
 def get_user_agent_hash(request):
-    return hashlib.sha256(request.META['HTTP_USER_AGENT'].encode()).hexdigest()
+    return hashlib.sha256(request.headers['User-Agent'].encode()).hexdigest()
 
 
 def assert_session_valid(request):
@@ -26,7 +26,7 @@ def assert_session_valid(request):
         if time.time() - last_used > settings.PRETIX_SESSION_TIMEOUT_RELATIVE:
             raise SessionReauthRequired()
 
-    if 'HTTP_USER_AGENT' in request.META:
+    if 'User-Agent' in request.headers:
         if 'pinned_user_agent' in request.session:
             if request.session.get('pinned_user_agent') != get_user_agent_hash(request):
                 raise SessionInvalid()

src/pretix/multidomain/middlewares.py

@@ -23,10 +23,10 @@ LOCAL_HOST_NAMES = ('testserver', 'localhost')
 class MultiDomainMiddleware(MiddlewareMixin):
     def process_request(self, request):
         # We try three options, in order of decreasing preference.
-        if settings.USE_X_FORWARDED_HOST and ('HTTP_X_FORWARDED_HOST' in request.META):
-            host = request.META['HTTP_X_FORWARDED_HOST']
-        elif 'HTTP_HOST' in request.META:
-            host = request.META['HTTP_HOST']
+        if settings.USE_X_FORWARDED_HOST and ('X-Forwarded-Host' in request.headers):
+            host = request.headers['X-Forwarded-Host']
+        elif 'Host' in request.headers:
+            host = request.headers['Host']
         else:
             # Reconstruct the host using the algorithm from PEP 333.
             host = request.META['SERVER_NAME']

src/pretix/plugins/badges/models.py

@@ -52,3 +52,6 @@ class BadgeItem(models.Model):
                                 on_delete=models.CASCADE)
     layout = models.ForeignKey('BadgeLayout', on_delete=models.CASCADE, related_name='item_assignments',
                                null=True, blank=True)
+
+    class Meta:
+        ordering = ('id',)

src/pretix/plugins/banktransfer/models.py

@@ -21,6 +21,9 @@ class BankImportJob(models.Model):
     created = models.DateTimeField(auto_now_add=True)
     state = models.CharField(max_length=32, choices=STATES, default=STATE_PENDING)
 
+    class Meta:
+        ordering = ('id',)
+
     @property
     def owner_kwargs(self):
         if self.event:
@@ -76,3 +79,4 @@ class BankTransaction(models.Model):
 
     class Meta:
         unique_together = ('event', 'organizer', 'checksum')
+        ordering = ('date', 'id')

src/pretix/plugins/stripe/payment.py

@@ -321,6 +321,10 @@ class StripeMethod(BasePaymentProvider):
                 amount=self._get_amount(payment),
                 currency=self.event.currency.lower(),
                 source=source,
+                description='{event}-{code}'.format(
+                    event=self.event.slug.upper(),
+                    code=payment.order.code
+                ),
                 metadata={
                     'order': str(payment.order.id),
                     'event': self.event.id,

src/pretix/plugins/ticketoutputpdf/models.py

@@ -73,3 +73,4 @@ class TicketLayoutItem(models.Model):
 
     class Meta:
         unique_together = (('item', 'layout', 'sales_channel'),)
+        ordering = ("id",)

src/pretix/presale/checkoutflow.py

@@ -28,7 +28,9 @@ from pretix.presale.signals import (
     checkout_all_optional, checkout_confirm_messages, checkout_flow_steps,
     contact_form_fields, order_meta_from_request, question_form_fields,
 )
-from pretix.presale.views import CartMixin, get_cart, get_cart_total
+from pretix.presale.views import (
+    CartMixin, get_cart, get_cart_is_free, get_cart_total,
+)
 from pretix.presale.views.cart import (
     cart_session, create_empty_cart_id, get_or_create_cart_id,
 )
@@ -338,20 +340,23 @@ class QuestionsStep(QuestionsViewMixin, CartMixin, TemplateFlowStep):
 
     @cached_property
     def invoice_form(self):
-        initial = {
-            'name_parts': {
-                k[21:].replace('-', '_'): v
-                for k, v in self.cart_session.get('widget_data', {}).items()
-                if k.startswith('invoice-address-name-')
-            },
-            'company': self.cart_session.get('widget_data', {}).get('invoice-address-company', ''),
-            'is_business': bool(self.cart_session.get('widget_data', {}).get('invoice-address-company', '')),
-            'street': self.cart_session.get('widget_data', {}).get('invoice-address-street', ''),
-            'zipcode': self.cart_session.get('widget_data', {}).get('invoice-address-zipcode', ''),
-            'city': self.cart_session.get('widget_data', {}).get('invoice-address-city', ''),
-            'country': self.cart_session.get('widget_data', {}).get('invoice-address-country', ''),
-        }
-        if not self.request.event.settings.invoice_address_asked and self.request.event.settings.invoice_name_required:
+        if not self.invoice_address.pk:
+            initial = {
+                'name_parts': {
+                    k[21:].replace('-', '_'): v
+                    for k, v in self.cart_session.get('widget_data', {}).items()
+                    if k.startswith('invoice-address-name-')
+                },
+                'company': self.cart_session.get('widget_data', {}).get('invoice-address-company', ''),
+                'is_business': bool(self.cart_session.get('widget_data', {}).get('invoice-address-company', '')),
+                'street': self.cart_session.get('widget_data', {}).get('invoice-address-street', ''),
+                'zipcode': self.cart_session.get('widget_data', {}).get('invoice-address-zipcode', ''),
+                'city': self.cart_session.get('widget_data', {}).get('invoice-address-city', ''),
+                'country': self.cart_session.get('widget_data', {}).get('invoice-address-country', ''),
+            }
+        else:
+            initial = {}
+        if not self.address_asked and self.request.event.settings.invoice_name_required:
             return InvoiceNameForm(data=self.request.POST if self.request.method == "POST" else None,
                                    event=self.request.event,
                                    request=self.request,
@@ -365,10 +370,17 @@ class QuestionsStep(QuestionsViewMixin, CartMixin, TemplateFlowStep):
                                   instance=self.invoice_address,
                                   validate_vat_id=self.eu_reverse_charge_relevant, all_optional=self.all_optional)
 
+    @cached_property
+    def address_asked(self):
+        return (
+            self.request.event.settings.invoice_address_asked
+            and (not self.request.event.settings.invoice_address_not_asked_free or not get_cart_is_free(self.request))
+        )
+
     def post(self, request):
         self.request = request
         failed = not self.save() or not self.contact_form.is_valid()
-        if request.event.settings.invoice_address_asked or self.request.event.settings.invoice_name_required:
+        if self.address_asked or self.request.event.settings.invoice_name_required:
             failed = failed or not self.invoice_form.is_valid()
         if failed:
             messages.error(request,
@@ -376,7 +388,7 @@ class QuestionsStep(QuestionsViewMixin, CartMixin, TemplateFlowStep):
             return self.render()
         self.cart_session['email'] = self.contact_form.cleaned_data['email']
         self.cart_session['contact_form_data'] = self.contact_form.cleaned_data
-        if request.event.settings.invoice_address_asked or self.request.event.settings.invoice_name_required:
+        if self.address_asked or self.request.event.settings.invoice_name_required:
             addr = self.invoice_form.save()
             self.cart_session['invoice_address'] = addr.pk
 
@@ -404,9 +416,11 @@ class QuestionsStep(QuestionsViewMixin, CartMixin, TemplateFlowStep):
             return False
 
         if not self.all_optional:
-            if request.event.settings.invoice_address_required and (not self.invoice_address or not self.invoice_address.street):
-                messages.warning(request, _('Please enter your invoicing address.'))
-                return False
+
+            if self.address_asked:
+                if request.event.settings.invoice_address_required and (not self.invoice_address or not self.invoice_address.street):
+                    messages.warning(request, _('Please enter your invoicing address.'))
+                    return False
 
             if request.event.settings.invoice_name_required and (not self.invoice_address or not self.invoice_address.name):
                 messages.warning(request, _('Please enter your name.'))
@@ -471,6 +485,7 @@ class QuestionsStep(QuestionsViewMixin, CartMixin, TemplateFlowStep):
         ctx['reverse_charge_relevant'] = self.eu_reverse_charge_relevant
         ctx['cart'] = self.get_cart()
         ctx['cart_session'] = self.cart_session
+        ctx['invoice_address_asked'] = self.address_asked
         return ctx
 
 
@@ -591,6 +606,13 @@ class ConfirmStep(CartMixin, AsyncAction, TemplateFlowStep):
     def is_completed(self, request, warn=False):
         pass
 
+    @cached_property
+    def address_asked(self):
+        return (
+            self.request.event.settings.invoice_address_asked
+            and (not self.request.event.settings.invoice_address_not_asked_free or not get_cart_is_free(self.request))
+        )
+
     def get_context_data(self, **kwargs):
         ctx = super().get_context_data(**kwargs)
         ctx['cart'] = self.get_cart(answers=True)
@@ -601,6 +623,7 @@ class ConfirmStep(CartMixin, AsyncAction, TemplateFlowStep):
         ctx['addr'] = self.invoice_address
         ctx['confirm_messages'] = self.confirm_messages
         ctx['cart_session'] = self.cart_session
+        ctx['invoice_address_asked'] = self.address_asked
 
         email = self.cart_session.get('contact_form_data', {}).get('email')
         if email != settings.PRETIX_EMAIL_NONE_VALUE:

src/pretix/presale/templates/pretixpresale/event/checkout_confirm.html

@@ -72,7 +72,7 @@
         {% endif %}
         {% eventsignal event "pretix.presale.signals.checkout_confirm_page_content" request=request %}
         <div class="row">
-            {% if request.event.settings.invoice_address_asked %}
+            {% if invoice_address_asked %}
                 <div class="col-md-6 col-xs-12">
                     <div class="panel panel-primary panel-contact">
                         <div class="panel-heading">
@@ -111,7 +111,7 @@
                     </div>
                 </div>
             {% endif %}
-            <div class="{% if request.event.settings.invoice_address_asked %}col-md-6{% endif %} col-xs-12">
+            <div class="{% if invoice_address_asked %}col-md-6{% endif %} col-xs-12">
                 <div class="panel panel-primary panel-contact">
                     <div class="panel-heading">
                         <div class="pull-right">
@@ -125,7 +125,7 @@
                         </h3>
                     </div>
                     <div class="panel-body">
-                        {% if not event.settings.invoice_address_asked and event.settings.invoice_name_required %}
+                        {% if not asked and event.settings.invoice_name_required %}
                             <dl class="dl-horizontal">
                                 <dt>{% trans "Name" %}</dt>
                                 <dd>{{ addr.name }}</dd>

src/pretix/presale/templates/pretixpresale/event/checkout_questions.html

@@ -22,13 +22,13 @@
                 <div id="contact">
                     <div class="panel-body">
                         {% bootstrap_form contact_form layout="horizontal" %}
-                        {% if not event.settings.invoice_address_asked and event.settings.invoice_name_required %}
+                        {% if not invoice_address_asked and event.settings.invoice_name_required %}
                             {% bootstrap_form invoice_form layout="horizontal" %}
                         {% endif %}
                     </div>
                 </div>
             </details>
-            {% if event.settings.invoice_address_asked %}
+            {% if invoice_address_asked %}
                 <details class="panel panel-default" {% if event.settings.invoice_address_required or event.settings.invoice_name_required %}open{% endif %}>
                     <summary class="panel-heading">
                         <h4 class="panel-title">

src/pretix/presale/templates/pretixpresale/event/order.html

@@ -159,7 +159,7 @@
     {% eventsignal event "pretix.presale.signals.order_info" order=order %}
     <div class="row">
         {% if invoices %}
-            <div class="col-xs-12 {% if request.event.settings.invoice_address_asked or request.event.settings.invoice_name_required %}col-md-6{% endif %}">
+            <div class="col-xs-12 {% if invoice_address_asked or request.event.settings.invoice_name_required %}col-md-6{% endif %}">
                 <div class="panel panel-primary">
                     <div class="panel-heading">
                         <h3 class="panel-title">
@@ -180,7 +180,7 @@
                 </div>
             </div>
         {% elif can_generate_invoice %}
-            <div class="col-xs-12 {% if request.event.settings.invoice_address_asked or request.event.settings.invoice_name_required %}col-md-6{% endif %}">
+            <div class="col-xs-12 {% if invoice_address_asked or request.event.settings.invoice_name_required %}col-md-6{% endif %}">
                 <div class="panel panel-primary">
                     <div class="panel-heading">
                         <h3 class="panel-title">
@@ -199,7 +199,7 @@
                 </div>
             </div>
         {% endif %}
-        {% if request.event.settings.invoice_address_asked or request.event.settings.invoice_name_required %}
+        {% if invoice_address_asked or request.event.settings.invoice_name_required %}
             <div class="col-xs-12 {% if invoices or can_generate_invoice %}col-md-6{% endif %}">
                 <div class="panel panel-primary">
                     <div class="panel-heading">
@@ -221,13 +221,13 @@
                     </div>
                     <div class="panel-body">
                         <dl class="dl-horizontal">
-                            {% if request.event.settings.invoice_address_asked %}
+                            {% if invoice_address_asked %}
                                 <dt>{% trans "Company" %}</dt>
                                 <dd>{{ order.invoice_address.company }}</dd>
                             {% endif %}
                             <dt>{% trans "Name" %}</dt>
                             <dd>{{ order.invoice_address.name }}</dd>
-                            {% if request.event.settings.invoice_address_asked %}
+                            {% if invoice_address_asked %}
                                 <dt>{% trans "Address" %}</dt>
                                 <dd>{{ order.invoice_address.street|linebreaksbr }}</dd>
                                 <dt>{% trans "ZIP code and city" %}</dt>

src/pretix/presale/templates/pretixpresale/event/order_cancel.html

@@ -15,22 +15,24 @@
         {% endblocktrans %}
         {% trans "This will invalidate all of your tickets." %}
     </p>
-    {% if can_auto_refund %}
-        <p>
-            <strong>
+    {% if refund_amount %}
+        {% if can_auto_refund %}
+            <p>
+                <strong>
+                    {% blocktrans trimmed with amount=refund_amount|money:request.event.currency %}
+                        The refund amount of {{ amount }} will automatically be sent back to your original payment method. Depending on the payment method,
+                        please allow for up to two weeks before this appears on your statement.
+                    {% endblocktrans %}
+                </strong>
+            </p>
+        {% else %}
+            <div class="alert alert-warning">
                 {% blocktrans trimmed with amount=refund_amount|money:request.event.currency %}
-                    The refund amount of {{ amount }} will automatically be sent back to your original payment method. Depending on the payment method,
-                    please allow for up to two weeks before this appears on your statement.
+                    With to the payment method you used, the refund amount of {{ amount }} <strong>can not be sent back to you automatically</strong>. Instead, the
+                    event organizer will need to initiate the transfer manually. Please be patient as this might take a bit longer.
                 {% endblocktrans %}
-            </strong>
-        </p>
-    {% else %}
-        <div class="alert alert-warning">
-            {% blocktrans trimmed with amount=refund_amount|money:request.event.currency %}
-                With to the payment method you used, the refund amount of {{ amount }} <strong>can not be sent back to you automatically</strong>. Instead, the
-                event organizer will need to initiate the transfer manually. Please be patient as this might take a bit longer.
-            {% endblocktrans %}
-        </div>
+            </div>
+        {% endif %}
     {% endif %}
 
     <form method="post" action="{% eventurl request.event "presale:event.order.cancel.do" secret=order.secret order=order.code %}" data-asynctask>

src/pretix/presale/templates/pretixpresale/event/order_modify.html

@@ -12,8 +12,8 @@
     <form class="form-horizontal" method="post" enctype="multipart/form-data">
         {% csrf_token %}
         <div class="panel-group" id="questions_accordion">
-            {% if event.settings.invoice_address_asked or event.settings.invoice_name_required %}
-                {% if event.settings.invoice_address_asked %}
+            {% if invoice_address_asked or event.settings.invoice_name_required %}
+                {% if invoice_address_asked %}
                     <div class="alert alert-info">
                         {% blocktrans trimmed %}
                             Modifying your invoice address will not automatically generate a new invoice.
@@ -25,7 +25,7 @@
                     <summary class="panel-heading">
                         <h4 class="panel-title">
                             <strong>
-                                {% if request.event.settings.invoice_address_asked %}
+                                {% if invoice_address_asked %}
                                     {% trans "Invoice information" %}{% if not event.settings.invoice_address_required %}
                                         {% trans "(optional)" %}
                                     {% endif %}

src/pretix/presale/views/__init__.py

@@ -205,6 +205,34 @@ def get_cart_total(request):
     return request._cart_total_cache
 
 
+def get_cart_invoice_address(request):
+    from pretix.presale.views.cart import cart_session
+
+    if not hasattr(request, '_checkout_flow_invoice_address'):
+        cs = cart_session(request)
+        iapk = cs.get('invoice_address')
+        if not iapk:
+            request._checkout_flow_invoice_address = InvoiceAddress()
+        else:
+            try:
+                request._checkout_flow_invoice_address = InvoiceAddress.objects.get(pk=iapk, order__isnull=True)
+            except InvoiceAddress.DoesNotExist:
+                request._checkout_flow_invoice_address = InvoiceAddress()
+    return request._checkout_flow_invoice_address
+
+
+def get_cart_is_free(request):
+    from pretix.presale.views.cart import cart_session
+
+    if not hasattr(request, '_cart_free_cache'):
+        cs = cart_session(request)
+        ia = get_cart_invoice_address(request)
+        total = get_cart_total(request)
+        fees = get_fees(request.event, request, total, ia, cs.get('payment'))
+        request._cart_free_cache = total + sum(f.value for f in fees) == Decimal('0.00')
+    return request._cart_free_cache
+
+
 class EventViewMixin:
     def get_context_data(self, **kwargs):
         context = super().get_context_data(**kwargs)

src/pretix/presale/views/locale.py

@@ -11,7 +11,7 @@ from .robots import NoSearchIndexViewMixin
 class LocaleSet(NoSearchIndexViewMixin, View):
 
     def get(self, request, *args, **kwargs):
-        url = request.GET.get('next', request.META.get('HTTP_REFERER', '/'))
+        url = request.GET.get('next', request.headers.get('Referer', '/'))
         url = url if is_safe_url(url, allowed_hosts=[request.get_host()]) else '/'
         resp = HttpResponseRedirect(url)
 

src/pretix/presale/views/order.py

@@ -30,7 +30,9 @@ from pretix.base.services.invoices import (
 from pretix.base.services.mail import SendMailException
 from pretix.base.services.orders import cancel_order, change_payment_provider
 from pretix.base.services.tickets import generate
-from pretix.base.signals import allow_ticket_download, register_ticket_outputs
+from pretix.base.signals import (
+    allow_ticket_download, order_modified, register_ticket_outputs,
+)
 from pretix.base.views.mixins import OrderQuestionsViewMixin
 from pretix.base.views.tasks import AsyncAction
 from pretix.helpers.safedownload import check_token
@@ -122,6 +124,9 @@ class OrderDetails(EventViewMixin, OrderDetailMixin, CartMixin, TemplateView):
                 'secret': self.order.secret
             }
         )
+        ctx['invoice_address_asked'] = self.request.event.settings.invoice_address_asked and (
+            self.order.total != Decimal('0.00') or not self.request.event.settings.invoice_address_not_asked_free
+        )
 
         if self.order.status == Order.STATUS_PENDING:
             ctx['pending_sum'] = self.order.pending_sum
@@ -526,7 +531,8 @@ class OrderModify(EventViewMixin, OrderDetailMixin, OrderQuestionsViewMixin, Tem
             messages.error(self.request,
                            _("We had difficulties processing your input. Please review the errors below."))
             return self.get(request, *args, **kwargs)
-        self.invoice_form.save()
+        if hasattr(self.invoice_form, 'save'):
+            self.invoice_form.save()
         self.order.log_action('pretix.event.order.modified', {
             'invoice_data': self.invoice_form.cleaned_data,
             'data': [{
@@ -536,6 +542,7 @@ class OrderModify(EventViewMixin, OrderDetailMixin, OrderQuestionsViewMixin, Tem
                 for k in f.changed_data
             } for f in self.forms]
         })
+        order_modified.send(sender=self.request.event, order=self.order)
         if self.invoice_form.has_changed():
             success_message = ('Your invoice address has been updated. Please contact us if you need us '
                                'to regenerate your invoice.')
@@ -579,7 +586,7 @@ class OrderCancel(EventViewMixin, OrderDetailMixin, TemplateView):
     def get_context_data(self, **kwargs):
         ctx = super().get_context_data(**kwargs)
         ctx['order'] = self.order
-        refund_amount = self.order.total - self.order.user_cancel_fee
+        refund_amount = self.order.payment_refund_sum - self.order.user_cancel_fee
         proposals = self.order.propose_auto_refunds(refund_amount)
         ctx['refund_amount'] = refund_amount
         ctx['can_auto_refund'] = sum(proposals.values()) == refund_amount

src/pretix/settings.py

@@ -329,6 +329,7 @@ CORE_MODULES = {
 }
 
 MIDDLEWARE = [
+    'pretix.api.middleware.IdempotencyMiddleware',
     'django.middleware.common.CommonMiddleware',
     'pretix.multidomain.middlewares.MultiDomainMiddleware',
     'pretix.multidomain.middlewares.SessionMiddleware',

src/pretix/static/pretixpresale/js/ui/main.js

@@ -153,6 +153,28 @@ $(function () {
         copy_answers(idx);
         return false;
     });
+    var copy_to_first_ticket = true;
+    $("input[id*=attendee_name_parts_], input[id*=attendee_email]").each(function () {
+        if ($(this).val()) {
+            copy_to_first_ticket = false;
+        }
+    })
+    $("input[id^=id_name_parts_], #id_email").change(function () {
+        console.log(copy_to_first_ticket);
+        console.log($(".questions-form").first().select("input[id*=attendee_email]"));
+        console.log($("#id_email").val());
+        if (copy_to_first_ticket) {
+            $(".questions-form").first().find("input[id*=attendee_email]").val($("#id_email").val());
+            $(".questions-form").first().find("input[id*=attendee_name_parts]").each(function () {
+                var parts = $(this).attr("id").split("_");
+                var num = parts[parts.length - 1];
+                $(this).val($("#id_name_parts_" + num).val());
+            });
+        }
+    });
+    $("input[id*=attendee_name_parts_], input[id*=attendee_email]").change(function () {
+        copy_to_first_ticket = false;
+    });
 
     // Subevent choice
     if ($(".subevent-toggle").length) {

src/pretix/static/pretixpresale/scss/_cart.scss

@@ -63,6 +63,14 @@
         padding-left: 10px;
         font-weight: bold;
     }
+
+    dl {
+      margin: 5px 0;
+        dt {
+            font-weight: normal;
+            margin-top: 5px;
+        }
+    }
 }
 
 @media(max-width: $screen-sm-max) {

src/pretix/testutils/settings.py

@@ -18,9 +18,16 @@ atexit.register(tmpdir.cleanup)
 EMAIL_BACKEND = 'django.core.mail.outbox'
 
 COMPRESS_ENABLED = COMPRESS_OFFLINE = False
+COMPRESS_CACHE_BACKEND = 'testcache'
 STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
 PRETIX_INSTANCE_NAME = 'pretix.eu'
 
+COMPRESS_PRECOMPILERS_ORIGINAL = COMPRESS_PRECOMPILERS
+COMPRESS_PRECOMPILERS = ()
+TEMPLATES[0]['OPTIONS']['loaders'] = (
+    ('django.template.loaders.cached.Loader', template_loaders),
+)
+
 DEBUG = True
 DEBUG_PROPAGATE_EXCEPTIONS = True
 

src/requirements/dev.txt

@@ -1,5 +1,5 @@
-django-debug-toolbar==1.9.1
-sqlparse==0.2.1  # pinned due to difficulties with django-debug-toolbar
+django-debug-toolbar==1.11
+sqlparse==0.3.*  # pinned due to difficulties with django-debug-toolbar
 # Testing requirements
 pycodestyle==2.5.*
 pyflakes==2.1.*
@@ -7,15 +7,16 @@ pep8-naming
 flake8==3.7.*
 codecov
 coverage
-pytest==3.6.*
+pytest==4.4.*
 pytest-django
-pytest-xdist
 isort
-pytest-rerunfailures==4.*
-pytest-mock==1.6.*
-pytest-cache
-pytest-sugar
+pytest-rerunfailures==7.*
+pytest-mock==1.10.*
 responses
 potypo
 freezegun
 
+# Not really required, just nice to have
+pytest-xdist==1.28.*
+pytest-cache
+pytest-sugar

src/requirements/production.txt

@@ -1,23 +1,23 @@
 # Functional requirements
-Django>=2.1,<2.2
-djangorestframework==3.8.*
-python-dateutil
+Django==2.2.*
+djangorestframework==3.9.*
+python-dateutil==2.8.*
 pytz
-django-bootstrap3==10.0.*
+django-bootstrap3==11.0.*
 django-formset-js-improved==0.5.0.2
 django-compressor==2.2.*
 django-hierarkey==1.0.*,>=1.0.3
-django-filter==2.0.*
+django-filter==2.1.*
 reportlab==3.5.*
 PyPDF2==1.26.*
 Pillow==5.*
 django-libsass
 libsass
-django-otp==0.4.*
+django-otp==0.5.*
 python-u2flib-server==4.*
 django-formtools==2.1
-celery>=4.1.1,<4.2.0
-kombu==4.2.*
+celery==4.3.*
+kombu==4.5.*
 django-statici18n==1.8.*
 inlinestyler==0.2.*
 BeautifulSoup4==4.7.*
@@ -28,7 +28,7 @@ dj-static
 csscompressor
 django-markup
 markdown<=2.2
-bleach==2.*
+bleach==3.1.*
 sentry-sdk==0.7.*
 babel
 django-i18nfield>=1.4.0
@@ -54,5 +54,5 @@ defusedcsv>=1.0.1
 vat_moss==0.11.0
 django-localflavor
 idna==2.6 # required by current requests
-django-redis==4.8.*
-redis==2.10.5
+django-redis==4.10.*
+redis==3.2.*

src/setup.cfg

@@ -18,6 +18,9 @@ skip = make_testdata.py,wsgi.py,bootstrap,celery_app.py,pretix/settings.py,tests
 [tool:pytest]
 DJANGO_SETTINGS_MODULE=tests.settings
 addopts =--reruns 3 -rw
+filterwarnings =
+    ignore:The 'warn' method is deprecated:DeprecationWarning
+    ignore:django.contrib.staticfiles.templatetags.static:DeprecationWarning
 
 [coverage:run]
 source = pretix

src/setup.py

@@ -87,25 +87,25 @@ setup(
 
     keywords='tickets web shop ecommerce',
     install_requires=[
-        'Django>=2.1,<2.2',
-        'djangorestframework==3.8.*',
-        'python-dateutil==2.4.*',
+        'Django==2.2.*',
+        'djangorestframework==3.9.*',
+        'python-dateutil==2.8.*',
         'pytz',
-        'django-bootstrap3==10.0.*',
+        'django-bootstrap3==11.0.*',
         'django-formset-js-improved==0.5.0.2',
         'django-compressor==2.2.*',
         'django-hierarkey==1.0.*,>=1.0.2',
-        'django-filter==2.0.*',
+        'django-filter==2.1.*',
         'reportlab==3.5.*',
         'Pillow==5.*',
         'PyPDF2==1.26.*',
         'django-libsass',
         'libsass',
-        'django-otp==0.4.*',
+        'django-otp==0.5.*',
         'python-u2flib-server==4.*',
         'django-formtools==2.1',
-        'celery>=4.1.1,<4.2.0',
-        'kombu==4.2.*',
+        'celery==4.3.*',
+        'kombu==4.5.*',
         'django-statici18n==1.8.*',
         'inlinestyler==0.2.*',
         'BeautifulSoup4==4.7.*',
@@ -116,13 +116,13 @@ setup(
         'csscompressor',
         'django-markup',
         'markdown<=2.2',
-        'bleach==2.*',
+        'bleach==3.1.*',
         'sentry-sdk==0.7.*',
         'babel',
         'paypalrestsdk==1.13.*',
         'pycparser==2.13',
-        'django-redis==4.8.*',
-        'redis==2.10.5',
+        'django-redis==4.10.*',
+        'redis==3.2.*',
         'stripe==2.0.*',
         'chardet<3.1.0,>=3.0.2',
         'mt-940==3.2',
@@ -144,21 +144,22 @@ setup(
     ],
     extras_require={
         'dev': [
-            'django-debug-toolbar==1.9.1',
-            'sqlparse==0.2.1',
+            'django-debug-toolbar==1.11',
+            'sqlparse==0.3.*',
             'pycodestyle==2.5.*',
             'pyflakes==2.1.*',
             'flake8==3.7.*',
             'pep8-naming',
             'coveralls',
             'coverage',
-            'pytest==3.6.*',
+            'pytest==4.4.*',
             'pytest-django',
-            'pytest-xdist',
+            'pytest-xdist==1.28.*',
             'isort',
-            'pytest-mock==1.6.*',
-            'pytest-rerunfailures',
+            'pytest-mock==1.10.*',
+            'pytest-rerunfailures==7.*',
             'responses',
+            'potypo',
             'freezegun',
         ],
         'memcached': ['pylibmc'],

src/tests/api/test_cart.py

@@ -76,8 +76,8 @@ def test_cp_list(token_client, organizer, event, item, taxrule, question):
         cr = CartPosition.objects.create(
             event=event, cart_id="aaa", item=item,
             price=23, attendee_name_parts={'full_name': 'Peter'},
-            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0),
-            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0)
+            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC),
+            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC)
         )
     res = dict(TEST_CARTPOSITION_RES)
     res["id"] = cr.pk
@@ -97,8 +97,8 @@ def test_cp_list_api(token_client, organizer, event, item, taxrule, question):
         cr = CartPosition.objects.create(
             event=event, cart_id="aaa@api", item=item,
             price=23, attendee_name_parts={'full_name': 'Peter'},
-            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0),
-            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0)
+            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC),
+            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC)
         )
     res = dict(TEST_CARTPOSITION_RES)
     res["id"] = cr.pk
@@ -118,8 +118,8 @@ def test_cp_detail(token_client, organizer, event, item, taxrule, question):
         cr = CartPosition.objects.create(
             event=event, cart_id="aaa@api", item=item,
             price=23, attendee_name_parts={'full_name': 'Peter'},
-            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0),
-            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0)
+            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC),
+            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC)
         )
     res = dict(TEST_CARTPOSITION_RES)
     res["id"] = cr.pk
@@ -139,8 +139,8 @@ def test_cp_delete(token_client, organizer, event, item, taxrule, question):
         cr = CartPosition.objects.create(
             event=event, cart_id="aaa@api", item=item,
             price=23, attendee_name_parts={'full_name': 'Peter'},
-            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0),
-            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0)
+            datetime=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC),
+            expires=datetime.datetime(2018, 6, 11, 10, 0, 0, 0, tzinfo=UTC)
         )
     res = dict(TEST_CARTPOSITION_RES)
     res["id"] = cr.pk
@@ -537,7 +537,7 @@ def test_cartpos_create_answer_validation(token_client, organizer, event, item,
     )
     assert resp.status_code == 400
     assert resp.data == {
-        'answers': [{'non_field_errors': ['Date has wrong format. Use one of these formats instead: YYYY[-MM[-DD]].']}]}
+        'answers': [{'non_field_errors': ['Date has wrong format. Use one of these formats instead: YYYY-MM-DD.']}]}
 
     question.type = Question.TYPE_DATETIME
     question.save()

src/tests/api/test_events.py

@@ -89,10 +89,10 @@ TEST_EVENT_RES = {
     "slug": "dummy",
     "has_subevents": False,
     "meta_data": {"type": "Conference"},
-    'plugins': {
+    'plugins': [
         'pretix.plugins.banktransfer',
         'pretix.plugins.ticketoutputpdf'
-    }
+    ]
 }
 
 
@@ -573,7 +573,7 @@ def test_event_update_plugins(token_client, organizer, event, free_item, free_qu
         format='json'
     )
     assert resp.status_code == 200
-    assert resp.data.get('plugins') == {
+    assert set(resp.data.get('plugins')) == {
         "pretix.plugins.ticketoutputpdf",
         "pretix.plugins.pretixdroid"
     }
@@ -588,9 +588,9 @@ def test_event_update_plugins(token_client, organizer, event, free_item, free_qu
         format='json'
     )
     assert resp.status_code == 200
-    assert resp.data.get('plugins') == {
+    assert resp.data.get('plugins') == [
         "pretix.plugins.banktransfer"
-    }
+    ]
 
     resp = token_client.patch(
         '/api/v1/organizers/{}/events/{}/'.format(organizer.slug, event.slug),

src/tests/api/test_idempotency.py

@@ -0,0 +1,154 @@
+import datetime
+import json
+
+import pytest
+from django.utils.timezone import now
+from pytz import UTC
+
+from pretix.api.models import ApiCall
+from pretix.base.models import Order
+
+PAYLOAD = {
+    "name": {
+        "en": "Demo Conference 2020 Test"
+    },
+    "live": False,
+    "testmode": True,
+    "currency": "EUR",
+    "date_from": "2018-12-27T10:00:00Z",
+    "date_to": "2018-12-28T10:00:00Z",
+    "date_admission": None,
+    "is_public": False,
+    "presale_start": None,
+    "presale_end": None,
+    "location": None,
+    "slug": "2030",
+}
+
+
+@pytest.mark.django_db
+def test_default(token_client, organizer):
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json')
+    assert resp.status_code == 201
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json')
+    assert resp.status_code == 400
+
+
+@pytest.mark.django_db
+def test_scoped_by_key(token_client, organizer):
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    d1 = resp
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    assert d1.data == json.loads(resp.content.decode())
+    assert d1._headers == resp._headers
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='bar')
+    assert resp.status_code == 400
+
+
+@pytest.mark.django_db
+def test_concurrent(token_client, organizer):
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    ApiCall.objects.all().update(locked=now())
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 409
+
+
+@pytest.mark.django_db
+def test_ignore_path_method_body(token_client, organizer):
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             {}, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    resp = token_client.patch('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                              {}, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    resp = token_client.post('/api/v1/organizers/{}/§invalid/'.format(organizer.slug),
+                             {}, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+
+
+@pytest.mark.django_db
+def test_scoped_by_token(token_client, device, organizer):
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 201
+    token_client.credentials(HTTP_AUTHORIZATION='Device ' + device.api_token)
+    resp = token_client.post('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 403
+
+
+@pytest.mark.django_db
+def test_ignore_get(token_client, organizer, event):
+    resp = token_client.get('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                            HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 200
+    d1 = resp.data
+    event.name = "foo"
+    event.save()
+    resp = token_client.get('/api/v1/organizers/{}/events/'.format(organizer.slug),
+                            HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 200
+    assert d1 != json.loads(resp.content.decode())
+
+
+@pytest.mark.django_db
+def test_ignore_outside_api(token_client, organizer):
+    resp = token_client.post('/control/login'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 200
+    resp = token_client.post('/control/invalid'.format(organizer.slug),
+                             PAYLOAD, format='json', HTTP_X_IDEMPOTENCY_KEY='foo')
+    assert resp.status_code == 301
+
+
+@pytest.fixture
+def order(event):
+    return Order.objects.create(
+        code='FOO', event=event, email='dummy@dummy.test',
+        status=Order.STATUS_PENDING, secret="k24fiuwvu8kxz3y1",
+        datetime=datetime.datetime(2017, 12, 1, 10, 0, 0, tzinfo=UTC),
+        expires=datetime.datetime(2017, 12, 10, 10, 0, 0, tzinfo=UTC),
+        total=23, locale='en'
+    )
+
+
+@pytest.mark.django_db
+def test_allow_retry_409(token_client, organizer, event, order):
+    order.status = Order.STATUS_EXPIRED
+    order.save()
+    with event.lock():
+        resp = token_client.post(
+            '/api/v1/organizers/{}/events/{}/orders/{}/mark_paid/'.format(
+                organizer.slug, event.slug, order.code
+            ), format='json', HTTP_X_IDEMPOTENCY_KEY='foo'
+        )
+        assert resp.status_code == 409
+        order.refresh_from_db()
+        assert order.status == Order.STATUS_EXPIRED
+    resp = token_client.post(
+        '/api/v1/organizers/{}/events/{}/orders/{}/mark_paid/'.format(
+            organizer.slug, event.slug, order.code
+        ), format='json', HTTP_X_IDEMPOTENCY_KEY='foo'
+    )
+    assert resp.status_code == 200
+    order.refresh_from_db()
+    assert order.status == Order.STATUS_PAID

src/tests/api/test_items.py

@@ -76,8 +76,8 @@ def cart_position(event, item, variations):
         c = CartPosition.objects.create(
             event=event,
             item=item,
-            datetime=datetime.now(),
-            expires=datetime.now() + timedelta(days=1),
+            datetime=testtime,
+            expires=testtime + timedelta(days=1),
             variation=variations[0],
             price=Decimal("23"),
             cart_id="z3fsn8jyufm5kpk768q69gkbyr5f4h6w"

src/tests/api/test_orders.py

@@ -2181,7 +2181,7 @@ def test_order_create_answer_validation(token_client, organizer, event, item, qu
     )
     assert resp.status_code == 400
     assert resp.data == {'positions': [{'answers': [
-        {'non_field_errors': ['Date has wrong format. Use one of these formats instead: YYYY[-MM[-DD]].']}]}]}
+        {'non_field_errors': ['Date has wrong format. Use one of these formats instead: YYYY-MM-DD.']}]}]}
 
     question.type = Question.TYPE_DATETIME
     question.save()
@@ -2428,6 +2428,7 @@ def test_order_create_paid_generate_invoice(token_client, organizer, event, item
     event.settings.invoice_generate = 'paid'
     res = copy.deepcopy(ORDER_CREATE_PAYLOAD)
     res['status'] = 'p'
+    res['payment_date'] = '2019-04-01 08:20:00Z'
     res['positions'][0]['item'] = item.pk
     res['positions'][0]['answers'][0]['question'] = question.pk
     resp = token_client.post(
@@ -2443,6 +2444,11 @@ def test_order_create_paid_generate_invoice(token_client, organizer, event, item
     assert p.provider == "banktransfer"
     assert p.amount == o.total
     assert p.state == "confirmed"
+    assert p.payment_date.year == 2019
+    assert p.payment_date.month == 4
+    assert p.payment_date.day == 1
+    assert p.payment_date.hour == 8
+    assert p.payment_date.minute == 20
 
 
 REFUND_CREATE_PAYLOAD = {

src/tests/api/test_permissions.py

@@ -95,6 +95,7 @@ event_permission_sub_urls = [
     ('patch', 'can_change_items', 'questions/1/options/1/', 404),
     ('delete', 'can_change_items', 'questions/1/options/1/', 404),
     ('post', 'can_change_orders', 'orders/', 400),
+    ('patch', 'can_change_orders', 'orders/ABC12/', 404),
     ('post', 'can_change_orders', 'orders/ABC12/mark_paid/', 404),
     ('post', 'can_change_orders', 'orders/ABC12/mark_pending/', 404),
     ('post', 'can_change_orders', 'orders/ABC12/mark_expired/', 404),
@@ -102,6 +103,9 @@ event_permission_sub_urls = [
     ('post', 'can_change_orders', 'orders/ABC12/approve/', 404),
     ('post', 'can_change_orders', 'orders/ABC12/deny/', 404),
     ('post', 'can_change_orders', 'orders/ABC12/extend/', 400),
+    ('post', 'can_change_orders', 'orders/ABC12/create_invoice/', 404),
+    ('post', 'can_change_orders', 'orders/ABC12/resend_link/', 404),
+    ('post', 'can_change_orders', 'orders/ABC12/regenerate_secrets/', 404),
     ('get', 'can_view_orders', 'orders/ABC12/payments/', 404),
     ('get', 'can_view_orders', 'orders/ABC12/payments/1/', 404),
     ('get', 'can_view_orders', 'orders/ABC12/refunds/', 404),

src/tests/base/test_models.py

@@ -1748,10 +1748,11 @@ class CachedFileTestCase(TestCase):
         cf.filename = "testfile.txt"
         cf.save()
         assert default_storage.exists(cf.file.name)
+        n = cf.file.name
         with default_storage.open(cf.file.name, 'r') as f:
             assert f.read().strip() == "file_content"
         cf.delete()
-        assert not default_storage.exists(cf.file.name)
+        assert not default_storage.exists(n)
 
 
 class CheckinListTestCase(TestCase):

src/tests/control/test_events.py

@@ -363,7 +363,7 @@ class EventsTest(SoupTest):
 
     def test_payment_settings_last_date_payment_after_presale_end(self):
         tr19 = self.event1.tax_rules.create(rate=Decimal('19.00'))
-        self.event1.presale_end = datetime.datetime.now()
+        self.event1.presale_end = now()
         self.event1.save(update_fields=['presale_end'])
         doc = self.post_doc('/control/event/%s/%s/settings/payment' % (self.orga1.slug, self.event1.slug), {
             'payment_term_days': '2',

src/tests/control/test_items.py

@@ -20,6 +20,7 @@ class ItemFormTest(SoupTest):
             organizer=self.orga1, name='30C3', slug='30c3',
             date_from=datetime.datetime(2013, 12, 26, tzinfo=datetime.timezone.utc),
         )
+        self.item1 = Item.objects.create(event=self.event1, name="Standard", default_price=0, position=1)
         t = Team.objects.create(organizer=self.orga1, can_change_event_settings=True, can_change_items=True)
         t.members.add(self.user)
         t.limit_events.add(self.event1)
@@ -83,6 +84,7 @@ class QuestionsTest(ItemFormTest):
         form_data = extract_form_fields(doc.select('.container-fluid form')[0])
         form_data['question_0'] = 'What is your shoe size?'
         form_data['type'] = 'N'
+        form_data['items'] = self.item1.id
         doc = self.post_doc('/control/event/%s/%s/questions/add' % (self.orga1.slug, self.event1.slug), form_data)
         assert doc.select(".alert-success")
         self.assertIn("shoe size", doc.select("#page-wrapper table")[0].text)
@@ -97,6 +99,7 @@ class QuestionsTest(ItemFormTest):
         form_data['form-MIN_NUM_FORMS'] = '0'
         form_data['form-MAX_NUM_FORMS'] = '1'
         form_data['form-0-id'] = o1.pk
+        form_data['items'] = self.item1.id
         form_data['form-0-answer_0'] = 'England'
         self.post_doc('/control/event/%s/%s/questions/%s/change' % (self.orga1.slug, self.event1.slug, c.id),
                       form_data)
@@ -113,6 +116,7 @@ class QuestionsTest(ItemFormTest):
         form_data['form-INITIAL_FORMS'] = '1'
         form_data['form-MIN_NUM_FORMS'] = '0'
         form_data['form-MAX_NUM_FORMS'] = '1'
+        form_data['items'] = self.item1.id
         form_data['form-0-id'] = o1.pk
         form_data['form-0-answer_0'] = 'England'
         form_data['form-0-DELETE'] = 'yes'
@@ -130,6 +134,7 @@ class QuestionsTest(ItemFormTest):
         form_data['form-INITIAL_FORMS'] = '0'
         form_data['form-MIN_NUM_FORMS'] = '0'
         form_data['form-MAX_NUM_FORMS'] = '1'
+        form_data['items'] = self.item1.id
         form_data['form-0-id'] = ''
         form_data['form-0-answer_0'] = 'Germany'
         self.post_doc('/control/event/%s/%s/questions/%s/change' % (self.orga1.slug, self.event1.slug, c.id),
@@ -142,6 +147,7 @@ class QuestionsTest(ItemFormTest):
         c = Question.objects.create(event=self.event1, question="What is your shoe size?", type="N", required=True)
         doc = self.get_doc('/control/event/%s/%s/questions/%s/change' % (self.orga1.slug, self.event1.slug, c.id))
         form_data = extract_form_fields(doc.select('.container-fluid form')[0])
+        form_data['items'] = self.item1.id
         form_data['question_0'] = 'How old are you?'
         doc = self.post_doc('/control/event/%s/%s/questions/%s/change' % (self.orga1.slug, self.event1.slug, c.id),
                             form_data)
@@ -218,6 +224,7 @@ class QuestionsTest(ItemFormTest):
         o1 = q1.options.create(answer='Germany')
         doc = self.get_doc('/control/event/%s/%s/questions/%s/change' % (self.orga1.slug, self.event1.slug, q2.id))
         form_data = extract_form_fields(doc.select('.container-fluid form')[0])
+        form_data['items'] = self.item1.id
         form_data['dependency_question'] = q1.pk
         form_data['dependency_value'] = o1.identifier
         doc = self.post_doc('/control/event/%s/%s/questions/%s/change' % (self.orga1.slug, self.event1.slug, q2.id),
@@ -309,7 +316,6 @@ class ItemsTest(ItemFormTest):
 
     def setUp(self):
         super().setUp()
-        self.item1 = Item.objects.create(event=self.event1, name="Standard", default_price=0, position=1)
         self.item2 = Item.objects.create(event=self.event1, name="Business", default_price=0, position=2,
                                          description="If your ticket is paid by your employer",
                                          active=True, available_until=now() + datetime.timedelta(days=4),

src/tests/control/test_permissions.py

@@ -66,6 +66,7 @@ event_urls = [
     "items/add",
     "items/1/",
     "items/1/variations",
+    "items/1/bundles",
     "items/1/up",
     "items/1/down",
     "items/1/delete",
@@ -112,6 +113,7 @@ event_urls = [
     "orders/ABC/refunds/1/cancel",
     "orders/ABC/refunds/1/process",
     "orders/ABC/refunds/1/done",
+    "orders/ABC/delete",
     "orders/ABC/",
     "orders/",
     "checkinlists/",
@@ -238,6 +240,9 @@ event_permission_urls = [
     ("can_change_items", "items/1/up", 404),
     ("can_change_items", "items/1/down", 404),
     ("can_change_items", "items/1/delete", 404),
+    ("can_change_items", "items/1/variations", 404),
+    ("can_change_items", "items/1/addons", 404),
+    ("can_change_items", "items/1/bundles", 404),
     # ("can_change_items", "categories/", 200),
     # We don't have to create categories and similar objects
     # for testing this, it is enough to test that a 404 error
@@ -272,6 +277,7 @@ event_permission_urls = [
     ("can_change_orders", "orders/FOO/change", 200),
     ("can_change_orders", "orders/FOO/approve", 200),
     ("can_change_orders", "orders/FOO/deny", 200),
+    ("can_change_orders", "orders/FOO/delete", 302),
     ("can_change_orders", "orders/FOO/comment", 405),
     ("can_change_orders", "orders/FOO/locale", 200),
     ("can_view_orders", "orders/FOO/answer/5/", 404),

src/tests/presale/test_cart.py

@@ -1023,6 +1023,18 @@ class CartTest(CartTestMixin, TestCase):
         self.assertIn('empty', doc.select('.alert-success')[0].text)
         self.assertFalse(CartPosition.objects.filter(cart_id=self.session_key, event=self.event).exists())
 
+    def test_remove_expired_voucher(self):
+        v = Voucher.objects.create(item=self.ticket, event=self.event, valid_until=now() - timedelta(days=1))
+        cp = CartPosition.objects.create(
+            event=self.event, cart_id=self.session_key, item=self.ticket,
+            price=23, expires=now() - timedelta(minutes=10), voucher=v
+        )
+        self.client.post('/%s/%s/cart/remove' % (self.orga.slug, self.event.slug), {
+            'id': cp.pk
+        }, follow=True)
+        objs = list(CartPosition.objects.filter(cart_id=self.session_key, event=self.event))
+        self.assertEqual(len(objs), 0)
+
     def test_voucher(self):
         v = Voucher.objects.create(item=self.ticket, event=self.event)
         self.client.post('/%s/%s/cart/add' % (self.orga.slug, self.event.slug), {

src/tests/presale/test_checkout.py

@@ -504,6 +504,7 @@ class CheckoutTestCase(BaseCheckoutTestCase, TestCase):
     def test_invoice_address_required(self):
         self.event.settings.invoice_address_asked = True
         self.event.settings.invoice_address_required = True
+        self.event.settings.invoice_address_not_asked_free = True
         self.event.settings.set('name_scheme', 'title_given_middle_family')
 
         CartPosition.objects.create(
@@ -552,6 +553,26 @@ class CheckoutTestCase(BaseCheckoutTestCase, TestCase):
         }
         assert ia.name_cached == 'Mr John Kennedy'
 
+    def test_invoice_address_hidden_for_free(self):
+        self.event.settings.invoice_address_asked = True
+        self.event.settings.invoice_address_required = True
+        self.event.settings.invoice_address_not_asked_free = True
+        self.event.settings.set('name_scheme', 'title_given_middle_family')
+
+        CartPosition.objects.create(
+            event=self.event, cart_id=self.session_key, item=self.ticket,
+            price=0, expires=now() + timedelta(minutes=10)
+        )
+        response = self.client.get('/%s/%s/checkout/questions/' % (self.orga.slug, self.event.slug), follow=True)
+        doc = BeautifulSoup(response.rendered_content, "lxml")
+        self.assertEqual(len(doc.select('input[name="city"]')), 0)
+
+        response = self.client.post('/%s/%s/checkout/questions/' % (self.orga.slug, self.event.slug), {
+            'email': 'admin@localhost'
+        }, follow=True)
+        self.assertRedirects(response, '/%s/%s/checkout/confirm/' % (self.orga.slug, self.event.slug),
+                             target_status_code=200)
+
     def test_invoice_address_optional(self):
         self.event.settings.invoice_address_asked = True
         self.event.settings.invoice_address_required = False

src/tests/presale/test_widget.py

@@ -4,7 +4,7 @@ from decimal import Decimal
 
 from bs4 import BeautifulSoup
 from django.conf import settings
-from django.test import TestCase
+from django.test import TestCase, override_settings
 from django.utils.timezone import now
 from freezegun import freeze_time
 
@@ -261,6 +261,7 @@ class WidgetCartTest(CartTestMixin, TestCase):
             "itemnum": 0,
         }
 
+    @override_settings(COMPRESS_PRECOMPILERS=settings.COMPRESS_PRECOMPILERS_ORIGINAL)
     def test_css_customized(self):
         response = self.client.get('/%s/%s/widget/v1.css' % (self.orga.slug, self.event.slug))
         c = b"".join(response.streaming_content).decode()