Allow to refund currencies with zero decimals

Adds a checkbox in each payment provider's settings controlling whether sending out expiry reminders should be prevented

.github/workflows/tests.yml

@@ -35,7 +35,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: harmon758/postgresql-action@v1
         with:
-          postgresql version: '11'
+          postgresql version: '15'
           postgresql db: 'pretix'
           postgresql user: 'postgres'
           postgresql password: 'postgres'

Dockerfile

@@ -33,8 +33,7 @@ RUN apt-get update && \
     mkdir /static && \
     mkdir /etc/supervisord && \
 	curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash - && \
-    apt-get install -y nodejs && \
+    apt-get install -y nodejs
-    curl -qL https://www.npmjs.com/install.sh | sh
 
 
 ENV LC_ALL=C.UTF-8 \

deployment/docker/production_settings.py

@@ -1,4 +1,4 @@
 from pretix.settings import *
 
 LOGGING['handlers']['mail_admins']['include_html'] = True
-STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
+STORAGES["staticfiles"]["BACKEND"] = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

doc/admin/config.rst

@@ -152,6 +152,10 @@ Example::
     password=abcd
     host=localhost
     port=3306
+    sslmode=require
+    sslrootcert=/etc/pretix/postgresql-ca.crt
+    sslcert=/etc/pretix/postgresql-client-crt.crt
+    sslkey=/etc/pretix/postgresql-client-key.key
 
 ``backend``
     One of ``sqlite3`` and ``postgresql``.
@@ -163,6 +167,11 @@ Example::
 ``user``, ``password``, ``host``, ``port``
     Connection details for the database connection. Empty by default.
 
+``sslmode``, ``sslrootcert``
+    Connection TLS details for the PostgreSQL database connection. Possible values of ``sslmode`` are ``disable``, ``allow``, ``prefer``, ``require``, ``verify-ca``, and ``verify-full``. ``sslrootcert`` should be the accessible path of the ca certificate. Both values are empty by default.
+
+``sslcert``, ``sslkey``
+    Connection mTLS details for the PostgreSQL database connection. It's also necessary to specify ``sslmode`` and ``sslrootcert`` parameters, please check the correct values from the TLS part. ``sslcert`` should be the accessible path of the client certificate.  ``sslkey`` should be the accessible path of the client key. All values are empty by default.
 .. _`config-replica`:
 
 Database replica settings
@@ -324,6 +333,10 @@ to speed up various operations::
             ["sentinel_host_3", 26379]
         ]
     password=password
+    ssl_cert_reqs=required
+    ssl_ca_certs=/etc/pretix/redis-ca.pem
+    ssl_keyfile=/etc/pretix/redis-client-crt.pem
+    ssl_certfile=/etc/pretix/redis-client-key.key
 
 ``location``
     The location of redis, as a URL of the form ``redis://[:password]@localhost:6379/0``
@@ -347,6 +360,22 @@ to speed up various operations::
     If your redis setup doesn't require a password or you already specified it in the location you can omit this option.
     If this is set it will be passed to redis as the connection option PASSWORD.
 
+``ssl_cert_reqs``
+    If this is set it will be passed to redis as the connection option ``SSL_CERT_REQS``.
+    Possible values are ``none``, ``optional``, and ``required``.
+
+``ssl_ca_certs``
+    If your redis setup doesn't require TLS you can omit this option.
+    If this is set it will be passed to redis as the connection option ``SSL_CA_CERTS``. Possible value is the ca path.
+
+``ssl_keyfile``
+    If your redis setup doesn't require mTLS you can omit this option.
+    If this is set it will be passed to redis as the connection option ``SSL_KEYFILE``. Possible value is the keyfile path.
+
+``ssl_certfile``
+    If your redis setup doesn't require mTLS you can omit this option.
+    If this is set it will be passed to redis as the connection option ``SSL_CERTFILE``. Possible value is the certfile path.
+
 If redis is not configured, pretix will store sessions and locks in the database. If memcached
 is configured, memcached will be used for caching instead of redis.
 
@@ -396,6 +425,8 @@ The two ``transport_options`` entries can be omitted in most cases.
 If they are present they need to be a valid JSON dictionary.
 For possible entries in that dictionary see the `Celery documentation`_.
 
+It is possible the use Redis with TLS/mTLS for the broker or the backend. To do so, it is necessary to specify the TLS identifier ``rediss``, the ssl mode ``ssl_cert_reqs`` and optionally specify the CA (TLS) ``ssl_ca_certs``, cert ``ssl_certfile`` and key ``ssl_keyfile`` (mTLS) path as encoded string. the following uri describes the format and possible parameters ``rediss://0.0.0.0:6379/1?ssl_cert_reqs=required&ssl_ca_certs=%2Fetc%2Fpretix%2Fredis-ca.pem&ssl_certfile=%2Fetc%2Fpretix%2Fredis-client-crt.pem&ssl_keyfile=%2Fetc%2Fpretix%2Fredis-client-key.key``
+
 To use redis with sentinels set the broker or backend to ``sentinel://sentinel_host_1:26379;sentinel_host_2:26379/0``
 and the respective transport_options to ``{"master_name":"mymaster"}``.
 If your redis instances behind the sentinel have a password use ``sentinel://:my_password@sentinel_host_1:26379;sentinel_host_2:26379/0``.

doc/admin/installation/enterprise.rst

@@ -68,7 +68,7 @@ generated key and installs the plugin from the URL we told you::
         mkdir -p /etc/ssh && \
         ssh-keyscan -t rsa -p 10022 code.rami.io >> /root/.ssh/known_hosts && \
         echo StrictHostKeyChecking=no >> /root/.ssh/config && \
-        DJANGO_SETTINGS_MODULE=pretix.settings pip3 install -U "git+ssh://git@code.rami.io:10022/pretix/pretix-slack.git@stable#egg=pretix-slack" && \
+        DJANGO_SETTINGS_MODULE= pip3 install -U "git+ssh://git@code.rami.io:10022/pretix/pretix-slack.git@stable#egg=pretix-slack" && \
         cd /pretix/src && \
         sudo -u pretixuser make production
     USER pretixuser

doc/api/deviceauth.rst

@@ -32,10 +32,16 @@ as well as the type of underlying hardware. Example:
        "token": "kpp4jn8g2ynzonp6",
        "hardware_brand": "Samsung",
        "hardware_model": "Galaxy S",
+       "os_name": "Android",
+       "os_version": "2.3.6",
        "software_brand": "pretixdroid",
-       "software_version": "4.0.0"
+       "software_version": "4.0.0",
+       "rsa_pubkey": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqh…nswIDAQAB\n-----END PUBLIC KEY-----\n"
    }
 
+The ``rsa_pubkey`` is optional any only required for certain fatures such as working with reusable
+media and NFC cryptography.
+
 Every initialization token can only be used once. On success, you will receive a response containing
 information on your device as well as your API token:
 
@@ -98,6 +104,8 @@ following endpoint:
    {
        "hardware_brand": "Samsung",
        "hardware_model": "Galaxy S",
+       "os_name": "Android",
+       "os_version": "2.3.6",
        "software_brand": "pretixdroid",
        "software_version": "4.1.0",
        "info": {"arbitrary": "data"}
@@ -133,9 +141,29 @@ The response will look like this:
          "id": 3,
          "name": "South entrance"
        }
-     }
+     },
+     "server": {
+       "version": {
+         "pretix": "3.6.0.dev0",
+         "pretix_numeric": 30060001000
+       }
+     },
+     "medium_key_sets": [
+       {
+         "public_id": 3456349,
+         "organizer": "foo",
+         "active": true,
+         "media_type": "nfc_mf0aes",
+         "uid_key": "base64-encoded-encrypted-key",
+         "diversification_key": "base64-encoded-encrypted-key",
+       }
+     ]
    }
 
+``"medium_key_sets`` will always be empty if you did not set an ``rsa_pubkey``.
+The individual keys in the key sets are encrypted with the device's ``rsa_pubkey``
+using ``RSA/ECB/PKCS1Padding``.
+
 Creating a new API key
 ----------------------
 

doc/api/resources/devices.rst

@@ -24,6 +24,8 @@ all_events                            boolean                    Whether this de
 limit_events                          list                       List of event slugs this device has access to
 hardware_brand                        string                     Device hardware manufacturer (read-only)
 hardware_model                        string                     Device hardware model (read-only)
+os_name                               string                     Device operating system name (read-only)
+os_version                            string                     Device operating system version (read-only)
 software_brand                        string                     Device software product (read-only)
 software_version                      string                     Device software version (read-only)
 created                               datetime                   Creation time
@@ -76,6 +78,8 @@ Device endpoints
             "security_profile": "full",
             "hardware_brand": "Zebra",
             "hardware_model": "TC25",
+            "os_name": "Android",
+            "os_version": "8.1.0",
             "software_brand": "pretixSCAN",
             "software_version": "1.5.1"
           }
@@ -123,6 +127,8 @@ Device endpoints
         "security_profile": "full",
         "hardware_brand": "Zebra",
         "hardware_model": "TC25",
+        "os_name": "Android",
+        "os_version": "8.1.0",
         "software_brand": "pretixSCAN",
         "software_version": "1.5.1"
       }
@@ -173,6 +179,8 @@ Device endpoints
         "initialized": null
         "hardware_brand": null,
         "hardware_model": null,
+        "os_name": null,
+        "os_version": null,
         "software_brand": null,
         "software_version": null
       }

doc/api/resources/discounts.rst

@@ -31,9 +31,9 @@ subevent_mode                            strings                    Determines h
                                                                     ``"same"`` (discount is only applied for groups within
                                                                     the same date), or ``"distinct"`` (discount is only applied
                                                                     for groups with no two same dates).
-condition_all_products                   boolean                    If ``true``, the discount applies to all items.
+condition_all_products                   boolean                    If ``true``, the discount condition applies to all items.
 condition_limit_products                 list of integers           If ``condition_all_products`` is not set, this is a list
-                                                                    of internal item IDs that the discount applies to.
+                                                                    of internal item IDs that the discount condition applies to.
 condition_apply_to_addons                boolean                    If ``true``, the discount applies to add-on products as well,
                                                                     otherwise it only applies to top-level items. The discount never
                                                                     applies to bundled products.
@@ -48,6 +48,17 @@ benefit_discount_matching_percent        decimal (string)           The percenta
 benefit_only_apply_to_cheapest_n_matches integer                    If set higher than 0, the discount will only be applied to
                                                                     the cheapest matches. Useful for a "3 for 2"-style discount.
                                                                     Cannot be combined with ``condition_min_value``.
+benefit_same_products                    boolean                    If ``true``, the discount benefit applies to the same set of items
+                                                                    as the condition (see above).
+benefit_limit_products                   list of integers           If ``benefit_same_products`` is not set, this is a list
+                                                                    of internal item IDs that the discount benefit applies to.
+benefit_apply_to_addons                  boolean                    (Only used if ``benefit_same_products`` is ``false``.)
+                                                                    If ``true``, the discount applies to add-on products as well,
+                                                                    otherwise it only applies to top-level items. The discount never
+                                                                    applies to bundled products.
+benefit_ignore_voucher_discounted        boolean                    (Only used if ``benefit_same_products`` is ``false``.)
+                                                                    If ``true``, the discount does not apply to products which have
+                                                                    been discounted by a voucher.
 ======================================== ========================== =======================================================
 
 
@@ -94,6 +105,10 @@ Endpoints
             "condition_ignore_voucher_discounted": false,
             "condition_min_count": 3,
             "condition_min_value": "0.00",
+            "benefit_same_products": true,
+            "benefit_limit_products": [],
+            "benefit_apply_to_addons": true,
+            "benefit_ignore_voucher_discounted": false,
             "benefit_discount_matching_percent": "100.00",
             "benefit_only_apply_to_cheapest_n_matches": 1
           }
@@ -146,6 +161,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }
@@ -184,6 +203,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }
@@ -211,6 +234,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }
@@ -267,6 +294,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }

doc/api/resources/invoices.rst

@@ -12,6 +12,7 @@ The invoice resource contains the following public fields:
 Field                                 Type                       Description
 ===================================== ========================== =======================================================
 number                                string                     Invoice number (with prefix)
+event                                 string                     The slug of the parent event
 order                                 string                     Order code of the order this invoice belongs to
 is_cancellation                       boolean                    ``true``, if this invoice is the cancellation of a
                                                                  different invoice.
@@ -121,9 +122,13 @@ internal_reference                    string                     Customer's refe
 
    The attribute ``lines.subevent`` has been added.
 
+.. versionchanged:: 2023.8
 
-Endpoints
+   The ``event`` attribute has been added. The organizer-level endpoint has been added.
----------
+
+
+List of all invoices
+--------------------
 
 .. http:get:: /api/v1/organizers/(organizer)/events/(event)/invoices/
 
@@ -152,6 +157,7 @@ Endpoints
         "results": [
           {
             "number": "SAMPLECONF-00001",
+            "event": "sampleconf",
             "order": "ABC12",
             "is_cancellation": false,
             "invoice_from_name": "Big Events LLC",
@@ -221,6 +227,50 @@ Endpoints
    :statuscode 401: Authentication failure
    :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
 
+.. http:get:: /api/v1/organizers/(organizer)/invoices/
+
+   Returns a list of all invoices within all events of a given organizer (with sufficient access permissions).
+
+   Supported query parameters and output format of this endpoint are identical to the list endpoint within an event.
+
+   **Example request**:
+
+   .. sourcecode:: http
+
+      GET /api/v1/organizers/bigevents/events/sampleconf/invoices/ HTTP/1.1
+      Host: pretix.eu
+      Accept: application/json, text/javascript
+
+   **Example response**:
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Vary: Accept
+      Content-Type: application/json
+
+      {
+        "count": 1,
+        "next": null,
+        "previous": null,
+        "results": [
+          {
+            "number": "SAMPLECONF-00001",
+            "event": "sampleconf",
+            "order": "ABC12",
+            ...
+        ]
+      }
+
+   :param organizer: The ``slug`` field of the organizer to fetch
+   :statuscode 200: no error
+   :statuscode 401: Authentication failure
+   :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
+
+
+Fetching individual invoices
+----------------------------
+
 .. http:get:: /api/v1/organizers/(organizer)/events/(event)/invoices/(number)/
 
    Returns information on one invoice, identified by its invoice number.
@@ -243,6 +293,7 @@ Endpoints
 
       {
         "number": "SAMPLECONF-00001",
+        "event": "sampleconf",
         "order": "ABC12",
         "is_cancellation": false,
         "invoice_from_name": "Big Events LLC",
@@ -337,6 +388,12 @@ Endpoints
    :statuscode 409: The file is not yet ready and will now be prepared. Retry the request after waiting for a few
                     seconds.
 
+
+Modifying invoices
+------------------
+
+Invoices cannot be edited directly, but the following actions can be triggered:
+
 .. http:post:: /api/v1/organizers/(organizer)/events/(event)/invoices/(invoice_no)/reissue/
 
    Cancels the invoice and creates a new one.

doc/api/resources/orders.rst

@@ -20,6 +20,7 @@ The order resource contains the following public fields:
 Field                                 Type                       Description
 ===================================== ========================== =======================================================
 code                                  string                     Order code
+event                                 string                     The slug of the parent event
 status                                string                     Order status, one of:
 
                                                                  * ``n`` – pending
@@ -130,6 +131,10 @@ last_modified                         datetime                   Last modificati
 
    The ``valid_if_pending`` attribute has been added.
 
+.. versionchanged:: 2023.8
+
+   The ``event`` attribute has been added. The organizer-level endpoint has been added.
+
 
 .. _order-position-resource:
 
@@ -289,6 +294,7 @@ List of all orders
         "results": [
           {
             "code": "ABC12",
+            "event": "sampleconf",
             "status": "p",
             "testmode": false,
             "secret": "k24fiuwvu8kxz3y1",
@@ -441,6 +447,48 @@ List of all orders
    :statuscode 401: Authentication failure
    :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
 
+.. http:get:: /api/v1/organizers/(organizer)/orders/
+
+   Returns a list of all orders within all events of a given organizer (with sufficient access permissions).
+
+   Supported query parameters and output format of this endpoint are identical to the list endpoint within an event,
+   with the exception that the ``pdf_data`` parameter is not supported here.
+
+   **Example request**:
+
+   .. sourcecode:: http
+
+      GET /api/v1/organizers/bigevents/orders/ HTTP/1.1
+      Host: pretix.eu
+      Accept: application/json, text/javascript
+
+   **Example response**:
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Vary: Accept
+      Content-Type: application/json
+      X-Page-Generated: 2017-12-01T10:00:00Z
+
+      {
+        "count": 1,
+        "next": null,
+        "previous": null,
+        "results": [
+          {
+            "code": "ABC12",
+            "event": "sampleconf",
+            ...
+          }
+        ]
+      }
+
+   :param organizer: The ``slug`` field of the organizer to fetch
+   :statuscode 200: no error
+   :statuscode 401: Authentication failure
+   :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
+
 Fetching individual orders
 --------------------------
 
@@ -466,6 +514,7 @@ Fetching individual orders
 
       {
         "code": "ABC12",
+        "event": "sampleconf",
         "status": "p",
         "testmode": false,
         "secret": "k24fiuwvu8kxz3y1",

doc/api/resources/reusablemedia.rst

@@ -18,7 +18,7 @@ The reusable medium resource contains the following public fields:
 Field                                 Type                       Description
 ===================================== ========================== =======================================================
 id                                    integer                    Internal ID of the medium
-type                                  string                     Type of medium, e.g. ``"barcode"`` or ``"nfc_uid"``.
+type                                  string                     Type of medium, e.g. ``"barcode"``, ``"nfc_uid"`` or ``"nfc_mf0aes"``.
 organizer                             string                     Organizer slug of the organizer who "owns" this medium.
 identifier                            string                     Unique identifier of the medium. The format depends on the ``type``.
 active                                boolean                    Whether this medium may be used.
@@ -37,6 +37,7 @@ Existing media types are:
 
 - ``barcode``
 - ``nfc_uid``
+- ``nfc_mf0aes``
 
 Endpoints
 ---------

doc/api/resources/sendmail_rules.rst

@@ -1,10 +1,10 @@
-Automated email rules
+Scheduled email rules
 =====================
 
 Resource description
 --------------------
 
-Automated email rules that specify emails that the system will send automatically at a specific point in time, e.g.
+Scheduled email rules that specify emails that the system will send automatically at a specific point in time, e.g.
 the day of the event.
 
 .. rst-class:: rest-resource-table
@@ -18,8 +18,19 @@ subject                               multi-lingual string       The subject of
 template                              multi-lingual string       The body of the email
 all_products                          boolean                    If ``true``, the email is sent to buyers of all products
 limit_products                        list of integers           List of product IDs, if ``all_products`` is not set
-include_pending                       boolean                    If ``true``, the email is sent to pending orders. If ``false``,
+[**DEPRECATED**] include_pending      boolean                    If ``true``, the email is sent to pending orders. If ``false``,
                                                                  only paid orders are considered.
+restrict_to_status                    list                       List of order states to restrict recipients to. Valid
+                                                                 entries are ``p`` for paid, ``e`` for expired, ``c`` for canceled,
+                                                                 ``n__pending_approval`` for pending approval,
+                                                                 ``n__not_pending_approval_and_not_valid_if_pending`` for payment
+                                                                 pending, ``n__valid_if_pending`` for payment pending but already confirmed,
+                                                                 and ``n__pending_overdue`` for pending with payment overdue.
+                                                                 The default is ``["p", "n__valid_if_pending"]``.
+checked_in_status                     string                     Check-in status to restrict recipients to. Valid strings are:
+                                                                 ``null`` for no filtering (default), ``checked_in`` for
+                                                                 limiting to attendees that are or have been checked in, and
+                                                                 ``no_checkin`` for limiting to attendees who have not checked in.
 date_is_absolute                      boolean                    If ``true``, the email is set at a specific point in time.
 send_date                             datetime                   If ``date_is_absolute`` is set: Date and time to send the email.
 send_offset_days                      integer                    If ``date_is_absolute`` is not set, this is the number of days
@@ -37,7 +48,10 @@ send_to                               string                     Can be ``"order
                                                                  or ``"both"``.
                                                                  date. Otherwise it is relative to the event start date.
 ===================================== ========================== =======================================================
+.. versionchanged:: 2023.7
 
+    The ``include_pending`` field has been  deprecated.
+    The ``restrict_to_status`` field has been added.
 
 Endpoints
 ---------
@@ -74,7 +88,12 @@ Endpoints
             "template": {"en": "Don't forget your tickets, download them at {url}"},
             "all_products": true,
             "limit_products": [],
-            "include_pending": false,
+            "restrict_to_status": [
+                "p",
+                "n__not_pending_approval_and_not_valid_if_pending",
+                "n__valid_if_pending"
+            ],
+            "checked_in_status": null,
             "send_date": null,
             "send_offset_days": 1,
             "send_offset_time": "18:00",
@@ -120,7 +139,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": null,
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",
@@ -157,7 +181,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": "checked_in",
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",
@@ -182,7 +211,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": "checked_in",
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",
@@ -235,7 +269,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": "checked_in",
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",

doc/api/resources/subevents.rst

@@ -68,6 +68,10 @@ last_modified                         datetime                   Last modificati
     The ``date_from_before``, ``date_from_after``, ``date_to_before``, and ``date_to_after`` query parameters have been
     added.
 
+.. versionchanged:: 2023.8.0
+
+    For the organizer-wide endpoint, the ``search`` query parameter has been modified to filter sub-events by their parent events slug too.
+
 Endpoints
 ---------
 
@@ -472,6 +476,7 @@ Endpoints
    :query date_to_after: If set to a date and time, only events that have an end date and end at or after the given time are returned.
    :query date_to_before: If set to a date and time, only events that have an end date and end at or before the given time are returned.
    :query ends_after: If set to a date and time, only events that happen during of after the given time are returned.
+   :query search: Only return events matching a given search query.
    :query sales_channel: If set to a sales channel identifier, the response will only contain subevents from events available on this sales channel.
    :param organizer: The ``slug`` field of a valid organizer
    :param event: The ``slug`` field of the event to fetch

doc/api/resources/taxrules.rst

@@ -20,11 +20,16 @@ internal_name                         string                     An optional nam
 rate                                  decimal (string)           Tax rate in percent
 price_includes_tax                    boolean                    If ``true`` (default), tax is assumed to be included in
                                                                  the specified product price
-eu_reverse_charge                     boolean                    If ``true``, EU reverse charge rules are applied
+eu_reverse_charge                     boolean                    If ``true``, EU reverse charge rules are applied. Will
+                                                                 be ignored if custom rules are set.
 home_country                          string                     Merchant country (required for reverse charge), can be
                                                                  ``null`` or empty string
 keep_gross_if_rate_changes            boolean                    If ``true``, changes of the tax rate based on custom
                                                                  rules keep the gross price constant (default is ``false``)
+custom_rules                          object                     Dynamic rules specification. Each list element
+                                                                 corresponds to one rule that will be processed in order.
+                                                                 The current version of the schema in use can be found
+                                                                 `here`_.
 ===================================== ========================== =======================================================
 
 
@@ -32,6 +37,10 @@ keep_gross_if_rate_changes            boolean                    If ``true``, ch
 
     The ``internal_name`` and ``keep_gross_if_rate_changes`` attributes have been added.
 
+.. versionchanged:: 2023.6
+
+    The ``custom_rules`` attribute has been added.
+
 Endpoints
 ---------
 
@@ -68,6 +77,7 @@ Endpoints
             "price_includes_tax": true,
             "eu_reverse_charge": false,
             "keep_gross_if_rate_changes": false,
+            "custom_rules": null,
             "home_country": "DE"
           }
         ]
@@ -108,6 +118,7 @@ Endpoints
         "price_includes_tax": true,
         "eu_reverse_charge": false,
         "keep_gross_if_rate_changes": false,
+        "custom_rules": null,
         "home_country": "DE"
       }
 
@@ -156,6 +167,7 @@ Endpoints
         "price_includes_tax": true,
         "eu_reverse_charge": false,
         "keep_gross_if_rate_changes": false,
+        "custom_rules": null,
         "home_country": "DE"
       }
 
@@ -203,6 +215,7 @@ Endpoints
         "price_includes_tax": true,
         "eu_reverse_charge": false,
         "keep_gross_if_rate_changes": false,
+        "custom_rules": null,
         "home_country": "DE"
       }
 
@@ -242,3 +255,5 @@ Endpoints
    :statuscode 204: no error
    :statuscode 401: Authentication failure
    :statuscode 403: The requested organizer/event/rule does not exist **or** you have no permission to change it **or** this tax rule cannot be deleted since it is currently in use.
+
+.. _here: https://github.com/pretix/pretix/blob/master/src/pretix/static/schema/tax-rules-custom.schema.json

doc/api/resources/webhooks.rst

@@ -67,6 +67,9 @@ The following values for ``action_types`` are valid with pretix core:
     * ``pretix.event.live.deactivated``
     * ``pretix.event.testmode.activated``
     * ``pretix.event.testmode.deactivated``
+    * ``pretix.customer.created``
+    * ``pretix.customer.changed``
+    * ``pretix.customer.anonymized``
 
 Installed plugins might register more valid values.
 

doc/development/api/general.rst

@@ -61,7 +61,7 @@ Backend
              item_formsets, order_search_filter_q, order_search_forms
 
 .. automodule:: pretix.base.signals
-   :members: logentry_display, logentry_object_link, requiredaction_display, timeline_events, orderposition_blocked_display
+   :members: logentry_display, logentry_object_link, requiredaction_display, timeline_events, orderposition_blocked_display, customer_created, customer_signed_in
 
 Vouchers
 """"""""

doc/development/api/payment.rst

@@ -70,6 +70,8 @@ The provider class
 
    .. autoattribute:: settings_form_fields
 
+   .. autoattribute:: walletqueries
+
    .. automethod:: settings_form_clean
 
    .. automethod:: settings_content_render

doc/development/implementation/i18n.rst

@@ -37,7 +37,7 @@ you to execute a piece of code with a different locale:
 This is very useful e.g. when sending an email to a user that has a different language than the user performing the
 action that causes the mail to be sent.
 
-.. _translation features: https://docs.djangoproject.com/en/1.9/topics/i18n/translation/
+.. _translation features: https://docs.djangoproject.com/en/4.2/topics/i18n/translation/
 .. _GNU gettext: https://www.gnu.org/software/gettext/
 .. _strings: https://django-i18nfield.readthedocs.io/en/latest/strings.html
 .. _database fields: https://django-i18nfield.readthedocs.io/en/latest/quickstart.html

doc/development/implementation/urlconfig.rst

@@ -15,25 +15,33 @@ and the admin panel is available at ``https://pretix.eu/control/event/bigorg/awe
 
 If the organizer now configures a custom domain like ``tickets.bigorg.com``, his event will
 from now on be available on ``https://tickets.bigorg.com/awesomecon/``. The former URL at
-``pretix.eu`` will redirect there. However, the admin panel will still only be available
+``pretix.eu`` will redirect there. It's also possible to do this for just an event, in which
-on ``pretix.eu`` for convenience and security reasons.
+case the event will be available on ``https://tickets.awesomecon.org/``.
+
+However, the admin panel will still only be available on ``pretix.eu`` for convenience and security reasons.
 
 URL routing
 -----------
 
 The hard part about implementing this URL routing in Django is that
 ``https://pretix.eu/bigorg/awesomecon/`` contains two parameters of nearly arbitrary content
-and ``https://tickets.bigorg.com/awesomecon/`` contains only one. The only robust way to do
+and ``https://tickets.bigorg.com/awesomecon/`` contains only one and ``https://tickets.awesomecon.org/`` does not contain any.
-this is by having *separate* URL configuration for those two cases. In pretix, we call the
+The only robust way to do this is by having *separate* URL configuration for those three cases.
-former our ``maindomain`` config and the latter our ``subdomain`` config. For pretix's core
-modules we do some magic to avoid duplicate configuration, but for a fairly simple plugin with
-only a handful of routes, we recommend just configuring the two URL sets separately.
 
+In pretix, we therefore do not have a global URL configuration, but three, living in the following modules:
+
+- ``pretix.multidomain.maindomain_urlconf``
+- ``pretix.multidomain.organizer_domain_urlconf``
+- ``pretix.multidomain.event_domain_urlconf``
+
+We provide some helper utilities to work with these to avoid duplicate configuration of the individual URLs.
 The file ``urls.py`` inside your plugin package will be loaded and scanned for URL configuration
 automatically and should be provided by any plugin that provides any view.
+However, unlike plain Django, we look not only for a ``urlpatterns`` attribute on the module but support other
+attributes like ``event_patterns`` and ``organizer_patterns`` as well.
 
-A very basic example that provides one view in the admin panel and one view in the frontend
+For example, for a simple plugin that adds one URL to the backend and one event-level URL to the frontend, you can
-could look like this::
+create the following configuration in your ``urls.py``::
 
     from django.urls import re_path
 
@@ -52,7 +60,7 @@ could look like this::
     As you can see, the view in the frontend is not included in the standard Django ``urlpatterns``
     setting but in a separate list with the name ``event_patterns``. This will automatically prepend
     the appropriate parameters to the regex (e.g. the event or the event and the organizer, depending
-    on the called domain).
+    on the called domain). For organizer-level views, ``organizer_patterns`` works the same way.
 
 If you only provide URLs in the admin area, you do not need to provide a ``event_patterns`` attribute.
 
@@ -71,11 +79,16 @@ is a python method that emulates a behavior similar to ``reverse``:
 
 .. autofunction:: pretix.multidomain.urlreverse.eventreverse
 
+If you need to communicate the URL externally, you can use a different method to ensure that it is always an absolute URL:
+
+.. autofunction:: pretix.multidomain.urlreverse.build_absolute_uri
+
 In addition, there is a template tag that works similar to ``url`` but takes an event or organizer object
 as its first argument and can be used like this::
 
     {% load eventurl %}
     <a href="{% eventurl request.event "presale:event.checkout" step="payment" %}">Pay</a>
+    <a href="{% abseventurl request.event "presale:event.checkout" step="payment" %}">Pay</a>
 
 
 Implementation details

doc/development/index.rst

@@ -12,3 +12,4 @@ Developer documentation
    api/index
    structure
    translation/index
+   nfc/index

doc/development/nfc/index.rst

@@ -0,0 +1,15 @@
+NFC media
+=========
+
+pretix supports using NFC chips as "reusable media", for example to store gift cards or tickets.
+
+Most of this implementation currently lives in our proprietary app pretixPOS, but in the future might also become part of our open-source pretixSCAN solution.
+Either way, we want this to be an open ecosystem and therefore document the exact mechanisms in use on the following pages.
+
+We support multiple implementations of NFC media, each documented on its own page:
+
+.. toctree::
+   :maxdepth: 2
+
+   uid
+   mf0aes

doc/development/nfc/mf0aes.rst

@@ -0,0 +1,113 @@
+Mifare Ultralight AES
+=====================
+
+We offer an implementation that provides a higher security level than the UID-based approach and uses the `Mifare Ultralight AES`_ chip sold by NXP.
+We believe the security model of this approach is adequate to the situation where this will usually be used and we'll outline known risks below.
+
+If you want to dive deeper into the properties of the Mifare Ultralight AES chip, we recommend reading the `data sheet`_.
+
+Random UIDs
+-----------
+
+Mifare Ultralight AES supports a feature that returns a randomized UID every time a non-authenticated user tries to
+read the UID. This has a strong privacy benefit, since no unauthorized entity can use the NFC chips to track users.
+On the other hand, this reduces interoperability of the system. For example, this prevents you from using the same NFC
+chips for a different purpose where you only need the UID. This will also prevent your guests from reading their UID
+themselves with their phones, which might be useful e.g. in debugging situations.
+
+Since there's no one-size-fits-all choice here, you can enable or disable this feature in the pretix organizer
+settings. If you change it, the change will apply to all newly encoded chips after the change.
+
+Key management
+--------------
+
+For every organizer, the server will generate create a "key set", which consists of a publicly known ID (random 32-bit integer) and two 16-byte keys ("diversification key" and "UID key").
+
+Using our :ref:`Device authentication mechanism <rest-deviceauth>`, an authorized device can submit a locally generated RSA public key to the server.
+This key can no longer changed on the server once it is set, thus protecting against the attack scenario of a leaked device API token.
+
+The server will then include key sets in the response to ``/api/v1/device/info``, encrypted with the device's RSA key.
+This includes all key sets generated for the organizer the device belongs to, as well as all keys of organizers that have granted sufficient access to this organizer.
+
+The device will decrypt the key sets using its RSA key and store the key sets locally.
+
+.. warning:: The device **will** have access to the raw key sets. Therefore, there is a risk of leaked master keys if an
+             authorized device is stolen or abused. Our implementation in pretixPOS attempts to make this very hard on
+             modern, non-rooted Android devices by keeping them encrypted with the RSA key and only storing the RSA key
+             in the hardware-backed keystore of the device. A sufficiently motivated attacker, however, will likely still
+             be able to extract the keys from a stolen device.
+
+Encoding a chip
+---------------
+
+When a new chip is encoded, the following steps will be taken:
+
+- The UID of the chip is retrieved.
+
+- A chip-specific key is generated using the mechanism documented in `AN10922`_ using the "diversification key" from the
+  organizer's key set as the CMAC key and the diversification input concatenated in the from of ``0x01 + UID + APPID + SYSTEMID``
+  with the following values:
+
+  - The UID of the chip as ``UID``
+
+  - ``"eu.pretix"`` (``0x65 0x75 0x2e 0x70 0x72 0x65 0x74 0x69 0x78``) as ``APPID``
+
+  - The ``public_id`` from the organizer's key set as a 4-byte big-endian value as ``SYSTEMID``
+
+- The chip-specific key is written to the chip as the "data protection key" (config pages 0x30 to 0x33)
+
+- The UID key from the organizer's key set is written to the chip as the "UID retrieval key" (config pages 0x34 to 0x37)
+
+- The config page 0x29 is set like this:
+
+  - ``RID_ACT`` (random UID) to ``1`` or ``0`` based on the organizer's configuration
+  - ``SEC_MSG_ACT`` (secure messaging) to ``1``
+  - ``AUTH0`` (first page that needs authentication) to 0x04 (first non-UID page)
+
+- The config page 0x2A is set like this:
+
+  - ``PROT`` to ``0`` (only write access restricted, not read access)
+  - ``AUTHLIM`` to ``256`` (maximum number of wrong authentications before "self-desctruction")
+  - Everything else to its default value (no lock bits are set)
+
+- The ``public_id`` of the key set will be written to page 0x04 as a big-endian value
+
+- The UID of the chip will be registered as a reusable medium on the server.
+
+.. warning:: During encoding, the chip-specific key and the UID key are transmitted in plain text over the air. The
+             security model therefore relies on the encoding of chips being performed in a trusted physical environment
+             to prevent a nearby attacker from sniffing the keys with a strong antenna.
+
+.. note:: If an attacker tries to authenticate with the chip 256 times using the wrong key, the chip will become
+          unusable. A chip may also become unusable if it is detached from the reader in the middle of the encoding
+          process (even though we've tried to implement it in a way that makes this unlikely).
+
+Usage
+-----
+
+When a chip is presented to the NFC reader, the following steps will be taken:
+
+- Command ``GET_VERSION`` is used to determine if it is a Mifare Ultralight AES chip (if not, abort).
+
+- Page 0x04 is read. If it is all zeroes, the chip is considered un-encoded (abort). If it contains a value that
+  corresponds to the ``public_id`` of a known key set, this key set is used for all further operations. If it contains
+  a different value, we consider this chip to belong to a different organizer or not to a pretix system at all (abort).
+
+- An authentication with the chip using the UID key is performed.
+
+- The UID of the chip will be read.
+
+- The chip-specific key will be derived using the mechanism described above in the encoding step.
+
+- An authentication with the chip using the chip-specific key is performed. If this is fully successful, this step
+  proves that the chip knows the same chip-specific key as we do and is therefore an authentic chip encoded by us and
+  we can trust its UID value.
+
+- The UID is transmitted to the server to fetch the correct medium.
+
+During these steps, the keys are never transmitted in plain text and can thus not be sniffed by a nearby attacker
+with a strong antenna.
+
+.. _Mifare Ultralight AES: https://www.nxp.com/products/rfid-nfc/mifare-hf/mifare-ultralight/mifare-ultralight-aes-enhanced-security-for-limited-use-contactless-applications:MF0AESx20
+.. _data sheet: https://www.nxp.com/docs/en/data-sheet/MF0AES(H)20.pdf
+.. _AN10922: https://www.nxp.com/docs/en/application-note/AN10922.pdf

doc/development/nfc/uid.rst

@@ -0,0 +1,10 @@
+UID-based
+=========
+
+With UID-based NFC, only the unique ID (UID) of the NFC chip is used for identification purposes.
+This can be used with virtually all NFC chips that provide compatibility with the NFC reader in use, typically at least all chips that comply with ISO/IEC 14443-3A.
+
+We make only one restriction: The UID may not start with ``08``, since that usually signifies a randomized UID that changes on every read (which would not be very useful).
+
+.. warning:: The UID-based approach provides only a very low level of security. It is easy to clone a chip with the same
+             UID and impersonate someone else.

doc/development/setup.rst

@@ -96,6 +96,20 @@ http://localhost:8000/control/ for the admin view.
           port (for example because you develop on `pretixdroid`_), you can check
           `Django's documentation`_ for more options.
 
+When running the local development webserver, ensure Celery is not configured
+in ``pretix.cfg``. i.e., you should remove anything such as::
+
+    [celery]
+    backend=redis://redis:6379/2
+    broker=redis://redis:6379/2
+
+If you choose to use Celery for development, you must also start a Celery worker
+process::
+
+    celery -A pretix.celery_app worker -l info
+
+However, beware that code changes will not auto-reload within Celery.
+
 .. _`checksandtests`:
 
 Code checks and unit tests

pyproject.toml

@@ -30,13 +30,13 @@ dependencies = [
   "babel",
   "BeautifulSoup4==4.12.*",
   "bleach==5.0.*",
-  "celery==5.2.*",
+  "celery==5.3.*",
   "chardet==5.1.*",
   "cryptography>=3.4.2",
   "css-inline==0.8.*",
   "defusedcsv>=1.1.0",
   "dj-static",
-  "Django==4.1.*",
+  "Django==4.2.*",
   "django-bootstrap3==23.1.*",
   "django-compressor==4.3.*",
   "django-countries==7.5.*",
@@ -59,10 +59,10 @@ dependencies = [
   "dnspython==2.3.*",
   "drf_ujson2==1.7.*",
   "geoip2==4.*",
-  "importlib_metadata==6.6.*",  # Polyfill, we can probably drop this once we require Python 3.10+
+  "importlib_metadata==6.*",  # Polyfill, we can probably drop this once we require Python 3.10+
   "isoweek",
   "jsonschema",
-  "kombu==5.2.*",
+  "kombu==5.3.*",
   "libsass==0.22.*",
   "lxml",
   "markdown==3.4.3",  # 3.3.5 requires importlib-metadata>=4.4, but django-bootstrap3 requires importlib-metadata<3.
@@ -90,7 +90,7 @@ dependencies = [
   "pytz-deprecation-shim==0.1.*",
   "pyuca",
   "qrcode==7.4.*",
-  "redis==4.5.*,>=4.5.4",
+  "redis==4.6.*",
   "reportlab==4.0.*",
   "requests==2.31.*",
   "sentry-sdk==1.15.*",
@@ -112,6 +112,7 @@ memcached = ["pylibmc"]
 dev = [
   "coverage",
   "coveralls",
+  "fakeredis==2.18.*",
   "flake8==6.0.*",
   "freezegun",
   "isort==5.12.*",

src/pretix/__init__.py

@@ -19,4 +19,4 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
-__version__ = "4.21.0.dev0"
+__version__ = "2023.8.0.dev0"

src/pretix/_base_settings.py

@@ -196,7 +196,14 @@ STATICFILES_DIRS = [
 
 STATICI18N_ROOT = os.path.join(BASE_DIR, "pretix/static")
 
-STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
+STORAGES = {
+    "default": {
+        "BACKEND": "django.core.files.storage.FileSystemStorage",
+    },
+    "staticfiles": {
+        "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
+    },
+}
 
 # if os.path.exists(os.path.join(DATA_DIR, 'static')):
 #     STATICFILES_DIRS.insert(0, os.path.join(DATA_DIR, 'static'))

src/pretix/api/auth/devicesecurity.py

@@ -223,6 +223,7 @@ class PretixPosSecurityProfile(AllowListSecurityProfile):
         ('POST', 'api-v1:checkinrpc.redeem'),
         ('GET', 'api-v1:checkinrpc.search'),
         ('POST', 'api-v1:reusablemedium-lookup'),
+        ('POST', 'api-v1:reusablemedium-list'),
     )
 
 

src/pretix/api/middleware.py

@@ -59,7 +59,7 @@ class IdempotencyMiddleware:
         auth_hash = sha1(auth_hash_parts.encode()).hexdigest()
         idempotency_key = request.headers.get('X-Idempotency-Key', '')
 
-        with transaction.atomic():
+        with transaction.atomic(durable=True):
             call, created = ApiCall.objects.select_for_update(of=OF_SELF).get_or_create(
                 auth_hash=auth_hash,
                 idempotency_key=idempotency_key,
@@ -75,7 +75,7 @@ class IdempotencyMiddleware:
 
         if created:
             resp = self.get_response(request)
-            with transaction.atomic():
+            with transaction.atomic(durable=True):
                 if resp.status_code in (409, 429, 500, 503):
                     # This is the exception: These calls are *meant* to be retried!
                     call.delete()

src/pretix/api/serializers/__init__.py

@@ -19,6 +19,8 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
+import json
+
 from rest_framework import serializers
 
 
@@ -46,3 +48,16 @@ class AsymmetricField(serializers.Field):
 
     def run_validation(self, data=serializers.empty):
         return self.write.run_validation(data)
+
+
+class CompatibleJSONField(serializers.JSONField):
+    def to_internal_value(self, data):
+        try:
+            return json.dumps(data)
+        except (TypeError, ValueError):
+            self.fail('invalid')
+
+    def to_representation(self, value):
+        if value:
+            return json.loads(value)
+        return value

src/pretix/api/serializers/discount.py

@@ -32,11 +32,13 @@ class DiscountSerializer(I18nAwareModelSerializer):
                   'available_until', 'subevent_mode', 'condition_all_products', 'condition_limit_products',
                   'condition_apply_to_addons', 'condition_min_count', 'condition_min_value',
                   'benefit_discount_matching_percent', 'benefit_only_apply_to_cheapest_n_matches',
-                  'condition_ignore_voucher_discounted')
+                  'benefit_same_products', 'benefit_limit_products', 'benefit_apply_to_addons',
+                  'benefit_ignore_voucher_discounted', 'condition_ignore_voucher_discounted')
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.fields['condition_limit_products'].queryset = self.context['event'].items.all()
+        self.fields['benefit_limit_products'].queryset = self.context['event'].items.all()
 
     def validate(self, data):
         data = super().validate(data)

src/pretix/api/serializers/event.py

@@ -46,6 +46,7 @@ from rest_framework import serializers
 from rest_framework.fields import ChoiceField, Field
 from rest_framework.relations import SlugRelatedField
 
+from pretix.api.serializers import CompatibleJSONField
 from pretix.api.serializers.i18n import I18nAwareModelSerializer
 from pretix.api.serializers.settings import SettingsSerializer
 from pretix.base.models import Device, Event, TaxRule, TeamAPIToken
@@ -53,6 +54,7 @@ from pretix.base.models.event import SubEvent
 from pretix.base.models.items import (
     ItemMetaProperty, SubEventItem, SubEventItemVariation,
 )
+from pretix.base.models.tax import CustomRulesValidator
 from pretix.base.services.seating import (
     SeatProtected, generate_seats, validate_plan_change,
 )
@@ -650,9 +652,16 @@ class SubEventSerializer(I18nAwareModelSerializer):
 
 
 class TaxRuleSerializer(CountryFieldMixin, I18nAwareModelSerializer):
+    custom_rules = CompatibleJSONField(
+        validators=[CustomRulesValidator()],
+        required=False,
+        allow_null=True,
+    )
+
     class Meta:
         model = TaxRule
-        fields = ('id', 'name', 'rate', 'price_includes_tax', 'eu_reverse_charge', 'home_country', 'internal_name', 'keep_gross_if_rate_changes')
+        fields = ('id', 'name', 'rate', 'price_includes_tax', 'eu_reverse_charge', 'home_country', 'internal_name',
+                  'keep_gross_if_rate_changes', 'custom_rules')
 
 
 class EventSettingsSerializer(SettingsSerializer):
@@ -719,6 +728,7 @@ class EventSettingsSerializer(SettingsSerializer):
         'payment_term_minutes',
         'payment_term_last',
         'payment_term_expire_automatically',
+        'payment_term_expire_delay_days',
         'payment_term_accept_late',
         'payment_explanation',
         'payment_pending_hidden',
@@ -807,6 +817,10 @@ class EventSettingsSerializer(SettingsSerializer):
         'reusable_media_type_nfc_uid',
         'reusable_media_type_nfc_uid_autocreate_giftcard',
         'reusable_media_type_nfc_uid_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes_random_uid',
     ]
     readonly_fields = [
         # These are read-only since they are currently only settable on organizers, not events
@@ -816,6 +830,10 @@ class EventSettingsSerializer(SettingsSerializer):
         'reusable_media_type_nfc_uid',
         'reusable_media_type_nfc_uid_autocreate_giftcard',
         'reusable_media_type_nfc_uid_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes_random_uid',
     ]
 
     def __init__(self, *args, **kwargs):
@@ -884,6 +902,8 @@ class DeviceEventSettingsSerializer(EventSettingsSerializer):
         'name_scheme',
         'reusable_media_type_barcode',
         'reusable_media_type_nfc_uid',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_random_uid',
         'system_question_order',
     ]
 

src/pretix/api/serializers/order.py

@@ -19,7 +19,6 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
-import json
 import logging
 import os
 from collections import Counter, defaultdict
@@ -28,6 +27,7 @@ from decimal import Decimal
 import pycountry
 from django.conf import settings
 from django.core.files import File
+from django.db import models
 from django.db.models import F, Q
 from django.utils.encoding import force_str
 from django.utils.timezone import now
@@ -39,6 +39,7 @@ from rest_framework.exceptions import ValidationError
 from rest_framework.relations import SlugRelatedField
 from rest_framework.reverse import reverse
 
+from pretix.api.serializers import CompatibleJSONField
 from pretix.api.serializers.event import SubEventSerializer
 from pretix.api.serializers.i18n import I18nAwareModelSerializer
 from pretix.api.serializers.item import (
@@ -283,11 +284,12 @@ class FailedCheckinSerializer(I18nAwareModelSerializer):
     raw_item = serializers.PrimaryKeyRelatedField(queryset=Item.objects.none(), required=False, allow_null=True)
     raw_variation = serializers.PrimaryKeyRelatedField(queryset=ItemVariation.objects.none(), required=False, allow_null=True)
     raw_subevent = serializers.PrimaryKeyRelatedField(queryset=SubEvent.objects.none(), required=False, allow_null=True)
+    nonce = serializers.CharField(required=False, allow_null=True)
 
     class Meta:
         model = Checkin
         fields = ('error_reason', 'error_explanation', 'raw_barcode', 'raw_item', 'raw_variation',
-                  'raw_subevent', 'datetime', 'type', 'position')
+                  'raw_subevent', 'nonce', 'datetime', 'type', 'position')
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -372,11 +374,15 @@ class PdfDataSerializer(serializers.Field):
                 self.context['vars_images'] = get_images(self.context['event'])
 
             for k, f in self.context['vars'].items():
-                try:
+                if 'evaluate_bulk' in f:
-                    res[k] = f['evaluate'](instance, instance.order, ev)
+                    # Will be evaluated later by our list serializers
-                except:
+                    res[k] = (f['evaluate_bulk'], instance)
-                    logger.exception('Evaluating PDF variable failed')
+                else:
-                    res[k] = '(error)'
+                    try:
+                        res[k] = f['evaluate'](instance, instance.order, ev)
+                    except:
+                        logger.exception('Evaluating PDF variable failed')
+                        res[k] = '(error)'
 
             if not hasattr(ev, '_cached_meta_data'):
                 ev._cached_meta_data = ev.meta_data
@@ -429,6 +435,38 @@ class PdfDataSerializer(serializers.Field):
             return res
 
 
+class OrderPositionListSerializer(serializers.ListSerializer):
+
+    def to_representation(self, data):
+        # We have a custom implementation of this method because PdfDataSerializer() might keep some elements unevaluated
+        # with a (callable, input) tuple. We'll loop over these entries and evaluate them bulk-wise to save on SQL queries.
+
+        if isinstance(self.parent, OrderSerializer) and isinstance(self.parent.parent, OrderListSerializer):
+            # Do not execute our custom code because it will be executed by OrderListSerializer later for the
+            # full result set.
+            return super().to_representation(data)
+
+        iterable = data.all() if isinstance(data, models.Manager) else data
+
+        data = []
+        evaluate_queue = defaultdict(list)
+
+        for item in iterable:
+            entry = self.child.to_representation(item)
+            if "pdf_data" in entry:
+                for k, v in entry["pdf_data"].items():
+                    if isinstance(v, tuple) and callable(v[0]):
+                        evaluate_queue[v[0]].append((v[1], entry, k))
+            data.append(entry)
+
+        for func, entries in evaluate_queue.items():
+            results = func([item for (item, entry, k) in entries])
+            for (item, entry, k), result in zip(entries, results):
+                entry["pdf_data"][k] = result
+
+        return data
+
+
 class OrderPositionSerializer(I18nAwareModelSerializer):
     checkins = CheckinSerializer(many=True, read_only=True)
     answers = AnswerSerializer(many=True)
@@ -440,6 +478,7 @@ class OrderPositionSerializer(I18nAwareModelSerializer):
     attendee_name = serializers.CharField(required=False)
 
     class Meta:
+        list_serializer_class = OrderPositionListSerializer
         model = OrderPosition
         fields = ('id', 'order', 'positionid', 'item', 'variation', 'price', 'attendee_name', 'attendee_name_parts',
                   'company', 'street', 'zipcode', 'city', 'country', 'state', 'discount',
@@ -468,6 +507,20 @@ class OrderPositionSerializer(I18nAwareModelSerializer):
     def validate(self, data):
         raise TypeError("this serializer is readonly")
 
+    def to_representation(self, data):
+        if isinstance(self.parent, (OrderListSerializer, OrderPositionListSerializer)):
+            # Do not execute our custom code because it will be executed by OrderListSerializer later for the
+            # full result set.
+            return super().to_representation(data)
+
+        entry = super().to_representation(data)
+        if "pdf_data" in entry:
+            for k, v in entry["pdf_data"].items():
+                if isinstance(v, tuple) and callable(v[0]):
+                    entry["pdf_data"][k] = v[0]([v[1]])[0]
+
+        return entry
+
 
 class RequireAttentionField(serializers.Field):
     def to_representation(self, instance: OrderPosition):
@@ -562,7 +615,7 @@ class PaymentURLField(serializers.URLField):
     def to_representation(self, instance: OrderPayment):
         if instance.state != OrderPayment.PAYMENT_STATE_CREATED:
             return None
-        return build_absolute_uri(self.context['event'], 'presale:event.order.pay', kwargs={
+        return build_absolute_uri(instance.order.event, 'presale:event.order.pay', kwargs={
             'order': instance.order.code,
             'secret': instance.order.secret,
             'payment': instance.pk,
@@ -607,13 +660,42 @@ class OrderRefundSerializer(I18nAwareModelSerializer):
 
 class OrderURLField(serializers.URLField):
     def to_representation(self, instance: Order):
-        return build_absolute_uri(self.context['event'], 'presale:event.order', kwargs={
+        return build_absolute_uri(instance.event, 'presale:event.order', kwargs={
             'order': instance.code,
             'secret': instance.secret,
         })
 
 
+class OrderListSerializer(serializers.ListSerializer):
+
+    def to_representation(self, data):
+        # We have a custom implementation of this method because PdfDataSerializer() might keep some elements
+        # unevaluated with a (callable, input) tuple. We'll loop over these entries and evaluate them bulk-wise to
+        # save on SQL queries.
+        iterable = data.all() if isinstance(data, models.Manager) else data
+
+        data = []
+        evaluate_queue = defaultdict(list)
+
+        for item in iterable:
+            entry = self.child.to_representation(item)
+            for p in entry.get("positions", []):
+                if "pdf_data" in p:
+                    for k, v in p["pdf_data"].items():
+                        if isinstance(v, tuple) and callable(v[0]):
+                            evaluate_queue[v[0]].append((v[1], p, k))
+            data.append(entry)
+
+        for func, entries in evaluate_queue.items():
+            results = func([item for (item, entry, k) in entries])
+            for (item, entry, k), result in zip(entries, results):
+                entry["pdf_data"][k] = result
+
+        return data
+
+
 class OrderSerializer(I18nAwareModelSerializer):
+    event = SlugRelatedField(slug_field='slug', read_only=True)
     invoice_address = InvoiceAddressSerializer(allow_null=True)
     positions = OrderPositionSerializer(many=True, read_only=True)
     fees = OrderFeeSerializer(many=True, read_only=True)
@@ -627,8 +709,9 @@ class OrderSerializer(I18nAwareModelSerializer):
 
     class Meta:
         model = Order
+        list_serializer_class = OrderListSerializer
         fields = (
-            'code', 'status', 'testmode', 'secret', 'email', 'phone', 'locale', 'datetime', 'expires', 'payment_date',
+            'code', 'event', 'status', 'testmode', 'secret', 'email', 'phone', 'locale', 'datetime', 'expires', 'payment_date',
             'payment_provider', 'fees', 'total', 'comment', 'custom_followup_at', 'invoice_address', 'positions', 'downloads',
             'checkin_attention', 'last_modified', 'payments', 'refunds', 'require_approval', 'sales_channel',
             'url', 'customer', 'valid_if_pending'
@@ -896,19 +979,6 @@ class OrderPositionCreateSerializer(I18nAwareModelSerializer):
         return data
 
 
-class CompatibleJSONField(serializers.JSONField):
-    def to_internal_value(self, data):
-        try:
-            return json.dumps(data)
-        except (TypeError, ValueError):
-            self.fail('invalid')
-
-    def to_representation(self, value):
-        if value:
-            return json.loads(value)
-        return value
-
-
 class WrappedList:
     def __init__(self, data):
         self._data = data
@@ -1525,6 +1595,7 @@ class InlineInvoiceLineSerializer(I18nAwareModelSerializer):
 
 
 class InvoiceSerializer(I18nAwareModelSerializer):
+    event = SlugRelatedField(slug_field='slug', read_only=True)
     order = serializers.SlugRelatedField(slug_field='code', read_only=True)
     refers = serializers.SlugRelatedField(slug_field='full_invoice_no', read_only=True)
     lines = InlineInvoiceLineSerializer(many=True)
@@ -1533,7 +1604,7 @@ class InvoiceSerializer(I18nAwareModelSerializer):
 
     class Meta:
         model = Invoice
-        fields = ('order', 'number', 'is_cancellation', 'invoice_from', 'invoice_from_name', 'invoice_from_zipcode',
+        fields = ('event', 'order', 'number', 'is_cancellation', 'invoice_from', 'invoice_from_name', 'invoice_from_zipcode',
                   'invoice_from_city', 'invoice_from_country', 'invoice_from_tax_id', 'invoice_from_vat_id',
                   'invoice_to', 'invoice_to_company', 'invoice_to_name', 'invoice_to_street', 'invoice_to_zipcode',
                   'invoice_to_city', 'invoice_to_state', 'invoice_to_country', 'invoice_to_vat_id', 'invoice_to_beneficiary',

src/pretix/api/serializers/organizer.py

@@ -94,6 +94,14 @@ class CustomerSerializer(I18nAwareModelSerializer):
             data['name_parts']['_scheme'] = self.context['request'].organizer.settings.name_scheme
         return data
 
+    def validate_email(self, value):
+        qs = Customer.objects.filter(organizer=self.context['organizer'], email__iexact=value)
+        if self.instance and self.instance.pk:
+            qs = qs.exclude(pk=self.instance.pk)
+        if qs.exists():
+            raise ValidationError(_("An account with this email address is already registered."))
+        return value
+
 
 class CustomerCreateSerializer(CustomerSerializer):
     send_email = serializers.BooleanField(default=False, required=False, allow_null=True)
@@ -251,6 +259,8 @@ class DeviceSerializer(serializers.ModelSerializer):
     unique_serial = serializers.CharField(read_only=True)
     hardware_brand = serializers.CharField(read_only=True)
     hardware_model = serializers.CharField(read_only=True)
+    os_name = serializers.CharField(read_only=True)
+    os_version = serializers.CharField(read_only=True)
     software_brand = serializers.CharField(read_only=True)
     software_version = serializers.CharField(read_only=True)
     created = serializers.DateTimeField(read_only=True)
@@ -263,7 +273,7 @@ class DeviceSerializer(serializers.ModelSerializer):
         fields = (
             'device_id', 'unique_serial', 'initialization_token', 'all_events', 'limit_events',
             'revoked', 'name', 'created', 'initialized', 'hardware_brand', 'hardware_model',
-            'software_brand', 'software_version', 'security_profile'
+            'os_name', 'os_version', 'software_brand', 'software_version', 'security_profile'
         )
 
 
@@ -390,6 +400,9 @@ class OrganizerSettingsSerializer(SettingsSerializer):
         'reusable_media_type_nfc_uid',
         'reusable_media_type_nfc_uid_autocreate_giftcard',
         'reusable_media_type_nfc_uid_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency',
     ]
 
     def __init__(self, *args, **kwargs):

src/pretix/api/urls.py

@@ -61,6 +61,8 @@ orga_router.register(r'membershiptypes', organizer.MembershipTypeViewSet)
 orga_router.register(r'reusablemedia', media.ReusableMediaViewSet)
 orga_router.register(r'teams', organizer.TeamViewSet)
 orga_router.register(r'devices', organizer.DeviceViewSet)
+orga_router.register(r'orders', order.OrganizerOrderViewSet)
+orga_router.register(r'invoices', order.InvoiceViewSet)
 orga_router.register(r'exporters', exporters.OrganizerExportersViewSet, basename='exporters')
 
 team_router = routers.DefaultRouter()
@@ -77,7 +79,7 @@ event_router.register(r'questions', item.QuestionViewSet)
 event_router.register(r'discounts', discount.DiscountViewSet)
 event_router.register(r'quotas', item.QuotaViewSet)
 event_router.register(r'vouchers', voucher.VoucherViewSet)
-event_router.register(r'orders', order.OrderViewSet)
+event_router.register(r'orders', order.EventOrderViewSet)
 event_router.register(r'orderpositions', order.OrderPositionViewSet)
 event_router.register(r'invoices', order.InvoiceViewSet)
 event_router.register(r'revokedsecrets', order.RevokedSecretViewSet, basename='revokedsecrets')

src/pretix/api/views/checkin.py

@@ -164,8 +164,21 @@ class CheckinListViewSet(viewsets.ModelViewSet):
                 secret=serializer.validated_data['raw_barcode']
             ).first()
 
+        clist = self.get_object()
+        if serializer.validated_data.get('nonce'):
+            if kwargs.get('position'):
+                prev = kwargs['position'].all_checkins.filter(nonce=serializer.validated_data['nonce']).first()
+            else:
+                prev = clist.checkins.filter(
+                    nonce=serializer.validated_data['nonce'],
+                    raw_barcode=serializer.validated_data['raw_barcode'],
+                ).first()
+            if prev:
+                # Ignore because nonce is already handled
+                return Response(serializer.data, status=201)
+
         c = serializer.save(
-            list=self.get_object(),
+            list=clist,
             successful=False,
             forced=True,
             force_sent=True,

src/pretix/api/views/device.py

@@ -19,8 +19,12 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
+import base64
 import logging
 
+from cryptography.hazmat.backends.openssl.backend import Backend
+from cryptography.hazmat.primitives.asymmetric import padding
+from cryptography.hazmat.primitives.serialization import load_pem_public_key
 from django.db.models import Exists, OuterRef, Q
 from django.db.models.functions import Coalesce
 from django.utils.timezone import now
@@ -34,6 +38,8 @@ from pretix.api.auth.device import DeviceTokenAuthentication
 from pretix.api.views.version import numeric_version
 from pretix.base.models import CheckinList, Device, SubEvent
 from pretix.base.models.devices import Gate, generate_api_token
+from pretix.base.models.media import MediumKeySet
+from pretix.base.services.media import get_keysets_for_organizer
 
 logger = logging.getLogger(__name__)
 
@@ -42,17 +48,73 @@ class InitializationRequestSerializer(serializers.Serializer):
     token = serializers.CharField(max_length=190)
     hardware_brand = serializers.CharField(max_length=190)
     hardware_model = serializers.CharField(max_length=190)
+    os_name = serializers.CharField(max_length=190, required=False, allow_null=True)
+    os_version = serializers.CharField(max_length=190, required=False, allow_null=True)
     software_brand = serializers.CharField(max_length=190)
     software_version = serializers.CharField(max_length=190)
     info = serializers.JSONField(required=False, allow_null=True)
+    rsa_pubkey = serializers.CharField(required=False, allow_null=True)
+
+    def validate(self, attrs):
+        if attrs.get('rsa_pubkey'):
+            try:
+                load_pem_public_key(
+                    attrs['rsa_pubkey'].encode(), Backend()
+                )
+            except:
+                raise ValidationError({'rsa_pubkey': ['Not a valid public key.']})
+        return attrs
 
 
 class UpdateRequestSerializer(serializers.Serializer):
     hardware_brand = serializers.CharField(max_length=190)
     hardware_model = serializers.CharField(max_length=190)
+    os_name = serializers.CharField(max_length=190, required=False, allow_null=True)
+    os_version = serializers.CharField(max_length=190, required=False, allow_null=True)
     software_brand = serializers.CharField(max_length=190)
     software_version = serializers.CharField(max_length=190)
     info = serializers.JSONField(required=False, allow_null=True)
+    rsa_pubkey = serializers.CharField(required=False, allow_null=True)
+
+    def validate(self, attrs):
+        if attrs.get('rsa_pubkey'):
+            try:
+                load_pem_public_key(
+                    attrs['rsa_pubkey'].encode(), Backend()
+                )
+            except:
+                raise ValidationError({'rsa_pubkey': ['Not a valid public key.']})
+        return attrs
+
+
+class RSAEncryptedField(serializers.Field):
+    def to_representation(self, value):
+        public_key = load_pem_public_key(
+            self.context['device'].rsa_pubkey.encode(), Backend()
+        )
+        cipher_text = public_key.encrypt(
+            # RSA/ECB/PKCS1Padding
+            value,
+            padding.PKCS1v15()
+        )
+        return base64.b64encode(cipher_text).decode()
+
+
+class MediumKeySetSerializer(serializers.ModelSerializer):
+    uid_key = RSAEncryptedField(read_only=True)
+    diversification_key = RSAEncryptedField(read_only=True)
+    organizer = serializers.SlugRelatedField(slug_field='slug', read_only=True)
+
+    class Meta:
+        model = MediumKeySet
+        fields = [
+            'public_id',
+            'organizer',
+            'active',
+            'media_type',
+            'uid_key',
+            'diversification_key',
+        ]
 
 
 class GateSerializer(serializers.ModelSerializer):
@@ -99,9 +161,12 @@ class InitializeView(APIView):
         device.initialized = now()
         device.hardware_brand = serializer.validated_data.get('hardware_brand')
         device.hardware_model = serializer.validated_data.get('hardware_model')
+        device.os_name = serializer.validated_data.get('os_name')
+        device.os_version = serializer.validated_data.get('os_version')
         device.software_brand = serializer.validated_data.get('software_brand')
         device.software_version = serializer.validated_data.get('software_version')
         device.info = serializer.validated_data.get('info')
+        device.rsa_pubkey = serializer.validated_data.get('rsa_pubkey')
         device.api_token = generate_api_token()
         device.save()
 
@@ -120,8 +185,15 @@ class UpdateView(APIView):
         device = request.auth
         device.hardware_brand = serializer.validated_data.get('hardware_brand')
         device.hardware_model = serializer.validated_data.get('hardware_model')
+        device.os_name = serializer.validated_data.get('os_name')
+        device.os_version = serializer.validated_data.get('os_version')
         device.software_brand = serializer.validated_data.get('software_brand')
         device.software_version = serializer.validated_data.get('software_version')
+        if serializer.validated_data.get('rsa_pubkey') and serializer.validated_data.get('rsa_pubkey') != device.rsa_pubkey:
+            if device.rsa_pubkey:
+                raise ValidationError({'rsa_pubkey': ['You cannot change the rsa_pubkey of the device once it is set.']})
+            else:
+                device.rsa_pubkey = serializer.validated_data.get('rsa_pubkey')
         device.info = serializer.validated_data.get('info')
         device.save()
         device.log_action('pretix.device.updated', data=serializer.validated_data, auth=device)
@@ -169,8 +241,12 @@ class InfoView(APIView):
                     'pretix': __version__,
                     'pretix_numeric': numeric_version(__version__),
                 }
-            }
+            },
-
+            'medium_key_sets': MediumKeySetSerializer(
+                get_keysets_for_organizer(device.organizer),
+                many=True,
+                context={'device': request.auth}
+            ).data if device.rsa_pubkey else []
         })
 
 

src/pretix/api/views/event.py

@@ -381,16 +381,29 @@ with scopes_disabled():
                 | Q(location__icontains=i18ncomp(value))
             )
 
+    class OrganizerSubEventFilter(SubEventFilter):
+        def search_qs(self, queryset, name, value):
+            return queryset.filter(
+                Q(name__icontains=i18ncomp(value))
+                | Q(event__slug__icontains=value)
+                | Q(location__icontains=i18ncomp(value))
+            )
+
 
 class SubEventViewSet(ConditionalListView, viewsets.ModelViewSet):
     serializer_class = SubEventSerializer
     queryset = SubEvent.objects.none()
     write_permission = 'can_change_event_settings'
     filter_backends = (DjangoFilterBackend, TotalOrderingFilter)
-    filterset_class = SubEventFilter
     ordering = ('date_from',)
     ordering_fields = ('id', 'date_from', 'last_modified')
 
+    @property
+    def filterset_class(self):
+        if getattr(self.request, 'event', None):
+            return SubEventFilter
+        return OrganizerSubEventFilter
+
     def get_queryset(self):
         if getattr(self.request, 'event', None):
             qs = self.request.event.subevents
@@ -415,6 +428,7 @@ class SubEventViewSet(ConditionalListView, viewsets.ModelViewSet):
             'subeventitem_set',
             'subeventitemvariation_set',
             'meta_values',
+            'meta_values__property',
             Prefetch(
                 'seat_category_mappings',
                 to_attr='_seat_category_mappings',

src/pretix/api/views/media.py

@@ -104,6 +104,12 @@ class ReusableMediaViewSet(viewsets.ModelViewSet):
             auth=self.request.auth,
             data=merge_dicts(self.request.data, {'id': inst.pk})
         )
+        mt = MEDIA_TYPES.get(serializer.validated_data["type"])
+        if mt:
+            m = mt.handle_new(self.request.organizer, inst, self.request.user, self.request.auth)
+            if m:
+                s = self.get_serializer(m)
+                return Response({"result": s.data})
 
     @transaction.atomic()
     def perform_update(self, serializer):

src/pretix/api/views/order.py

@@ -44,6 +44,7 @@ from rest_framework.exceptions import (
     APIException, NotFound, PermissionDenied, ValidationError,
 )
 from rest_framework.mixins import CreateModelMixin
+from rest_framework.permissions import SAFE_METHODS
 from rest_framework.response import Response
 
 from pretix.api.models import OAuthAccessToken
@@ -115,12 +116,16 @@ with scopes_disabled():
 
         @scopes_disabled()
         def subevent_after_qs(self, qs, name, value):
+            if getattr(self.request, 'event', None):
+                subevents = self.request.event.subevents
+            else:
+                subevents = SubEvent.objects.filter(event__organizer=self.request.organizer)
+
             qs = qs.filter(
                 pk__in=Subquery(
                     OrderPosition.all.filter(
-                        subevent_id__in=SubEvent.objects.filter(
+                        subevent_id__in=subevents.filter(
                             Q(date_to__gt=value) | Q(date_from__gt=value, date_to__isnull=True),
-                            event=self.request.event
                         ).values_list('id'),
                     ).values_list('order_id')
                 )
@@ -128,12 +133,16 @@ with scopes_disabled():
             return qs
 
         def subevent_before_qs(self, qs, name, value):
+            if getattr(self.request, 'event', None):
+                subevents = self.request.event.subevents
+            else:
+                subevents = SubEvent.objects.filter(event__organizer=self.request.organizer)
+
             qs = qs.filter(
                 pk__in=Subquery(
                     OrderPosition.all.filter(
-                        subevent_id__in=SubEvent.objects.filter(
+                        subevent_id__in=subevents.filter(
                             Q(date_from__lt=value),
-                            event=self.request.event
                         ).values_list('id'),
                     ).values_list('order_id')
                 )
@@ -185,7 +194,7 @@ with scopes_disabled():
             )
 
 
-class OrderViewSet(viewsets.ModelViewSet):
+class OrderViewSetMixin:
     serializer_class = OrderSerializer
     queryset = Order.objects.none()
     filter_backends = (DjangoFilterBackend, TotalOrderingFilter)
@@ -193,19 +202,12 @@ class OrderViewSet(viewsets.ModelViewSet):
     ordering_fields = ('datetime', 'code', 'status', 'last_modified')
     filterset_class = OrderFilter
     lookup_field = 'code'
-    permission = 'can_view_orders'
-    write_permission = 'can_change_orders'
 
-    def get_serializer_context(self):
+    def get_base_queryset(self):
-        ctx = super().get_serializer_context()
+        raise NotImplementedError()
-        ctx['event'] = self.request.event
-        ctx['pdf_data'] = self.request.query_params.get('pdf_data', 'false') == 'true'
-        ctx['exclude'] = self.request.query_params.getlist('exclude')
-        ctx['include'] = self.request.query_params.getlist('include')
-        return ctx
 
     def get_queryset(self):
-        qs = self.request.event.orders
+        qs = self.get_base_queryset()
         if 'fees' not in self.request.GET.getlist('exclude'):
             if self.request.query_params.get('include_canceled_fees', 'false') == 'true':
                 fqs = OrderFee.all
@@ -227,11 +229,12 @@ class OrderViewSet(viewsets.ModelViewSet):
             opq = OrderPosition.all
         else:
             opq = OrderPosition.objects
-        if request.query_params.get('pdf_data', 'false') == 'true':
+        if request.query_params.get('pdf_data', 'false') == 'true' and getattr(request, 'event', None):
             prefetch_related_objects([request.organizer], 'meta_properties')
             prefetch_related_objects(
                 [request.event],
-                Prefetch('meta_values', queryset=EventMetaValue.objects.select_related('property'), to_attr='meta_values_cached'),
+                Prefetch('meta_values', queryset=EventMetaValue.objects.select_related('property'),
+                         to_attr='meta_values_cached'),
                 'questions',
                 'item_meta_properties',
             )
@@ -266,13 +269,12 @@ class OrderViewSet(viewsets.ModelViewSet):
                 )
             )
 
-    def _get_output_provider(self, identifier):
+    def get_serializer_context(self):
-        responses = register_ticket_outputs.send(self.request.event)
+        ctx = super().get_serializer_context()
-        for receiver, response in responses:
+        ctx['exclude'] = self.request.query_params.getlist('exclude')
-            prov = response(self.request.event)
+        ctx['include'] = self.request.query_params.getlist('include')
-            if prov.identifier == identifier:
+        ctx['pdf_data'] = False
-                return prov
+        return ctx
-        raise NotFound('Unknown output provider.')
 
     @scopes_disabled()  # we are sure enough that get_queryset() is correct, so we save some perforamnce
     def list(self, request, **kwargs):
@@ -289,6 +291,45 @@ class OrderViewSet(viewsets.ModelViewSet):
         serializer = self.get_serializer(queryset, many=True)
         return Response(serializer.data, headers={'X-Page-Generated': date})
 
+
+class OrganizerOrderViewSet(OrderViewSetMixin, viewsets.ReadOnlyModelViewSet):
+    def get_base_queryset(self):
+        perm = "can_view_orders" if self.request.method in SAFE_METHODS else "can_change_orders"
+        if isinstance(self.request.auth, (TeamAPIToken, Device)):
+            return Order.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.auth.get_events_with_permission(perm, request=self.request)
+            )
+        elif self.request.user.is_authenticated:
+            return Order.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.user.get_events_with_permission(perm, request=self.request)
+            )
+        else:
+            raise PermissionDenied()
+
+
+class EventOrderViewSet(OrderViewSetMixin, viewsets.ModelViewSet):
+    permission = 'can_view_orders'
+    write_permission = 'can_change_orders'
+
+    def get_serializer_context(self):
+        ctx = super().get_serializer_context()
+        ctx['event'] = self.request.event
+        ctx['pdf_data'] = self.request.query_params.get('pdf_data', 'false') == 'true'
+        return ctx
+
+    def get_base_queryset(self):
+        return self.request.event.orders
+
+    def _get_output_provider(self, identifier):
+        responses = register_ticket_outputs.send(self.request.event)
+        for receiver, response in responses:
+            prov = response(self.request.event)
+            if prov.identifier == identifier:
+                return prov
+        raise NotFound('Unknown output provider.')
+
     @action(detail=True, url_name='download', url_path='download/(?P<output>[^/]+)')
     def download(self, request, output, **kwargs):
         provider = self._get_output_provider(output)
@@ -945,6 +986,7 @@ with scopes_disabled():
                 | Q(addon_to__attendee_email__icontains=value)
                 | Q(order__code__istartswith=value)
                 | Q(order__invoice_address__name_cached__icontains=value)
+                | Q(order__invoice_address__company__icontains=value)
                 | Q(order__email__icontains=value)
                 | Q(pk__in=matching_media)
             )
@@ -1781,11 +1823,24 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
     write_permission = 'can_change_orders'
 
     def get_queryset(self):
-        return self.request.event.invoices.prefetch_related('lines').select_related('order', 'refers').annotate(
+        perm = "can_view_orders" if self.request.method in SAFE_METHODS else "can_change_orders"
+        if getattr(self.request, 'event', None):
+            qs = self.request.event.invoices
+        elif isinstance(self.request.auth, (TeamAPIToken, Device)):
+            qs = Invoice.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.auth.get_events_with_permission(perm, request=self.request)
+            )
+        elif self.request.user.is_authenticated:
+            qs = Invoice.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.user.get_events_with_permission(perm, request=self.request)
+            )
+        return qs.prefetch_related('lines').select_related('order', 'refers').annotate(
             nr=Concat('prefix', 'invoice_no')
         )
 
-    @action(detail=True, )
+    @action(detail=True)
     def download(self, request, **kwargs):
         invoice = self.get_object()
 
@@ -1804,7 +1859,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
         return resp
 
     @action(detail=True, methods=['POST'])
-    def regenerate(self, request, **kwarts):
+    def regenerate(self, request, **kwargs):
         inv = self.get_object()
         if inv.canceled:
             raise ValidationError('The invoice has already been canceled.')
@@ -1814,7 +1869,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
             raise PermissionDenied('The invoice file is no longer stored on the server.')
         elif inv.sent_to_organizer:
             raise PermissionDenied('The invoice file has already been exported.')
-        elif now().astimezone(self.request.event.timezone).date() - inv.date > datetime.timedelta(days=1):
+        elif now().astimezone(inv.event.timezone).date() - inv.date > datetime.timedelta(days=1):
             raise PermissionDenied('The invoice file is too old to be regenerated.')
         else:
             inv = regenerate_invoice(inv)
@@ -1829,7 +1884,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
             return Response(status=204)
 
     @action(detail=True, methods=['POST'])
-    def reissue(self, request, **kwarts):
+    def reissue(self, request, **kwargs):
         inv = self.get_object()
         if inv.canceled:
             raise ValidationError('The invoice has already been canceled.')

src/pretix/api/webhooks.py

@@ -202,6 +202,21 @@ class ParametrizedWaitingListEntryWebhookEvent(ParametrizedWebhookEvent):
         }
 
 
+class ParametrizedCustomerWebhookEvent(ParametrizedWebhookEvent):
+
+    def build_payload(self, logentry: LogEntry):
+        customer = logentry.content_object
+        if not customer:
+            return None
+
+        return {
+            'notification_id': logentry.pk,
+            'organizer': customer.organizer.slug,
+            'customer': customer.identifier,
+            'action': logentry.action_type,
+        }
+
+
 @receiver(register_webhook_events, dispatch_uid="base_register_default_webhook_events")
 def register_default_webhook_events(sender, **kwargs):
     return (
@@ -350,6 +365,18 @@ def register_default_webhook_events(sender, **kwargs):
             'pretix.event.orders.waitinglist.voucher_assigned',
             _('Waiting list entry received voucher'),
         ),
+        ParametrizedCustomerWebhookEvent(
+            'pretix.customer.created',
+            _('Customer account created'),
+        ),
+        ParametrizedCustomerWebhookEvent(
+            'pretix.customer.changed',
+            _('Customer account changed'),
+        ),
+        ParametrizedCustomerWebhookEvent(
+            'pretix.customer.anonymized',
+            _('Customer account anonymized'),
+        ),
     )
 
 

src/pretix/base/cache.py

@@ -62,27 +62,27 @@ class NamespacedCache:
             prefix = int(time.time())
             self.cache.set(self.prefixkey, prefix)
 
-    def set(self, key: str, value: str, timeout: int=300):
+    def set(self, key: str, value: any, timeout: int=300):
         return self.cache.set(self._prefix_key(key), value, timeout)
 
-    def get(self, key: str) -> str:
+    def get(self, key: str) -> any:
         return self.cache.get(self._prefix_key(key, known_prefix=self._last_prefix))
 
-    def get_or_set(self, key: str, default: Callable, timeout=300) -> str:
+    def get_or_set(self, key: str, default: Callable, timeout=300) -> any:
         return self.cache.get_or_set(
             self._prefix_key(key, known_prefix=self._last_prefix),
             default=default,
             timeout=timeout
         )
 
-    def get_many(self, keys: List[str]) -> Dict[str, str]:
+    def get_many(self, keys: List[str]) -> Dict[str, any]:
         values = self.cache.get_many([self._prefix_key(key) for key in keys])
         newvalues = {}
         for k, v in values.items():
             newvalues[self._strip_prefix(k)] = v
         return newvalues
 
-    def set_many(self, values: Dict[str, str], timeout=300):
+    def set_many(self, values: Dict[str, any], timeout=300):
         newvalues = {}
         for k, v in values.items():
             newvalues[self._prefix_key(k)] = v

src/pretix/base/email.py

@@ -134,8 +134,11 @@ class TemplateBasedMailRenderer(BaseHTMLMailRenderer):
     def template_name(self):
         raise NotImplementedError()
 
+    def compile_markdown(self, plaintext):
+        return markdown_compile_email(plaintext)
+
     def render(self, plain_body: str, plain_signature: str, subject: str, order, position) -> str:
-        body_md = markdown_compile_email(plain_body)
+        body_md = self.compile_markdown(plain_body)
         htmlctx = {
             'site': settings.PRETIX_INSTANCE_NAME,
             'site_url': settings.SITE_URL,
@@ -153,7 +156,7 @@ class TemplateBasedMailRenderer(BaseHTMLMailRenderer):
 
         if plain_signature:
             signature_md = plain_signature.replace('\n', '<br>\n')
-            signature_md = markdown_compile_email(signature_md)
+            signature_md = self.compile_markdown(signature_md)
             htmlctx['signature'] = signature_md
 
         if order:

src/pretix/base/exporter.py

@@ -140,7 +140,7 @@ class BaseExporter:
         """
         return {}
 
-    def render(self, form_data: dict) -> Tuple[str, str, bytes]:
+    def render(self, form_data: dict) -> Tuple[str, str, Optional[bytes]]:
         """
         Render the exported file and return a tuple consisting of a filename, a file type
         and file content.

src/pretix/base/exporters/orderlist.py

@@ -549,7 +549,9 @@ class OrderListExporter(MultiSheetListExporter):
             headers.append(_('End date'))
         headers += [
             _('Product'),
+            _('Product ID'),
             _('Variation'),
+            _('Variation ID'),
             _('Price'),
             _('Tax rate'),
             _('Tax rule'),
@@ -656,7 +658,9 @@ class OrderListExporter(MultiSheetListExporter):
                         row.append('')
                 row += [
                     str(op.item),
+                    str(op.item_id),
                     str(op.variation) if op.variation else '',
+                    str(op.variation_id) if op.variation_id else '',
                     op.price,
                     op.tax_rate,
                     str(op.tax_rule) if op.tax_rule else '',

src/pretix/base/media.py

@@ -49,6 +49,9 @@ class BaseMediaType:
     def handle_unknown(self, organizer, identifier, user, auth):
         pass
 
+    def handle_new(self, organizer, medium, user, auth):
+        pass
+
     def __str__(self):
         return str(self.verbose_name)
 
@@ -108,9 +111,43 @@ class NfcUidMediaType(BaseMediaType):
                 return m
 
 
+class NfcMf0aesMediaType(BaseMediaType):
+    identifier = 'nfc_mf0aes'
+    verbose_name = 'NFC Mifare Ultralight AES'
+    medium_created_by_server = False
+    supports_giftcard = True
+    supports_orderposition = False
+
+    def handle_new(self, organizer, medium, user, auth):
+        from pretix.base.models import GiftCard
+
+        if organizer.settings.get(f'reusable_media_type_{self.identifier}_autocreate_giftcard', as_type=bool):
+            with transaction.atomic():
+                gc = GiftCard.objects.create(
+                    issuer=organizer,
+                    expires=organizer.default_gift_card_expiry,
+                    currency=organizer.settings.get(f'reusable_media_type_{self.identifier}_autocreate_giftcard_currency'),
+                )
+                medium.linked_giftcard = gc
+                medium.save()
+                medium.log_action(
+                    'pretix.reusable_medium.linked_giftcard.changed',
+                    user=user, auth=auth,
+                    data={
+                        'linked_giftcard': gc.pk
+                    }
+                )
+                gc.log_action(
+                    'pretix.giftcards.created',
+                    user=user, auth=auth,
+                )
+                return medium
+
+
 MEDIA_TYPES = {
     m.identifier: m for m in [
         BarcodePlainMediaType(),
         NfcUidMediaType(),
+        NfcMf0aesMediaType(),
     ]
 }

src/pretix/base/middleware.py

@@ -26,7 +26,7 @@ from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
 from django.conf import settings
 from django.http import Http404, HttpRequest, HttpResponse
 from django.middleware.common import CommonMiddleware
-from django.urls import get_script_prefix
+from django.urls import get_script_prefix, resolve
 from django.utils import timezone, translation
 from django.utils.cache import patch_vary_headers
 from django.utils.deprecation import MiddlewareMixin
@@ -230,6 +230,8 @@ class SecurityMiddleware(MiddlewareMixin):
     )
 
     def process_response(self, request, resp):
+        url = resolve(request.path_info)
+
         if settings.DEBUG and resp.status_code >= 400:
             # Don't use CSP on debug error page as it breaks of Django's fancy error
             # pages
@@ -249,20 +251,28 @@ class SecurityMiddleware(MiddlewareMixin):
 
         h = {
             'default-src': ["{static}"],
-            'script-src': ['{static}', 'https://checkout.stripe.com', 'https://js.stripe.com'],
+            'script-src': ['{static}'],
             'object-src': ["'none'"],
-            'frame-src': ['{static}', 'https://checkout.stripe.com', 'https://js.stripe.com'],
+            'frame-src': ['{static}'],
             'style-src': ["{static}", "{media}"],
-            'connect-src': ["{dynamic}", "{media}", "https://checkout.stripe.com"],
+            'connect-src': ["{dynamic}", "{media}"],
-            'img-src': ["{static}", "{media}", "data:", "https://*.stripe.com"] + img_src,
+            'img-src': ["{static}", "{media}", "data:"] + img_src,
             'font-src': ["{static}"],
             'media-src': ["{static}", "data:"],
             # form-action is not only used to match on form actions, but also on URLs
             # form-actions redirect to. In the context of e.g. payment providers or
-            # single-sign-on this can be nearly anything so we cannot really restrict
+            # single-sign-on this can be nearly anything, so we cannot really restrict
             # this. However, we'll restrict it to HTTPS.
             'form-action': ["{dynamic}", "https:"] + (['http:'] if settings.SITE_URL.startswith('http://') else []),
         }
+        # Only include pay.google.com for wallet detection purposes on the Payment selection page
+        if (
+                url.url_name == "event.order.pay.change" or
+                (url.url_name == "event.checkout" and url.kwargs['step'] == "payment")
+        ):
+            h['script-src'].append('https://pay.google.com')
+            h['frame-src'].append('https://pay.google.com')
+            h['connect-src'].append('https://google.com/pay')
         if settings.LOG_CSP:
             h['report-uri'] = ["/csp_report/"]
         if 'Content-Security-Policy' in resp:

src/pretix/base/migrations/0243_device_os_name_and_os_version.py

@@ -0,0 +1,23 @@
+# Generated by Django 4.1.9 on 2023-06-26 10:59
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pretixbase', '0242_auto_20230512_1008'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='device',
+            name='os_name',
+            field=models.CharField(max_length=190, null=True),
+        ),
+        migrations.AddField(
+            model_name='device',
+            name='os_version',
+            field=models.CharField(max_length=190, null=True),
+        ),
+    ]

src/pretix/base/migrations/0244_mediumkeyset.py

@@ -0,0 +1,35 @@
+# Generated by Django 3.2.18 on 2023-05-17 11:32
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pretixbase', '0243_device_os_name_and_os_version'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='device',
+            name='rsa_pubkey',
+            field=models.TextField(null=True),
+        ),
+        migrations.CreateModel(
+            name='MediumKeySet',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False)),
+                ('public_id', models.BigIntegerField(unique=True)),
+                ('media_type', models.CharField(max_length=100)),
+                ('active', models.BooleanField(default=True)),
+                ('uid_key', models.BinaryField()),
+                ('diversification_key', models.BinaryField()),
+                ('organizer', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='medium_key_sets', to='pretixbase.organizer')),
+            ],
+        ),
+        migrations.AddConstraint(
+            model_name='mediumkeyset',
+            constraint=models.UniqueConstraint(condition=models.Q(('active', True)), fields=('organizer', 'media_type'), name='keyset_unique_active'),
+        ),
+    ]

src/pretix/base/migrations/0245_discount_benefit_products.py

@@ -0,0 +1,34 @@
+# Generated by Django 4.2.4 on 2023-08-28 12:30
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("pretixbase", "0244_mediumkeyset"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_apply_to_addons",
+            field=models.BooleanField(default=True),
+        ),
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_ignore_voucher_discounted",
+            field=models.BooleanField(default=False),
+        ),
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_limit_products",
+            field=models.ManyToManyField(
+                related_name="benefit_discounts", to="pretixbase.item"
+            ),
+        ),
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_same_products",
+            field=models.BooleanField(default=True),
+        ),
+    ]

src/pretix/base/models/_transactions.py

@@ -97,7 +97,7 @@ def _transactions_mark_order_dirty(order_id, using=None):
     if getattr(dirty_transactions, 'order_ids', None) is None:
         dirty_transactions.order_ids = set()
 
-    if _check_for_dirty_orders not in [func for savepoint_id, func in conn.run_on_commit]:
+    if _check_for_dirty_orders not in [func for (savepoint_id, func, *__) in conn.run_on_commit]:
         transaction.on_commit(_check_for_dirty_orders, using)
         dirty_transactions.order_ids.clear()  # This is necessary to clean up after old threads with rollbacked transactions
 

src/pretix/base/models/devices.py

@@ -143,6 +143,14 @@ class Device(LoggedModel):
         max_length=190,
         null=True, blank=True
     )
+    os_name = models.CharField(
+        max_length=190,
+        null=True, blank=True
+    )
+    os_version = models.CharField(
+        max_length=190,
+        null=True, blank=True
+    )
     software_brand = models.CharField(
         max_length=190,
         null=True, blank=True
@@ -158,6 +166,10 @@ class Device(LoggedModel):
         null=True,
         blank=False
     )
+    rsa_pubkey = models.TextField(
+        null=True,
+        blank=True,
+    )
     info = models.JSONField(
         null=True, blank=True,
     )

src/pretix/base/models/discount.py

@@ -99,7 +99,7 @@ class Discount(LoggedModel):
     )
     condition_apply_to_addons = models.BooleanField(
         default=True,
-        verbose_name=_("Apply to add-on products"),
+        verbose_name=_("Count add-on products"),
         help_text=_("Discounts never apply to bundled products"),
     )
     condition_ignore_voucher_discounted = models.BooleanField(
@@ -107,7 +107,7 @@ class Discount(LoggedModel):
         verbose_name=_("Ignore products discounted by a voucher"),
         help_text=_("If this option is checked, products that already received a discount through a voucher will not "
                     "be considered for this discount. However, products that use a voucher only to e.g. unlock a "
-                    "hidden product or gain access to sold-out quota will still receive the discount."),
+                    "hidden product or gain access to sold-out quota will still be considered."),
     )
     condition_min_count = models.PositiveIntegerField(
         verbose_name=_('Minimum number of matching products'),
@@ -120,6 +120,19 @@ class Discount(LoggedModel):
         default=Decimal('0.00'),
     )
 
+    benefit_same_products = models.BooleanField(
+        default=True,
+        verbose_name=_("Apply discount to same set of products"),
+        help_text=_("By default, the discount is applied across the same selection of products than the condition for "
+                    "the discount given above. If you want, you can however also select a different selection of "
+                    "products.")
+    )
+    benefit_limit_products = models.ManyToManyField(
+        'Item',
+        verbose_name=_("Apply discount to specific products"),
+        related_name='benefit_discounts',
+        blank=True
+    )
     benefit_discount_matching_percent = models.DecimalField(
         verbose_name=_('Percentual discount on matching products'),
         decimal_places=2,
@@ -139,6 +152,18 @@ class Discount(LoggedModel):
         blank=True,
         validators=[MinValueValidator(1)],
     )
+    benefit_apply_to_addons = models.BooleanField(
+        default=True,
+        verbose_name=_("Apply to add-on products"),
+        help_text=_("Discounts never apply to bundled products"),
+    )
+    benefit_ignore_voucher_discounted = models.BooleanField(
+        default=False,
+        verbose_name=_("Ignore products discounted by a voucher"),
+        help_text=_("If this option is checked, products that already received a discount through a voucher will not "
+                    "be discounted. However, products that use a voucher only to e.g. unlock a hidden product or gain "
+                    "access to sold-out quota will still receive the discount."),
+    )
 
     # more feature ideas:
     # - max_usages_per_order
@@ -187,6 +212,14 @@ class Discount(LoggedModel):
                   'on a minimum value.')
             )
 
+        if data.get('subevent_mode') == cls.SUBEVENT_MODE_DISTINCT and not data.get('benefit_same_products'):
+            raise ValidationError(
+                {'benefit_same_products': [
+                    _('You cannot apply the discount to a different set of products if the discount is only valid '
+                      'for bookings of different dates.')
+                ]}
+            )
+
     def allow_delete(self):
         return not self.orderposition_set.exists()
 
@@ -197,6 +230,7 @@ class Discount(LoggedModel):
             'condition_min_value': self.condition_min_value,
             'benefit_only_apply_to_cheapest_n_matches': self.benefit_only_apply_to_cheapest_n_matches,
             'subevent_mode': self.subevent_mode,
+            'benefit_same_products': self.benefit_same_products,
         })
 
     def is_available_by_time(self, now_dt=None) -> bool:
@@ -207,14 +241,14 @@ class Discount(LoggedModel):
             return False
         return True
 
-    def _apply_min_value(self, positions, idx_group, result):
+    def _apply_min_value(self, positions, condition_idx_group, benefit_idx_group, result):
-        if self.condition_min_value and sum(positions[idx][2] for idx in idx_group) < self.condition_min_value:
+        if self.condition_min_value and sum(positions[idx][2] for idx in condition_idx_group) < self.condition_min_value:
             return
 
         if self.condition_min_count or self.benefit_only_apply_to_cheapest_n_matches:
             raise ValueError('Validation invariant violated.')
 
-        for idx in idx_group:
+        for idx in benefit_idx_group:
             previous_price = positions[idx][2]
             new_price = round_decimal(
                 previous_price * (Decimal('100.00') - self.benefit_discount_matching_percent) / Decimal('100.00'),
@@ -222,8 +256,8 @@ class Discount(LoggedModel):
             )
             result[idx] = new_price
 
-    def _apply_min_count(self, positions, idx_group, result):
+    def _apply_min_count(self, positions, condition_idx_group, benefit_idx_group, result):
-        if len(idx_group) < self.condition_min_count:
+        if len(condition_idx_group) < self.condition_min_count:
             return
 
         if not self.condition_min_count or self.condition_min_value:
@@ -233,15 +267,17 @@ class Discount(LoggedModel):
             if not self.condition_min_count:
                 raise ValueError('Validation invariant violated.')
 
-            idx_group = sorted(idx_group, key=lambda idx: (positions[idx][2], -idx))  # sort by line_price
+            condition_idx_group = sorted(condition_idx_group, key=lambda idx: (positions[idx][2], -idx))  # sort by line_price
+            benefit_idx_group = sorted(benefit_idx_group, key=lambda idx: (positions[idx][2], -idx))  # sort by line_price
 
             # Prevent over-consuming of items, i.e. if our discount is "buy 2, get 1 free", we only
             # want to match multiples of 3
-            consume_idx = idx_group[:len(idx_group) // self.condition_min_count * self.condition_min_count]
+            n_groups = min(len(condition_idx_group) // self.condition_min_count, len(benefit_idx_group))
-            benefit_idx = idx_group[:len(idx_group) // self.condition_min_count * self.benefit_only_apply_to_cheapest_n_matches]
+            consume_idx = condition_idx_group[:n_groups * self.condition_min_count]
+            benefit_idx = benefit_idx_group[:n_groups * self.benefit_only_apply_to_cheapest_n_matches]
         else:
-            consume_idx = idx_group
+            consume_idx = condition_idx_group
-            benefit_idx = idx_group
+            benefit_idx = benefit_idx_group
 
         for idx in benefit_idx:
             previous_price = positions[idx][2]
@@ -276,7 +312,7 @@ class Discount(LoggedModel):
             limit_products = {p.pk for p in self.condition_limit_products.all()}
 
         # First, filter out everything not even covered by our product scope
-        initial_candidates = [
+        condition_candidates = [
             idx
             for idx, (item_id, subevent_id, line_price_gross, is_addon_to, voucher_discount) in positions.items()
             if (
@@ -286,11 +322,25 @@ class Discount(LoggedModel):
             )
         ]
 
+        if self.benefit_same_products:
+            benefit_candidates = list(condition_candidates)
+        else:
+            benefit_products = {p.pk for p in self.benefit_limit_products.all()}
+            benefit_candidates = [
+                idx
+                for idx, (item_id, subevent_id, line_price_gross, is_addon_to, voucher_discount) in positions.items()
+                if (
+                    item_id in benefit_products and
+                    (self.benefit_apply_to_addons or not is_addon_to) and
+                    (not self.benefit_ignore_voucher_discounted or voucher_discount is None or voucher_discount == Decimal('0.00'))
+                )
+            ]
+
         if self.subevent_mode == self.SUBEVENT_MODE_MIXED:  # also applies to non-series events
             if self.condition_min_count:
-                self._apply_min_count(positions, initial_candidates, result)
+                self._apply_min_count(positions, condition_candidates, benefit_candidates, result)
             else:
-                self._apply_min_value(positions, initial_candidates, result)
+                self._apply_min_value(positions, condition_candidates, benefit_candidates, result)
 
         elif self.subevent_mode == self.SUBEVENT_MODE_SAME:
             def key(idx):
@@ -299,17 +349,18 @@ class Discount(LoggedModel):
             # Build groups of candidates with the same subevent, then apply our regular algorithm
             # to each group
 
-            _groups = groupby(sorted(initial_candidates, key=key), key=key)
+            _groups = groupby(sorted(condition_candidates, key=key), key=key)
-            candidate_groups = [list(g) for k, g in _groups]
+            candidate_groups = [(k, list(g)) for k, g in _groups]
 
-            for g in candidate_groups:
+            for subevent_id, g in candidate_groups:
+                benefit_g = [idx for idx in benefit_candidates if positions[idx][1] == subevent_id]
                 if self.condition_min_count:
-                    self._apply_min_count(positions, g, result)
+                    self._apply_min_count(positions, g, benefit_g, result)
                 else:
-                    self._apply_min_value(positions, g, result)
+                    self._apply_min_value(positions, g, benefit_g, result)
 
         elif self.subevent_mode == self.SUBEVENT_MODE_DISTINCT:
-            if self.condition_min_value:
+            if self.condition_min_value or not self.benefit_same_products:
                 raise ValueError('Validation invariant violated.')
 
             # Build optimal groups of candidates with distinct subevents, then apply our regular algorithm
@@ -336,7 +387,7 @@ class Discount(LoggedModel):
                 candidates = []
                 cardinality = None
                 for se, l in subevent_to_idx.items():
-                    l = [ll for ll in l if ll in initial_candidates and ll not in current_group]
+                    l = [ll for ll in l if ll in condition_candidates and ll not in current_group]
                     if cardinality and len(l) != cardinality:
                         continue
                     if se not in {positions[idx][1] for idx in current_group}:
@@ -373,5 +424,5 @@ class Discount(LoggedModel):
                                 break
 
             for g in candidate_groups:
-                self._apply_min_count(positions, g, result)
+                self._apply_min_count(positions, g, g, result)
         return result

src/pretix/base/models/event.py

@@ -907,14 +907,18 @@ class Event(EventMixin, LoggedModel):
             self.items.filter(hidden_if_available_id=oldid).update(hidden_if_available=q)
 
         for d in Discount.objects.filter(event=other).prefetch_related('condition_limit_products'):
-            items = list(d.condition_limit_products.all())
+            c_items = list(d.condition_limit_products.all())
+            b_items = list(d.benefit_limit_products.all())
             d.pk = None
             d.event = self
             d.save(force_insert=True)
             d.log_action('pretix.object.cloned')
-            for i in items:
+            for i in c_items:
                 if i.pk in item_map:
                     d.condition_limit_products.add(item_map[i.pk])
+            for i in b_items:
+                if i.pk in item_map:
+                    d.benefit_limit_products.add(item_map[i.pk])
 
         question_map = {}
         for q in Question.objects.filter(event=other).prefetch_related('items', 'options'):
@@ -1277,6 +1281,9 @@ class Event(EventMixin, LoggedModel):
         return not self.orders.exists() and not self.invoices.exists()
 
     def delete_sub_objects(self):
+        from .checkin import Checkin
+
+        Checkin.all.filter(successful=False, list__event=self).delete()
         self.cartposition_set.filter(addon_to__isnull=False).delete()
         self.cartposition_set.all().delete()
         self.vouchers.all().delete()

src/pretix/base/models/items.py

@@ -43,6 +43,7 @@ from typing import Optional, Tuple
 from zoneinfo import ZoneInfo
 
 import dateutil.parser
+import django_redis
 from dateutil.tz import datetime_exists
 from django.conf import settings
 from django.core.exceptions import ValidationError
@@ -57,7 +58,6 @@ from django.utils.functional import cached_property
 from django.utils.timezone import is_naive, make_aware, now
 from django.utils.translation import gettext_lazy as _, pgettext_lazy
 from django_countries.fields import Country
-from django_redis import get_redis_connection
 from django_scopes import ScopedManager
 from i18nfield.fields import I18nCharField, I18nTextField
 
@@ -1463,7 +1463,7 @@ class Question(LoggedModel):
         (TYPE_PHONENUMBER, _("Phone number")),
     )
     UNLOCALIZED_TYPES = [TYPE_DATE, TYPE_TIME, TYPE_DATETIME]
-    ASK_DURING_CHECKIN_UNSUPPORTED = [TYPE_PHONENUMBER]
+    ASK_DURING_CHECKIN_UNSUPPORTED = []
 
     event = models.ForeignKey(
         Event,
@@ -1910,8 +1910,13 @@ class Quota(LoggedModel):
 
     def rebuild_cache(self, now_dt=None):
         if settings.HAS_REDIS:
-            rc = get_redis_connection("redis")
+            rc = django_redis.get_redis_connection("redis")
-            rc.hdel(f'quotas:{self.event_id}:availabilitycache', str(self.pk))
+            p = rc.pipeline()
+            p.hdel(f'quotas:{self.event_id}:availabilitycache', str(self.pk))
+            p.hdel(f'quotas:{self.event_id}:availabilitycache:nocw', str(self.pk))
+            p.hdel(f'quotas:{self.event_id}:availabilitycache:igcl', str(self.pk))
+            p.hdel(f'quotas:{self.event_id}:availabilitycache:nocw:igcl', str(self.pk))
+            p.execute()
             self.availability(now_dt=now_dt)
 
     def availability(

src/pretix/base/models/log.py

@@ -88,9 +88,7 @@ class LogEntry(models.Model):
 
     class Meta:
         ordering = ('-datetime', '-id')
-        index_together = [
+        indexes = [models.Index(fields=["datetime", "id"])]
-            ['datetime', 'id']
-        ]
 
     def display(self):
         from ..signals import logentry_display

src/pretix/base/models/media.py

@@ -121,5 +121,30 @@ class ReusableMedium(LoggedModel):
 
     class Meta:
         unique_together = (("identifier", "type", "organizer"),)
-        index_together = (("identifier", "type", "organizer"), ("updated", "id"))
+        indexes = [
+            models.Index(fields=("identifier", "type", "organizer")),
+            models.Index(fields=("updated", "id")),
+        ]
         ordering = "identifier", "type", "organizer"
+
+
+class MediumKeySet(models.Model):
+    organizer = models.ForeignKey('Organizer', on_delete=models.CASCADE, related_name='medium_key_sets')
+    public_id = models.BigIntegerField(
+        unique=True,
+    )
+    media_type = models.CharField(max_length=100)
+    active = models.BooleanField(default=True)
+    uid_key = models.BinaryField()
+    diversification_key = models.BinaryField()
+
+    objects = ScopedManager(organizer='organizer')
+
+    class Meta:
+        constraints = [
+            models.UniqueConstraint(
+                fields=["organizer", "media_type"],
+                condition=Q(active=True),
+                name="keyset_unique_active",
+            ),
+        ]

src/pretix/base/models/orders.py

@@ -270,9 +270,9 @@ class Order(LockModel, LoggedModel):
         verbose_name = _("Order")
         verbose_name_plural = _("Orders")
         ordering = ("-datetime", "-pk")
-        index_together = [
+        indexes = [
-            ["datetime", "id"],
+            models.Index(fields=["datetime", "id"]),
-            ["last_modified", "id"],
+            models.Index(fields=["last_modified", "id"]),
         ]
 
     def __str__(self):
@@ -896,6 +896,33 @@ class Order(LockModel, LoggedModel):
             ), tz)
         return term_last
 
+    @property
+    def payment_term_expire_date(self):
+        delay = self.event.settings.get('payment_term_expire_delay_days', as_type=int)
+        if not delay:  # performance saver + backwards compatibility
+            return self.expires
+
+        term_last = self.payment_term_last
+        if term_last and self.expires > term_last:  # backwards compatibility
+            return self.expires
+
+        expires = self.expires.date() + timedelta(days=delay)
+        if self.event.settings.get('payment_term_weekdays'):
+            if expires.weekday() == 5:
+                expires += timedelta(days=2)
+            elif expires.weekday() == 6:
+                expires += timedelta(days=1)
+
+        tz = ZoneInfo(self.event.settings.timezone)
+        expires = make_aware(datetime.combine(
+            expires,
+            time(hour=23, minute=59, second=59)
+        ), tz)
+        if term_last:
+            return min(expires, term_last)
+        else:
+            return expires
+
     def _can_be_paid(self, count_waitinglist=True, ignore_date=False, force=False) -> Union[bool, str]:
         error_messages = {
             'late_lastdate': _("The payment can not be accepted as the last date of payments configured in the "
@@ -1645,12 +1672,13 @@ class OrderPayment(models.Model):
         if status_change:
             self.order.create_transactions()
 
-    def fail(self, info=None, user=None, auth=None, log_data=None):
+    def fail(self, info=None, user=None, auth=None, log_data=None, send_mail=True):
         """
         Marks the order as failed and sets info to ``info``, but only if the order is in ``created`` or ``pending``
         state. This is equivalent to setting ``state`` to ``OrderPayment.PAYMENT_STATE_FAILED`` and logging a failure,
-        but it adds strong database logging since we do not want to report a failure for an order that has just
+        but it adds strong database locking since we do not want to report a failure for an order that has just
         been marked as paid.
+        :param send_mail: Whether an email should be sent to the user about this event (default: ``True``).
         """
         with transaction.atomic():
             locked_instance = OrderPayment.objects.select_for_update(of=OF_SELF).get(pk=self.pk)
@@ -1675,6 +1703,17 @@ class OrderPayment(models.Model):
             'info': info,
             'data': log_data,
         }, user=user, auth=auth)
+
+        if send_mail:
+            with language(self.order.locale, self.order.event.settings.region):
+                email_subject = self.order.event.settings.mail_subject_order_payment_failed
+                email_template = self.order.event.settings.mail_text_order_payment_failed
+                email_context = get_email_context(event=self.order.event, order=self.order)
+                self.order.send_mail(
+                    email_subject, email_template, email_context,
+                    'pretix.event.order.email.payment_failed', user=user, auth=auth,
+                )
+
         return True
 
     def confirm(self, count_waitinglist=True, send_mail=True, force=False, user=None, auth=None, mail_text='',
@@ -2717,8 +2756,8 @@ class Transaction(models.Model):
 
     class Meta:
         ordering = 'datetime', 'pk'
-        index_together = [
+        indexes = [
-            ['datetime', 'id']
+            models.Index(fields=['datetime', 'id'])
         ]
 
     def save(self, *args, **kwargs):

src/pretix/base/models/tax.py

@@ -22,9 +22,12 @@
 import json
 from decimal import Decimal
 
+import jsonschema
+from django.contrib.staticfiles import finders
 from django.core.exceptions import ValidationError
 from django.core.validators import MaxValueValidator, MinValueValidator
 from django.db import models
+from django.utils.deconstruct import deconstructible
 from django.utils.formats import localize
 from django.utils.translation import gettext_lazy as _, pgettext
 from i18nfield.fields import I18nCharField
@@ -135,6 +138,25 @@ def cc_to_vat_prefix(country_code):
     return country_code
 
 
+@deconstructible
+class CustomRulesValidator:
+    def __call__(self, value):
+        if not isinstance(value, dict):
+            try:
+                val = json.loads(value)
+            except ValueError:
+                raise ValidationError(_('Your layout file is not a valid JSON file.'))
+        else:
+            val = value
+        with open(finders.find('schema/tax-rules-custom.schema.json'), 'r') as f:
+            schema = json.loads(f.read())
+        try:
+            jsonschema.validate(val, schema)
+        except jsonschema.ValidationError as e:
+            e = str(e).replace('%', '%%')
+            raise ValidationError(_('Your set of rules is not valid. Error message: {}').format(e))
+
+
 class TaxRule(LoggedModel):
     event = models.ForeignKey('Event', related_name='tax_rules', on_delete=models.CASCADE)
     internal_name = models.CharField(
@@ -318,10 +340,17 @@ class TaxRule(LoggedModel):
         rules = self._custom_rules
         if invoice_address:
             for r in rules:
-                if r['country'] == 'EU' and not is_eu_country(invoice_address.country):
+                if r['country'] == 'ZZ':    # Rule: Any country
-                    continue
+                    pass
-                if r['country'] not in ('ZZ', 'EU') and r['country'] != str(invoice_address.country):
+                elif r['country'] == 'EU':  # Rule: Any EU country
-                    continue
+                    if not is_eu_country(invoice_address.country):
+                        continue
+                elif '-' in r['country']:   # Rule: Specific country and state
+                    if r['country'] != str(invoice_address.country) + '-' + str(invoice_address.state):
+                        continue
+                else:                       # Rule: Specific country
+                    if r['country'] != str(invoice_address.country):
+                        continue
                 if r['address_type'] == 'individual' and invoice_address.is_business:
                     continue
                 if r['address_type'] in ('business', 'business_vat_id') and not invoice_address.is_business:

src/pretix/base/orderimport.py

@@ -28,6 +28,7 @@ import pycountry
 from django.conf import settings
 from django.core.exceptions import ValidationError
 from django.core.validators import EmailValidator
+from django.db.models import Q
 from django.utils import formats
 from django.utils.functional import cached_property
 from django.utils.translation import (
@@ -42,8 +43,8 @@ from phonenumbers import SUPPORTED_REGIONS
 from pretix.base.channels import get_all_sales_channels
 from pretix.base.forms.questions import guess_country
 from pretix.base.models import (
-    ItemVariation, OrderPosition, Question, QuestionAnswer, QuestionOption,
+    Customer, ItemVariation, OrderPosition, Question, QuestionAnswer,
-    Seat, SubEvent,
+    QuestionOption, Seat, SubEvent,
 )
 from pretix.base.services.pricing import get_price
 from pretix.base.settings import (
@@ -804,7 +805,7 @@ class QuestionColumn(ImportColumn):
                 return self.q.clean_answer(value)
 
     def assign(self, value, order, position, invoice_address, **kwargs):
-        if value:
+        if value is not None:
             if not hasattr(order, '_answers'):
                 order._answers = []
             if isinstance(value, QuestionOption):
@@ -826,6 +827,28 @@ class QuestionColumn(ImportColumn):
                 a.options.add(*a._options)
 
 
+class CustomerColumn(ImportColumn):
+    identifier = 'customer'
+    verbose_name = gettext_lazy('Customer')
+
+    def clean(self, value, previous_values):
+        if value:
+            try:
+                value = self.event.organizer.customers.get(
+                    Q(identifier=value) | Q(email__iexact=value) | Q(external_identifier=value)
+                )
+            except Customer.MultipleObjectsReturned:
+                value = self.event.organizer.customers.get(
+                    Q(identifier=value)
+                )
+            except Customer.DoesNotExist:
+                raise ValidationError(_('No matching customer was found.'))
+        return value
+
+    def assign(self, value, order, position, invoice_address, **kwargs):
+        order.customer = value
+
+
 def get_all_columns(event):
     default = []
     if event.has_subevents:
@@ -837,6 +860,10 @@ def get_all_columns(event):
         Variation(event),
         InvoiceAddressCompany(event),
     ]
+    if event.settings.customer_accounts:
+        default += [
+            CustomerColumn(event),
+        ]
     scheme = PERSON_NAME_SCHEMES.get(event.settings.name_scheme)
     for n, l, w in scheme['fields']:
         default.append(InvoiceAddressNamePart(event, n, l))

src/pretix/base/payment.py

@@ -60,7 +60,7 @@ from pretix.base.channels import get_all_sales_channels
 from pretix.base.forms import PlaceholderValidator
 from pretix.base.models import (
     CartPosition, Event, GiftCard, InvoiceAddress, Order, OrderPayment,
-    OrderRefund, Quota,
+    OrderRefund, Quota, TaxRule,
 )
 from pretix.base.reldate import RelativeDateField, RelativeDateWrapper
 from pretix.base.settings import SettingsSandbox
@@ -78,6 +78,16 @@ from pretix.presale.views.cart import cart_session, get_or_create_cart_id
 logger = logging.getLogger(__name__)
 
 
+class WalletQueries:
+    APPLEPAY = 'applepay'
+    GOOGLEPAY = 'googlepay'
+
+    WALLETS = (
+        (APPLEPAY, pgettext_lazy('payment', 'Apple Pay')),
+        (GOOGLEPAY, pgettext_lazy('payment', 'Google Pay')),
+    )
+
+
 class PaymentProviderForm(Form):
     def clean(self):
         cleaned_data = super().clean()
@@ -431,11 +441,31 @@ class BasePaymentProvider:
                      'Share this link with customers who should use this payment method.'
                  ),
              )),
+            ('_prevent_reminder_mail',
+             forms.BooleanField(
+                 label=_('Do not send a payment reminder mail'),
+                 help_text=_('Users will not receive a reminder mail to pay for their order before it expires if '
+                             'they have chosen this payment method.'),
+                 required=False,
+             )),
         ])
         d['_restricted_countries']._as_type = list
         d['_restrict_to_sales_channels']._as_type = list
         return d
 
+    @property
+    def walletqueries(self):
+        """
+        .. warning:: This property is considered **experimental**. It might change or get removed at any time without
+                     prior notice.
+
+        A list of wallet payment methods that should be dynamically joined to the public name of the payment method,
+        if they are available to the user.
+        The detection is made on a best effort basis with no guarantees of correctness and actual availability.
+        Wallets that pretix can check for are exposed through ``pretix.base.payment.WalletQueries``.
+        """
+        return []
+
     def settings_form_clean(self, cleaned_data):
         """
         Overriding this method allows you to inject custom validation into the settings form.
@@ -474,6 +504,14 @@ class BasePaymentProvider:
         if order.status == Order.STATUS_PAID:
             return _('paid')
 
+    def prevent_reminder_mail(self, order: Order, payment: OrderPayment) -> bool:
+        """
+        This is called when a periodic task runs and sends out reminder mails to orders that have not been paid yet
+        and are soon expiring.
+        The default implementation returns the content of the _prevent_reminder_mail configuration variable (a boolean value).
+        """
+        return self.settings.get('_prevent_reminder_mail', as_type=bool, default=False)
+
     @property
     def payment_form_fields(self) -> dict:
         """
@@ -719,7 +757,7 @@ class BasePaymentProvider:
         the amount of money that should be paid.
 
         If you need any special behavior, you can return a string containing the URL the user will be redirected to.
-        If you are done with your process you should return the user to the order's detail page. Redirection is not
+        If you are done with your process you should return the user to the order's detail page. Redirection is only
         allowed if you set ``execute_payment_needs_user`` to ``True``.
 
         If the payment is completed, you should call ``payment.confirm()``. Please note that this might
@@ -1015,7 +1053,11 @@ class FreeOrderProvider(BasePaymentProvider):
 
         cart = get_cart(request)
         total = get_cart_total(request)
-        total += sum([f.value for f in get_fees(self.event, request, total, None, None, cart)])
+        try:
+            total += sum([f.value for f in get_fees(self.event, request, total, None, None, cart)])
+        except TaxRule.SaleNotAllowed:
+            # ignore for now, will fail on order creation
+            pass
         return total == 0
 
     def order_change_allowed(self, order: Order) -> bool:

src/pretix/base/pdf.py

@@ -43,16 +43,18 @@ import subprocess
 import tempfile
 import unicodedata
 import uuid
-from collections import OrderedDict
+from collections import OrderedDict, defaultdict
 from functools import partial
 from io import BytesIO
 
 import jsonschema
+import reportlab.rl_config
 from bidi.algorithm import get_display
 from django.conf import settings
 from django.contrib.staticfiles import finders
 from django.core.exceptions import ValidationError
 from django.db.models import Max, Min
+from django.db.models.fields.files import FieldFile
 from django.dispatch import receiver
 from django.utils.deconstruct import deconstructible
 from django.utils.formats import date_format
@@ -60,7 +62,8 @@ from django.utils.html import conditional_escape
 from django.utils.timezone import now
 from django.utils.translation import gettext_lazy as _, pgettext
 from i18nfield.strings import LazyI18nString
-from pypdf import PdfReader
+from pypdf import PdfReader, PdfWriter, Transformation
+from pypdf.generic import RectangleObject
 from reportlab.graphics import renderPDF
 from reportlab.graphics.barcode.qr import QrCodeWidget
 from reportlab.graphics.shapes import Drawing
@@ -85,6 +88,9 @@ from pretix.presale.style import get_fonts
 
 logger = logging.getLogger(__name__)
 
+if not settings.DEBUG:
+    reportlab.rl_config.shapeChecking = 0
+
 
 DEFAULT_VARIABLES = OrderedDict((
     ("secret", {
@@ -102,7 +108,10 @@ DEFAULT_VARIABLES = OrderedDict((
     ("positionid", {
         "label": _("Order position number"),
         "editor_sample": "1",
-        "evaluate": lambda orderposition, order, event: str(orderposition.positionid)
+        "evaluate": lambda orderposition, order, event: str(orderposition.positionid),
+        # There is no performance gain in using evaluate_bulk here, but we want to make sure it is used somewhere
+        # in core to make sure we notice if the implementation of the API breaks.
+        "evaluate_bulk": lambda orderpositions: [str(p.positionid) for p in orderpositions],
     }),
     ("order_positionid", {
         "label": _("Order code and position number"),
@@ -355,14 +364,9 @@ DEFAULT_VARIABLES = OrderedDict((
     }),
     ("addons", {
         "label": _("List of Add-Ons"),
-        "editor_sample": _("Add-on 1\nAdd-on 2"),
+        "editor_sample": _("Add-on 1\n2x Add-on 2"),
         "evaluate": lambda op, order, ev: "\n".join([
-            '{} - {}'.format(p.item.name, p.variation.value) if p.variation else str(p.item.name)
+            str(p) for p in generate_compressed_addon_list(op, order, ev)
-            for p in (
-                op.addons.all() if 'addons' in getattr(op, '_prefetched_objects_cache', {})
-                else op.addons.select_related('item', 'variation')
-            )
-            if not p.canceled
         ])
     }),
     ("organizer", {
@@ -696,6 +700,30 @@ def get_seat(op: OrderPosition):
     return None
 
 
+def generate_compressed_addon_list(op, order, event):
+    itemcount = defaultdict(int)
+    addons = [p for p in (
+        op.addons.all() if 'addons' in getattr(op, '_prefetched_objects_cache', {})
+        else op.addons.select_related('item', 'variation')
+    ) if not p.canceled]
+    for pos in addons:
+        itemcount[pos.item, pos.variation] += 1
+
+    addonlist = []
+    for (item, variation), count in itemcount.items():
+        if variation:
+            if count > 1:
+                addonlist.append('{}x {} - {}'.format(count, item.name, variation.value))
+            else:
+                addonlist.append('{} - {}'.format(item.name, variation.value))
+        else:
+            if count > 1:
+                addonlist.append('{}x {}'.format(count, item.name))
+            else:
+                addonlist.append(item.name)
+    return addonlist
+
+
 class Renderer:
 
     def __init__(self, event, layout, background_file):
@@ -860,22 +888,37 @@ class Renderer:
                 image_file = None
 
         if image_file:
-            ir = ThumbnailingImageReader(image_file)
             try:
+                ir = ThumbnailingImageReader(image_file)
                 ir.resize(float(o['width']) * mm, float(o['height']) * mm, 300)
+                canvas.drawImage(
+                    image=ir,
+                    x=float(o['left']) * mm,
+                    y=float(o['bottom']) * mm,
+                    width=float(o['width']) * mm,
+                    height=float(o['height']) * mm,
+                    preserveAspectRatio=True,
+                    anchor='c',  # centered in frame
+                    mask='auto'
+                )
+                if isinstance(image_file, FieldFile):
+                    # ThumbnailingImageReader "closes" the file, so it's no use to use the same file pointer
+                    # in case we need it again. For FieldFile, fortunately, there is an easy way to make the file
+                    # refresh itself when it is used next.
+                    del image_file.file
             except:
-                logger.exception("Can not resize image")
+                logger.exception("Can not load or resize image")
-                pass
+                canvas.saveState()
-            canvas.drawImage(
+                canvas.setFillColorRGB(.8, .8, .8, alpha=1)
-                image=ir,
+                canvas.rect(
-                x=float(o['left']) * mm,
+                    x=float(o['left']) * mm,
-                y=float(o['bottom']) * mm,
+                    y=float(o['bottom']) * mm,
-                width=float(o['width']) * mm,
+                    width=float(o['width']) * mm,
-                height=float(o['height']) * mm,
+                    height=float(o['height']) * mm,
-                preserveAspectRatio=True,
+                    stroke=0,
-                anchor='c',  # centered in frame
+                    fill=1,
-                mask='auto'
+                )
-            )
+                canvas.restoreState()
         else:
             canvas.saveState()
             canvas.setFillColorRGB(.8, .8, .8, alpha=1)
@@ -929,7 +972,7 @@ class Renderer:
 
         # reportlab does not support unicode combination characters
         # It's important we do this before we use ArabicReshaper
-        text = unicodedata.normalize("NFKC", text)
+        text = unicodedata.normalize("NFC", text)
 
         # reportlab does not support RTL, ligature-heavy scripts like Arabic. Therefore, we use ArabicReshaper
         # to resolve all ligatures and python-bidi to switch RTL texts.
@@ -982,7 +1025,10 @@ class Renderer:
                 elif o['type'] == "poweredby":
                     self._draw_poweredby(canvas, op, o)
                 if self.bg_pdf:
-                    page_size = (self.bg_pdf.pages[0].mediabox[2], self.bg_pdf.pages[0].mediabox[3])
+                    page_size = (
+                        self.bg_pdf.pages[0].mediabox[2] - self.bg_pdf.pages[0].mediabox[0],
+                        self.bg_pdf.pages[0].mediabox[3] - self.bg_pdf.pages[0].mediabox[1]
+                    )
                     if self.bg_pdf.pages[0].get('/Rotate') in (90, 270):
                         # swap dimensions due to pdf being rotated
                         page_size = page_size[::-1]
@@ -1010,14 +1056,12 @@ class Renderer:
                 with open(os.path.join(d, 'out.pdf'), 'rb') as f:
                     return BytesIO(f.read())
         else:
-            from pypdf import PdfReader, PdfWriter, Transformation
-            from pypdf.generic import RectangleObject
             buffer.seek(0)
             new_pdf = PdfReader(buffer)
             output = PdfWriter()
 
             for i, page in enumerate(new_pdf.pages):
-                bg_page = copy.copy(self.bg_pdf.pages[i])
+                bg_page = copy.deepcopy(self.bg_pdf.pages[i])
                 bg_rotation = bg_page.get('/Rotate')
                 if bg_rotation:
                     # /Rotate is clockwise, transformation.rotate is counter-clockwise
@@ -1054,6 +1098,56 @@ class Renderer:
             return outbuffer
 
 
+def merge_background(fg_pdf, bg_pdf, out_file, compress):
+    if settings.PDFTK:
+        with tempfile.TemporaryDirectory() as d:
+            fg_filename = os.path.join(d, 'fg.pdf')
+            bg_filename = os.path.join(d, 'bg.pdf')
+            fg_pdf.write(fg_filename)
+            bg_pdf.write(bg_filename)
+            pdftk_cmd = [
+                settings.PDFTK,
+                fg_filename,
+                'multibackground',
+                bg_filename,
+                'output',
+                '-',
+            ]
+            if compress:
+                pdftk_cmd.append('compress')
+            subprocess.run(pdftk_cmd, check=True, stdout=out_file)
+    else:
+        output = PdfWriter()
+        for i, page in enumerate(fg_pdf.pages):
+            bg_page = copy.deepcopy(bg_pdf.pages[i])
+            bg_rotation = bg_page.get('/Rotate')
+            if bg_rotation:
+                # /Rotate is clockwise, transformation.rotate is counter-clockwise
+                t = Transformation().rotate(bg_rotation)
+                w = float(page.mediabox.getWidth())
+                h = float(page.mediabox.getHeight())
+                if bg_rotation in (90, 270):
+                    # offset due to rotation base
+                    if bg_rotation == 90:
+                        t = t.translate(h, 0)
+                    else:
+                        t = t.translate(0, w)
+                    # rotate mediabox as well
+                    page.mediabox = RectangleObject((
+                        page.mediabox.left.as_numeric(),
+                        page.mediabox.bottom.as_numeric(),
+                        page.mediabox.top.as_numeric(),
+                        page.mediabox.right.as_numeric(),
+                    ))
+                    page.trimbox = page.mediabox
+                elif bg_rotation == 180:
+                    t = t.translate(w, h)
+                page.add_transformation(t)
+            bg_page.merge_page(page)
+            output.add_page(bg_page)
+        output.write(out_file)
+
+
 @deconstructible
 class PdfLayoutValidator:
     def __call__(self, value):

src/pretix/base/services/cart.py

@@ -141,9 +141,10 @@ error_messages = {
     'price_not_a_number': gettext_lazy('The entered price is not a number.'),
     'price_too_high': gettext_lazy('The entered price is to high.'),
     'voucher_invalid': gettext_lazy('This voucher code is not known in our database.'),
-    'voucher_min_usages': gettext_lazy(
+    'voucher_min_usages': ngettext_lazy(
-        'The voucher code "%(voucher)s" can only be used if you select at least %(number)s '
+        'The voucher code "%(voucher)s" can only be used if you select at least %(number)s matching products.',
-        'matching products.'
+        'The voucher code "%(voucher)s" can only be used if you select at least %(number)s matching products.',
+        'number'
     ),
     'voucher_min_usages_removed': ngettext_lazy(
         'The voucher code "%(voucher)s" can only be used if you select at least %(number)s matching products. '
@@ -317,6 +318,9 @@ class CartManager:
     def _delete_out_of_timeframe(self):
         err = None
         for cp in self.positions:
+            if not cp.pk:
+                continue
+
             if cp.subevent and cp.subevent.presale_start and self.now_dt < cp.subevent.presale_start:
                 err = error_messages['some_subevent_not_started']
                 cp.addons.all().delete()
@@ -1074,6 +1078,7 @@ class CartManager:
         quotas_ok = _get_quota_availability(self._quota_diff, self.now_dt)
         err = None
         new_cart_positions = []
+        deleted_positions = set()
 
         err = err or self._check_min_max_per_product()
 
@@ -1085,7 +1090,10 @@ class CartManager:
                 if op.position.expires > self.now_dt:
                     for q in op.position.quotas:
                         quotas_ok[q] += 1
-                op.position.addons.all().delete()
+                addons = op.position.addons.all()
+                deleted_positions |= {a.pk for a in addons}
+                addons.delete()
+                deleted_positions.add(op.position.pk)
                 op.position.delete()
 
             elif isinstance(op, (self.AddOperation, self.ExtendOperation)):
@@ -1235,20 +1243,28 @@ class CartManager:
                     if op.seat and not op.seat.is_available(ignore_cart=op.position, sales_channel=self._sales_channel,
                                                             ignore_voucher_id=op.position.voucher_id):
                         err = err or error_messages['seat_unavailable']
-                        op.position.addons.all().delete()
+
+                        addons = op.position.addons.all()
+                        deleted_positions |= {a.pk for a in addons}
+                        deleted_positions.add(op.position.pk)
+                        addons.delete()
                         op.position.delete()
                     elif available_count == 1:
                         op.position.expires = self._expiry
                         op.position.listed_price = op.listed_price
                         op.position.price_after_voucher = op.price_after_voucher
                         # op.position.price will be updated by recompute_final_prices_and_taxes()
-                        try:
+                        if op.position.pk not in deleted_positions:
-                            op.position.save(force_update=True, update_fields=['expires', 'listed_price', 'price_after_voucher'])
+                            try:
-                        except DatabaseError:
+                                op.position.save(force_update=True, update_fields=['expires', 'listed_price', 'price_after_voucher'])
-                            # Best effort... The position might have been deleted in the meantime!
+                            except DatabaseError:
-                            pass
+                                # Best effort... The position might have been deleted in the meantime!
+                                pass
                     elif available_count == 0:
-                        op.position.addons.all().delete()
+                        addons = op.position.addons.all()
+                        deleted_positions |= {a.pk for a in addons}
+                        deleted_positions.add(op.position.pk)
+                        addons.delete()
                         op.position.delete()
                     else:
                         raise AssertionError("ExtendOperation cannot affect more than one item")

src/pretix/base/services/checkin.py

@@ -86,8 +86,8 @@ def _build_time(t=None, value=None, ev=None, now_dt=None):
         return ev.date_admission or ev.date_from
 
 
-def _logic_annotate_for_graphic_explain(rules, ev, rule_data):
+def _logic_annotate_for_graphic_explain(rules, ev, rule_data, now_dt):
-    logic_environment = _get_logic_environment(ev)
+    logic_environment = _get_logic_environment(ev, now_dt)
     event = ev if isinstance(ev, Event) else ev.event
 
     def _evaluate_inners(r):
@@ -152,7 +152,7 @@ def _logic_explain(rules, ev, rule_data, now_dt=None):
     get in before 17:00". In the middle of the night it would switch to "You can only get in after 09:00".
     """
     now_dt = now_dt or now()
-    logic_environment = _get_logic_environment(ev)
+    logic_environment = _get_logic_environment(ev, now_dt)
     _var_values = {'False': False, 'True': True}
     _var_explanations = {}
 
@@ -229,7 +229,7 @@ def _logic_explain(rules, ev, rule_data, now_dt=None):
     for vname, data in _var_explanations.items():
         var, operator, rhs = data['var'], data['operator'], data['rhs']
         if var == 'now':
-            compare_to = _build_time(*rhs[0]['buildTime'], ev=ev).astimezone(ev.timezone)
+            compare_to = _build_time(*rhs[0]['buildTime'], ev=ev, now_dt=now_dt).astimezone(ev.timezone)
             tolerance = timedelta(minutes=float(rhs[1])) if len(rhs) > 1 and rhs[1] else timedelta(seconds=0)
             if operator == 'isBefore':
                 compare_to += tolerance
@@ -337,7 +337,7 @@ def _logic_explain(rules, ev, rule_data, now_dt=None):
     return ', '.join(var_texts[v] for v in paths_with_min_weight[0] if not _var_values[v])
 
 
-def _get_logic_environment(ev):
+def _get_logic_environment(ev, now_dt):
     # Every change to our supported JSON logic must be done
     # * in pretix.base.services.checkin
     # * in pretix.base.models.checkin
@@ -354,7 +354,7 @@ def _get_logic_environment(ev):
     logic.add_operation('objectList', lambda *objs: list(objs))
     logic.add_operation('lookup', lambda model, pk, str: int(pk))
     logic.add_operation('inList', lambda a, b: a in b)
-    logic.add_operation('buildTime', partial(_build_time, ev=ev))
+    logic.add_operation('buildTime', partial(_build_time, ev=ev, now_dt=now_dt))
     logic.add_operation('isBefore', is_before)
     logic.add_operation('isAfter', lambda t1, t2, tol=None: is_before(t2, t1, tol))
     return logic
@@ -861,7 +861,7 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
 
         if type == Checkin.TYPE_ENTRY and clist.rules:
             rule_data = LazyRuleVars(op, clist, dt)
-            logic = _get_logic_environment(op.subevent or clist.event)
+            logic = _get_logic_environment(op.subevent or clist.event, now_dt=dt)
             if not logic.apply(clist.rules, rule_data):
                 if force:
                     force_used = True
@@ -886,7 +886,7 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
         if isinstance(auth, Device):
             device = auth
 
-        last_cis = list(op.checkins.order_by('-datetime').filter(list=clist).only('type', 'nonce'))
+        last_cis = list(op.checkins.order_by('-datetime').filter(list=clist).only('type', 'nonce', 'position_id'))
         entry_allowed = (
             type == Checkin.TYPE_EXIT or
             clist.allow_multiple_entries or

src/pretix/base/services/export.py

@@ -26,7 +26,7 @@ from typing import Any, Dict, Union
 from celery.exceptions import MaxRetriesExceededError
 from django.conf import settings
 from django.core.files.base import ContentFile
-from django.db import connection, transaction
+from django.db import close_old_connections, connection, transaction
 from django.dispatch import receiver
 from django.utils.timezone import now, override
 from django.utils.translation import gettext
@@ -86,9 +86,12 @@ def export(self, event: Event, fileid: str, provider: str, form_data: Dict[str,
                         gettext('Your export did not contain any data.')
                     )
                 file.filename, file.type, data = d
+
+                close_old_connections()  # This task can run very long, we might need a new DB connection
+
                 f = ContentFile(data)
                 file.file.save(cachedfile_name(file, file.filename), f)
-    return file.pk
+    return str(file.pk)
 
 
 @app.task(base=ProfiledOrganizerUserTask, throws=(ExportError,), bind=True)
@@ -154,9 +157,12 @@ def multiexport(self, organizer: Organizer, user: User, device: int, token: int,
                         gettext('Your export did not contain any data.')
                     )
                 file.filename, file.type, data = d
+
+                close_old_connections()  # This task can run very long, we might need a new DB connection
+
                 f = ContentFile(data)
                 file.file.save(cachedfile_name(file, file.filename), f)
-    return file.pk
+    return str(file.pk)
 
 
 def _run_scheduled_export(schedule, context: Union[Event, Organizer], exporter, config_url, retry_func, has_permission):
@@ -214,6 +220,11 @@ def _run_scheduled_export(schedule, context: Union[Event, Organizer], exporter,
                 raise ExportError(
                     gettext('Your exported data exceeded the size limit for scheduled exports.')
                 )
+
+            conn = transaction.get_connection()
+            if not conn.in_atomic_block:  # atomic execution only happens during tests or with celery always_eager on
+                close_old_connections()  # This task can run very long, we might need a new DB connection
+
             f = ContentFile(data)
             file.file.save(cachedfile_name(file, file.filename), f)
         except ExportEmptyError as e:

src/pretix/base/services/media.py

@@ -0,0 +1,72 @@
+#
+# This file is part of pretix (Community Edition).
+#
+# Copyright (C) 2014-2020 Raphael Michel and contributors
+# Copyright (C) 2020-2021 rami.io GmbH and contributors
+#
+# This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
+# Public License as published by the Free Software Foundation in version 3 of the License.
+#
+# ADDITIONAL TERMS APPLY: Pursuant to Section 7 of the GNU Affero General Public License, additional terms are
+# applicable granting you additional permissions and placing additional restrictions on your usage of this software.
+# Please refer to the pretix LICENSE file to obtain the full terms applicable to this work. If you did not receive
+# this file, see <https://pretix.eu/about/en/license>.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
+# <https://www.gnu.org/licenses/>.
+#
+import secrets
+
+from django.db import IntegrityError
+from django.db.models import Q
+from django_scopes import scopes_disabled
+
+from pretix.base.models import GiftCardAcceptance
+from pretix.base.models.media import MediumKeySet
+
+
+def create_nfc_mf0aes_keyset(organizer):
+    for i in range(20):
+        public_id = secrets.randbelow(2 ** 32)
+        uid_key = secrets.token_bytes(16)
+        diversification_key = secrets.token_bytes(16)
+        try:
+            return MediumKeySet.objects.create(
+                organizer=organizer,
+                media_type="nfc_mf0aes",
+                public_id=public_id,
+                diversification_key=diversification_key,
+                uid_key=uid_key,
+                active=True,
+            )
+        except IntegrityError:  # either race condition with another thread or duplicate public ID
+            try:
+                return MediumKeySet.objects.get(
+                    organizer=organizer,
+                    media_type="nfc_mf0aes",
+                    active=True,
+                )
+            except MediumKeySet.DoesNotExist:
+                continue  # duplicate public ID, let's try again
+
+
+@scopes_disabled()
+def get_keysets_for_organizer(organizer):
+    sets = list(MediumKeySet.objects.filter(
+        Q(organizer=organizer) | Q(organizer__in=GiftCardAcceptance.objects.filter(
+            acceptor=organizer,
+            active=True,
+            reusable_media=True,
+        ).values_list("issuer_id", flat=True))
+    ))
+    if organizer.settings.reusable_media_type_nfc_mf0aes and not any(
+        ks.organizer == organizer and ks.media_type == "nfc_mf0aes" for ks in sets
+    ):
+        new_set = create_nfc_mf0aes_keyset(organizer)
+        if new_set:
+            sets.append(new_set)
+    return sets

src/pretix/base/services/orders.py

@@ -935,7 +935,10 @@ def _create_order(event: Event, email: str, positions: List[CartPosition], now_d
             raise OrderError(e.message)
 
         require_approval = any(p.requires_approval(invoice_address=address) for p in positions)
-        fees = _get_fees(positions, payment_requests, address, meta_info, event, require_approval=require_approval)
+        try:
+            fees = _get_fees(positions, payment_requests, address, meta_info, event, require_approval=require_approval)
+        except TaxRule.SaleNotAllowed:
+            raise OrderError(error_messages['country_blocked'])
         total = pending_sum = sum([c.price for c in positions]) + sum([c.value for c in fees])
 
         order = Order(
@@ -968,7 +971,10 @@ def _create_order(event: Event, email: str, positions: List[CartPosition], now_d
 
         for fee in fees:
             fee.order = order
-            fee._calculate_tax()
+            try:
+                fee._calculate_tax()
+            except TaxRule.SaleNotAllowed:
+                raise OrderError(error_messages['country_blocked'])
             if fee.tax_rule and not fee.tax_rule.pk:
                 fee.tax_rule = None  # TODO: deprecate
             fee.save()
@@ -1264,12 +1270,12 @@ def expire_orders(sender, **kwargs):
         Exists(
             OrderFee.objects.filter(order_id=OuterRef('pk'), fee_type=OrderFee.FEE_TYPE_CANCELLATION)
         )
-    ).select_related('event').order_by('event_id')
+    ).prefetch_related('event').order_by('event_id')
     for o in qs:
         if o.event_id != event_id:
             expire = o.event.settings.get('payment_term_expire_automatically', as_type=bool)
             event_id = o.event_id
-        if expire:
+        if expire and now() >= o.payment_term_expire_date:
             mark_order_expired(o)
 
 
@@ -1283,9 +1289,18 @@ def send_expiry_warnings(sender, **kwargs):
     event_id = None
 
     for o in Order.objects.filter(
-        expires__gte=today, expiry_reminder_sent=False, status=Order.STATUS_PENDING,
+            expires__gte=today, expiry_reminder_sent=False, status=Order.STATUS_PENDING,
-        datetime__lte=now() - timedelta(hours=2), require_approval=False
+            datetime__lte=now() - timedelta(hours=2), require_approval=False
     ).only('pk', 'event_id', 'expires').order_by('event_id'):
+
+        lp = o.payments.last()
+        if (
+                lp and
+                lp.state in [OrderPayment.PAYMENT_STATE_CREATED, OrderPayment.PAYMENT_STATE_PENDING] and
+                lp.payment_provider.prevent_reminder_mail(o, lp)
+        ):
+            continue
+
         if event_id != o.event_id:
             settings = o.event.settings
             days = cache.get_or_set('{}:{}:setting_mail_days_order_expire_warning'.format('event', o.event_id),
@@ -2470,6 +2485,11 @@ class OrderChangeManager:
             split_order.status = Order.STATUS_PAID
         else:
             split_order.status = Order.STATUS_PENDING
+            if self.order.status == Order.STATUS_PAID:
+                split_order.set_expires(
+                    now(),
+                    list(set(p.subevent_id for p in split_positions))
+                )
         split_order.save()
 
         if offset_amount > Decimal('0.00'):

src/pretix/base/services/pricing.py

@@ -171,7 +171,7 @@ def apply_discounts(event: Event, sales_channel: str,
         Q(available_until__isnull=True) | Q(available_until__gte=now()),
         sales_channels__contains=sales_channel,
         active=True,
-    ).prefetch_related('condition_limit_products').order_by('position', 'pk')
+    ).prefetch_related('condition_limit_products', 'benefit_limit_products').order_by('position', 'pk')
     for discount in discount_qs:
         result = discount.apply({
             idx: (item_id, subevent_id, line_price_gross, is_addon_to, voucher_discount)

src/pretix/base/services/quotas.py

@@ -24,13 +24,13 @@ import time
 from collections import Counter, defaultdict
 from itertools import zip_longest
 
+import django_redis
 from django.conf import settings
 from django.db import models
 from django.db.models import (
     Case, Count, F, Func, Max, OuterRef, Q, Subquery, Sum, Value, When,
 )
 from django.utils.timezone import now
-from django_redis import get_redis_connection
 
 from pretix.base.models import (
     CartPosition, Checkin, Order, OrderPosition, Quota, Voucher,
@@ -102,6 +102,12 @@ class QuotaAvailability:
         self.count_waitinglist = defaultdict(int)
         self.count_cart = defaultdict(int)
 
+        self._cache_key_suffix = ""
+        if not self._count_waitinglist:
+            self._cache_key_suffix += ":nocw"
+        if self._ignore_closed:
+            self._cache_key_suffix += ":igcl"
+
         self.sizes = {}
 
     def queue(self, *quota):
@@ -121,17 +127,14 @@ class QuotaAvailability:
             if self._full_results:
                 raise ValueError("You cannot combine full_results and allow_cache.")
 
-            elif not self._count_waitinglist:
-                raise ValueError("If you set allow_cache, you need to set count_waitinglist.")
-
             elif settings.HAS_REDIS:
-                rc = get_redis_connection("redis")
+                rc = django_redis.get_redis_connection("redis")
                 quotas_by_event = defaultdict(list)
                 for q in [_q for _q in self._queue if _q.id in quota_ids_set]:
                     quotas_by_event[q.event_id].append(q)
 
                 for eventid, evquotas in quotas_by_event.items():
-                    d = rc.hmget(f'quotas:{eventid}:availabilitycache', [str(q.pk) for q in evquotas])
+                    d = rc.hmget(f'quotas:{eventid}:availabilitycache{self._cache_key_suffix}', [str(q.pk) for q in evquotas])
                     for redisval, q in zip(d, evquotas):
                         if redisval is not None:
                             data = [rv for rv in redisval.decode().split(',')]
@@ -164,12 +167,12 @@ class QuotaAvailability:
         if not settings.HAS_REDIS or not quotas:
             return
 
-        rc = get_redis_connection("redis")
+        rc = django_redis.get_redis_connection("redis")
         # We write the computed availability to redis in a per-event hash as
         #
         #   quota_id -> (availability_state, availability_number, timestamp).
         #
-        # We store this in a hash instead of inidividual values to avoid making two many redis requests
+        # We store this in a hash instead of individual values to avoid making too many redis requests
         # which would introduce latency.
 
         # The individual entries in the hash are "valid" for 120 seconds. This means in a typical peak scenario with
@@ -179,16 +182,16 @@ class QuotaAvailability:
         # these quotas. We choose 10 seconds since that should be well above the duration of a write.
 
         lock_name = '_'.join([str(p) for p in sorted([q.pk for q in quotas])])
-        if rc.exists(f'quotas:availabilitycachewrite:{lock_name}'):
+        if rc.exists(f'quotas:availabilitycachewrite:{lock_name}{self._cache_key_suffix}'):
             return
-        rc.setex(f'quotas:availabilitycachewrite:{lock_name}', '1', 10)
+        rc.setex(f'quotas:availabilitycachewrite:{lock_name}{self._cache_key_suffix}', '1', 10)
 
         update = defaultdict(list)
         for q in quotas:
             update[q.event_id].append(q)
 
         for eventid, quotas in update.items():
-            rc.hmset(f'quotas:{eventid}:availabilitycache', {
+            rc.hmset(f'quotas:{eventid}:availabilitycache{self._cache_key_suffix}', {
                 str(q.id): ",".join(
                     [str(i) for i in self.results[q]] +
                     [str(int(time.time()))]
@@ -197,7 +200,7 @@ class QuotaAvailability:
             # To make sure old events do not fill up our redis instance, we set an expiry on the cache. However, we set it
             # on 7 days even though we mostly ignore values older than 2 monites. The reasoning is that we have some places
             # where we set allow_cache_stale and use the old entries anyways to save on performance.
-            rc.expire(f'quotas:{eventid}:availabilitycache', 3600 * 24 * 7)
+            rc.expire(f'quotas:{eventid}:availabilitycache{self._cache_key_suffix}', 3600 * 24 * 7)
 
         # We used to also delete item_quota_cache:* from the event cache here, but as the cache
         # gets more complex, this does not seem worth it. The cache is only present for up to

src/pretix/base/services/shredder.py

@@ -20,6 +20,7 @@
 # <https://www.gnu.org/licenses/>.
 #
 
+
 # This file is based on an earlier version of pretix which was released under the Apache License 2.0. The full text of
 # the Apache License 2.0 can be obtained at <http://www.apache.org/licenses/LICENSE-2.0>.
 #
@@ -31,7 +32,7 @@
 # Unless required by applicable law or agreed to in writing, software distributed under the Apache License 2.0 is
 # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 # License for the specific language governing permissions and limitations under the License.
-
+import inspect
 import json
 from datetime import timedelta
 from tempfile import NamedTemporaryFile
@@ -41,10 +42,13 @@ from zipfile import ZipFile
 from dateutil.parser import parse
 from django.conf import settings
 from django.utils.crypto import get_random_string
+from django.utils.formats import date_format
 from django.utils.timezone import now
 from django.utils.translation import gettext_lazy as _
 
-from pretix.base.models import CachedFile, Event, cachedfile_name
+from pretix.base.i18n import language
+from pretix.base.models import CachedFile, Event, User, cachedfile_name
+from pretix.base.services.mail import SendMailException, mail
 from pretix.base.services.tasks import ProfiledEventTask
 from pretix.base.shredder import ShredError
 from pretix.celery_app import app
@@ -101,8 +105,20 @@ def export(event: Event, shredders: List[str], session_key=None, cfid=None) -> N
     return cf.pk
 
 
-@app.task(base=ProfiledEventTask, throws=(ShredError,))
+@app.task(base=ProfiledEventTask, throws=(ShredError,), bind=True)
-def shred(event: Event, fileid: str, confirm_code: str) -> None:
+def shred(self, event: Event, fileid: str, confirm_code: str, user: int=None, locale: str='en') -> None:
+    steps = []
+
+    if user:
+        user = User.objects.get(pk=user)
+
+    def set_progress(val):
+        if not self.request.called_directly:
+            self.update_state(
+                state='PROGRESS',
+                meta={'value': val, 'steps': steps}
+            )
+
     known_shredders = event.get_data_shredders()
     try:
         cf = CachedFile.objects.get(pk=fileid)
@@ -124,8 +140,52 @@ def shred(event: Event, fileid: str, confirm_code: str) -> None:
     if event.logentry_set.filter(datetime__gte=parse(indexdata['time'])):
         raise ShredError(_("Something happened in your event after the export, please try again."))
 
-    for shredder in shredders:
+    event.log_action(
-        shredder.shred_data()
+        'pretix.event.shredder.started', user=user, data={
+            'indexdata': indexdata
+        }
+    )
+
+    for i, shredder in enumerate(shredders):
+        with language(locale):
+            steps.append({'label': str(shredder.verbose_name), 'done': False})
+        set_progress(i * 100 / len(shredders))
+        if 'progress_callback' in inspect.signature(shredder.shred_data).parameters:
+            shredder.shred_data(
+                progress_callback=lambda y: set_progress(
+                    i * 100 / len(shredders) + min(max(y, 0), 100) / 100 * 100 / len(shredders)
+                )
+            )
+        else:
+            shredder.shred_data()
+        steps[-1]['done'] = True
 
     cf.file.delete(save=False)
     cf.delete()
+
+    event.log_action(
+        'pretix.event.shredder.completed', user=user, data={
+            'indexdata': indexdata
+        }
+    )
+
+    if user:
+        with language(user.locale):
+            try:
+                mail(
+                    user.email,
+                    _('Data shredding completed'),
+                    'pretixbase/email/shred_completed.txt',
+                    {
+                        'user': user,
+                        'organizer': event.organizer.name,
+                        'event': str(event.name),
+                        'start_time': date_format(parse(indexdata['time']).astimezone(event.timezone), 'SHORT_DATETIME_FORMAT'),
+                        'shredders': ', '.join([str(s.verbose_name) for s in shredders])
+                    },
+                    event=None,
+                    user=user,
+                    locale=user.locale,
+                )
+            except SendMailException:
+                pass  # Already logged

src/pretix/base/services/waitinglist.py

@@ -22,13 +22,15 @@
 import sys
 from datetime import timedelta
 
-from django.db.models import Exists, F, OuterRef, Q, Sum
+from django.db.models import (
+    Exists, F, OuterRef, Prefetch, Q, Sum, prefetch_related_objects,
+)
 from django.dispatch import receiver
 from django.utils.timezone import now
 from django_scopes import scopes_disabled
 
 from pretix.base.models import (
-    Event, SeatCategoryMapping, User, WaitingListEntry,
+    Event, EventMetaValue, SeatCategoryMapping, User, WaitingListEntry,
 )
 from pretix.base.models.waitinglist import WaitingListException
 from pretix.base.services.tasks import EventTask
@@ -59,8 +61,21 @@ def assign_automatically(event: Event, user_id: int=None, subevent_id: int=None)
         ).aggregate(free=Sum(F('max_usages') - F('redeemed')))['free'] or 0
         seats_available[(m.product_id, m.subevent_id)] = num_free_seets_for_product - num_valid_vouchers_for_product
 
-    qs = WaitingListEntry.objects.filter(
+    prefetch_related_objects(
-        event=event, voucher__isnull=True
+        [event.organizer],
+        'meta_properties'
+    )
+    prefetch_related_objects(
+        [event],
+        Prefetch(
+            'meta_values',
+            EventMetaValue.objects.select_related('property'),
+            to_attr='meta_values_cached'
+        )
+    )
+
+    qs = event.waitinglistentries.filter(
+        voucher__isnull=True
     ).select_related('item', 'variation', 'subevent').prefetch_related(
         'item__quotas', 'variation__quotas'
     ).order_by('-priority', 'created')

src/pretix/base/settings.py

@@ -259,6 +259,46 @@ DEFAULTS = {
             label=_("Gift card currency"),
         )
     },
+    'reusable_media_type_nfc_mf0aes': {
+        'default': 'False',
+        'type': bool,
+        'form_class': forms.BooleanField,
+        'serializer_class': serializers.BooleanField,
+        'form_kwargs': dict(
+            label=_("Active"),
+        )
+    },
+    'reusable_media_type_nfc_mf0aes_autocreate_giftcard': {
+        'default': 'False',
+        'type': bool,
+        'form_class': forms.BooleanField,
+        'serializer_class': serializers.BooleanField,
+        'form_kwargs': dict(
+            label=_("Automatically create a new gift card if a new chip is encoded"),
+        )
+    },
+    'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency': {
+        'default': 'EUR',
+        'type': str,
+        'form_class': forms.ChoiceField,
+        'serializer_class': serializers.ChoiceField,
+        'serializer_kwargs': dict(
+            choices=[(c.alpha_3, c.alpha_3 + " - " + c.name) for c in settings.CURRENCIES],
+        ),
+        'form_kwargs': dict(
+            choices=[(c.alpha_3, c.alpha_3 + " - " + c.name) for c in settings.CURRENCIES],
+            label=_("Gift card currency"),
+        )
+    },
+    'reusable_media_type_nfc_mf0aes_random_uid': {
+        'default': 'False',
+        'type': bool,
+        'form_class': forms.BooleanField,
+        'serializer_class': serializers.BooleanField,
+        'form_kwargs': dict(
+            label=_("Use UID protection feature of NFC chip"),
+        )
+    },
     'max_items_per_order': {
         'default': '10',
         'type': int,
@@ -893,6 +933,28 @@ DEFAULTS = {
                         "the pool and can be ordered by other people."),
         )
     },
+    'payment_term_expire_delay_days': {
+        'default': '0',
+        'type': int,
+        'form_class': forms.IntegerField,
+        'serializer_class': serializers.IntegerField,
+        'form_kwargs': dict(
+            label=_('Expiration delay'),
+            help_text=_("The order will only actually expire this many days after the expiration date communicated "
+                        "to the customer. If you select \"Only end payment terms on weekdays\" above, this will also "
+                        "be respected. However, this will not delay beyond the \"last date of payments\" "
+                        "configured above, which is always enforced."),
+            # Every order in between the official expiry date and the delayed expiry date has a performance penalty
+            # for the cron job, so we limit this feature to 30 days to prevent arbitrary numbers of orders needing
+            # to be checked.
+            min_value=0,
+            max_value=30,
+        ),
+        'serializer_kwargs': dict(
+            min_value=0,
+            max_value=30,
+        ),
+    },
     'payment_pending_hidden': {
         'default': 'False',
         'type': bool,
@@ -971,10 +1033,10 @@ DEFAULTS = {
             widget=forms.RadioSelect,
             choices=(
                 ('False', _('Do not generate invoices')),
-                ('admin', _('Only manually in admin panel')),
+                ('paid', _('Automatically after payment or when required by payment method')),
+                ('True', _('Automatically before payment for all created orders')),
                 ('user', _('Automatically on user request')),
-                ('True', _('Automatically for all created orders')),
+                ('admin', _('Only manually in admin panel')),
-                ('paid', _('Automatically on payment or when required by payment method')),
             ),
             help_text=_("Invoices will never be automatically generated for free orders.")
         )
@@ -2279,6 +2341,24 @@ missing additional payment of **{pending_sum}**.
 You can view the payment information and the status of your order at
 {url}
 
+Best regards,  
+Your {event} team"""))  # noqa: W291
+    },
+    'mail_subject_order_payment_failed': {
+        'type': LazyI18nString,
+        'default': LazyI18nString.from_gettext(gettext_noop("Payment failed for your order: {code}")),
+    },
+    'mail_text_order_payment_failed': {
+        'type': LazyI18nString,
+        'default': LazyI18nString.from_gettext(gettext_noop("""Hello,
+
+your payment attempt for your order for {event} has failed.
+
+Your order is still valid and you can try to pay again using the same or a different payment method. Please complete your payment before {expire_date}.
+
+You can retry the payment and view the status of your order at
+{url}
+
 Best regards,  
 Your {event} team"""))  # noqa: W291
     },
@@ -3635,14 +3715,10 @@ def validate_organizer_settings(organizer, settings_dict):
     # This is not doing anything for the time being.
     # But earlier we called validate_event_settings for the organizer, too - and that didn't do anything for
     # organizer-settings either.
-    #
+    if settings_dict.get('reusable_media_type_nfc_mf0aes') and settings_dict.get('reusable_media_type_nfc_uid'):
-    # N.B.: When actually fleshing out this stub, adding it to the OrganizerUpdateForm should be considered.
-    """
-    if settings_dict.get('reusable_media_type_ntag_pretix1') and settings_dict.get('reusable_media_type_nfc_uid'):
         raise ValidationError({
             'reusable_media_type_nfc_uid': _('This needs to be disabled if other NFC-based types are active.')
         })
-    """
 
 
 def global_settings_object(holder):

src/pretix/base/shredder.py

@@ -32,11 +32,12 @@
 # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 # License for the specific language governing permissions and limitations under the License.
 
+import copy
 import json
 import os
+import time
 from typing import List, Tuple
 
-from django.db import transaction
 from django.db.models import Max, Q
 from django.db.models.functions import Greatest
 from django.dispatch import receiver
@@ -99,11 +100,13 @@ class BaseDataShredder:
         """
         raise NotImplementedError()  # NOQA
 
-    def shred_data(self):
+    def shred_data(self, progress_callback=None):
         """
         This method is called to actually remove the data from the system. You should remove any database objects
         here.
 
+        You can call ``progress_callback`` with an integer value between 0 and 100 to communicate back your progress.
+
         You should never delete ``LogEntry`` objects, but you might modify them to remove personal data. In this
         case, set the ``LogEntry.shredded`` attribute to ``True`` to show that this is no longer original log data.
         """
@@ -151,6 +154,7 @@ class BaseDataShredder:
 
 def shred_log_fields(logentry, banlist=None, whitelist=None):
     d = logentry.parsed_data
+    initial_data = copy.copy(d)
     shredded = False
     if whitelist:
         for k, v in d.items():
@@ -162,9 +166,63 @@ def shred_log_fields(logentry, banlist=None, whitelist=None):
             if f in d:
                 d[f] = '█'
                 shredded = True
-    logentry.data = json.dumps(d)
+    if d != initial_data:
-    logentry.shredded = logentry.shredded or shredded
+        logentry.data = json.dumps(d)
-    logentry.save(update_fields=['data', 'shredded'])
+        logentry.shredded = logentry.shredded or shredded
+        logentry.save(update_fields=['data', 'shredded'])
+
+
+def slow_update(qs, batch_size=1000, sleep_time=.5, progress_callback=None, progress_offset=0, progress_total=None, **update):
+    """
+    Doing UPDATE queries on hundreds of thousands of rows can cause outages due to high write load on the database.
+    This provides a throttled way to update rows. The condition for this to work properly is that the queryset has a
+    filter condition that no longer applies after the update!
+    Otherwise, this will be an endless loop!
+    """
+    total_updated = 0
+    while True:
+        updated = qs.order_by().filter(
+            pk__in=qs.order_by().values_list('pk', flat=True)[:batch_size]
+        ).update(**update)
+        total_updated += updated
+        if not updated:
+            break
+        if total_updated >= 0.8 * batch_size:
+            time.sleep(sleep_time)
+        if progress_callback and progress_total:
+            progress_callback((progress_offset + total_updated) / progress_total)
+
+    return total_updated
+
+
+def slow_delete(qs, batch_size=1000, sleep_time=.5, progress_callback=None, progress_offset=0, progress_total=None):
+    """
+    Doing DELETE queries on hundreds of thousands of rows can cause outages due to high write load on the database.
+    This provides a throttled way to update rows.
+    """
+    total_deleted = 0
+    while True:
+        deleted = qs.order_by().filter(
+            pk__in=qs.order_by().values_list('pk', flat=True)[:batch_size]
+        ).delete()[0]
+        total_deleted += deleted
+        if not deleted:
+            break
+        if total_deleted >= 0.8 * batch_size:
+            time.sleep(sleep_time)
+        if progress_callback and progress_total:
+            progress_callback((progress_offset + total_deleted) / progress_total)
+    return total_deleted
+
+
+def _progress_helper(queryset, progress_callback, offset, total):
+    if not progress_callback:
+        yield from queryset
+    else:
+        for i, o in enumerate(queryset):
+            yield o
+            if i % 10 == 0:
+                progress_callback((i + offset) / total * 100)
 
 
 class PhoneNumberShredder(BaseDataShredder):
@@ -177,18 +235,26 @@ class PhoneNumberShredder(BaseDataShredder):
             o.code: o.phone for o in self.event.orders.filter(phone__isnull=False)
         }, cls=CustomJSONEncoder, indent=4)
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_orders = self.event.orders.all()
-        for o in self.event.orders.all():
+        qs_orders_cnt = qs_orders.count()
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.phone.changed")
+        qs_le_cnt = qs_le.count()
+        total = qs_le_cnt + qs_orders_cnt
+
+        for o in _progress_helper(qs_orders, progress_callback, 0, total):
+            changed = bool(o.phone)
             o.phone = None
             d = o.meta_info_data
             if d:
                 if 'contact_form_data' in d and 'phone' in d['contact_form_data']:
+                    changed = True
                     del d['contact_form_data']['phone']
-                o.meta_info = json.dumps(d)
+                    o.meta_info = json.dumps(d)
-            o.save(update_fields=['meta_info', 'phone'])
+            if changed:
+                o.save(update_fields=['meta_info', 'phone'])
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.phone.changed"):
+        for le in _progress_helper(qs_le, progress_callback, qs_orders_cnt, total):
             shred_log_fields(le, banlist=['old_phone', 'new_phone'])
 
 
@@ -207,37 +273,66 @@ class EmailAddressShredder(BaseDataShredder):
             for op in OrderPosition.all.filter(order__event=self.event, attendee_email__isnull=False)
         }, indent=4)
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_op = OrderPosition.all.filter(order__event=self.event, attendee_email__isnull=False)
-        OrderPosition.all.filter(order__event=self.event, attendee_email__isnull=False).update(attendee_email=None)
+        qs_op_cnt = qs_op.count()
 
-        for o in self.event.orders.all():
+        qs_orders = self.event.orders.all()
+        qs_orders_cnt = qs_orders.count()
+
+        qs_le = self.event.logentry_set.filter(
+            Q(action_type__contains="order.email") | Q(action_type__contains="position.email") |
+            Q(action_type="pretix.event.order.contact.changed") |
+            Q(action_type="pretix.event.order.modified")
+        ).exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_op_cnt + qs_orders_cnt + qs_le_cnt
+
+        slow_update(
+            qs_op,
+            attendee_email=None,
+            progress_callback=progress_callback,
+            progress_offset=0,
+            progress_total=total,
+            # Updates to order position table are slow, since PostgreSQL needs to update many indexes, so let's
+            # take them really slowly to not overwhelm the database.
+            batch_size=100,
+            sleep_time=2,
+        )
+
+        for o in _progress_helper(qs_orders, progress_callback, qs_op_cnt, total):
+            changed = bool(o.email) or bool(o.customer)
             o.email = None
             o.customer = None
             d = o.meta_info_data
             if d:
                 if 'contact_form_data' in d and 'email' in d['contact_form_data']:
                     del d['contact_form_data']['email']
-                o.meta_info = json.dumps(d)
+                    changed = True
-            o.save(update_fields=['meta_info', 'email', 'customer'])
+                    o.meta_info = json.dumps(d)
+                if 'contact_form_data' in d and 'email_repeat' in d['contact_form_data']:
+                    del d['contact_form_data']['email_repeat']
+                    changed = True
+            if changed:
+                if d:
+                    o.meta_info = json.dumps(d)
+                o.save(update_fields=['meta_info', 'email', 'customer'])
 
-        for le in self.event.logentry_set.filter(
+        for le in _progress_helper(qs_le, progress_callback, qs_op_cnt + qs_orders_cnt, total):
-            Q(action_type__contains="order.email") | Q(action_type__contains="position.email"),
+            if le.action_type == "pretix.event.order.modified":
-        ):
+                d = le.parsed_data
-            shred_log_fields(le, banlist=['recipient', 'message', 'subject', 'full_mail'])
+                if 'data' in d:
-
+                    for row in d['data']:
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.contact.changed"):
+                        if 'attendee_email' in row:
-            shred_log_fields(le, banlist=['old_email', 'new_email'])
+                            row['attendee_email'] = '█'
-
+                    le.data = json.dumps(d)
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
+                    le.shredded = True
-            d = le.parsed_data
+                    le.save(update_fields=['data', 'shredded'])
-            if 'data' in d:
+            else:
-                for row in d['data']:
+                shred_log_fields(le, banlist=[
-                    if 'attendee_email' in row:
+                    'recipient', 'message', 'subject', 'full_mail', 'old_email', 'new_email'
-                        row['attendee_email'] = '█'
+                ])
-                le.data = json.dumps(d)
-                le.shredded = True
-                le.save(update_fields=['data', 'shredded'])
 
 
 class WaitingListShredder(BaseDataShredder):
@@ -251,16 +346,35 @@ class WaitingListShredder(BaseDataShredder):
             for wle in self.event.waitinglistentries.all()
         ], indent=4)
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_wle = self.event.waitinglistentries.exclude(email='█')
-        self.event.waitinglistentries.update(name_cached=None, name_parts={'_shredded': True}, email='█', phone='█')
+        qs_wle_cnt = qs_wle.count()
 
-        for wle in self.event.waitinglistentries.select_related('voucher').filter(voucher__isnull=False):
+        qs_voucher = self.event.waitinglistentries.select_related('voucher').filter(voucher__isnull=False)
+        qs_voucher_cnt = qs_voucher.count()
+
+        qs_le = self.event.logentry_set.filter(action_type="pretix.voucher.added.waitinglist").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_voucher_cnt + qs_wle_cnt + qs_le_cnt
+
+        slow_update(
+            qs_wle,
+            name_cached=None,
+            name_parts={'_shredded': True},
+            email='█',
+            phone='█',
+            progress_callback=progress_callback,
+            progress_offset=0,
+            progress_total=total,
+        )
+
+        for wle in _progress_helper(qs_voucher, progress_callback, qs_wle_cnt, total):
             if '@' in wle.voucher.comment:
                 wle.voucher.comment = '█'
                 wle.voucher.save(update_fields=['comment'])
 
-        for le in self.event.logentry_set.filter(action_type="pretix.voucher.added.waitinglist").exclude(data=""):
+        for le in _progress_helper(qs_le, progress_callback, qs_wle_cnt + qs_voucher_cnt, total):
             d = le.parsed_data
             if 'name' in d:
                 d['name'] = '█'
@@ -298,17 +412,41 @@ class AttendeeInfoShredder(BaseDataShredder):
             )
         }, indent=4)
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_op = OrderPosition.all.filter(
-        OrderPosition.all.filter(
             order__event=self.event
         ).filter(
-            Q(attendee_name_cached__isnull=False) | Q(attendee_name_parts__isnull=False) |
+            Q(attendee_name_cached__isnull=False) |
-            Q(company__isnull=False) | Q(street__isnull=False) | Q(zipcode__isnull=False) | Q(city__isnull=False)
+            Q(company__isnull=False) |
-        ).update(attendee_name_cached=None, attendee_name_parts={'_shredded': True}, company=None, street=None,
+            Q(street__isnull=False) |
-                 zipcode=None, city=None)
+            Q(zipcode__isnull=False) |
+            Q(city__isnull=False)
+        )
+        qs_op_cnt = qs_op.count()
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_op_cnt + qs_le_cnt
+
+        slow_update(
+            qs_op,
+            attendee_name_cached=None,
+            attendee_name_parts={'_shredded': True},
+            company=None,
+            street=None,
+            zipcode=None,
+            city=None,
+            progress_callback=progress_callback,
+            progress_total=total,
+            progress_offset=0,
+            # Updates to order position table are slow, since PostgreSQL needs to update many indexes, so let's
+            # take them really slowly to not overwhelm the database.
+            batch_size=100,
+            sleep_time=2,
+        )
+
+        for le in _progress_helper(qs_le, progress_callback, qs_op_cnt, total):
             d = le.parsed_data
             if 'data' in d:
                 for i, row in enumerate(d['data']):
@@ -343,11 +481,18 @@ class InvoiceAddressShredder(BaseDataShredder):
             for ia in InvoiceAddress.objects.filter(order__event=self.event)
         }, indent=4)
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_ia = InvoiceAddress.objects.filter(order__event=self.event)
-        InvoiceAddress.objects.filter(order__event=self.event).delete()
+        qs_ia_cnt = qs_ia.count()
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_ia_cnt + qs_le_cnt
+
+        slow_delete(qs_ia, progress_callback=progress_callback, progress_total=total, progress_offset=0)
+
+        for le in _progress_helper(qs_le, progress_callback, qs_ia_cnt, total):
             d = le.parsed_data
             if 'invoice_data' in d and not isinstance(d['invoice_data'], bool):
                 for field in d['invoice_data']:
@@ -375,11 +520,18 @@ class QuestionAnswerShredder(BaseDataShredder):
             ).data
         yield 'question-answers.json', 'application/json', json.dumps(d, indent=4)
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_qa = QuestionAnswer.objects.filter(orderposition__order__event=self.event)
-        QuestionAnswer.objects.filter(orderposition__order__event=self.event).delete()
+        qs_qa_cnt = qs_qa.count()
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_qa_cnt + qs_le_cnt
+
+        slow_delete(qs_qa, progress_callback=progress_callback, progress_total=total, progress_offset=0)
+
+        for le in _progress_helper(qs_le, progress_callback, qs_qa_cnt, total):
             d = le.parsed_data
             if 'data' in d:
                 for i, row in enumerate(d['data']):
@@ -408,9 +560,11 @@ class InvoiceShredder(BaseDataShredder):
             yield 'invoices/{}.pdf'.format(i.number), 'application/pdf', i.file.read()
             i.file.close()
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_i = self.event.invoices.filter(shredded=False)
-        for i in self.event.invoices.filter(shredded=False):
+        total = qs_i.count()
+
+        for i in _progress_helper(qs_i, progress_callback, 0, total):
             if i.file:
                 i.file.delete()
                 i.shredded = True
@@ -430,10 +584,17 @@ class CachedTicketShredder(BaseDataShredder):
     def generate_files(self) -> List[Tuple[str, str, str]]:
         pass
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_1 = CachedTicket.objects.filter(order_position__order__event=self.event)
-        CachedTicket.objects.filter(order_position__order__event=self.event).delete()
+        qs_1_cnt = qs_1.count()
-        CachedCombinedTicket.objects.filter(order__event=self.event).delete()
+
+        qs_2 = CachedCombinedTicket.objects.filter(order__event=self.event)
+        qs_2_cnt = qs_2.count()
+
+        total = qs_1_cnt + qs_2_cnt
+
+        slow_delete(qs_1, progress_callback=progress_callback, progress_total=total, progress_offset=0)
+        slow_delete(qs_2, progress_callback=progress_callback, progress_total=total, progress_offset=qs_1_cnt)
 
 
 class PaymentInfoShredder(BaseDataShredder):
@@ -446,14 +607,21 @@ class PaymentInfoShredder(BaseDataShredder):
     def generate_files(self) -> List[Tuple[str, str, str]]:
         pass
 
-    @transaction.atomic
+    def shred_data(self, progress_callback=None):
-    def shred_data(self):
+        qs_p = OrderPayment.objects.filter(order__event=self.event)
+        qs_p_count = qs_p.count()
+        qs_r = OrderRefund.objects.filter(order__event=self.event)
+        qs_r_count = qs_r.count()
+
+        total = qs_p_count + qs_r_count
+
         provs = self.event.get_payment_providers()
-        for obj in OrderPayment.objects.filter(order__event=self.event):
+        for obj in _progress_helper(qs_p, progress_callback, 0, total):
             pprov = provs.get(obj.provider)
             if pprov:
                 pprov.shred_payment_info(obj)
-        for obj in OrderRefund.objects.filter(order__event=self.event):
+
+        for obj in _progress_helper(qs_r, progress_callback, qs_p_count, total):
             pprov = provs.get(obj.provider)
             if pprov:
                 pprov.shred_payment_info(obj)

src/pretix/base/signals.py

@@ -683,12 +683,16 @@ dictionaries as values that contain keys like in the following example::
         "product": {
             "label": _("Product name"),
             "editor_sample": _("Sample product"),
-            "evaluate": lambda orderposition, order, event: str(orderposition.item)
+            "evaluate": lambda orderposition, order, event: str(orderposition.item),
+            "evaluate_bulk": lambda orderpositions: [str(op.item) for op in orderpositions],
         }
     }
 
 The ``evaluate`` member will be called with the order position, order and event as arguments. The event might
 also be a subevent, if applicable.
+
+The ``evaluate_bulk`` member is optional but can significantly improve performance in some situations because you
+can perform database fetches in bulk instead of single queries for every position.
 """
 
 
@@ -787,3 +791,23 @@ return a dictionary mapping names of attributes in the settings store to DRF ser
 
 As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
 """
+
+customer_created = GlobalSignal()
+"""
+Arguments: ``customer``
+
+This signal is sent out every time a customer account is created. The ``customer``
+object is given as the first argument.
+
+The ``sender`` keyword argument will contain the organizer.
+"""
+
+customer_signed_in = GlobalSignal()
+"""
+Arguments: ``customer``
+
+This signal is sent out every time a customer signs in. The ``customer`` object
+is given as the first argument.
+
+The ``sender`` keyword argument will contain the organizer.
+"""

src/pretix/base/templates/pretixbase/email/shred_completed.txt

@@ -0,0 +1,17 @@
+{% load i18n %}
+{% load i18n %}{% blocktrans with url=url|safe %}Hello,
+
+we hereby confirm that the following data shredding job has been completed:
+
+Organizer: {{ organizer }}
+
+Event: {{ event }}
+
+Data selection: {{ shredders }}
+
+Start time: {{ start_time }} (new data added after this time might not have been deleted)
+
+Best regards,
+
+Your pretix team
+{% endblocktrans %}

src/pretix/base/templatetags/rich_text.py

@@ -48,6 +48,8 @@ from django.utils.http import url_has_allowed_host_and_scheme
 from django.utils.safestring import mark_safe
 from markdown import Extension
 from markdown.inlinepatterns import SubstituteTagInlineProcessor
+from markdown.postprocessors import Postprocessor
+from markdown.treeprocessors import UnescapeTreeprocessor
 from tlds import tld_set
 
 register = template.Library()
@@ -108,6 +110,8 @@ URL_RE = SimpleLazyObject(lambda: build_url_re(tlds=sorted(tld_set, key=len, rev
 
 EMAIL_RE = SimpleLazyObject(lambda: build_email_re(tlds=sorted(tld_set, key=len, reverse=True)))
 
+DOT_ESCAPE = "|escaped-dot-sGnY9LMK|"
+
 
 def safelink_callback(attrs, new=False):
     """
@@ -185,25 +189,130 @@ class EmailNl2BrExtension(Extension):
         md.inlinePatterns.register(br_tag, 'nl', 5)
 
 
-def markdown_compile_email(source):
+class LinkifyPostprocessor(Postprocessor):
+    def __init__(self, linker):
+        self.linker = linker
+        super().__init__()
+
+    def run(self, text):
+        return self.linker.linkify(text)
+
+
+class CleanPostprocessor(Postprocessor):
+    def __init__(self, tags, attributes, protocols, strip):
+        self.tags = tags
+        self.attributes = attributes
+        self.protocols = protocols
+        self.strip = strip
+        super().__init__()
+
+    def run(self, text):
+        return bleach.clean(
+            text,
+            tags=self.tags,
+            attributes=self.attributes,
+            protocols=self.protocols,
+            strip=self.strip
+        )
+
+
+class CustomUnescapeTreeprocessor(UnescapeTreeprocessor):
+    """
+    This un-escapes everything except \\.
+    """
+
+    def _unescape(self, m):
+        if m.group(1) == "46":  # 46 is the ASCII position of .
+            return DOT_ESCAPE
+        return chr(int(m.group(1)))
+
+
+class CustomUnescapePostprocessor(Postprocessor):
+    """
+    Restore escaped .
+    """
+
+    def run(self, text):
+        return text.replace(DOT_ESCAPE, ".")
+
+
+class LinkifyAndCleanExtension(Extension):
+    r"""
+    We want to do:
+
+    input --> markdown --> bleach clean --> linkify --> output
+
+    Internally, the markdown library does:
+
+    source --> parse --> (tree|inline)processors --> serializing --> postprocessors
+
+    All escaped characters such as \. will be turned to something like <STX>46<ETX> in the processors
+    step and then will be converted to . back again in the last tree processor, before serialization.
+    Therefore, linkify does not see the escaped character anymore. This is annoying for the one case
+    where you want to type "rich_text.py" and *not* have it turned into a link, since you can't type
+    "rich_text\.py" either.
+
+    A simple solution would be to run linkify before markdown, but that may cause other issues when
+    linkify messes with the markdown syntax and it makes handling our attributes etc. harder.
+
+    So we do a weird hack where we modify the unescape processor to unescape everything EXCEPT for the
+    dot and then unescape that one manually after linkify. However, to make things even harder, the bleach
+    clean step removes any invisible characters, so we need to cheat a bit more.
+    """
+
+    def __init__(self, linker, tags, attributes, protocols, strip):
+        self.linker = linker
+        self.tags = tags
+        self.attributes = attributes
+        self.protocols = protocols
+        self.strip = strip
+        super().__init__()
+
+    def extendMarkdown(self, md):
+        md.treeprocessors.deregister('unescape')
+        md.treeprocessors.register(
+            CustomUnescapeTreeprocessor(md),
+            'unescape',
+            0
+        )
+        md.postprocessors.register(
+            CleanPostprocessor(self.tags, self.attributes, self.protocols, self.strip),
+            'clean',
+            2
+        )
+        md.postprocessors.register(
+            LinkifyPostprocessor(self.linker),
+            'linkify',
+            1
+        )
+        md.postprocessors.register(
+            CustomUnescapePostprocessor(self.linker),
+            'unescape_dot',
+            0
+        )
+
+
+def markdown_compile_email(source, allowed_tags=ALLOWED_TAGS, allowed_attributes=ALLOWED_ATTRIBUTES):
     linker = bleach.Linker(
         url_re=URL_RE,
         email_re=EMAIL_RE,
         callbacks=DEFAULT_CALLBACKS + [truelink_callback, abslink_callback],
         parse_email=True
     )
-    return linker.linkify(bleach.clean(
+    return markdown.markdown(
-        markdown.markdown(
+        source,
-            source,
+        extensions=[
-            extensions=[
+            'markdown.extensions.sane_lists',
-                'markdown.extensions.sane_lists',
+            EmailNl2BrExtension(),
-                EmailNl2BrExtension(),
+            LinkifyAndCleanExtension(
-            ]
+                linker,
-        ),
+                tags=allowed_tags,
-        tags=ALLOWED_TAGS,
+                attributes=allowed_attributes,
-        attributes=ALLOWED_ATTRIBUTES,
+                protocols=ALLOWED_PROTOCOLS,
-        protocols=ALLOWED_PROTOCOLS,
+                strip=False,
-    ))
+            )
+        ]
+    )
 
 
 class SnippetExtension(markdown.extensions.Extension):
@@ -213,23 +322,24 @@ class SnippetExtension(markdown.extensions.Extension):
         md.parser.blockprocessors.deregister('quote')
 
 
-def markdown_compile(source, snippet=False):
+def markdown_compile(source, linker, snippet=False):
     tags = ALLOWED_TAGS_SNIPPET if snippet else ALLOWED_TAGS
     exts = [
         'markdown.extensions.sane_lists',
-        'markdown.extensions.nl2br'
+        'markdown.extensions.nl2br',
+        LinkifyAndCleanExtension(
+            linker,
+            tags=tags,
+            attributes=ALLOWED_ATTRIBUTES,
+            protocols=ALLOWED_PROTOCOLS,
+            strip=snippet,
+        )
     ]
     if snippet:
         exts.append(SnippetExtension())
-    return bleach.clean(
+    return markdown.markdown(
-        markdown.markdown(
+        source,
-            source,
+        extensions=exts
-            extensions=exts
-        ),
-        strip=snippet,
-        tags=tags,
-        attributes=ALLOWED_ATTRIBUTES,
-        protocols=ALLOWED_PROTOCOLS,
     )
 
 
@@ -245,7 +355,7 @@ def rich_text(text: str, **kwargs):
         callbacks=DEFAULT_CALLBACKS + ([truelink_callback, safelink_callback] if kwargs.get('safelinks', True) else [truelink_callback, abslink_callback]),
         parse_email=True
     )
-    body_md = linker.linkify(markdown_compile(text))
+    body_md = markdown_compile(text, linker)
     return mark_safe(body_md)
 
 
@@ -261,5 +371,5 @@ def rich_text_snippet(text: str, **kwargs):
         callbacks=DEFAULT_CALLBACKS + ([truelink_callback, safelink_callback] if kwargs.get('safelinks', True) else [truelink_callback, abslink_callback]),
         parse_email=True
     )
-    body_md = linker.linkify(markdown_compile(text, snippet=True))
+    body_md = markdown_compile(text, linker, snippet=True)
     return mark_safe(body_md)

src/pretix/base/views/tasks.py

@@ -115,7 +115,8 @@ class AsyncMixin:
         elif state == 'PROGRESS':
             data.update({
                 'started': True,
-                'percentage': info.get('value', 0) if isinstance(info, dict) else 0
+                'percentage': info.get('value', 0) if isinstance(info, dict) else 0,
+                'steps': info.get('steps', []) if isinstance(info, dict) else None,
             })
         elif state == 'STARTED':
             data.update({

src/pretix/control/forms/discounts.py

@@ -50,11 +50,16 @@ class DiscountForm(I18nModelForm):
             'condition_ignore_voucher_discounted',
             'benefit_discount_matching_percent',
             'benefit_only_apply_to_cheapest_n_matches',
+            'benefit_same_products',
+            'benefit_limit_products',
+            'benefit_apply_to_addons',
+            'benefit_ignore_voucher_discounted',
         ]
         field_classes = {
             'available_from': SplitDateTimeField,
             'available_until': SplitDateTimeField,
             'condition_limit_products': ItemMultipleChoiceField,
+            'benefit_limit_products': ItemMultipleChoiceField,
         }
         widgets = {
             'subevent_mode': forms.RadioSelect,
@@ -64,11 +69,14 @@ class DiscountForm(I18nModelForm):
                 'data-inverse-dependency': '<[name$=all_products]',
                 'class': 'scrolling-multiple-choice',
             }),
+            'benefit_limit_products': forms.CheckboxSelectMultiple(attrs={
+                'class': 'scrolling-multiple-choice',
+            }),
             'benefit_only_apply_to_cheapest_n_matches': forms.NumberInput(
                 attrs={
                     'data-display-dependency': '#id_condition_min_count',
                 }
-            )
+            ),
         }
 
     def __init__(self, *args, **kwargs):
@@ -85,6 +93,7 @@ class DiscountForm(I18nModelForm):
             widget=forms.CheckboxSelectMultiple,
         )
         self.fields['condition_limit_products'].queryset = self.event.items.all()
+        self.fields['benefit_limit_products'].queryset = self.event.items.all()
         self.fields['condition_min_count'].required = False
         self.fields['condition_min_count'].widget.is_required = False
         self.fields['condition_min_value'].required = False

src/pretix/control/forms/event.py

@@ -38,6 +38,7 @@ from decimal import Decimal
 from urllib.parse import urlencode, urlparse
 from zoneinfo import ZoneInfo
 
+import pycountry
 from django import forms
 from django.conf import settings
 from django.core.exceptions import NON_FIELD_ERRORS, ValidationError
@@ -48,7 +49,7 @@ from django.forms import (
 )
 from django.urls import reverse
 from django.utils.functional import cached_property
-from django.utils.html import escape
+from django.utils.html import escape, format_html
 from django.utils.safestring import mark_safe
 from django.utils.timezone import get_current_timezone_name
 from django.utils.translation import gettext, gettext_lazy as _, pgettext_lazy
@@ -65,7 +66,8 @@ from pretix.base.models import Event, Organizer, TaxRule, Team
 from pretix.base.models.event import EventFooterLink, EventMetaValue, SubEvent
 from pretix.base.reldate import RelativeDateField, RelativeDateTimeField
 from pretix.base.settings import (
-    PERSON_NAME_SCHEMES, PERSON_NAME_TITLE_GROUPS, validate_event_settings,
+    COUNTRIES_WITH_STATE_IN_ADDRESS, DEFAULTS, PERSON_NAME_SCHEMES,
+    PERSON_NAME_TITLE_GROUPS, validate_event_settings,
 )
 from pretix.base.validators import multimail_validate
 from pretix.control.forms import (
@@ -749,6 +751,7 @@ class PaymentSettingsForm(EventSettingsValidationMixin, SettingsForm):
         'payment_term_minutes',
         'payment_term_last',
         'payment_term_expire_automatically',
+        'payment_term_expire_delay_days',
         'payment_term_accept_late',
         'payment_pending_hidden',
         'payment_explanation',
@@ -897,6 +900,27 @@ class InvoiceSettingsForm(EventSettingsValidationMixin, SettingsForm):
         )
         self.fields['invoice_numbers_counter_length'].validators.append(MaxValueValidator(15))
 
+        pps = [str(pp.verbose_name) for pp in event.get_payment_providers().values() if pp.requires_invoice_immediately]
+        if pps:
+            generate_paid_help_text = _('An invoice will be issued before payment if the customer selects one of the following payment methods: {list}').format(
+                list=', '.join(pps)
+            )
+        else:
+            generate_paid_help_text = _('None of the currently configured payment methods will cause an invoice to be issued before payment.')
+
+        generate_choices = list(DEFAULTS['invoice_generate']['form_kwargs']['choices'])
+        idx = [i for i, t in enumerate(generate_choices) if t[0] == 'paid'][0]
+        generate_choices[idx] = (
+            'paid',
+            format_html(
+                '{} <span class="label label-success">{}</span><br><span class="text-muted">{}</span>',
+                generate_choices[idx][1],
+                _('Recommended'),
+                generate_paid_help_text
+            )
+        )
+        self.fields['invoice_generate'].choices = generate_choices
+
 
 def contains_web_channel_validate(val):
     if "web" not in val:
@@ -1115,6 +1139,16 @@ class MailSettingsForm(SettingsForm):
         help_text=_("This email only applies to payment methods that can receive incomplete payments, "
                     "such as bank transfer."),
     )
+    mail_subject_order_payment_failed = I18nFormField(
+        label=_("Subject"),
+        required=False,
+        widget=I18nTextInput,
+    )
+    mail_text_order_payment_failed = I18nFormField(
+        label=_("Text"),
+        required=False,
+        widget=I18nTextarea,
+    )
     mail_subject_waiting_list = I18nFormField(
         label=_("Subject"),
         required=False,
@@ -1288,6 +1322,8 @@ class MailSettingsForm(SettingsForm):
         'mail_subject_order_pending_warning': ['event', 'order'],
         'mail_text_order_incomplete_payment': ['event', 'order', 'pending_sum'],
         'mail_subject_order_incomplete_payment': ['event', 'order'],
+        'mail_text_order_payment_failed': ['event', 'order'],
+        'mail_subject_order_payment_failed': ['event', 'order'],
         'mail_text_order_custom_mail': ['event', 'order'],
         'mail_text_download_reminder': ['event', 'order'],
         'mail_subject_download_reminder': ['event', 'order'],
@@ -1415,9 +1451,20 @@ class CountriesAndEU(CachedCountries):
     cache_subkey = 'with_any_or_eu'
 
 
+class CountriesAndEUAndStates(CountriesAndEU):
+    def __iter__(self):
+        for country_code, country_name in super().__iter__():
+            yield country_code, country_name
+            if country_code in COUNTRIES_WITH_STATE_IN_ADDRESS:
+                types, form = COUNTRIES_WITH_STATE_IN_ADDRESS[country_code]
+                yield from sorted(((state.code, country_name + " - " + state.name)
+                                   for state in pycountry.subdivisions.get(country_code=country_code)
+                                   if state.type in types), key=lambda s: s[1])
+
+
 class TaxRuleLineForm(I18nForm):
     country = LazyTypedChoiceField(
-        choices=CountriesAndEU(),
+        choices=CountriesAndEUAndStates(),
         required=False
     )
     address_type = forms.ChoiceField(

src/pretix/control/forms/filter.py

@@ -1732,8 +1732,8 @@ class CheckinListAttendeeFilterForm(FilterForm):
         '-timestamp': (OrderBy(F('last_entry'), nulls_last=True, descending=True), '-order__code'),
         'item': ('item__name', 'variation__value', 'order__code'),
         '-item': ('-item__name', '-variation__value', '-order__code'),
-        'seat': ('seat__sorting_rank', 'seat__guid'),
+        'seat': ('seat__sorting_rank', 'seat__seat_guid'),
-        '-seat': ('-seat__sorting_rank', '-seat__guid'),
+        '-seat': ('-seat__sorting_rank', '-seat__seat_guid'),
         'date': ('subevent__date_from', 'subevent__id', 'order__code'),
         '-date': ('-subevent__date_from', 'subevent__id', '-order__code'),
         'name': {'_order': F('display_name').asc(nulls_first=True),
@@ -1940,7 +1940,7 @@ class VoucherFilterForm(FilterForm):
             'item__category__position',
             'item__category',
             'item__position',
-            'item__variation__position',
+            'variation__position',
             'quota__name',
         ),
         'subevent': 'subevent__date_from',
@@ -1950,7 +1950,7 @@ class VoucherFilterForm(FilterForm):
             '-item__category__position',
             '-item__category',
             '-item__position',
-            '-item__variation__position',
+            '-variation__position',
             '-quota__name',
         )
     }
@@ -2397,6 +2397,61 @@ class CheckinFilterForm(FilterForm):
         return qs
 
 
+class CheckinListFilterForm(FilterForm):
+    orders = {
+        "name": ("name", "subevent__date_from", "pk"),
+        "-name": ("-name", "-subevent__date_from", "-pk"),
+        "subevent": ("subevent__date_from", "name", "pk"),
+        "-subevent": ("-subevent__date_from", "-name", "-pk"),
+    }
+    subevent = forms.ModelChoiceField(
+        label=pgettext_lazy('subevent', 'Date'),
+        queryset=SubEvent.objects.none(),
+        required=False,
+        empty_label=pgettext_lazy('subevent', 'All dates')
+    )
+
+    def __init__(self, *args, **kwargs):
+        self.event = kwargs.pop("event")
+        super().__init__(*args, **kwargs)
+        if self.event.has_subevents:
+            self.fields['subevent'].queryset = self.event.subevents.all()
+            self.fields['subevent'].widget = Select2(
+                attrs={
+                    'data-model-select2': 'event',
+                    'data-select2-url': reverse('control:event.subevents.select2', kwargs={
+                        'event': self.event.slug,
+                        'organizer': self.event.organizer.slug,
+                    }),
+                    'data-placeholder': pgettext_lazy('subevent', 'All dates')
+                }
+            )
+            self.fields['subevent'].widget.choices = self.fields['subevent'].choices
+        elif 'subevent':
+            del self.fields['subevent']
+
+    def filter_qs(self, qs):
+        fdata = self.cleaned_data
+
+        if fdata.get('subevent'):
+            qs = qs.filter(subevent=fdata.get('subevent'))
+
+        if fdata.get("ordering"):
+            ob = self.orders[fdata.get('ordering')]
+            if isinstance(ob, dict):
+                ob = dict(ob)
+                o = ob.pop('_order')
+                qs = qs.annotate(**ob).order_by(*get_deterministic_ordering(OrderPosition, [o]))
+            elif isinstance(ob, (list, tuple)):
+                qs = qs.order_by(*get_deterministic_ordering(OrderPosition, ob))
+            else:
+                qs = qs.order_by(*get_deterministic_ordering(OrderPosition, [ob]))
+        else:
+            qs = qs.order_by("subevent__date_from", "name", "pk")
+
+        return qs.distinct()
+
+
 class DeviceFilterForm(FilterForm):
     orders = {
         'name': Upper('name'),

src/pretix/control/forms/global_settings.py

@@ -86,12 +86,14 @@ class GlobalSettingsForm(SettingsForm):
             ('leaflet_tiles', forms.CharField(
                 required=False,
                 label=_("Leaflet tiles URL pattern"),
-                help_text=_("e.g. {sample}").format(sample="https://a.tile.openstreetmap.org/{z}/{x}/{y}.png")
+                help_text=_("e.g. {sample}").format(sample="https://tile.openstreetmap.org/{z}/{x}/{y}.png")
             )),
             ('leaflet_tiles_attribution', forms.CharField(
                 required=False,
                 label=_("Leaflet tiles attribution"),
-                help_text=_("e.g. {sample}").format(sample='&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors')
+                help_text=_("e.g. {sample}").format(
+                    sample='&copy; &lt;a href=&quot;https://www.openstreetmap.org/copyright&quot;&gt;OpenStreetMap&lt;/a&gt; contributors'
+                )
             )),
         ])
         responses = register_global_settings.send(self)

src/pretix/control/forms/item.py

@@ -461,11 +461,6 @@ class ItemCreateForm(I18nModelForm):
                 )
 
         if self.cleaned_data.get('copy_from'):
-            for mv in self.cleaned_data['copy_from'].meta_values.all():
-                mv.pk = None
-                mv.item = instance
-                mv.save(force_insert=True)
-
             for question in self.cleaned_data['copy_from'].questions.all():
                 question.items.add(instance)
                 question.log_action('pretix.event.question.changed', user=self.user, data={

src/pretix/control/forms/orders.py

@@ -159,7 +159,7 @@ class ReactivateOrderForm(ForceQuotaConfirmationForm):
     pass
 
 
-class CancelForm(ForceQuotaConfirmationForm):
+class CancelForm(forms.Form):
     send_email = forms.BooleanField(
         required=False,
         label=_('Notify customer by email'),
@@ -188,6 +188,7 @@ class CancelForm(ForceQuotaConfirmationForm):
     )
 
     def __init__(self, *args, **kwargs):
+        self.instance = kwargs.pop("instance")
         super().__init__(*args, **kwargs)
         change_decimal_field(self.fields['cancellation_fee'], self.instance.event.currency)
         self.fields['cancellation_fee'].widget.attrs['placeholder'] = floatformat(
@@ -205,6 +206,20 @@ class CancelForm(ForceQuotaConfirmationForm):
         return val
 
 
+class DenyForm(forms.Form):
+    send_email = forms.BooleanField(
+        required=False,
+        label=_('Notify customer by email'),
+        initial=True
+    )
+    comment = forms.CharField(
+        label=_('Comment (will be sent to the user)'),
+        help_text=_('Will be included in the notification email when the respective placeholder is present in the '
+                    'configured email text.'),
+        required=False,
+    )
+
+
 class MarkPaidForm(ConfirmPaymentForm):
     send_email = forms.BooleanField(
         required=False,

src/pretix/control/forms/organizer.py

@@ -412,6 +412,10 @@ class OrganizerSettingsForm(SettingsForm):
         'reusable_media_type_nfc_uid',
         'reusable_media_type_nfc_uid_autocreate_giftcard',
         'reusable_media_type_nfc_uid_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes_random_uid',
     ]
 
     organizer_logo_image = ExtFileField(

src/pretix/control/forms/vouchers.py

@@ -340,6 +340,9 @@ class VoucherBulkForm(VoucherForm):
 
     def clean_send_recipients(self):
         raw = self.cleaned_data['send_recipients']
+        if self.cleaned_data.get('send', None) is False:
+            # No need to validate addresses if the section was turned off
+            return []
         if not raw:
             return []
         r = raw.split('\n')

src/pretix/control/logdisplay.py

@@ -341,6 +341,7 @@ def pretixcontrol_logentry_display(sender: Event, logentry: LogEntry, **kwargs):
         'pretix.giftcards.acceptance.added': _('Gift card acceptance for another organizer has been added.'),
         'pretix.giftcards.acceptance.removed': _('Gift card acceptance for another organizer has been removed.'),
         'pretix.giftcards.acceptance.acceptor.invited': _('A new gift card acceptor has been invited.'),
+        'pretix.giftcards.acceptance.acceptor.removed': _('A gift card acceptor has been removed.'),
         'pretix.giftcards.acceptance.issuer.removed': _('A gift card issuer has been removed or declined.'),
         'pretix.giftcards.acceptance.issuer.accepted': _('A new gift card issuer has been accepted.'),
         'pretix.webhook.created': _('The webhook has been created.'),
@@ -368,10 +369,13 @@ def pretixcontrol_logentry_display(sender: Event, logentry: LogEntry, **kwargs):
         'pretix.reusable_medium.created.auto': _('The reusable medium has been created automatically.'),
         'pretix.reusable_medium.changed': _('The reusable medium has been changed.'),
         'pretix.reusable_medium.linked_orderposition.changed': _('The medium has been connected to a new ticket.'),
+        'pretix.reusable_medium.linked_giftcard.changed': _('The medium has been connected to a new gift card.'),
         'pretix.email.error': _('Sending of an email has failed.'),
         'pretix.event.comment': _('The event\'s internal comment has been updated.'),
         'pretix.event.canceled': _('The event has been canceled.'),
         'pretix.event.deleted': _('An event has been deleted.'),
+        'pretix.event.shredder.started': _('A removal process for personal data has been started.'),
+        'pretix.event.shredder.completed': _('A removal process for personal data has been completed.'),
         'pretix.event.order.modified': _('The order details have been changed.'),
         'pretix.event.order.unpaid': _('The order has been marked as unpaid.'),
         'pretix.event.order.secret.changed': _('The order\'s secret has been changed.'),
@@ -430,6 +434,7 @@ def pretixcontrol_logentry_display(sender: Event, logentry: LogEntry, **kwargs):
                                                                     'the order has been received and requires '
                                                                     'approval.'),
         'pretix.event.order.email.resend': _('An email with a link to the order detail page has been resent to the user.'),
+        'pretix.event.order.email.payment_failed': _('An email has been sent to notify the user that the payment failed.'),
         'pretix.event.order.payment.confirmed': _('Payment {local_id} has been confirmed.'),
         'pretix.event.order.payment.canceled': _('Payment {local_id} has been canceled.'),
         'pretix.event.order.payment.canceled.failed': _('Canceling payment {local_id} has failed.'),

src/pretix/control/signals.py

@@ -304,6 +304,10 @@ an instance of a form class that you bind yourself when appropriate. Your form w
 as part of the standard validation and rendering cycle and rendered using default bootstrap
 styles. It is advisable to set a prefix for your form to avoid clashes with other plugins.
 
+Your forms may also have two special properties: ``template`` with a template that will be
+included to render the form, and ``title``, which will be used as a headline. Your template
+will be passed a ``form`` variable with your form.
+
 As with all plugin signals, the ``sender`` keyword argument will contain the event.
 """
 

src/pretix/control/templates/pretixcontrol/base.html

@@ -470,6 +470,8 @@
                         <div class="progress-bar progress-bar-success">
                         </div>
                     </div>
+                    <div class="steps">
+                    </div>
                 </div>
             </div>
         </div>

src/pretix/control/templates/pretixcontrol/checkin/lists.html

@@ -1,5 +1,7 @@
 {% extends "pretixcontrol/items/base.html" %}
 {% load i18n %}
+{% load bootstrap3 %}
+{% load urlreplace %}
 {% block title %}{% trans "Check-in lists" %}{% endblock %}
 {% block inside %}
     <h1>{% trans "Check-in lists" %}</h1>
@@ -27,11 +29,26 @@
         {% endblocktrans %}
     </p>
     {% if request.event.has_subevents %}
-        <form class="form-inline helper-display-inline" action="" method="get">
+        <div class="panel panel-default">
-            <form class="form-inline helper-display-inline" action="" method="get">
+            <div class="panel-heading">
-                {% include "pretixcontrol/event/fragment_subevent_choice_simple.html" %}
+                <h3 class="panel-title">{% trans "Filter" %}</h3>
+            </div>
+            <form class="panel-body filter-form" action="" method="get">
+                <div class="row">
+                    {% if filter_form.subevent %}
+                        <div class="col-md-12 col-sm-12 col-xs-12">
+                            {% bootstrap_field filter_form.subevent %}
+                        </div>
+                    {% endif %}
+                </div>
+                <div class="text-right">
+                    <button class="btn btn-primary btn-lg" type="submit">
+                        <span class="fa fa-filter"></span>
+                        {% trans "Filter" %}
+                    </button>
+                </div>
             </form>
-        </form>
+        </div>
     {% endif %}
     {% if checkinlists|length == 0 %}
         <div class="empty-collection">
@@ -47,7 +64,8 @@
 
             {% if "can_change_event_settings" in request.eventpermset %}
                 <a href="{% url "control:event.orders.checkinlists.add" organizer=request.event.organizer.slug event=request.event.slug %}"
-                   class="btn btn-primary btn-lg"><i class="fa fa-plus"></i> {% trans "Create a new check-in list" %}</a>
+                   class="btn btn-primary btn-lg"><i class="fa fa-plus"></i> {% trans "Create a new check-in list" %}
+                </a>
             {% endif %}
             {% if can_change_organizer_settings %}
                 <a href="{% url "control:organizer.devices" organizer=request.organizer.slug %}"
@@ -69,10 +87,18 @@
             <table class="table table-hover table-quotas">
                 <thead>
                 <tr>
-                    <th>{% trans "Check-in lists" %}</th>
+                    <th>
+                        {% trans "Name" %}
+                        <a href="?{% url_replace request 'ordering' '-name' %}"><i class="fa fa-caret-down"></i></a>
+                        <a href="?{% url_replace request 'ordering' 'name' %}"><i class="fa fa-caret-up"></i></a>
+                    </th>
                     <th>{% trans "Checked in" %}</th>
                     {% if request.event.has_subevents %}
-                        <th>{% trans "Date" context "subevent" %}</th>
+                        <th>
+                            {% trans "Date" context "subevent" %}
+                            <a href="?{% url_replace request 'ordering' '-subevent' %}"><i class="fa fa-caret-down"></i></a>
+                            <a href="?{% url_replace request 'ordering' 'subevent' %}"><i class="fa fa-caret-up"></i></a>
+                        </th>
                     {% endif %}
                     <th class="iconcol">{% trans "Automated check-in" %}</th>
                     <th>{% trans "Products" %}</th>
@@ -83,7 +109,8 @@
                 {% for cl in checkinlists %}
                     <tr>
                         <td>
-                            <strong><a href="{% url "control:event.orders.checkinlists.show" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}">{{ cl.name }}</a></strong>
+                            <strong><a
+                                    href="{% url "control:event.orders.checkinlists.show" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}">{{ cl.name }}</a></strong>
                         </td>
                         <td>
                             <div class="quotabox availability">
@@ -92,14 +119,16 @@
                                     </div>
                                 </div>
                                 <div class="numbers">
-                                    {{ cl.checkin_count|default_if_none:"0" }} / {{ cl.position_count|default_if_none:"0" }}
+                                    {{ cl.checkin_count|default_if_none:"0" }} /
+                                    {{ cl.position_count|default_if_none:"0" }}
                                 </div>
                             </div>
                         </td>
                         {% if request.event.has_subevents %}
                             {% if cl.subevent %}
                                 <td>
-                                    {{ cl.subevent.name }} – {{ cl.subevent.get_date_range_display }} {{ cl.subevent.date_from|date:"TIME_FORMAT" }}
+                                    {{ cl.subevent.name }} – {{ cl.subevent.get_date_range_display }}
+                                    {{ cl.subevent.date_from|date:"TIME_FORMAT" }}
                                 </td>
                             {% else %}
                                 <td>
@@ -127,17 +156,20 @@
                             {% endif %}
                         </td>
                         <td class="text-right flip">
-                            <a href="{% url "control:event.orders.checkinlists.show" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}" class="btn btn-default btn-sm"><i class="fa fa-eye"></i></a>
+                            <a href="{% url "control:event.orders.checkinlists.show" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}"
+                               class="btn btn-default btn-sm"><i class="fa fa-eye"></i></a>
                             {% if "can_change_event_settings" in request.eventpermset %}
                                 <a href="{% url "control:event.orders.checkinlists.add" organizer=request.event.organizer.slug event=request.event.slug %}?copy_from={{ cl.id }}"
-                                        class="btn btn-sm btn-default" title="{% trans "Clone" %}" data-toggle="tooltip">
+                                   class="btn btn-sm btn-default" title="{% trans "Clone" %}" data-toggle="tooltip">
                                     <span class="fa fa-copy"></span>
                                 </a>
                                 <a href="{% url "control:event.orders.checkinlists.simulator" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}"
                                    title="{% trans "Check-in simulator" %}" data-toggle="tooltip"
                                    class="btn btn-default btn-sm"><i class="fa fa-flask"></i></a>
-                                <a href="{% url "control:event.orders.checkinlists.edit" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}" class="btn btn-default btn-sm"><i class="fa fa-wrench"></i></a>
+                                <a href="{% url "control:event.orders.checkinlists.edit" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}"
-                                <a href="{% url "control:event.orders.checkinlists.delete" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}" class="btn btn-danger btn-sm"><i class="fa fa-trash"></i></a>
+                                   class="btn btn-default btn-sm"><i class="fa fa-wrench"></i></a>
+                                <a href="{% url "control:event.orders.checkinlists.delete" organizer=request.event.organizer.slug event=request.event.slug list=cl.id %}"
+                                   class="btn btn-danger btn-sm"><i class="fa fa-trash"></i></a>
                             {% endif %}
                         </td>
                     </tr>

src/pretix/control/templates/pretixcontrol/email_setup_simple.html

@@ -20,7 +20,7 @@
             </div>
             <div class="panel-body form-horizontal">
                 {% if spf_warning %}
-                    <div class="alert alert-warning">
+                    <div class="alert alert-danger">
                         <p>
                             {{ spf_warning }}
                         </p>
@@ -70,10 +70,18 @@
             </div>
         </div>
 
-        <div class="form-group submit-group">
+        {% if spf_warning %}
-            <button type="submit" class="btn btn-primary btn-save">
+            <div class="form-group submit-group">
-                {% trans "Save" %}
+                <a href="" class="btn btn-default btn-save">
-            </button>
+                    {% trans "Cancel" %}
-        </div>
+                </a>
+            </div>
+        {% else %}
+            <div class="form-group submit-group">
+                <button type="submit" class="btn btn-primary btn-save">
+                    {% trans "Save" %}
+                </button>
+            </div>
+        {% endif %}
     </form>
 {% endblock %}

src/pretix/control/templates/pretixcontrol/event/fragment_qr_dropdown.html

@@ -0,0 +1,27 @@
+{% load i18n %}
+<ul class="dropdown-menu dropdown-menu-right">
+    <li>
+        <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="png" %}{% if url %}?url={{ url|urlencode }}{% endif %}"
+           target="_blank" download>
+            {% blocktrans with filetype="PNG" %}Download QR code as {{ filetype }} image{% endblocktrans %}
+        </a>
+    </li>
+    <li>
+        <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="svg" %}{% if url %}?url={{ url|urlencode }}{% endif %}"
+           target="_blank" download>
+            {% blocktrans with filetype="SVG" %}Download QR code as {{ filetype }} image{% endblocktrans %}
+        </a>
+    </li>
+    <li>
+        <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="jpeg" %}{% if url %}?url={{ url|urlencode }}{% endif %}"
+           target="_blank" download>
+            {% blocktrans with filetype="JPG" %}Download QR code as {{ filetype }} image{% endblocktrans %}
+        </a>
+    </li>
+    <li>
+        <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="gif" %}{% if url %}?url={{ url|urlencode }}{% endif %}"
+           target="_blank" download>
+            {% blocktrans with filetype="GIF" %}Download QR code as {{ filetype }} image{% endblocktrans %}
+        </a>
+    </li>
+</ul>

src/pretix/control/templates/pretixcontrol/event/index.html

@@ -27,28 +27,7 @@
             <button type="button" class="btn btn-default btn-xs dropdown-toggle" data-toggle="dropdown" title="{% trans "Create QR code" %}" aria-haspopup="true" aria-expanded="false">
                 <i class="fa fa-qrcode" aria-hidden="true"></i>
             </button>
-            <ul class="dropdown-menu dropdown-menu-right">
+            {% include "pretixcontrol/event/fragment_qr_dropdown.html" with url=0 %}
-                <li>
-                    <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="png" %}" target="_blank" download>
-                        {% blocktrans with filetype="PNG" %}Download QR code as {{ filetype }} image{% endblocktrans %}
-                    </a>
-                </li>
-                <li>
-                    <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="svg" %}" target="_blank" download>
-                        {% blocktrans with filetype="SVG" %}Download QR code as {{ filetype }} image{% endblocktrans %}
-                    </a>
-                </li>
-                <li>
-                    <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="jpeg" %}" target="_blank" download>
-                        {% blocktrans with filetype="JPG" %}Download QR code as {{ filetype }} image{% endblocktrans %}
-                    </a>
-                </li>
-                <li>
-                    <a href="{% url "control:event.qrcode" event=request.event.slug organizer=request.organizer.slug filetype="gif" %}" target="_blank" download>
-                        {% blocktrans with filetype="GIF" %}Download QR code as {{ filetype }} image{% endblocktrans %}
-                    </a>
-                </li>
-            </ul>
         </div>
         <div class="clearfix"></div>
     </div>

src/pretix/control/templates/pretixcontrol/event/mail.html

@@ -105,6 +105,9 @@
                     {% blocktrans asvar title_payment_reminder %}Payment reminder{% endblocktrans %}
                     {% include "pretixcontrol/event/mail_settings_fragment.html" with pid="order_expirew" title=title_payment_reminder items="mail_days_order_expire_warning,mail_subject_order_expire_warning,mail_text_order_expire_warning,mail_subject_order_pending_warning,mail_text_order_pending_warning,mail_subject_order_incomplete_payment,mail_text_order_incomplete_payment" exclude="mail_days_order_expire_warning" %}
 
+                    {% blocktrans asvar title_payment_failed %}Payment failed{% endblocktrans %}
+                    {% include "pretixcontrol/event/mail_settings_fragment.html" with pid="payment_failed" title=title_payment_failed items="mail_subject_order_payment_failed,mail_text_order_payment_failed" %}
+
                     {% blocktrans asvar title_waiting_list_notification %}Waiting list notification{% endblocktrans %}
                     {% include "pretixcontrol/event/mail_settings_fragment.html" with pid="waiting_list" title=title_waiting_list_notification items="mail_subject_waiting_list,mail_text_waiting_list" %}
 

src/pretix/control/templates/pretixcontrol/event/payment.html

@@ -65,6 +65,8 @@
                 {% bootstrap_field form.payment_term_minutes layout="control" %}
                 {% bootstrap_field form.payment_term_last layout="control" %}
                 {% bootstrap_field form.payment_term_expire_automatically layout="control" %}
+                {% trans "days" context "unit" as days %}
+                {% bootstrap_field form.payment_term_expire_delay_days layout="control" addon_after=days %}
                 {% bootstrap_field form.payment_term_accept_late layout="control" %}
                 {% bootstrap_field form.payment_pending_hidden layout="control" %}
             </fieldset>

src/pretix/control/templates/pretixcontrol/event/settings.html

@@ -337,7 +337,7 @@
                 <div class="alert alert-info">
                     {% blocktrans trimmed %}
                         The waiting list currently is not compatible with some advanced features of pretix such as
-                        add-on products or product bundles.
+                        hidden products, add-on products or product bundles.
                     {% endblocktrans %}
                 </div>
                 <div class="alert alert-info">

src/pretix/control/templates/pretixcontrol/item/index.html

@@ -186,8 +186,12 @@
                             {% bootstrap_field form.media_type layout="control" %}
                         {% endif %}
                         {% for f in plugin_forms %}
-                            {% if f.is_layouts %}
+                            {% if f.is_layouts and not f.title %}
-                                {% bootstrap_form f layout="control" %}
+                                {% if f.template and not "template" in f.fields %}
+                                    {% include f.template with form=f %}
+                                {% else %}
+                                    {% bootstrap_form f layout="control" %}
+                                {% endif %}
                             {% endif %}
                         {% endfor %}
                     </fieldset>
@@ -256,11 +260,27 @@
                         {% endif %}
                         {% bootstrap_field form.show_quota_left layout="control" %}
                         {% for f in plugin_forms %}
-                            {% if not f.is_layouts %}
+                            {% if not f.is_layouts and not f.title %}
-                                {% bootstrap_form f layout="control" %}
+                                {% if f.template and not "template" in f.fields %}
+                                    {% include f.template with form=f %}
+                                {% else %}
+                                    {% bootstrap_form f layout="control" %}
+                                {% endif %}
                             {% endif %}
                         {% endfor %}
                     </fieldset>
+                    {% for f in plugin_forms %}
+                        {% if not f.is_layouts and f.title %}
+                            <fieldset>
+                                <legend>{{ f.title }}</legend>
+                                {% if f.template and not "template" in f.fields %}
+                                    {% include f.template with form=f %}
+                                {% else %}
+                                    {% bootstrap_form f layout="control" %}
+                                {% endif %}
+                            </fieldset>
+                        {% endif %}
+                    {% endfor %}
                 </div>
                 <div class="form-group submit-group">
                     <button type="submit" class="btn btn-primary btn-save">

src/pretix/control/templates/pretixcontrol/items/discount.html

@@ -48,6 +48,12 @@
                 </fieldset>
                 <fieldset>
                     <legend>{% trans "Benefit" context "discount" %}</legend>
+                    {% bootstrap_field form.benefit_same_products layout="control" %}
+                    <div data-display-dependency="#id_benefit_same_products" data-inverse>
+                        {% bootstrap_field form.benefit_limit_products layout="control" %}
+                        {% bootstrap_field form.benefit_apply_to_addons layout="control" %}
+                        {% bootstrap_field form.benefit_ignore_voucher_discounted layout="control" %}
+                    </div>
                     {% bootstrap_field form.benefit_discount_matching_percent layout="control" addon_after="%" %}
                     {% bootstrap_field form.benefit_only_apply_to_cheapest_n_matches layout="control" %}
                 </fieldset>

src/pretix/control/templates/pretixcontrol/order/deny.html

@@ -1,5 +1,6 @@
 {% extends "pretixcontrol/event/base.html" %}
 {% load i18n %}
+{% load bootstrap3 %}
 {% block title %}
     {% trans "Deny order" %}
 {% endblock %}
@@ -13,16 +14,7 @@
 
     <form method="post" href="">
         {% csrf_token %}
-        <div class="checkbox">
+        {% bootstrap_form form %}
-            <label>
-                <input type="checkbox" name="send_email" value="on" checked="checked">
-                {% trans "Notify user by e-mail" %}
-            </label>
-        </div>
-        <p>
-            <label>{% trans "Comment (will be sent to the user)" %}</label>
-            <textarea name="comment" class="form-control" rows="5"></textarea>
-        </p>
         <div class="row checkout-button-row">
             <div class="col-md-4">
                 <a class="btn btn-block btn-default btn-lg"

src/pretix/control/templates/pretixcontrol/order/index.html

@@ -110,7 +110,7 @@
             <input type="hidden" name="start-action" value="do_nothing">
             <input type="hidden" name="start-mode" value="partial">
             {% localize off %}
-            <input type="hidden" name="start-partial_amount" value="{{ overpaid|floatformat:2 }}">
+            <input type="hidden" name="start-partial_amount" value="{{ overpaid|money_numberfield:request.event.currency }}">
             {% endlocalize %}
             <input type="hidden" name="comment" value="{% trans "Refund for overpayment" %}">
             <div class="alert alert-warning">

src/pretix/control/templates/pretixcontrol/order/refund_choose.html

@@ -51,9 +51,9 @@
                                     <div class="input-group">
                                         <input type="text" name="refund-{{ p.pk }}"
                                                 {% if p.propose_refund %}
-                                                    value="{{ p.propose_refund|floatformat:2 }}"
+                                                    value="{{ p.propose_refund|money_numberfield:request.event.currency }}"
                                                 {% else %}
-                                                    placeholder="{{ p.propose_refund|floatformat:2 }}"
+                                                    placeholder="{{ p.propose_refund|money_numberfield:request.event.currency }}"
                                                 {% endif %}
                                                 title="" class="form-control">
                                         <span class="input-group-addon">
@@ -63,7 +63,7 @@
                                 {% elif p.full_refund_possible %}
                                     <label class="checkbox">
                                         <input type="checkbox" name="refund-{{ p.pk }}"
-                                                value="{{ p.amount|floatformat:2 }}"
+                                                value="{{ p.amount|money_numberfield:request.event.currency }}"
                                                 {% if p.propose_refund == p.amount %}checked{% endif %}>
                                         {% trans "Full amount" %} ({{ p.amount|money:request.event.currency }})
                                     </label>
@@ -101,7 +101,7 @@
                             <td class="text-right flip refund-amount">
                                 <div class="input-group">
                                     <input type="text" name="newrefund-{{ prov }}"
-                                            placeholder="{{ 0|floatformat:2 }}"
+                                            placeholder="{{ 0|money_numberfield:request.event.currency }}"
                                             title="" class="form-control">
                                     <span class="input-group-addon">
                                         {{ request.event.currency }}
@@ -119,7 +119,7 @@
                         <td class="text-right flip refund-amount">
                             <div class="input-group">
                                 <input type="text" name="refund-offsetting"
-                                        title="" class="form-control" placeholder="{{ 0|floatformat:2 }}">
+                                        title="" class="form-control" placeholder="{{ 0|money_numberfield:request.event.currency }}">
                                 <span class="input-group-addon">
                                     {{ request.event.currency }}
                                 </span>
@@ -144,9 +144,9 @@
                                 <input type="text" name="refund-new-giftcard"
                                         title="" class="form-control"
                                         {% if giftcard_proposal %}
-                                            value="{{ giftcard_proposal|floatformat:2 }}"
+                                            value="{{ giftcard_proposal|money_numberfield:request.event.currency }}"
                                         {% else %}
-                                            placeholder="{{ giftcard_proposal|floatformat:2 }}"
+                                            placeholder="{{ giftcard_proposal|money_numberfield:request.event.currency }}"
                                         {% endif %}
                                 >
                                 <span class="input-group-addon">
@@ -172,9 +172,9 @@
                             <div class="input-group">
                                 <input type="text" name="refund-manual"
                                         {% if remainder %}
-                                            value="{{ remainder|floatformat:2 }}"
+                                            value="{{ remainder|money_numberfield:request.event.currency }}"
                                         {% else %}
-                                            placeholder="{{ remainder|floatformat:2 }}"
+                                            placeholder="{{ remainder|money_numberfield:request.event.currency }}"
                                         {% endif %}
                                         title="" class="form-control">
                                 <span class="input-group-addon">