API: Add webhooks for customer events

Co-authored-by: Raphael Michel <michel@rami.io>

.github/workflows/tests.yml

@@ -35,7 +35,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: harmon758/postgresql-action@v1
         with:
-          postgresql version: '11'
+          postgresql version: '15'
           postgresql db: 'pretix'
           postgresql user: 'postgres'
           postgresql password: 'postgres'

Dockerfile

@@ -33,8 +33,7 @@ RUN apt-get update && \
     mkdir /static && \
     mkdir /etc/supervisord && \
 	curl -fsSL https://deb.nodesource.com/setup_16.x | sudo -E bash - && \
-    apt-get install -y nodejs && \
-    curl -qL https://www.npmjs.com/install.sh | sh
+    apt-get install -y nodejs
 
 
 ENV LC_ALL=C.UTF-8 \

deployment/docker/production_settings.py

@@ -1,4 +1,4 @@
 from pretix.settings import *
 
 LOGGING['handlers']['mail_admins']['include_html'] = True
-STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
+STORAGES["staticfiles"]["BACKEND"] = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

doc/admin/config.rst

@@ -152,6 +152,10 @@ Example::
     password=abcd
     host=localhost
     port=3306
+    sslmode=require
+    sslrootcert=/etc/pretix/postgresql-ca.crt
+    sslcert=/etc/pretix/postgresql-client-crt.crt
+    sslkey=/etc/pretix/postgresql-client-key.key
 
 ``backend``
     One of ``sqlite3`` and ``postgresql``.
@@ -163,6 +167,11 @@ Example::
 ``user``, ``password``, ``host``, ``port``
     Connection details for the database connection. Empty by default.
 
+``sslmode``, ``sslrootcert``
+    Connection TLS details for the PostgreSQL database connection. Possible values of ``sslmode`` are ``disable``, ``allow``, ``prefer``, ``require``, ``verify-ca``, and ``verify-full``. ``sslrootcert`` should be the accessible path of the ca certificate. Both values are empty by default.
+
+``sslcert``, ``sslkey``
+    Connection mTLS details for the PostgreSQL database connection. It's also necessary to specify ``sslmode`` and ``sslrootcert`` parameters, please check the correct values from the TLS part. ``sslcert`` should be the accessible path of the client certificate.  ``sslkey`` should be the accessible path of the client key. All values are empty by default.
 .. _`config-replica`:
 
 Database replica settings
@@ -324,6 +333,10 @@ to speed up various operations::
             ["sentinel_host_3", 26379]
         ]
     password=password
+    ssl_cert_reqs=required
+    ssl_ca_certs=/etc/pretix/redis-ca.pem
+    ssl_keyfile=/etc/pretix/redis-client-crt.pem
+    ssl_certfile=/etc/pretix/redis-client-key.key
 
 ``location``
     The location of redis, as a URL of the form ``redis://[:password]@localhost:6379/0``
@@ -347,6 +360,22 @@ to speed up various operations::
     If your redis setup doesn't require a password or you already specified it in the location you can omit this option.
     If this is set it will be passed to redis as the connection option PASSWORD.
 
+``ssl_cert_reqs``
+    If this is set it will be passed to redis as the connection option ``SSL_CERT_REQS``.
+    Possible values are ``none``, ``optional``, and ``required``.
+
+``ssl_ca_certs``
+    If your redis setup doesn't require TLS you can omit this option.
+    If this is set it will be passed to redis as the connection option ``SSL_CA_CERTS``. Possible value is the ca path.
+
+``ssl_keyfile``
+    If your redis setup doesn't require mTLS you can omit this option.
+    If this is set it will be passed to redis as the connection option ``SSL_KEYFILE``. Possible value is the keyfile path.
+
+``ssl_certfile``
+    If your redis setup doesn't require mTLS you can omit this option.
+    If this is set it will be passed to redis as the connection option ``SSL_CERTFILE``. Possible value is the certfile path.
+
 If redis is not configured, pretix will store sessions and locks in the database. If memcached
 is configured, memcached will be used for caching instead of redis.
 
@@ -396,6 +425,8 @@ The two ``transport_options`` entries can be omitted in most cases.
 If they are present they need to be a valid JSON dictionary.
 For possible entries in that dictionary see the `Celery documentation`_.
 
+It is possible the use Redis with TLS/mTLS for the broker or the backend. To do so, it is necessary to specify the TLS identifier ``rediss``, the ssl mode ``ssl_cert_reqs`` and optionally specify the CA (TLS) ``ssl_ca_certs``, cert ``ssl_certfile`` and key ``ssl_keyfile`` (mTLS) path as encoded string. the following uri describes the format and possible parameters ``rediss://0.0.0.0:6379/1?ssl_cert_reqs=required&ssl_ca_certs=%2Fetc%2Fpretix%2Fredis-ca.pem&ssl_certfile=%2Fetc%2Fpretix%2Fredis-client-crt.pem&ssl_keyfile=%2Fetc%2Fpretix%2Fredis-client-key.key``
+
 To use redis with sentinels set the broker or backend to ``sentinel://sentinel_host_1:26379;sentinel_host_2:26379/0``
 and the respective transport_options to ``{"master_name":"mymaster"}``.
 If your redis instances behind the sentinel have a password use ``sentinel://:my_password@sentinel_host_1:26379;sentinel_host_2:26379/0``.

doc/admin/installation/docker_smallscale.rst

@@ -26,7 +26,7 @@ installation guides):
 * `Docker`_
 * A SMTP server to send out mails, e.g. `Postfix`_ on your machine or some third-party server you have credentials for
 * A HTTP reverse proxy, e.g. `nginx`_ or Apache to allow HTTPS connections
-* A `PostgreSQL`_ 9.6+ database server
+* A `PostgreSQL`_ 11+ database server
 * A `redis`_ server
 
 We also recommend that you use a firewall, although this is not a pretix-specific recommendation. If you're new to
@@ -321,11 +321,11 @@ workers, e.g. ``docker run … taskworker -Q notifications --concurrency 32``.
 
 
 .. _Docker: https://docs.docker.com/engine/installation/linux/debian/
-.. _Postfix: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-16-04
+.. _Postfix: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-22-04
 .. _nginx: https://botleg.com/stories/https-with-lets-encrypt-and-nginx/
 .. _Let's Encrypt: https://letsencrypt.org/
 .. _pretix.eu: https://pretix.eu/
-.. _PostgreSQL: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-20-04
+.. _PostgreSQL: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-22-04
 .. _redis: https://blog.programster.org/debian-8-install-redis-server/
 .. _ufw: https://en.wikipedia.org/wiki/Uncomplicated_Firewall
 .. _redis website: https://redis.io/topics/security

doc/admin/installation/enterprise.rst

@@ -68,7 +68,7 @@ generated key and installs the plugin from the URL we told you::
         mkdir -p /etc/ssh && \
         ssh-keyscan -t rsa -p 10022 code.rami.io >> /root/.ssh/known_hosts && \
         echo StrictHostKeyChecking=no >> /root/.ssh/config && \
-        DJANGO_SETTINGS_MODULE=pretix.settings pip3 install -U "git+ssh://git@code.rami.io:10022/pretix/pretix-slack.git@stable#egg=pretix-slack" && \
+        DJANGO_SETTINGS_MODULE= pip3 install -U "git+ssh://git@code.rami.io:10022/pretix/pretix-slack.git@stable#egg=pretix-slack" && \
         cd /pretix/src && \
         sudo -u pretixuser make production
     USER pretixuser

doc/admin/installation/manual_smallscale.rst

@@ -21,6 +21,7 @@ Requirements
 Please set up the following systems beforehand, we'll not explain them here in detail (but see these links for external
 installation guides):
 
+* A python 3.9+ installation
 * A SMTP server to send out mails, e.g. `Postfix`_ on your machine or some third-party server you have credentials for
 * A HTTP reverse proxy, e.g. `nginx`_ or Apache to allow HTTPS connections
 * A `PostgreSQL`_ 11+ database server
@@ -323,11 +324,11 @@ Then, proceed like after any plugin installation::
     (venv)$ python -m pretix updatestyles
     # systemctl restart pretix-web pretix-worker
 
-.. _Postfix: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-16-04
+.. _Postfix: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-as-a-send-only-smtp-server-on-ubuntu-22-04
 .. _nginx: https://botleg.com/stories/https-with-lets-encrypt-and-nginx/
 .. _Let's Encrypt: https://letsencrypt.org/
 .. _pretix.eu: https://pretix.eu/
-.. _PostgreSQL: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-20-04
+.. _PostgreSQL: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-22-04
 .. _redis: https://blog.programster.org/debian-8-install-redis-server/
 .. _ufw: https://en.wikipedia.org/wiki/Uncomplicated_Firewall
 .. _strong encryption settings: https://mozilla.github.io/server-side-tls/ssl-config-generator/

doc/api/deviceauth.rst

@@ -32,10 +32,16 @@ as well as the type of underlying hardware. Example:
        "token": "kpp4jn8g2ynzonp6",
        "hardware_brand": "Samsung",
        "hardware_model": "Galaxy S",
+       "os_name": "Android",
+       "os_version": "2.3.6",
        "software_brand": "pretixdroid",
-       "software_version": "4.0.0"
+       "software_version": "4.0.0",
+       "rsa_pubkey": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqh…nswIDAQAB\n-----END PUBLIC KEY-----\n"
    }
 
+The ``rsa_pubkey`` is optional any only required for certain fatures such as working with reusable
+media and NFC cryptography.
+
 Every initialization token can only be used once. On success, you will receive a response containing
 information on your device as well as your API token:
 
@@ -98,6 +104,8 @@ following endpoint:
    {
        "hardware_brand": "Samsung",
        "hardware_model": "Galaxy S",
+       "os_name": "Android",
+       "os_version": "2.3.6",
        "software_brand": "pretixdroid",
        "software_version": "4.1.0",
        "info": {"arbitrary": "data"}
@@ -133,9 +141,29 @@ The response will look like this:
          "id": 3,
          "name": "South entrance"
        }
-     }
+     },
+     "server": {
+       "version": {
+         "pretix": "3.6.0.dev0",
+         "pretix_numeric": 30060001000
+       }
+     },
+     "medium_key_sets": [
+       {
+         "public_id": 3456349,
+         "organizer": "foo",
+         "active": true,
+         "media_type": "nfc_mf0aes",
+         "uid_key": "base64-encoded-encrypted-key",
+         "diversification_key": "base64-encoded-encrypted-key",
+       }
+     ]
    }
 
+``"medium_key_sets`` will always be empty if you did not set an ``rsa_pubkey``.
+The individual keys in the key sets are encrypted with the device's ``rsa_pubkey``
+using ``RSA/ECB/PKCS1Padding``.
+
 Creating a new API key
 ----------------------
 

doc/api/resources/devices.rst

@@ -24,6 +24,8 @@ all_events                            boolean                    Whether this de
 limit_events                          list                       List of event slugs this device has access to
 hardware_brand                        string                     Device hardware manufacturer (read-only)
 hardware_model                        string                     Device hardware model (read-only)
+os_name                               string                     Device operating system name (read-only)
+os_version                            string                     Device operating system version (read-only)
 software_brand                        string                     Device software product (read-only)
 software_version                      string                     Device software version (read-only)
 created                               datetime                   Creation time
@@ -76,6 +78,8 @@ Device endpoints
             "security_profile": "full",
             "hardware_brand": "Zebra",
             "hardware_model": "TC25",
+            "os_name": "Android",
+            "os_version": "8.1.0",
             "software_brand": "pretixSCAN",
             "software_version": "1.5.1"
           }
@@ -123,6 +127,8 @@ Device endpoints
         "security_profile": "full",
         "hardware_brand": "Zebra",
         "hardware_model": "TC25",
+        "os_name": "Android",
+        "os_version": "8.1.0",
         "software_brand": "pretixSCAN",
         "software_version": "1.5.1"
       }
@@ -173,6 +179,8 @@ Device endpoints
         "initialized": null
         "hardware_brand": null,
         "hardware_model": null,
+        "os_name": null,
+        "os_version": null,
         "software_brand": null,
         "software_version": null
       }

doc/api/resources/discounts.rst

@@ -31,9 +31,9 @@ subevent_mode                            strings                    Determines h
                                                                     ``"same"`` (discount is only applied for groups within
                                                                     the same date), or ``"distinct"`` (discount is only applied
                                                                     for groups with no two same dates).
-condition_all_products                   boolean                    If ``true``, the discount applies to all items.
+condition_all_products                   boolean                    If ``true``, the discount condition applies to all items.
 condition_limit_products                 list of integers           If ``condition_all_products`` is not set, this is a list
-                                                                    of internal item IDs that the discount applies to.
+                                                                    of internal item IDs that the discount condition applies to.
 condition_apply_to_addons                boolean                    If ``true``, the discount applies to add-on products as well,
                                                                     otherwise it only applies to top-level items. The discount never
                                                                     applies to bundled products.
@@ -48,6 +48,17 @@ benefit_discount_matching_percent        decimal (string)           The percenta
 benefit_only_apply_to_cheapest_n_matches integer                    If set higher than 0, the discount will only be applied to
                                                                     the cheapest matches. Useful for a "3 for 2"-style discount.
                                                                     Cannot be combined with ``condition_min_value``.
+benefit_same_products                    boolean                    If ``true``, the discount benefit applies to the same set of items
+                                                                    as the condition (see above).
+benefit_limit_products                   list of integers           If ``benefit_same_products`` is not set, this is a list
+                                                                    of internal item IDs that the discount benefit applies to.
+benefit_apply_to_addons                  boolean                    (Only used if ``benefit_same_products`` is ``false``.)
+                                                                    If ``true``, the discount applies to add-on products as well,
+                                                                    otherwise it only applies to top-level items. The discount never
+                                                                    applies to bundled products.
+benefit_ignore_voucher_discounted        boolean                    (Only used if ``benefit_same_products`` is ``false``.)
+                                                                    If ``true``, the discount does not apply to products which have
+                                                                    been discounted by a voucher.
 ======================================== ========================== =======================================================
 
 
@@ -94,6 +105,10 @@ Endpoints
             "condition_ignore_voucher_discounted": false,
             "condition_min_count": 3,
             "condition_min_value": "0.00",
+            "benefit_same_products": true,
+            "benefit_limit_products": [],
+            "benefit_apply_to_addons": true,
+            "benefit_ignore_voucher_discounted": false,
             "benefit_discount_matching_percent": "100.00",
             "benefit_only_apply_to_cheapest_n_matches": 1
           }
@@ -146,6 +161,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }
@@ -184,6 +203,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }
@@ -211,6 +234,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }
@@ -267,6 +294,10 @@ Endpoints
         "condition_ignore_voucher_discounted": false,
         "condition_min_count": 3,
         "condition_min_value": "0.00",
+        "benefit_same_products": true,
+        "benefit_limit_products": [],
+        "benefit_apply_to_addons": true,
+        "benefit_ignore_voucher_discounted": false,
         "benefit_discount_matching_percent": "100.00",
         "benefit_only_apply_to_cheapest_n_matches": 1
       }

doc/api/resources/events.rst

@@ -70,6 +70,11 @@ Endpoints
 
     The ``public_url`` field has been added.
 
+.. versionchanged:: 5.0
+
+    The ``date_from_before``, ``date_from_after``, ``date_to_before``, and ``date_to_after`` query parameters have been
+    added.
+
 .. http:get:: /api/v1/organizers/(organizer)/events/
 
    Returns a list of all events within a given organizer the authenticated user/token has access to.
@@ -141,6 +146,10 @@ Endpoints
    :query has_subevents: If set to ``true``/``false``, only events with a matching value of ``has_subevents`` are returned.
    :query is_future: If set to ``true`` (``false``), only events that happen currently or in the future are (not) returned. Event series are never (always) returned.
    :query is_past: If set to ``true`` (``false``), only events that are over are (not) returned. Event series are never (always) returned.
+   :query date_from_after: If set to a date and time, only events that start at or after the given time are returned.
+   :query date_from_before: If set to a date and time, only events that start at or before the given time are returned.
+   :query date_to_after: If set to a date and time, only events that have an end date and end at or after the given time are returned.
+   :query date_to_before: If set to a date and time, only events that have an end date and end at or before the given time are returned.
    :query ends_after: If set to a date and time, only events that happen during of after the given time are returned. Event series are never returned.
    :query string ordering: Manually set the ordering of results. Valid fields to be used are ``date_from`` and
                            ``slug``. Keep in mind that ``date_from`` of event series does not really tell you anything.

doc/api/resources/exporters.rst

@@ -111,7 +111,7 @@ Listing available exporters
             "input_parameters": [
               {
                 "name": "events",
-                "required": true
+                "required": false
               },
               {
                 "name": "_format",

doc/api/resources/invoices.rst

@@ -12,6 +12,7 @@ The invoice resource contains the following public fields:
 Field                                 Type                       Description
 ===================================== ========================== =======================================================
 number                                string                     Invoice number (with prefix)
+event                                 string                     The slug of the parent event
 order                                 string                     Order code of the order this invoice belongs to
 is_cancellation                       boolean                    ``true``, if this invoice is the cancellation of a
                                                                  different invoice.
@@ -121,9 +122,13 @@ internal_reference                    string                     Customer's refe
 
    The attribute ``lines.subevent`` has been added.
 
+.. versionchanged:: 2023.8
 
-Endpoints
----------
+   The ``event`` attribute has been added. The organizer-level endpoint has been added.
+
+
+List of all invoices
+--------------------
 
 .. http:get:: /api/v1/organizers/(organizer)/events/(event)/invoices/
 
@@ -152,6 +157,7 @@ Endpoints
         "results": [
           {
             "number": "SAMPLECONF-00001",
+            "event": "sampleconf",
             "order": "ABC12",
             "is_cancellation": false,
             "invoice_from_name": "Big Events LLC",
@@ -221,6 +227,50 @@ Endpoints
    :statuscode 401: Authentication failure
    :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
 
+.. http:get:: /api/v1/organizers/(organizer)/invoices/
+
+   Returns a list of all invoices within all events of a given organizer (with sufficient access permissions).
+
+   Supported query parameters and output format of this endpoint are identical to the list endpoint within an event.
+
+   **Example request**:
+
+   .. sourcecode:: http
+
+      GET /api/v1/organizers/bigevents/events/sampleconf/invoices/ HTTP/1.1
+      Host: pretix.eu
+      Accept: application/json, text/javascript
+
+   **Example response**:
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Vary: Accept
+      Content-Type: application/json
+
+      {
+        "count": 1,
+        "next": null,
+        "previous": null,
+        "results": [
+          {
+            "number": "SAMPLECONF-00001",
+            "event": "sampleconf",
+            "order": "ABC12",
+            ...
+        ]
+      }
+
+   :param organizer: The ``slug`` field of the organizer to fetch
+   :statuscode 200: no error
+   :statuscode 401: Authentication failure
+   :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
+
+
+Fetching individual invoices
+----------------------------
+
 .. http:get:: /api/v1/organizers/(organizer)/events/(event)/invoices/(number)/
 
    Returns information on one invoice, identified by its invoice number.
@@ -243,6 +293,7 @@ Endpoints
 
       {
         "number": "SAMPLECONF-00001",
+        "event": "sampleconf",
         "order": "ABC12",
         "is_cancellation": false,
         "invoice_from_name": "Big Events LLC",
@@ -337,6 +388,12 @@ Endpoints
    :statuscode 409: The file is not yet ready and will now be prepared. Retry the request after waiting for a few
                     seconds.
 
+
+Modifying invoices
+------------------
+
+Invoices cannot be edited directly, but the following actions can be triggered:
+
 .. http:post:: /api/v1/organizers/(organizer)/events/(event)/invoices/(invoice_no)/reissue/
 
    Cancels the invoice and creates a new one.

doc/api/resources/orders.rst

@@ -20,6 +20,7 @@ The order resource contains the following public fields:
 Field                                 Type                       Description
 ===================================== ========================== =======================================================
 code                                  string                     Order code
+event                                 string                     The slug of the parent event
 status                                string                     Order status, one of:
 
                                                                  * ``n`` – pending
@@ -130,6 +131,10 @@ last_modified                         datetime                   Last modificati
 
    The ``valid_if_pending`` attribute has been added.
 
+.. versionchanged:: 2023.8
+
+   The ``event`` attribute has been added. The organizer-level endpoint has been added.
+
 
 .. _order-position-resource:
 
@@ -289,6 +294,7 @@ List of all orders
         "results": [
           {
             "code": "ABC12",
+            "event": "sampleconf",
             "status": "p",
             "testmode": false,
             "secret": "k24fiuwvu8kxz3y1",
@@ -441,6 +447,48 @@ List of all orders
    :statuscode 401: Authentication failure
    :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
 
+.. http:get:: /api/v1/organizers/(organizer)/orders/
+
+   Returns a list of all orders within all events of a given organizer (with sufficient access permissions).
+
+   Supported query parameters and output format of this endpoint are identical to the list endpoint within an event,
+   with the exception that the ``pdf_data`` parameter is not supported here.
+
+   **Example request**:
+
+   .. sourcecode:: http
+
+      GET /api/v1/organizers/bigevents/orders/ HTTP/1.1
+      Host: pretix.eu
+      Accept: application/json, text/javascript
+
+   **Example response**:
+
+   .. sourcecode:: http
+
+      HTTP/1.1 200 OK
+      Vary: Accept
+      Content-Type: application/json
+      X-Page-Generated: 2017-12-01T10:00:00Z
+
+      {
+        "count": 1,
+        "next": null,
+        "previous": null,
+        "results": [
+          {
+            "code": "ABC12",
+            "event": "sampleconf",
+            ...
+          }
+        ]
+      }
+
+   :param organizer: The ``slug`` field of the organizer to fetch
+   :statuscode 200: no error
+   :statuscode 401: Authentication failure
+   :statuscode 403: The requested organizer/event does not exist **or** you have no permission to view this resource.
+
 Fetching individual orders
 --------------------------
 
@@ -466,6 +514,7 @@ Fetching individual orders
 
       {
         "code": "ABC12",
+        "event": "sampleconf",
         "status": "p",
         "testmode": false,
         "secret": "k24fiuwvu8kxz3y1",

doc/api/resources/reusablemedia.rst

@@ -18,7 +18,8 @@ The reusable medium resource contains the following public fields:
 Field                                 Type                       Description
 ===================================== ========================== =======================================================
 id                                    integer                    Internal ID of the medium
-type                                  string                     Type of medium, e.g. ``"barcode"`` or ``"nfc_uid"``.
+type                                  string                     Type of medium, e.g. ``"barcode"``, ``"nfc_uid"`` or ``"nfc_mf0aes"``.
+organizer                             string                     Organizer slug of the organizer who "owns" this medium.
 identifier                            string                     Unique identifier of the medium. The format depends on the ``type``.
 active                                boolean                    Whether this medium may be used.
 created                               datetime                   Date of creation
@@ -36,6 +37,7 @@ Existing media types are:
 
 - ``barcode``
 - ``nfc_uid``
+- ``nfc_mf0aes``
 
 Endpoints
 ---------
@@ -67,6 +69,7 @@ Endpoints
         "results": [
           {
             "id": 1,
+            "organizer": "bigevents",
             "identifier": "ABCDEFGH",
             "created": "2021-04-06T13:44:22.809377Z",
             "updated": "2021-04-06T13:44:22.809377Z",
@@ -123,6 +126,7 @@ Endpoints
 
       {
         "id": 1,
+        "organizer": "bigevents",
         "identifier": "ABCDEFGH",
         "created": "2021-04-06T13:44:22.809377Z",
         "updated": "2021-04-06T13:44:22.809377Z",
@@ -152,6 +156,9 @@ Endpoints
    Look up a new reusable medium by its identifier. In some cases, this might lead to the automatic creation of a new
    medium behind the scenes.
 
+   This endpoint, and this endpoint only, might return media from a different organizer if there is a cross-acceptance
+   agreement. In this case, only linked gift cards will be returned, no order position or customer records,
+
    **Example request**:
 
    .. sourcecode:: http
@@ -176,6 +183,7 @@ Endpoints
 
       {
         "id": 1,
+        "organizer": "bigevents",
         "identifier": "ABCDEFGH",
         "created": "2021-04-06T13:44:22.809377Z",
         "updated": "2021-04-06T13:44:22.809377Z",
@@ -235,6 +243,7 @@ Endpoints
 
       {
         "id": 1,
+        "organizer": "bigevents",
         "identifier": "ABCDEFGH",
         "created": "2021-04-06T13:44:22.809377Z",
         "updated": "2021-04-06T13:44:22.809377Z",
@@ -291,6 +300,7 @@ Endpoints
 
       {
         "id": 1,
+        "organizer": "bigevents",
         "identifier": "ABCDEFGH",
         "created": "2021-04-06T13:44:22.809377Z",
         "updated": "2021-04-06T13:44:22.809377Z",

doc/api/resources/sendmail_rules.rst

@@ -18,8 +18,19 @@ subject                               multi-lingual string       The subject of
 template                              multi-lingual string       The body of the email
 all_products                          boolean                    If ``true``, the email is sent to buyers of all products
 limit_products                        list of integers           List of product IDs, if ``all_products`` is not set
-include_pending                       boolean                    If ``true``, the email is sent to pending orders. If ``false``,
+[**DEPRECATED**] include_pending      boolean                    If ``true``, the email is sent to pending orders. If ``false``,
                                                                  only paid orders are considered.
+restrict_to_status                    list                       List of order states to restrict recipients to. Valid
+                                                                 entries are ``p`` for paid, ``e`` for expired, ``c`` for canceled,
+                                                                 ``n__pending_approval`` for pending approval,
+                                                                 ``n__not_pending_approval_and_not_valid_if_pending`` for payment
+                                                                 pending, ``n__valid_if_pending`` for payment pending but already confirmed,
+                                                                 and ``n__pending_overdue`` for pending with payment overdue.
+                                                                 The default is ``["p", "n__valid_if_pending"]``.
+checked_in_status                     string                     Check-in status to restrict recipients to. Valid strings are:
+                                                                 ``null`` for no filtering (default), ``checked_in`` for
+                                                                 limiting to attendees that are or have been checked in, and
+                                                                 ``no_checkin`` for limiting to attendees who have not checked in.
 date_is_absolute                      boolean                    If ``true``, the email is set at a specific point in time.
 send_date                             datetime                   If ``date_is_absolute`` is set: Date and time to send the email.
 send_offset_days                      integer                    If ``date_is_absolute`` is not set, this is the number of days
@@ -37,7 +48,10 @@ send_to                               string                     Can be ``"order
                                                                  or ``"both"``.
                                                                  date. Otherwise it is relative to the event start date.
 ===================================== ========================== =======================================================
+.. versionchanged:: 2023.7
 
+    The ``include_pending`` field has been  deprecated.
+    The ``restrict_to_status`` field has been added.
 
 Endpoints
 ---------
@@ -74,7 +88,12 @@ Endpoints
             "template": {"en": "Don't forget your tickets, download them at {url}"},
             "all_products": true,
             "limit_products": [],
-            "include_pending": false,
+            "restrict_to_status": [
+                "p",
+                "n__not_pending_approval_and_not_valid_if_pending",
+                "n__valid_if_pending"
+            ],
+            "checked_in_status": null,
             "send_date": null,
             "send_offset_days": 1,
             "send_offset_time": "18:00",
@@ -120,7 +139,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": null,
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",
@@ -157,7 +181,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": "checked_in",
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",
@@ -182,7 +211,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": "checked_in",
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",
@@ -235,7 +269,12 @@ Endpoints
         "template": {"en": "Don't forget your tickets, download them at {url}"},
         "all_products": true,
         "limit_products": [],
-        "include_pending": false,
+        "restrict_to_status": [
+            "p",
+            "n__not_pending_approval_and_not_valid_if_pending",
+            "n__valid_if_pending"
+        ],
+        "checked_in_status": "checked_in",
         "send_date": null,
         "send_offset_days": 1,
         "send_offset_time": "18:00",

doc/api/resources/subevents.rst

@@ -63,6 +63,11 @@ last_modified                         datetime                   Last modificati
 
     The ``search`` query parameter has been added to filter sub-events by their name or location in any language.
 
+.. versionchanged:: 5.0
+
+    The ``date_from_before``, ``date_from_after``, ``date_to_before``, and ``date_to_after`` query parameters have been
+    added.
+
 Endpoints
 ---------
 
@@ -130,6 +135,10 @@ Endpoints
    :query active: If set to ``true``/``false``, only events with a matching value of ``active`` are returned.
    :query is_future: If set to ``true`` (``false``), only events that happen currently or in the future are (not) returned.
    :query is_past: If set to ``true`` (``false``), only events that are over are (not) returned.
+   :query date_from_after: If set to a date and time, only events that start at or after the given time are returned.
+   :query date_from_before: If set to a date and time, only events that start at or before the given time are returned.
+   :query date_to_after: If set to a date and time, only events that have an end date and end at or after the given time are returned.
+   :query date_to_before: If set to a date and time, only events that have an end date and end at or before the given time are returned.
    :query ends_after: If set to a date and time, only events that happen during of after the given time are returned.
    :query search: Only return events matching a given search query.
    :param organizer: The ``slug`` field of a valid organizer
@@ -458,6 +467,10 @@ Endpoints
    :query event__live: If set to ``true``/``false``, only events with a matching value of ``live`` on the parent event are returned.
    :query is_future: If set to ``true`` (``false``), only events that happen currently or in the future are (not) returned.
    :query is_past: If set to ``true`` (``false``), only events that are over are (not) returned.
+   :query date_from_after: If set to a date and time, only events that start at or after the given time are returned.
+   :query date_from_before: If set to a date and time, only events that start at or before the given time are returned.
+   :query date_to_after: If set to a date and time, only events that have an end date and end at or after the given time are returned.
+   :query date_to_before: If set to a date and time, only events that have an end date and end at or before the given time are returned.
    :query ends_after: If set to a date and time, only events that happen during of after the given time are returned.
    :query sales_channel: If set to a sales channel identifier, the response will only contain subevents from events available on this sales channel.
    :param organizer: The ``slug`` field of a valid organizer

doc/api/resources/taxrules.rst

@@ -20,11 +20,16 @@ internal_name                         string                     An optional nam
 rate                                  decimal (string)           Tax rate in percent
 price_includes_tax                    boolean                    If ``true`` (default), tax is assumed to be included in
                                                                  the specified product price
-eu_reverse_charge                     boolean                    If ``true``, EU reverse charge rules are applied
+eu_reverse_charge                     boolean                    If ``true``, EU reverse charge rules are applied. Will
+                                                                 be ignored if custom rules are set.
 home_country                          string                     Merchant country (required for reverse charge), can be
                                                                  ``null`` or empty string
 keep_gross_if_rate_changes            boolean                    If ``true``, changes of the tax rate based on custom
                                                                  rules keep the gross price constant (default is ``false``)
+custom_rules                          object                     Dynamic rules specification. Each list element
+                                                                 corresponds to one rule that will be processed in order.
+                                                                 The current version of the schema in use can be found
+                                                                 `here`_.
 ===================================== ========================== =======================================================
 
 
@@ -32,6 +37,10 @@ keep_gross_if_rate_changes            boolean                    If ``true``, ch
 
     The ``internal_name`` and ``keep_gross_if_rate_changes`` attributes have been added.
 
+.. versionchanged:: 2023.6
+
+    The ``custom_rules`` attribute has been added.
+
 Endpoints
 ---------
 
@@ -68,6 +77,7 @@ Endpoints
             "price_includes_tax": true,
             "eu_reverse_charge": false,
             "keep_gross_if_rate_changes": false,
+            "custom_rules": null,
             "home_country": "DE"
           }
         ]
@@ -108,6 +118,7 @@ Endpoints
         "price_includes_tax": true,
         "eu_reverse_charge": false,
         "keep_gross_if_rate_changes": false,
+        "custom_rules": null,
         "home_country": "DE"
       }
 
@@ -156,6 +167,7 @@ Endpoints
         "price_includes_tax": true,
         "eu_reverse_charge": false,
         "keep_gross_if_rate_changes": false,
+        "custom_rules": null,
         "home_country": "DE"
       }
 
@@ -203,6 +215,7 @@ Endpoints
         "price_includes_tax": true,
         "eu_reverse_charge": false,
         "keep_gross_if_rate_changes": false,
+        "custom_rules": null,
         "home_country": "DE"
       }
 
@@ -242,3 +255,5 @@ Endpoints
    :statuscode 204: no error
    :statuscode 401: Authentication failure
    :statuscode 403: The requested organizer/event/rule does not exist **or** you have no permission to change it **or** this tax rule cannot be deleted since it is currently in use.
+
+.. _here: https://github.com/pretix/pretix/blob/master/src/pretix/static/schema/tax-rules-custom.schema.json

doc/api/resources/webhooks.rst

@@ -50,6 +50,10 @@ The following values for ``action_types`` are valid with pretix core:
     * ``pretix.event.order.payment.confirmed``
     * ``pretix.event.order.approved``
     * ``pretix.event.order.denied``
+    * ``pretix.event.orders.waitinglist.added``
+    * ``pretix.event.orders.waitinglist.changed``
+    * ``pretix.event.orders.waitinglist.deleted``
+    * ``pretix.event.orders.waitinglist.voucher_assigned``
     * ``pretix.event.checkin``
     * ``pretix.event.checkin.reverted``
     * ``pretix.event.added``
@@ -63,6 +67,9 @@ The following values for ``action_types`` are valid with pretix core:
     * ``pretix.event.live.deactivated``
     * ``pretix.event.testmode.activated``
     * ``pretix.event.testmode.deactivated``
+    * ``pretix.customer.created``
+    * ``pretix.customer.changed``
+    * ``pretix.customer.anonymized``
 
 Installed plugins might register more valid values.
 

doc/development/api/general.rst

@@ -61,7 +61,7 @@ Backend
              item_formsets, order_search_filter_q, order_search_forms
 
 .. automodule:: pretix.base.signals
-   :members: logentry_display, logentry_object_link, requiredaction_display, timeline_events, orderposition_blocked_display
+   :members: logentry_display, logentry_object_link, requiredaction_display, timeline_events, orderposition_blocked_display, customer_created, customer_signed_in
 
 Vouchers
 """"""""

doc/development/api/payment.rst

@@ -70,6 +70,8 @@ The provider class
 
    .. autoattribute:: settings_form_fields
 
+   .. autoattribute:: walletqueries
+
    .. automethod:: settings_form_clean
 
    .. automethod:: settings_content_render

doc/development/implementation/i18n.rst

@@ -37,7 +37,7 @@ you to execute a piece of code with a different locale:
 This is very useful e.g. when sending an email to a user that has a different language than the user performing the
 action that causes the mail to be sent.
 
-.. _translation features: https://docs.djangoproject.com/en/1.9/topics/i18n/translation/
+.. _translation features: https://docs.djangoproject.com/en/4.2/topics/i18n/translation/
 .. _GNU gettext: https://www.gnu.org/software/gettext/
 .. _strings: https://django-i18nfield.readthedocs.io/en/latest/strings.html
 .. _database fields: https://django-i18nfield.readthedocs.io/en/latest/quickstart.html

doc/development/implementation/urlconfig.rst

@@ -15,25 +15,33 @@ and the admin panel is available at ``https://pretix.eu/control/event/bigorg/awe
 
 If the organizer now configures a custom domain like ``tickets.bigorg.com``, his event will
 from now on be available on ``https://tickets.bigorg.com/awesomecon/``. The former URL at
-``pretix.eu`` will redirect there. However, the admin panel will still only be available
-on ``pretix.eu`` for convenience and security reasons.
+``pretix.eu`` will redirect there. It's also possible to do this for just an event, in which
+case the event will be available on ``https://tickets.awesomecon.org/``.
+
+However, the admin panel will still only be available on ``pretix.eu`` for convenience and security reasons.
 
 URL routing
 -----------
 
 The hard part about implementing this URL routing in Django is that
 ``https://pretix.eu/bigorg/awesomecon/`` contains two parameters of nearly arbitrary content
-and ``https://tickets.bigorg.com/awesomecon/`` contains only one. The only robust way to do
-this is by having *separate* URL configuration for those two cases. In pretix, we call the
-former our ``maindomain`` config and the latter our ``subdomain`` config. For pretix's core
-modules we do some magic to avoid duplicate configuration, but for a fairly simple plugin with
-only a handful of routes, we recommend just configuring the two URL sets separately.
+and ``https://tickets.bigorg.com/awesomecon/`` contains only one and ``https://tickets.awesomecon.org/`` does not contain any.
+The only robust way to do this is by having *separate* URL configuration for those three cases.
 
+In pretix, we therefore do not have a global URL configuration, but three, living in the following modules:
+
+- ``pretix.multidomain.maindomain_urlconf``
+- ``pretix.multidomain.organizer_domain_urlconf``
+- ``pretix.multidomain.event_domain_urlconf``
+
+We provide some helper utilities to work with these to avoid duplicate configuration of the individual URLs.
 The file ``urls.py`` inside your plugin package will be loaded and scanned for URL configuration
 automatically and should be provided by any plugin that provides any view.
+However, unlike plain Django, we look not only for a ``urlpatterns`` attribute on the module but support other
+attributes like ``event_patterns`` and ``organizer_patterns`` as well.
 
-A very basic example that provides one view in the admin panel and one view in the frontend
-could look like this::
+For example, for a simple plugin that adds one URL to the backend and one event-level URL to the frontend, you can
+create the following configuration in your ``urls.py``::
 
     from django.urls import re_path
 
@@ -52,7 +60,7 @@ could look like this::
     As you can see, the view in the frontend is not included in the standard Django ``urlpatterns``
     setting but in a separate list with the name ``event_patterns``. This will automatically prepend
     the appropriate parameters to the regex (e.g. the event or the event and the organizer, depending
-    on the called domain).
+    on the called domain). For organizer-level views, ``organizer_patterns`` works the same way.
 
 If you only provide URLs in the admin area, you do not need to provide a ``event_patterns`` attribute.
 
@@ -71,11 +79,16 @@ is a python method that emulates a behavior similar to ``reverse``:
 
 .. autofunction:: pretix.multidomain.urlreverse.eventreverse
 
+If you need to communicate the URL externally, you can use a different method to ensure that it is always an absolute URL:
+
+.. autofunction:: pretix.multidomain.urlreverse.build_absolute_uri
+
 In addition, there is a template tag that works similar to ``url`` but takes an event or organizer object
 as its first argument and can be used like this::
 
     {% load eventurl %}
     <a href="{% eventurl request.event "presale:event.checkout" step="payment" %}">Pay</a>
+    <a href="{% abseventurl request.event "presale:event.checkout" step="payment" %}">Pay</a>
 
 
 Implementation details

doc/development/index.rst

@@ -12,3 +12,4 @@ Developer documentation
    api/index
    structure
    translation/index
+   nfc/index

doc/development/nfc/index.rst

@@ -0,0 +1,15 @@
+NFC media
+=========
+
+pretix supports using NFC chips as "reusable media", for example to store gift cards or tickets.
+
+Most of this implementation currently lives in our proprietary app pretixPOS, but in the future might also become part of our open-source pretixSCAN solution.
+Either way, we want this to be an open ecosystem and therefore document the exact mechanisms in use on the following pages.
+
+We support multiple implementations of NFC media, each documented on its own page:
+
+.. toctree::
+   :maxdepth: 2
+
+   uid
+   mf0aes

doc/development/nfc/mf0aes.rst

@@ -0,0 +1,113 @@
+Mifare Ultralight AES
+=====================
+
+We offer an implementation that provides a higher security level than the UID-based approach and uses the `Mifare Ultralight AES`_ chip sold by NXP.
+We believe the security model of this approach is adequate to the situation where this will usually be used and we'll outline known risks below.
+
+If you want to dive deeper into the properties of the Mifare Ultralight AES chip, we recommend reading the `data sheet`_.
+
+Random UIDs
+-----------
+
+Mifare Ultralight AES supports a feature that returns a randomized UID every time a non-authenticated user tries to
+read the UID. This has a strong privacy benefit, since no unauthorized entity can use the NFC chips to track users.
+On the other hand, this reduces interoperability of the system. For example, this prevents you from using the same NFC
+chips for a different purpose where you only need the UID. This will also prevent your guests from reading their UID
+themselves with their phones, which might be useful e.g. in debugging situations.
+
+Since there's no one-size-fits-all choice here, you can enable or disable this feature in the pretix organizer
+settings. If you change it, the change will apply to all newly encoded chips after the change.
+
+Key management
+--------------
+
+For every organizer, the server will generate create a "key set", which consists of a publicly known ID (random 32-bit integer) and two 16-byte keys ("diversification key" and "UID key").
+
+Using our :ref:`Device authentication mechanism <rest-deviceauth>`, an authorized device can submit a locally generated RSA public key to the server.
+This key can no longer changed on the server once it is set, thus protecting against the attack scenario of a leaked device API token.
+
+The server will then include key sets in the response to ``/api/v1/device/info``, encrypted with the device's RSA key.
+This includes all key sets generated for the organizer the device belongs to, as well as all keys of organizers that have granted sufficient access to this organizer.
+
+The device will decrypt the key sets using its RSA key and store the key sets locally.
+
+.. warning:: The device **will** have access to the raw key sets. Therefore, there is a risk of leaked master keys if an
+             authorized device is stolen or abused. Our implementation in pretixPOS attempts to make this very hard on
+             modern, non-rooted Android devices by keeping them encrypted with the RSA key and only storing the RSA key
+             in the hardware-backed keystore of the device. A sufficiently motivated attacker, however, will likely still
+             be able to extract the keys from a stolen device.
+
+Encoding a chip
+---------------
+
+When a new chip is encoded, the following steps will be taken:
+
+- The UID of the chip is retrieved.
+
+- A chip-specific key is generated using the mechanism documented in `AN10922`_ using the "diversification key" from the
+  organizer's key set as the CMAC key and the diversification input concatenated in the from of ``0x01 + UID + APPID + SYSTEMID``
+  with the following values:
+
+  - The UID of the chip as ``UID``
+
+  - ``"eu.pretix"`` (``0x65 0x75 0x2e 0x70 0x72 0x65 0x74 0x69 0x78``) as ``APPID``
+
+  - The ``public_id`` from the organizer's key set as a 4-byte big-endian value as ``SYSTEMID``
+
+- The chip-specific key is written to the chip as the "data protection key" (config pages 0x30 to 0x33)
+
+- The UID key from the organizer's key set is written to the chip as the "UID retrieval key" (config pages 0x34 to 0x37)
+
+- The config page 0x29 is set like this:
+
+  - ``RID_ACT`` (random UID) to ``1`` or ``0`` based on the organizer's configuration
+  - ``SEC_MSG_ACT`` (secure messaging) to ``1``
+  - ``AUTH0`` (first page that needs authentication) to 0x04 (first non-UID page)
+
+- The config page 0x2A is set like this:
+
+  - ``PROT`` to ``0`` (only write access restricted, not read access)
+  - ``AUTHLIM`` to ``256`` (maximum number of wrong authentications before "self-desctruction")
+  - Everything else to its default value (no lock bits are set)
+
+- The ``public_id`` of the key set will be written to page 0x04 as a big-endian value
+
+- The UID of the chip will be registered as a reusable medium on the server.
+
+.. warning:: During encoding, the chip-specific key and the UID key are transmitted in plain text over the air. The
+             security model therefore relies on the encoding of chips being performed in a trusted physical environment
+             to prevent a nearby attacker from sniffing the keys with a strong antenna.
+
+.. note:: If an attacker tries to authenticate with the chip 256 times using the wrong key, the chip will become
+          unusable. A chip may also become unusable if it is detached from the reader in the middle of the encoding
+          process (even though we've tried to implement it in a way that makes this unlikely).
+
+Usage
+-----
+
+When a chip is presented to the NFC reader, the following steps will be taken:
+
+- Command ``GET_VERSION`` is used to determine if it is a Mifare Ultralight AES chip (if not, abort).
+
+- Page 0x04 is read. If it is all zeroes, the chip is considered un-encoded (abort). If it contains a value that
+  corresponds to the ``public_id`` of a known key set, this key set is used for all further operations. If it contains
+  a different value, we consider this chip to belong to a different organizer or not to a pretix system at all (abort).
+
+- An authentication with the chip using the UID key is performed.
+
+- The UID of the chip will be read.
+
+- The chip-specific key will be derived using the mechanism described above in the encoding step.
+
+- An authentication with the chip using the chip-specific key is performed. If this is fully successful, this step
+  proves that the chip knows the same chip-specific key as we do and is therefore an authentic chip encoded by us and
+  we can trust its UID value.
+
+- The UID is transmitted to the server to fetch the correct medium.
+
+During these steps, the keys are never transmitted in plain text and can thus not be sniffed by a nearby attacker
+with a strong antenna.
+
+.. _Mifare Ultralight AES: https://www.nxp.com/products/rfid-nfc/mifare-hf/mifare-ultralight/mifare-ultralight-aes-enhanced-security-for-limited-use-contactless-applications:MF0AESx20
+.. _data sheet: https://www.nxp.com/docs/en/data-sheet/MF0AES(H)20.pdf
+.. _AN10922: https://www.nxp.com/docs/en/application-note/AN10922.pdf

doc/development/nfc/uid.rst

@@ -0,0 +1,10 @@
+UID-based
+=========
+
+With UID-based NFC, only the unique ID (UID) of the NFC chip is used for identification purposes.
+This can be used with virtually all NFC chips that provide compatibility with the NFC reader in use, typically at least all chips that comply with ISO/IEC 14443-3A.
+
+We make only one restriction: The UID may not start with ``08``, since that usually signifies a randomized UID that changes on every read (which would not be very useful).
+
+.. warning:: The UID-based approach provides only a very low level of security. It is easy to clone a chip with the same
+             UID and impersonate someone else.

doc/development/setup.rst

@@ -96,6 +96,20 @@ http://localhost:8000/control/ for the admin view.
           port (for example because you develop on `pretixdroid`_), you can check
           `Django's documentation`_ for more options.
 
+When running the local development webserver, ensure Celery is not configured
+in ``pretix.cfg``. i.e., you should remove anything such as::
+
+    [celery]
+    backend=redis://redis:6379/2
+    broker=redis://redis:6379/2
+
+If you choose to use Celery for development, you must also start a Celery worker
+process::
+
+    celery -A pretix.celery_app worker -l info
+
+However, beware that code changes will not auto-reload within Celery.
+
 .. _`checksandtests`:
 
 Code checks and unit tests

doc/plugins/epaybl.rst

@@ -0,0 +1,143 @@
+ePayBL
+======
+
+.. note::
+
+    Since ePayBL is only available to german federal, provincial and communal entities, the following page is also
+    only provided in german. Should you require assistance with ePayBL and do not speak this language, please feel free
+    reach out to support@pretix.eu.
+
+
+Einführung
+----------
+
+.. note::
+
+    Sollten Sie lediglich schnell entscheiden wollen, welcher Kontierungsmodus in den Einstellungen des pretix
+    ePayBL-plugins gewählt werden soll, so springen Sie direkt zur Sektion :ref:`Kontierungsmodus`.
+
+
+`ePayBL`_ - das ePayment-System von Bund und Länder - ist das am weitesten verbreitete Zahlungssystem für Bundes-, Länder-
+sowie kommunale Aufgabenträger. Während es nur wie eines von vielen anderen Zahlungssystemen scheint, so bietet es
+seinen Nutzern besondere Vorteile, wie die automatische Erfassung von Zahlungsbelegen, dem Übertragen von Buchungen in
+Haushaltskassen/-systeme sowie die automatische Erfassung von Kontierungen und Steuermerkmalen.
+
+Rein technisch gesehen ist ePayBL hierbei nicht ein eigenständiger Zahlungsdienstleister sondern nur ein eine Komponente
+im komplexen System, dass die Zahlungsabwicklung für Kommunen und Behörden ist.
+
+Im folgenden der schematische Aufbau einer Umgebung, in welcher ePayBL zum Einsatz kommt:
+
+.. figure:: img/epaybl_flowchart.png
+   :class: screenshot
+
+   Quelle: Integrationshandbuch ePayBL-Konnektor, DResearch Digital Media Systems GmbH
+
+
+In diesem Schaubild stellt pretix, bzw. die von Ihnen als Veranstalter angelegten Ticketshops, das Fachverfahren dar.
+
+ePayBL stellt das Bindeglied zwischen den Fachverfahren, Haushaltssystemen und dem eigentlichen Zahlungsdienstleister,
+dem sog. ZV-Provider dar. Dieser ZV-Provider ist die Stelle, welche die eigentlichen Kundengelder einzieht und an den
+Händler auszahlt. Das Gros der Zahlungsdienstleister unterstützt pretix hierbei auch direkt; sprich: Sollten Sie die
+Anbindung an Ihre Haushaltssysteme nicht benötigen, kann eine direkte Anbindung in der Regel ebenso - und dies bei meist
+vermindertem Aufwand - vorgenommen werden.
+
+In der Vergangenheit zeigte sich jedoch schnell, dass nicht jeder IT-Dienstleister immer sofort die neueste Version von
+ePayBL seinen Nutzern angeboten hat. Die Gründe hierfür sind mannigfaltig: Von fest vorgegebenen Update-Zyklen bis hin
+zu Systeme mit speziellen Anpassungen, kann leider nicht davon ausgegangen werden, dass alle ePayBL-Systeme exakt gleich
+ansprechbar sind - auch wenn es sich dabei eigentlich um einen standardisierten Dienst handelt.
+
+Aus diesem Grund gibt es mit dem ePayBL-Konnektor eine weitere Abstraktionsschicht welche optional zwischen den
+Fachverfahren und dem ePayBL-Server sitzt. Dieser Konnektor wird so gepflegt, dass er zum einen eine dauerhaft
+gleichartige Schnittstelle den Fachverfahren bietet aber gleichzeitig auch mit jeder Version des ePayBL-Servers
+kommunizieren kann - egal wie neu oder alt, wie regulär oder angepasst diese ist.
+
+Im Grunde müsste daher eigentlich immer gesagt werden, dass pretix eine Anbindung an den ePayBL-Konnektor bietet; nicht
+an "ePayBL" oder den "ePayBL-Server". Diese Unterscheidung kann bei der Ersteinrichtung und Anforderung von Zugangsdaten
+von Relevanz sein. Da in der Praxis jedoch beide Begriffe gleichbedeutend genutzt werden, wird im Folgenden auch nur von
+einer ePayBL-Anbindung die Rede sein - auch wenn explizit der Konnektor gemeint ist.
+
+
+.. _`Kontierungsmodus`:
+
+Kontierungsmodus
+----------------
+
+ePayBL ist ein Produkt, welches für die Abwicklung von Online-Zahlungsvorgängen in der Verwaltung geschaffen wurde. Ein
+Umfeld, in dem klar definiert ist, was ein Kunde gerade bezahlt und wohin das Geld genau fließt. Diese Annahmen lassen
+sich in einem Ticketshop wie pretix jedoch nur teilweise genauso abbilden.
+
+Die ePayBL-Integration für pretix bietet daher zwei unterschiedliche Modi an, wie Buchungen erfasst und an ePayBL und
+damit auch an die dahinterliegenden Haushaltssysteme gemeldet werden können.
+
+Kontierung pro Position/Artikel
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Dieser Modus versucht den klassischen, behördentypischen ePayBL-Zahlungsvorgang abzubilden: Jede einzelne Position, die
+ein Kunde in den Warenkorb legt, wird auch genauso 1:1 an ePayBL und die Hintergrundsysteme übermittelt.
+
+Hierbei muss zwingend auch für jede Position ein Kennzeichen für Haushaltsstelle und Objektnummer, sowie optional ein
+Kontierungsobjekt (``HREF``; bspw. ``stsl=Steuerschlüssel;psp=gsb:Geschäftsbereich,auft:Innenauftrag,kst:Kostenstelle;``
+) übermittelt werden.
+
+Diese Daten sind vom Veranstalter entsprechend für jeden in der Veranstaltung angelegten Artikel innerhalb des Tabs
+"Zusätzliche Einstellungen" der Produkteinstellungen zu hinterlegen.
+
+Während diese Einstellung eine größtmögliche Menge an Kontierungsdaten überträgt und auch ein separates Verbuchen von
+Leistungen auf unterschiedliche Haushaltsstellen erlaubt, so hat diese Option auch einen großen Nachteil: Der Kunde kann
+nur eine Zahlung für seine Bestellung leisten.
+
+Während sich dies nicht nach einem großen Problem anhört, so kann dies beim Kunden zu Frust führen. pretix bietet die
+Option an, dass ein Veranstalter eine Bestellung jederzeit verändern kann: Ändern von Preisen von Positionen in einer
+aufgegebenen Bestellung, Zubuchen und Entfernen von Bestellpositionen, etc. Hat der Kunde seine ursprüngliche Bestellung
+jedoch schon bezahlt, kann pretix nicht mehr die komplette Bestellung mit den passenden Kontierungen übertragen - es
+müsste nur ein Differenz-Abbild zwischen Ursprungsbestellung und aktueller Bestellung übertragen werden. Aber auch wenn
+eine "Nachmeldung" möglich wäre, so wäre ein konkretes Auflösen für was jetzt genau gezahlt wird, nicht mehr möglich.
+
+Daher gilt bei der Nutzung der Kontierung pro Position/Artikel: Der Kunde kann nur eine (erfolgreiche) Zahlung auf seine
+Bestellung leisten.
+
+Eine weitere Einschränkung dieses Modus ist, dass aktuell keine Gebühren-Positionen (Versandkosten, Zahlungs-, Storno-
+oder Servicegebühren) in diesem Modus übertragen werden können. Bitte wenden Sie sich an uns, wenn Sie diese
+Funktionalität benötigen.
+
+
+Kontierung pro Zahlvorgang
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Dieser Modus verabschiedet sich vom behördlichen "Jede Position gehört genau zu einem Haushaltskonto und muss genau
+zugeordnet werden". Stattdessen werden alle Bestellpositionen - inklusive eventuell definierter Gebühren - vermengt und
+nur als ein großer Warenkorb, genauer gesagt: eine einzige Position an ePayBL sowie die Hintergrundsysteme gemeldet.
+
+Während im "pro Postion/Artikel"-Modus jeder Artikel einzeln übermittelt wird und damit auch korrekt pro Artikel der
+jeweilige Brutto- und Nettopreis, sowie der anfallende Steuerbetrag und ein Steuerkennzeichen (mit Hilfe des optionalen
+``HREF``-Attributs) übermittelt werden, ist dies im "pro Zahlvorgang"-Modus nicht möglich.
+
+Stattdessen übermittelt pretix nur einen Betrag für den gesamten Warenkorb: Bruttopreis == Nettopreis. Der Steuerbetrag
+wird hierbei als 0 übermittelt.
+
+Die Angabe einer Haushaltsstelle und Objektnummer, sowie optional der ``HREF``-Kontierungsinformationen ist jedoch
+weiterhin notwendig - allerdings nicht mehr individuell für jeden Artikel/jede Position sondern nur für die gesamte
+Bestellung. Diese Daten sind direkt in den ePayBL-Einstellungen der Veranstaltung unter Einstellungen -> Zahlung ->
+ePayBL vorzunehmen
+
+In der Praxis bedeutet dies, dass in einem angeschlossenen Haushaltssystem nicht nachvollzogen kann, welche Positionen
+konkret erworben und bezahlt wurden - stattdessen kann nur der Fakt, dass etwas verkauft wurde erfasst werden.
+
+Je nach Aufbau und Vorgaben der Finanzbuchhaltung kann dies jedoch ausreichend sein - wenn bspw. eine Ferienfahrt
+angeboten wird und seitens der Haushaltssysteme nicht erfasst werden muss, wie viel vom Gesamtbetrag einer Bestellung
+auf die Ferienfahrt an sich, auf einen Zubringerbus und einen Satz Bettwäsche entfallen ist, sondern (vereinfacht
+gesagt) es ausreichend ist, dass "Eine Summe X für die Haushaltsstelle/Objektnummer geflossen ist".
+
+Dieser Modus der Kontierung bietet Ihnen auch als Vorteil gegenüber dem vorhergehenden an, dass die Bestellungen der
+Kunden jederzeit erweitert und verändert werden können - auch wenn die Ursprungsbestellung schon bezahlt wurde und nur
+noch eine Differenz gezahlt wird.
+
+
+Einschränkungen
+---------------
+
+Zum aktuellen Zeitpunkt erlaubt die pretix-Anbindung an ePayBL nicht das durchführen von Erstattungen von bereits
+geleisteten Zahlungen. Der Prozess hierfür unterscheidet sich von Behörde zu Behörde und muss daher händisch
+durchgeführt werden.
+
+.. _ePayBL: https://www.epaybl.de/

doc/plugins/index.rst

@@ -18,6 +18,7 @@ If you want to **create** a plugin, please go to the
    campaigns
    certificates
    digital
+   epaybl
    exhibitors
    shipping
    imported_secrets

pyproject.toml

@@ -30,13 +30,13 @@ dependencies = [
   "babel",
   "BeautifulSoup4==4.12.*",
   "bleach==5.0.*",
-  "celery==5.2.*",
+  "celery==5.3.*",
   "chardet==5.1.*",
   "cryptography>=3.4.2",
   "css-inline==0.8.*",
   "defusedcsv>=1.1.0",
   "dj-static",
-  "Django==4.1.*",
+  "Django==4.2.*",
   "django-bootstrap3==23.1.*",
   "django-compressor==4.3.*",
   "django-countries==7.5.*",
@@ -59,10 +59,10 @@ dependencies = [
   "dnspython==2.3.*",
   "drf_ujson2==1.7.*",
   "geoip2==4.*",
-  "importlib_metadata==6.6.*",  # Polyfill, we can probably drop this once we require Python 3.10+
+  "importlib_metadata==6.*",  # Polyfill, we can probably drop this once we require Python 3.10+
   "isoweek",
   "jsonschema",
-  "kombu==5.2.*",
+  "kombu==5.3.*",
   "libsass==0.22.*",
   "lxml",
   "markdown==3.4.3",  # 3.3.5 requires importlib-metadata>=4.4, but django-bootstrap3 requires importlib-metadata<3.
@@ -73,9 +73,10 @@ dependencies = [
   "packaging",
   "paypalrestsdk==1.13.*",
   "paypal-checkout-serversdk==1.0.*",
-  "PyJWT==2.6.*",
+  "PyJWT==2.7.*",
   "phonenumberslite==8.13.*",
   "Pillow==9.5.*",
+  "pretix-plugin-build",
   "protobuf==4.23.*",
   "psycopg2-binary",
   "pycountry",
@@ -89,9 +90,9 @@ dependencies = [
   "pytz-deprecation-shim==0.1.*",
   "pyuca",
   "qrcode==7.4.*",
-  "redis==4.5.*,>=4.5.4",
+  "redis==4.6.*",
   "reportlab==4.0.*",
-  "requests==2.30.*",
+  "requests==2.31.*",
   "sentry-sdk==1.15.*",
   "sepaxml==2.6.*",
   "slimit",
@@ -111,6 +112,7 @@ memcached = ["pylibmc"]
 dev = [
   "coverage",
   "coveralls",
+  "fakeredis==2.18.*",
   "flake8==6.0.*",
   "freezegun",
   "isort==5.12.*",

setup.py

@@ -29,7 +29,6 @@ sys.path.append(str(Path.cwd() / 'src'))
 
 
 def _CustomBuild(*args, **kwargs):
-    print(sys.path)
     from pretix._build import CustomBuild
     return CustomBuild(*args, **kwargs)
 

src/pretix/__init__.py

@@ -19,4 +19,4 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
-__version__ = "4.21.0.dev0"
+__version__ = "2023.8.0.dev0"

src/pretix/_base_settings.py

@@ -196,7 +196,14 @@ STATICFILES_DIRS = [
 
 STATICI18N_ROOT = os.path.join(BASE_DIR, "pretix/static")
 
-STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
+STORAGES = {
+    "default": {
+        "BACKEND": "django.core.files.storage.FileSystemStorage",
+    },
+    "staticfiles": {
+        "BACKEND": "django.contrib.staticfiles.storage.ManifestStaticFilesStorage",
+    },
+}
 
 # if os.path.exists(os.path.join(DATA_DIR, 'static')):
 #     STATICFILES_DIRS.insert(0, os.path.join(DATA_DIR, 'static'))

src/pretix/_build.py

@@ -45,6 +45,10 @@ def npm_install():
 
 class CustomBuild(build):
     def run(self):
+        if "src" not in os.listdir(".") or "pretix" not in os.listdir("src"):
+            # Only run this command on the pretix module, not on other modules even if it's registered globally
+            # in some cases
+            return build.run(self)
         if "PRETIX_DOCKER_BUILD" in os.environ:
             return  # this is a hack to allow calling this file early in our docker build to make use of caching
         os.environ.setdefault("DJANGO_SETTINGS_MODULE", "pretix._build_settings")
@@ -68,6 +72,10 @@ class CustomBuild(build):
 
 class CustomBuildExt(build_ext):
     def run(self):
+        if "src" not in os.listdir(".") or "pretix" not in os.listdir("src"):
+            # Only run this command on the pretix module, not on other modules even if it's registered globally
+            # in some cases
+            return build_ext.run(self)
         if "PRETIX_DOCKER_BUILD" in os.environ:
             return  # this is a hack to allow calling this file early in our docker build to make use of caching
         npm_install()

src/pretix/api/auth/devicesecurity.py

@@ -223,6 +223,7 @@ class PretixPosSecurityProfile(AllowListSecurityProfile):
         ('POST', 'api-v1:checkinrpc.redeem'),
         ('GET', 'api-v1:checkinrpc.search'),
         ('POST', 'api-v1:reusablemedium-lookup'),
+        ('POST', 'api-v1:reusablemedium-list'),
     )
 
 

src/pretix/api/middleware.py

@@ -59,7 +59,7 @@ class IdempotencyMiddleware:
         auth_hash = sha1(auth_hash_parts.encode()).hexdigest()
         idempotency_key = request.headers.get('X-Idempotency-Key', '')
 
-        with transaction.atomic():
+        with transaction.atomic(durable=True):
             call, created = ApiCall.objects.select_for_update(of=OF_SELF).get_or_create(
                 auth_hash=auth_hash,
                 idempotency_key=idempotency_key,
@@ -75,7 +75,7 @@ class IdempotencyMiddleware:
 
         if created:
             resp = self.get_response(request)
-            with transaction.atomic():
+            with transaction.atomic(durable=True):
                 if resp.status_code in (409, 429, 500, 503):
                     # This is the exception: These calls are *meant* to be retried!
                     call.delete()

src/pretix/api/serializers/__init__.py

@@ -19,6 +19,8 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
+import json
+
 from rest_framework import serializers
 
 
@@ -46,3 +48,16 @@ class AsymmetricField(serializers.Field):
 
     def run_validation(self, data=serializers.empty):
         return self.write.run_validation(data)
+
+
+class CompatibleJSONField(serializers.JSONField):
+    def to_internal_value(self, data):
+        try:
+            return json.dumps(data)
+        except (TypeError, ValueError):
+            self.fail('invalid')
+
+    def to_representation(self, value):
+        if value:
+            return json.loads(value)
+        return value

src/pretix/api/serializers/discount.py

@@ -32,11 +32,13 @@ class DiscountSerializer(I18nAwareModelSerializer):
                   'available_until', 'subevent_mode', 'condition_all_products', 'condition_limit_products',
                   'condition_apply_to_addons', 'condition_min_count', 'condition_min_value',
                   'benefit_discount_matching_percent', 'benefit_only_apply_to_cheapest_n_matches',
-                  'condition_ignore_voucher_discounted')
+                  'benefit_same_products', 'benefit_limit_products', 'benefit_apply_to_addons',
+                  'benefit_ignore_voucher_discounted', 'condition_ignore_voucher_discounted')
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.fields['condition_limit_products'].queryset = self.context['event'].items.all()
+        self.fields['benefit_limit_products'].queryset = self.context['event'].items.all()
 
     def validate(self, data):
         data = super().validate(data)

src/pretix/api/serializers/event.py

@@ -46,6 +46,7 @@ from rest_framework import serializers
 from rest_framework.fields import ChoiceField, Field
 from rest_framework.relations import SlugRelatedField
 
+from pretix.api.serializers import CompatibleJSONField
 from pretix.api.serializers.i18n import I18nAwareModelSerializer
 from pretix.api.serializers.settings import SettingsSerializer
 from pretix.base.models import Device, Event, TaxRule, TeamAPIToken
@@ -53,6 +54,7 @@ from pretix.base.models.event import SubEvent
 from pretix.base.models.items import (
     ItemMetaProperty, SubEventItem, SubEventItemVariation,
 )
+from pretix.base.models.tax import CustomRulesValidator
 from pretix.base.services.seating import (
     SeatProtected, generate_seats, validate_plan_change,
 )
@@ -650,9 +652,16 @@ class SubEventSerializer(I18nAwareModelSerializer):
 
 
 class TaxRuleSerializer(CountryFieldMixin, I18nAwareModelSerializer):
+    custom_rules = CompatibleJSONField(
+        validators=[CustomRulesValidator()],
+        required=False,
+        allow_null=True,
+    )
+
     class Meta:
         model = TaxRule
-        fields = ('id', 'name', 'rate', 'price_includes_tax', 'eu_reverse_charge', 'home_country', 'internal_name', 'keep_gross_if_rate_changes')
+        fields = ('id', 'name', 'rate', 'price_includes_tax', 'eu_reverse_charge', 'home_country', 'internal_name',
+                  'keep_gross_if_rate_changes', 'custom_rules')
 
 
 class EventSettingsSerializer(SettingsSerializer):
@@ -719,6 +728,7 @@ class EventSettingsSerializer(SettingsSerializer):
         'payment_term_minutes',
         'payment_term_last',
         'payment_term_expire_automatically',
+        'payment_term_expire_delay_days',
         'payment_term_accept_late',
         'payment_explanation',
         'payment_pending_hidden',
@@ -807,6 +817,10 @@ class EventSettingsSerializer(SettingsSerializer):
         'reusable_media_type_nfc_uid',
         'reusable_media_type_nfc_uid_autocreate_giftcard',
         'reusable_media_type_nfc_uid_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes_random_uid',
     ]
     readonly_fields = [
         # These are read-only since they are currently only settable on organizers, not events
@@ -816,6 +830,10 @@ class EventSettingsSerializer(SettingsSerializer):
         'reusable_media_type_nfc_uid',
         'reusable_media_type_nfc_uid_autocreate_giftcard',
         'reusable_media_type_nfc_uid_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes_random_uid',
     ]
 
     def __init__(self, *args, **kwargs):
@@ -884,6 +902,8 @@ class DeviceEventSettingsSerializer(EventSettingsSerializer):
         'name_scheme',
         'reusable_media_type_barcode',
         'reusable_media_type_nfc_uid',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_random_uid',
         'system_question_order',
     ]
 

src/pretix/api/serializers/exporters.py

@@ -93,7 +93,7 @@ class JobRunSerializer(serializers.Serializer):
         if events is not None and not isinstance(ex, OrganizerLevelExportMixin):
             self.fields["events"] = serializers.SlugRelatedField(
                 queryset=events,
-                required=True,
+                required=False,
                 allow_empty=False,
                 slug_field='slug',
                 many=True
@@ -156,8 +156,9 @@ class JobRunSerializer(serializers.Serializer):
     def to_internal_value(self, data):
         if isinstance(data, QueryDict):
             data = data.copy()
+
         for k, v in self.fields.items():
-            if isinstance(v, serializers.ManyRelatedField) and k not in data:
+            if isinstance(v, serializers.ManyRelatedField) and k not in data and k != "events":
                 data[k] = []
 
         for fk in self.fields.keys():

src/pretix/api/serializers/media.py

@@ -60,6 +60,8 @@ class NestedGiftCardSerializer(GiftCardSerializer):
 
 
 class ReusableMediaSerializer(I18nAwareModelSerializer):
+    organizer = serializers.SlugRelatedField(slug_field='slug', read_only=True)
+
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
 
@@ -111,6 +113,7 @@ class ReusableMediaSerializer(I18nAwareModelSerializer):
         model = ReusableMedium
         fields = (
             'id',
+            'organizer',
             'created',
             'updated',
             'type',

src/pretix/api/serializers/order.py

@@ -19,7 +19,6 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
-import json
 import logging
 import os
 from collections import Counter, defaultdict
@@ -28,6 +27,7 @@ from decimal import Decimal
 import pycountry
 from django.conf import settings
 from django.core.files import File
+from django.db import models
 from django.db.models import F, Q
 from django.utils.encoding import force_str
 from django.utils.timezone import now
@@ -39,6 +39,7 @@ from rest_framework.exceptions import ValidationError
 from rest_framework.relations import SlugRelatedField
 from rest_framework.reverse import reverse
 
+from pretix.api.serializers import CompatibleJSONField
 from pretix.api.serializers.event import SubEventSerializer
 from pretix.api.serializers.i18n import I18nAwareModelSerializer
 from pretix.api.serializers.item import (
@@ -283,11 +284,12 @@ class FailedCheckinSerializer(I18nAwareModelSerializer):
     raw_item = serializers.PrimaryKeyRelatedField(queryset=Item.objects.none(), required=False, allow_null=True)
     raw_variation = serializers.PrimaryKeyRelatedField(queryset=ItemVariation.objects.none(), required=False, allow_null=True)
     raw_subevent = serializers.PrimaryKeyRelatedField(queryset=SubEvent.objects.none(), required=False, allow_null=True)
+    nonce = serializers.CharField(required=False, allow_null=True)
 
     class Meta:
         model = Checkin
         fields = ('error_reason', 'error_explanation', 'raw_barcode', 'raw_item', 'raw_variation',
-                  'raw_subevent', 'datetime', 'type', 'position')
+                  'raw_subevent', 'nonce', 'datetime', 'type', 'position')
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
@@ -372,11 +374,15 @@ class PdfDataSerializer(serializers.Field):
                 self.context['vars_images'] = get_images(self.context['event'])
 
             for k, f in self.context['vars'].items():
-                try:
-                    res[k] = f['evaluate'](instance, instance.order, ev)
-                except:
-                    logger.exception('Evaluating PDF variable failed')
-                    res[k] = '(error)'
+                if 'evaluate_bulk' in f:
+                    # Will be evaluated later by our list serializers
+                    res[k] = (f['evaluate_bulk'], instance)
+                else:
+                    try:
+                        res[k] = f['evaluate'](instance, instance.order, ev)
+                    except:
+                        logger.exception('Evaluating PDF variable failed')
+                        res[k] = '(error)'
 
             if not hasattr(ev, '_cached_meta_data'):
                 ev._cached_meta_data = ev.meta_data
@@ -429,6 +435,38 @@ class PdfDataSerializer(serializers.Field):
             return res
 
 
+class OrderPositionListSerializer(serializers.ListSerializer):
+
+    def to_representation(self, data):
+        # We have a custom implementation of this method because PdfDataSerializer() might keep some elements unevaluated
+        # with a (callable, input) tuple. We'll loop over these entries and evaluate them bulk-wise to save on SQL queries.
+
+        if isinstance(self.parent, OrderSerializer) and isinstance(self.parent.parent, OrderListSerializer):
+            # Do not execute our custom code because it will be executed by OrderListSerializer later for the
+            # full result set.
+            return super().to_representation(data)
+
+        iterable = data.all() if isinstance(data, models.Manager) else data
+
+        data = []
+        evaluate_queue = defaultdict(list)
+
+        for item in iterable:
+            entry = self.child.to_representation(item)
+            if "pdf_data" in entry:
+                for k, v in entry["pdf_data"].items():
+                    if isinstance(v, tuple) and callable(v[0]):
+                        evaluate_queue[v[0]].append((v[1], entry, k))
+            data.append(entry)
+
+        for func, entries in evaluate_queue.items():
+            results = func([item for (item, entry, k) in entries])
+            for (item, entry, k), result in zip(entries, results):
+                entry["pdf_data"][k] = result
+
+        return data
+
+
 class OrderPositionSerializer(I18nAwareModelSerializer):
     checkins = CheckinSerializer(many=True, read_only=True)
     answers = AnswerSerializer(many=True)
@@ -440,6 +478,7 @@ class OrderPositionSerializer(I18nAwareModelSerializer):
     attendee_name = serializers.CharField(required=False)
 
     class Meta:
+        list_serializer_class = OrderPositionListSerializer
         model = OrderPosition
         fields = ('id', 'order', 'positionid', 'item', 'variation', 'price', 'attendee_name', 'attendee_name_parts',
                   'company', 'street', 'zipcode', 'city', 'country', 'state', 'discount',
@@ -468,6 +507,20 @@ class OrderPositionSerializer(I18nAwareModelSerializer):
     def validate(self, data):
         raise TypeError("this serializer is readonly")
 
+    def to_representation(self, data):
+        if isinstance(self.parent, (OrderListSerializer, OrderPositionListSerializer)):
+            # Do not execute our custom code because it will be executed by OrderListSerializer later for the
+            # full result set.
+            return super().to_representation(data)
+
+        entry = super().to_representation(data)
+        if "pdf_data" in entry:
+            for k, v in entry["pdf_data"].items():
+                if isinstance(v, tuple) and callable(v[0]):
+                    entry["pdf_data"][k] = v[0]([v[1]])[0]
+
+        return entry
+
 
 class RequireAttentionField(serializers.Field):
     def to_representation(self, instance: OrderPosition):
@@ -562,7 +615,7 @@ class PaymentURLField(serializers.URLField):
     def to_representation(self, instance: OrderPayment):
         if instance.state != OrderPayment.PAYMENT_STATE_CREATED:
             return None
-        return build_absolute_uri(self.context['event'], 'presale:event.order.pay', kwargs={
+        return build_absolute_uri(instance.order.event, 'presale:event.order.pay', kwargs={
             'order': instance.order.code,
             'secret': instance.order.secret,
             'payment': instance.pk,
@@ -607,13 +660,42 @@ class OrderRefundSerializer(I18nAwareModelSerializer):
 
 class OrderURLField(serializers.URLField):
     def to_representation(self, instance: Order):
-        return build_absolute_uri(self.context['event'], 'presale:event.order', kwargs={
+        return build_absolute_uri(instance.event, 'presale:event.order', kwargs={
             'order': instance.code,
             'secret': instance.secret,
         })
 
 
+class OrderListSerializer(serializers.ListSerializer):
+
+    def to_representation(self, data):
+        # We have a custom implementation of this method because PdfDataSerializer() might keep some elements
+        # unevaluated with a (callable, input) tuple. We'll loop over these entries and evaluate them bulk-wise to
+        # save on SQL queries.
+        iterable = data.all() if isinstance(data, models.Manager) else data
+
+        data = []
+        evaluate_queue = defaultdict(list)
+
+        for item in iterable:
+            entry = self.child.to_representation(item)
+            for p in entry.get("positions", []):
+                if "pdf_data" in p:
+                    for k, v in p["pdf_data"].items():
+                        if isinstance(v, tuple) and callable(v[0]):
+                            evaluate_queue[v[0]].append((v[1], p, k))
+            data.append(entry)
+
+        for func, entries in evaluate_queue.items():
+            results = func([item for (item, entry, k) in entries])
+            for (item, entry, k), result in zip(entries, results):
+                entry["pdf_data"][k] = result
+
+        return data
+
+
 class OrderSerializer(I18nAwareModelSerializer):
+    event = SlugRelatedField(slug_field='slug', read_only=True)
     invoice_address = InvoiceAddressSerializer(allow_null=True)
     positions = OrderPositionSerializer(many=True, read_only=True)
     fees = OrderFeeSerializer(many=True, read_only=True)
@@ -627,8 +709,9 @@ class OrderSerializer(I18nAwareModelSerializer):
 
     class Meta:
         model = Order
+        list_serializer_class = OrderListSerializer
         fields = (
-            'code', 'status', 'testmode', 'secret', 'email', 'phone', 'locale', 'datetime', 'expires', 'payment_date',
+            'code', 'event', 'status', 'testmode', 'secret', 'email', 'phone', 'locale', 'datetime', 'expires', 'payment_date',
             'payment_provider', 'fees', 'total', 'comment', 'custom_followup_at', 'invoice_address', 'positions', 'downloads',
             'checkin_attention', 'last_modified', 'payments', 'refunds', 'require_approval', 'sales_channel',
             'url', 'customer', 'valid_if_pending'
@@ -896,19 +979,6 @@ class OrderPositionCreateSerializer(I18nAwareModelSerializer):
         return data
 
 
-class CompatibleJSONField(serializers.JSONField):
-    def to_internal_value(self, data):
-        try:
-            return json.dumps(data)
-        except (TypeError, ValueError):
-            self.fail('invalid')
-
-    def to_representation(self, value):
-        if value:
-            return json.loads(value)
-        return value
-
-
 class WrappedList:
     def __init__(self, data):
         self._data = data
@@ -1525,6 +1595,7 @@ class InlineInvoiceLineSerializer(I18nAwareModelSerializer):
 
 
 class InvoiceSerializer(I18nAwareModelSerializer):
+    event = SlugRelatedField(slug_field='slug', read_only=True)
     order = serializers.SlugRelatedField(slug_field='code', read_only=True)
     refers = serializers.SlugRelatedField(slug_field='full_invoice_no', read_only=True)
     lines = InlineInvoiceLineSerializer(many=True)
@@ -1533,7 +1604,7 @@ class InvoiceSerializer(I18nAwareModelSerializer):
 
     class Meta:
         model = Invoice
-        fields = ('order', 'number', 'is_cancellation', 'invoice_from', 'invoice_from_name', 'invoice_from_zipcode',
+        fields = ('event', 'order', 'number', 'is_cancellation', 'invoice_from', 'invoice_from_name', 'invoice_from_zipcode',
                   'invoice_from_city', 'invoice_from_country', 'invoice_from_tax_id', 'invoice_from_vat_id',
                   'invoice_to', 'invoice_to_company', 'invoice_to_name', 'invoice_to_street', 'invoice_to_zipcode',
                   'invoice_to_city', 'invoice_to_state', 'invoice_to_country', 'invoice_to_vat_id', 'invoice_to_beneficiary',

src/pretix/api/serializers/organizer.py

@@ -36,9 +36,9 @@ from pretix.api.serializers.settings import SettingsSerializer
 from pretix.base.auth import get_auth_backends
 from pretix.base.i18n import get_language_without_region
 from pretix.base.models import (
-    Customer, Device, GiftCard, GiftCardTransaction, Membership,
-    MembershipType, OrderPosition, Organizer, ReusableMedium, SeatingPlan,
-    Team, TeamAPIToken, TeamInvite, User,
+    Customer, Device, GiftCard, GiftCardAcceptance, GiftCardTransaction,
+    Membership, MembershipType, OrderPosition, Organizer, ReusableMedium,
+    SeatingPlan, Team, TeamAPIToken, TeamInvite, User,
 )
 from pretix.base.models.seating import SeatingPlanLayoutValidator
 from pretix.base.services.mail import SendMailException, mail
@@ -94,6 +94,14 @@ class CustomerSerializer(I18nAwareModelSerializer):
             data['name_parts']['_scheme'] = self.context['request'].organizer.settings.name_scheme
         return data
 
+    def validate_email(self, value):
+        qs = Customer.objects.filter(organizer=self.context['organizer'], email__iexact=value)
+        if self.instance and self.instance.pk:
+            qs = qs.exclude(pk=self.instance.pk)
+        if qs.exists():
+            raise ValidationError(_("An account with this email address is already registered."))
+        return value
+
 
 class CustomerCreateSerializer(CustomerSerializer):
     send_email = serializers.BooleanField(default=False, required=False, allow_null=True)
@@ -183,8 +191,11 @@ class GiftCardSerializer(I18nAwareModelSerializer):
             qs = GiftCard.objects.filter(
                 secret=s
             ).filter(
-                Q(issuer=self.context["organizer"]) | Q(
-                    issuer__gift_card_collector_acceptance__collector=self.context["organizer"])
+                Q(issuer=self.context["organizer"]) |
+                Q(issuer__in=GiftCardAcceptance.objects.filter(
+                    acceptor=self.context["organizer"],
+                    active=True,
+                ).values_list('issuer', flat=True))
             )
             if self.instance:
                 qs = qs.exclude(pk=self.instance.pk)
@@ -248,6 +259,8 @@ class DeviceSerializer(serializers.ModelSerializer):
     unique_serial = serializers.CharField(read_only=True)
     hardware_brand = serializers.CharField(read_only=True)
     hardware_model = serializers.CharField(read_only=True)
+    os_name = serializers.CharField(read_only=True)
+    os_version = serializers.CharField(read_only=True)
     software_brand = serializers.CharField(read_only=True)
     software_version = serializers.CharField(read_only=True)
     created = serializers.DateTimeField(read_only=True)
@@ -260,7 +273,7 @@ class DeviceSerializer(serializers.ModelSerializer):
         fields = (
             'device_id', 'unique_serial', 'initialization_token', 'all_events', 'limit_events',
             'revoked', 'name', 'created', 'initialized', 'hardware_brand', 'hardware_model',
-            'software_brand', 'software_version', 'security_profile'
+            'os_name', 'os_version', 'software_brand', 'software_version', 'security_profile'
         )
 
 
@@ -387,6 +400,9 @@ class OrganizerSettingsSerializer(SettingsSerializer):
         'reusable_media_type_nfc_uid',
         'reusable_media_type_nfc_uid_autocreate_giftcard',
         'reusable_media_type_nfc_uid_autocreate_giftcard_currency',
+        'reusable_media_type_nfc_mf0aes',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard',
+        'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency',
     ]
 
     def __init__(self, *args, **kwargs):

src/pretix/api/urls.py

@@ -61,6 +61,8 @@ orga_router.register(r'membershiptypes', organizer.MembershipTypeViewSet)
 orga_router.register(r'reusablemedia', media.ReusableMediaViewSet)
 orga_router.register(r'teams', organizer.TeamViewSet)
 orga_router.register(r'devices', organizer.DeviceViewSet)
+orga_router.register(r'orders', order.OrganizerOrderViewSet)
+orga_router.register(r'invoices', order.InvoiceViewSet)
 orga_router.register(r'exporters', exporters.OrganizerExportersViewSet, basename='exporters')
 
 team_router = routers.DefaultRouter()
@@ -77,7 +79,7 @@ event_router.register(r'questions', item.QuestionViewSet)
 event_router.register(r'discounts', discount.DiscountViewSet)
 event_router.register(r'quotas', item.QuotaViewSet)
 event_router.register(r'vouchers', voucher.VoucherViewSet)
-event_router.register(r'orders', order.OrderViewSet)
+event_router.register(r'orders', order.EventOrderViewSet)
 event_router.register(r'orderpositions', order.OrderPositionViewSet)
 event_router.register(r'invoices', order.InvoiceViewSet)
 event_router.register(r'revokedsecrets', order.RevokedSecretViewSet, basename='revokedsecrets')

src/pretix/api/views/checkin.py

@@ -164,8 +164,21 @@ class CheckinListViewSet(viewsets.ModelViewSet):
                 secret=serializer.validated_data['raw_barcode']
             ).first()
 
+        clist = self.get_object()
+        if serializer.validated_data.get('nonce'):
+            if kwargs.get('position'):
+                prev = kwargs['position'].all_checkins.filter(nonce=serializer.validated_data['nonce']).first()
+            else:
+                prev = clist.checkins.filter(
+                    nonce=serializer.validated_data['nonce'],
+                    raw_barcode=serializer.validated_data['raw_barcode'],
+                ).first()
+            if prev:
+                # Ignore because nonce is already handled
+                return Response(serializer.data, status=201)
+
         c = serializer.save(
-            list=self.get_object(),
+            list=clist,
             successful=False,
             forced=True,
             force_sent=True,
@@ -396,7 +409,7 @@ def _checkin_list_position_queryset(checkinlists, ignore_status=False, ignore_pr
 
 def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force, checkin_type, ignore_unpaid, nonce,
                     untrusted_input, user, auth, expand, pdf_data, request, questions_supported, canceled_supported,
-                    source_type='barcode', legacy_url_support=False):
+                    source_type='barcode', legacy_url_support=False, simulate=False):
     if not checkinlists:
         raise ValidationError('No check-in list passed.')
 
@@ -433,6 +446,8 @@ def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force,
     )
     raw_barcode_for_checkin = None
     from_revoked_secret = False
+    if simulate:
+        common_checkin_args['__fake_arg_to_prevent_this_from_being_saved'] = True
 
     # 1. Gather a list of positions that could be the one we looking for, either from their ID, secret or
     #    parent secret
@@ -472,13 +487,14 @@ def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force,
             revoked_matches = list(
                 RevokedTicketSecret.objects.filter(event_id__in=list_by_event.keys(), secret=raw_barcode))
             if len(revoked_matches) == 0:
-                checkinlists[0].event.log_action('pretix.event.checkin.unknown', data={
-                    'datetime': datetime,
-                    'type': checkin_type,
-                    'list': checkinlists[0].pk,
-                    'barcode': raw_barcode,
-                    'searched_lists': [cl.pk for cl in checkinlists]
-                }, user=user, auth=auth)
+                if not simulate:
+                    checkinlists[0].event.log_action('pretix.event.checkin.unknown', data={
+                        'datetime': datetime,
+                        'type': checkin_type,
+                        'list': checkinlists[0].pk,
+                        'barcode': raw_barcode,
+                        'searched_lists': [cl.pk for cl in checkinlists]
+                    }, user=user, auth=auth)
 
                 for cl in checkinlists:
                     for k, s in cl.event.ticket_secret_generators.items():
@@ -492,12 +508,13 @@ def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force,
                         except:
                             pass
 
-                Checkin.objects.create(
-                    position=None,
-                    successful=False,
-                    error_reason=Checkin.REASON_INVALID,
-                    **common_checkin_args,
-                )
+                if not simulate:
+                    Checkin.objects.create(
+                        position=None,
+                        successful=False,
+                        error_reason=Checkin.REASON_INVALID,
+                        **common_checkin_args,
+                    )
 
                 if force and legacy_url_support and isinstance(auth, Device):
                     # There was a bug in libpretixsync: If you scanned a ticket in offline mode that was
@@ -539,19 +556,20 @@ def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force,
                 from_revoked_secret = True
             else:
                 op = revoked_matches[0].position
-                op.order.log_action('pretix.event.checkin.revoked', data={
-                    'datetime': datetime,
-                    'type': checkin_type,
-                    'list': list_by_event[revoked_matches[0].event_id].pk,
-                    'barcode': raw_barcode
-                }, user=user, auth=auth)
-                common_checkin_args['list'] = list_by_event[revoked_matches[0].event_id]
-                Checkin.objects.create(
-                    position=op,
-                    successful=False,
-                    error_reason=Checkin.REASON_REVOKED,
-                    **common_checkin_args
-                )
+                if not simulate:
+                    op.order.log_action('pretix.event.checkin.revoked', data={
+                        'datetime': datetime,
+                        'type': checkin_type,
+                        'list': list_by_event[revoked_matches[0].event_id].pk,
+                        'barcode': raw_barcode
+                    }, user=user, auth=auth)
+                    common_checkin_args['list'] = list_by_event[revoked_matches[0].event_id]
+                    Checkin.objects.create(
+                        position=op,
+                        successful=False,
+                        error_reason=Checkin.REASON_REVOKED,
+                        **common_checkin_args
+                    )
                 return Response({
                     'status': 'error',
                     'reason': Checkin.REASON_REVOKED,
@@ -588,24 +606,25 @@ def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force,
             # We choose the first match (regardless of product) for the logging since it's most likely to be the
             # base product according to our order_by above.
             op = op_candidates[0]
-            op.order.log_action('pretix.event.checkin.denied', data={
-                'position': op.id,
-                'positionid': op.positionid,
-                'errorcode': Checkin.REASON_AMBIGUOUS,
-                'reason_explanation': None,
-                'force': force,
-                'datetime': datetime,
-                'type': checkin_type,
-                'list': list_by_event[op.order.event_id].pk,
-            }, user=user, auth=auth)
-            common_checkin_args['list'] = list_by_event[op.order.event_id]
-            Checkin.objects.create(
-                position=op,
-                successful=False,
-                error_reason=Checkin.REASON_AMBIGUOUS,
-                error_explanation=None,
-                **common_checkin_args,
-            )
+            if not simulate:
+                op.order.log_action('pretix.event.checkin.denied', data={
+                    'position': op.id,
+                    'positionid': op.positionid,
+                    'errorcode': Checkin.REASON_AMBIGUOUS,
+                    'reason_explanation': None,
+                    'force': force,
+                    'datetime': datetime,
+                    'type': checkin_type,
+                    'list': list_by_event[op.order.event_id].pk,
+                }, user=user, auth=auth)
+                common_checkin_args['list'] = list_by_event[op.order.event_id]
+                Checkin.objects.create(
+                    position=op,
+                    successful=False,
+                    error_reason=Checkin.REASON_AMBIGUOUS,
+                    error_explanation=None,
+                    **common_checkin_args,
+                )
             return Response({
                 'status': 'error',
                 'reason': Checkin.REASON_AMBIGUOUS,
@@ -652,6 +671,7 @@ def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force,
                 raw_barcode=raw_barcode_for_checkin,
                 raw_source_type=source_type,
                 from_revoked_secret=from_revoked_secret,
+                simulate=simulate,
             )
         except RequiredQuestionsError as e:
             return Response({
@@ -664,23 +684,24 @@ def _redeem_process(*, checkinlists, raw_barcode, answers_data, datetime, force,
                 'list': MiniCheckinListSerializer(list_by_event[op.order.event_id]).data,
             }, status=400)
         except CheckInError as e:
-            op.order.log_action('pretix.event.checkin.denied', data={
-                'position': op.id,
-                'positionid': op.positionid,
-                'errorcode': e.code,
-                'reason_explanation': e.reason,
-                'force': force,
-                'datetime': datetime,
-                'type': checkin_type,
-                'list': list_by_event[op.order.event_id].pk,
-            }, user=user, auth=auth)
-            Checkin.objects.create(
-                position=op,
-                successful=False,
-                error_reason=e.code,
-                error_explanation=e.reason,
-                **common_checkin_args,
-            )
+            if not simulate:
+                op.order.log_action('pretix.event.checkin.denied', data={
+                    'position': op.id,
+                    'positionid': op.positionid,
+                    'errorcode': e.code,
+                    'reason_explanation': e.reason,
+                    'force': force,
+                    'datetime': datetime,
+                    'type': checkin_type,
+                    'list': list_by_event[op.order.event_id].pk,
+                }, user=user, auth=auth)
+                Checkin.objects.create(
+                    position=op,
+                    successful=False,
+                    error_reason=e.code,
+                    error_explanation=e.reason,
+                    **common_checkin_args,
+                )
             return Response({
                 'status': 'error',
                 'reason': e.code,

src/pretix/api/views/device.py

@@ -19,8 +19,12 @@
 # You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
 # <https://www.gnu.org/licenses/>.
 #
+import base64
 import logging
 
+from cryptography.hazmat.backends.openssl.backend import Backend
+from cryptography.hazmat.primitives.asymmetric import padding
+from cryptography.hazmat.primitives.serialization import load_pem_public_key
 from django.db.models import Exists, OuterRef, Q
 from django.db.models.functions import Coalesce
 from django.utils.timezone import now
@@ -34,6 +38,8 @@ from pretix.api.auth.device import DeviceTokenAuthentication
 from pretix.api.views.version import numeric_version
 from pretix.base.models import CheckinList, Device, SubEvent
 from pretix.base.models.devices import Gate, generate_api_token
+from pretix.base.models.media import MediumKeySet
+from pretix.base.services.media import get_keysets_for_organizer
 
 logger = logging.getLogger(__name__)
 
@@ -42,17 +48,73 @@ class InitializationRequestSerializer(serializers.Serializer):
     token = serializers.CharField(max_length=190)
     hardware_brand = serializers.CharField(max_length=190)
     hardware_model = serializers.CharField(max_length=190)
+    os_name = serializers.CharField(max_length=190, required=False, allow_null=True)
+    os_version = serializers.CharField(max_length=190, required=False, allow_null=True)
     software_brand = serializers.CharField(max_length=190)
     software_version = serializers.CharField(max_length=190)
     info = serializers.JSONField(required=False, allow_null=True)
+    rsa_pubkey = serializers.CharField(required=False, allow_null=True)
+
+    def validate(self, attrs):
+        if attrs.get('rsa_pubkey'):
+            try:
+                load_pem_public_key(
+                    attrs['rsa_pubkey'].encode(), Backend()
+                )
+            except:
+                raise ValidationError({'rsa_pubkey': ['Not a valid public key.']})
+        return attrs
 
 
 class UpdateRequestSerializer(serializers.Serializer):
     hardware_brand = serializers.CharField(max_length=190)
     hardware_model = serializers.CharField(max_length=190)
+    os_name = serializers.CharField(max_length=190, required=False, allow_null=True)
+    os_version = serializers.CharField(max_length=190, required=False, allow_null=True)
     software_brand = serializers.CharField(max_length=190)
     software_version = serializers.CharField(max_length=190)
     info = serializers.JSONField(required=False, allow_null=True)
+    rsa_pubkey = serializers.CharField(required=False, allow_null=True)
+
+    def validate(self, attrs):
+        if attrs.get('rsa_pubkey'):
+            try:
+                load_pem_public_key(
+                    attrs['rsa_pubkey'].encode(), Backend()
+                )
+            except:
+                raise ValidationError({'rsa_pubkey': ['Not a valid public key.']})
+        return attrs
+
+
+class RSAEncryptedField(serializers.Field):
+    def to_representation(self, value):
+        public_key = load_pem_public_key(
+            self.context['device'].rsa_pubkey.encode(), Backend()
+        )
+        cipher_text = public_key.encrypt(
+            # RSA/ECB/PKCS1Padding
+            value,
+            padding.PKCS1v15()
+        )
+        return base64.b64encode(cipher_text).decode()
+
+
+class MediumKeySetSerializer(serializers.ModelSerializer):
+    uid_key = RSAEncryptedField(read_only=True)
+    diversification_key = RSAEncryptedField(read_only=True)
+    organizer = serializers.SlugRelatedField(slug_field='slug', read_only=True)
+
+    class Meta:
+        model = MediumKeySet
+        fields = [
+            'public_id',
+            'organizer',
+            'active',
+            'media_type',
+            'uid_key',
+            'diversification_key',
+        ]
 
 
 class GateSerializer(serializers.ModelSerializer):
@@ -99,9 +161,12 @@ class InitializeView(APIView):
         device.initialized = now()
         device.hardware_brand = serializer.validated_data.get('hardware_brand')
         device.hardware_model = serializer.validated_data.get('hardware_model')
+        device.os_name = serializer.validated_data.get('os_name')
+        device.os_version = serializer.validated_data.get('os_version')
         device.software_brand = serializer.validated_data.get('software_brand')
         device.software_version = serializer.validated_data.get('software_version')
         device.info = serializer.validated_data.get('info')
+        device.rsa_pubkey = serializer.validated_data.get('rsa_pubkey')
         device.api_token = generate_api_token()
         device.save()
 
@@ -120,8 +185,15 @@ class UpdateView(APIView):
         device = request.auth
         device.hardware_brand = serializer.validated_data.get('hardware_brand')
         device.hardware_model = serializer.validated_data.get('hardware_model')
+        device.os_name = serializer.validated_data.get('os_name')
+        device.os_version = serializer.validated_data.get('os_version')
         device.software_brand = serializer.validated_data.get('software_brand')
         device.software_version = serializer.validated_data.get('software_version')
+        if serializer.validated_data.get('rsa_pubkey') and serializer.validated_data.get('rsa_pubkey') != device.rsa_pubkey:
+            if device.rsa_pubkey:
+                raise ValidationError({'rsa_pubkey': ['You cannot change the rsa_pubkey of the device once it is set.']})
+            else:
+                device.rsa_pubkey = serializer.validated_data.get('rsa_pubkey')
         device.info = serializer.validated_data.get('info')
         device.save()
         device.log_action('pretix.device.updated', data=serializer.validated_data, auth=device)
@@ -169,8 +241,12 @@ class InfoView(APIView):
                     'pretix': __version__,
                     'pretix_numeric': numeric_version(__version__),
                 }
-            }
-
+            },
+            'medium_key_sets': MediumKeySetSerializer(
+                get_keysets_for_organizer(device.organizer),
+                many=True,
+                context={'device': request.auth}
+            ).data if device.rsa_pubkey else []
         })
 
 

src/pretix/api/views/event.py

@@ -71,6 +71,8 @@ with scopes_disabled():
         ends_after = django_filters.rest_framework.IsoDateTimeFilter(method='ends_after_qs')
         sales_channel = django_filters.rest_framework.CharFilter(method='sales_channel_qs')
         search = django_filters.rest_framework.CharFilter(method='search_qs')
+        date_from = django_filters.rest_framework.IsoDateTimeFromToRangeFilter()
+        date_to = django_filters.rest_framework.IsoDateTimeFromToRangeFilter()
 
         class Meta:
             model = Event
@@ -336,6 +338,8 @@ with scopes_disabled():
         modified_since = django_filters.IsoDateTimeFilter(field_name='last_modified', lookup_expr='gte')
         sales_channel = django_filters.rest_framework.CharFilter(method='sales_channel_qs')
         search = django_filters.rest_framework.CharFilter(method='search_qs')
+        date_from = django_filters.rest_framework.IsoDateTimeFromToRangeFilter()
+        date_to = django_filters.rest_framework.IsoDateTimeFromToRangeFilter()
 
         class Meta:
             model = SubEvent
@@ -411,6 +415,7 @@ class SubEventViewSet(ConditionalListView, viewsets.ModelViewSet):
             'subeventitem_set',
             'subeventitemvariation_set',
             'meta_values',
+            'meta_values__property',
             Prefetch(
                 'seat_category_mappings',
                 to_attr='_seat_category_mappings',

src/pretix/api/views/exporters.py

@@ -133,7 +133,12 @@ class EventExportersViewSet(ExportersMixin, viewsets.ViewSet):
     def exporters(self):
         exporters = []
         responses = register_data_exporters.send(self.request.event)
-        for ex in sorted([response(self.request.event, self.request.organizer) for r, response in responses if response], key=lambda ex: str(ex.verbose_name)):
+        raw_exporters = [response(self.request.event, self.request.organizer) for r, response in responses if response]
+        raw_exporters = [
+            ex for ex in raw_exporters
+            if ex.available_for_user(self.request.user if self.request.user and self.request.user.is_authenticated else None)
+        ]
+        for ex in sorted(raw_exporters, key=lambda ex: str(ex.verbose_name)):
             ex._serializer = JobRunSerializer(exporter=ex)
             exporters.append(ex)
         return exporters
@@ -166,7 +171,7 @@ class OrganizerExportersViewSet(ExportersMixin, viewsets.ViewSet):
             if (
                 not isinstance(ex, OrganizerLevelExportMixin) or
                 perm_holder.has_organizer_permission(self.request.organizer, ex.organizer_required_permission, self.request)
-            )
+            ) and ex.available_for_user(self.request.user if self.request.user and self.request.user.is_authenticated else None)
         ]
         for ex in sorted(raw_exporters, key=lambda ex: str(ex.verbose_name)):
             ex._serializer = JobRunSerializer(exporter=ex, events=events)

src/pretix/api/views/media.py

@@ -39,7 +39,8 @@ from pretix.api.serializers.media import (
 )
 from pretix.base.media import MEDIA_TYPES
 from pretix.base.models import (
-    Checkin, GiftCard, GiftCardTransaction, OrderPosition, ReusableMedium,
+    Checkin, GiftCard, GiftCardAcceptance, GiftCardTransaction, OrderPosition,
+    ReusableMedium,
 )
 from pretix.helpers import OF_SELF
 from pretix.helpers.dicts import merge_dicts
@@ -103,6 +104,12 @@ class ReusableMediaViewSet(viewsets.ModelViewSet):
             auth=self.request.auth,
             data=merge_dicts(self.request.data, {'id': inst.pk})
         )
+        mt = MEDIA_TYPES.get(serializer.validated_data["type"])
+        if mt:
+            m = mt.handle_new(self.request.organizer, inst, self.request.user, self.request.auth)
+            if m:
+                s = self.get_serializer(m)
+                return Response({"result": s.data})
 
     @transaction.atomic()
     def perform_update(self, serializer):
@@ -135,12 +142,28 @@ class ReusableMediaViewSet(viewsets.ModelViewSet):
             s = self.get_serializer(m)
             return Response({"result": s.data})
         except ReusableMedium.DoesNotExist:
-            mt = MEDIA_TYPES.get(s.validated_data["type"])
-            if mt:
-                m = mt.handle_unknown(request.organizer, s.validated_data["identifier"], request.user, request.auth)
-                if m:
-                    s = self.get_serializer(m)
-                    return Response({"result": s.data})
+            try:
+                with scopes_disabled():
+                    m = ReusableMedium.objects.get(
+                        organizer__in=GiftCardAcceptance.objects.filter(
+                            acceptor=request.organizer,
+                            active=True,
+                            reusable_media=True,
+                        ).values_list('issuer', flat=True),
+                        type=s.validated_data["type"],
+                        identifier=s.validated_data["identifier"],
+                    )
+                m.linked_orderposition = None  # not relevant for cross-organizer
+                m.customer = None  # not relevant for cross-organizer
+                s = self.get_serializer(m)
+                return Response({"result": s.data})
+            except ReusableMedium.DoesNotExist:
+                mt = MEDIA_TYPES.get(s.validated_data["type"])
+                if mt:
+                    m = mt.handle_unknown(request.organizer, s.validated_data["identifier"], request.user, request.auth)
+                    if m:
+                        s = self.get_serializer(m)
+                        return Response({"result": s.data})
 
             return Response({"result": None})
 

src/pretix/api/views/order.py

@@ -44,6 +44,7 @@ from rest_framework.exceptions import (
     APIException, NotFound, PermissionDenied, ValidationError,
 )
 from rest_framework.mixins import CreateModelMixin
+from rest_framework.permissions import SAFE_METHODS
 from rest_framework.response import Response
 
 from pretix.api.models import OAuthAccessToken
@@ -185,7 +186,7 @@ with scopes_disabled():
             )
 
 
-class OrderViewSet(viewsets.ModelViewSet):
+class OrderViewSetMixin:
     serializer_class = OrderSerializer
     queryset = Order.objects.none()
     filter_backends = (DjangoFilterBackend, TotalOrderingFilter)
@@ -193,19 +194,12 @@ class OrderViewSet(viewsets.ModelViewSet):
     ordering_fields = ('datetime', 'code', 'status', 'last_modified')
     filterset_class = OrderFilter
     lookup_field = 'code'
-    permission = 'can_view_orders'
-    write_permission = 'can_change_orders'
 
-    def get_serializer_context(self):
-        ctx = super().get_serializer_context()
-        ctx['event'] = self.request.event
-        ctx['pdf_data'] = self.request.query_params.get('pdf_data', 'false') == 'true'
-        ctx['exclude'] = self.request.query_params.getlist('exclude')
-        ctx['include'] = self.request.query_params.getlist('include')
-        return ctx
+    def get_base_queryset(self):
+        raise NotImplementedError()
 
     def get_queryset(self):
-        qs = self.request.event.orders
+        qs = self.get_base_queryset()
         if 'fees' not in self.request.GET.getlist('exclude'):
             if self.request.query_params.get('include_canceled_fees', 'false') == 'true':
                 fqs = OrderFee.all
@@ -227,11 +221,12 @@ class OrderViewSet(viewsets.ModelViewSet):
             opq = OrderPosition.all
         else:
             opq = OrderPosition.objects
-        if request.query_params.get('pdf_data', 'false') == 'true':
+        if request.query_params.get('pdf_data', 'false') == 'true' and getattr(request, 'event', None):
             prefetch_related_objects([request.organizer], 'meta_properties')
             prefetch_related_objects(
                 [request.event],
-                Prefetch('meta_values', queryset=EventMetaValue.objects.select_related('property'), to_attr='meta_values_cached'),
+                Prefetch('meta_values', queryset=EventMetaValue.objects.select_related('property'),
+                         to_attr='meta_values_cached'),
                 'questions',
                 'item_meta_properties',
             )
@@ -266,13 +261,12 @@ class OrderViewSet(viewsets.ModelViewSet):
                 )
             )
 
-    def _get_output_provider(self, identifier):
-        responses = register_ticket_outputs.send(self.request.event)
-        for receiver, response in responses:
-            prov = response(self.request.event)
-            if prov.identifier == identifier:
-                return prov
-        raise NotFound('Unknown output provider.')
+    def get_serializer_context(self):
+        ctx = super().get_serializer_context()
+        ctx['exclude'] = self.request.query_params.getlist('exclude')
+        ctx['include'] = self.request.query_params.getlist('include')
+        ctx['pdf_data'] = False
+        return ctx
 
     @scopes_disabled()  # we are sure enough that get_queryset() is correct, so we save some perforamnce
     def list(self, request, **kwargs):
@@ -289,6 +283,45 @@ class OrderViewSet(viewsets.ModelViewSet):
         serializer = self.get_serializer(queryset, many=True)
         return Response(serializer.data, headers={'X-Page-Generated': date})
 
+
+class OrganizerOrderViewSet(OrderViewSetMixin, viewsets.ReadOnlyModelViewSet):
+    def get_base_queryset(self):
+        perm = "can_view_orders" if self.request.method in SAFE_METHODS else "can_change_orders"
+        if isinstance(self.request.auth, (TeamAPIToken, Device)):
+            return Order.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.auth.get_events_with_permission(perm)
+            )
+        elif self.request.user.is_authenticated:
+            return Order.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.user.get_events_with_permission(perm)
+            )
+        else:
+            raise PermissionDenied()
+
+
+class EventOrderViewSet(OrderViewSetMixin, viewsets.ModelViewSet):
+    permission = 'can_view_orders'
+    write_permission = 'can_change_orders'
+
+    def get_serializer_context(self):
+        ctx = super().get_serializer_context()
+        ctx['event'] = self.request.event
+        ctx['pdf_data'] = self.request.query_params.get('pdf_data', 'false') == 'true'
+        return ctx
+
+    def get_base_queryset(self):
+        return self.request.event.orders
+
+    def _get_output_provider(self, identifier):
+        responses = register_ticket_outputs.send(self.request.event)
+        for receiver, response in responses:
+            prov = response(self.request.event)
+            if prov.identifier == identifier:
+                return prov
+        raise NotFound('Unknown output provider.')
+
     @action(detail=True, url_name='download', url_path='download/(?P<output>[^/]+)')
     def download(self, request, output, **kwargs):
         provider = self._get_output_provider(output)
@@ -661,7 +694,16 @@ class OrderViewSet(viewsets.ModelViewSet):
 
         with language(order.locale, self.request.event.settings.region):
             payment = order.payments.last()
-
+            # OrderCreateSerializer creates at most one payment
+            if payment and payment.state == OrderPayment.PAYMENT_STATE_CONFIRMED:
+                order.log_action(
+                    'pretix.event.order.payment.confirmed', {
+                        'local_id': payment.local_id,
+                        'provider': payment.provider,
+                    },
+                    user=request.user if request.user.is_authenticated else None,
+                    auth=request.auth,
+                )
             order_placed.send(self.request.event, order=order)
             if order.status == Order.STATUS_PAID:
                 order_paid.send(self.request.event, order=order)
@@ -936,6 +978,7 @@ with scopes_disabled():
                 | Q(addon_to__attendee_email__icontains=value)
                 | Q(order__code__istartswith=value)
                 | Q(order__invoice_address__name_cached__icontains=value)
+                | Q(order__invoice_address__company__icontains=value)
                 | Q(order__email__icontains=value)
                 | Q(pk__in=matching_media)
             )
@@ -1772,11 +1815,24 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
     write_permission = 'can_change_orders'
 
     def get_queryset(self):
-        return self.request.event.invoices.prefetch_related('lines').select_related('order', 'refers').annotate(
+        perm = "can_view_orders" if self.request.method in SAFE_METHODS else "can_change_orders"
+        if getattr(self.request, 'event', None):
+            qs = self.request.event.invoices
+        elif isinstance(self.request.auth, (TeamAPIToken, Device)):
+            qs = Invoice.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.auth.get_events_with_permission(perm)
+            )
+        elif self.request.user.is_authenticated:
+            qs = Invoice.objects.filter(
+                event__organizer=self.request.organizer,
+                event__in=self.request.user.get_events_with_permission(perm)
+            )
+        return qs.prefetch_related('lines').select_related('order', 'refers').annotate(
             nr=Concat('prefix', 'invoice_no')
         )
 
-    @action(detail=True, )
+    @action(detail=True)
     def download(self, request, **kwargs):
         invoice = self.get_object()
 
@@ -1795,7 +1851,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
         return resp
 
     @action(detail=True, methods=['POST'])
-    def regenerate(self, request, **kwarts):
+    def regenerate(self, request, **kwargs):
         inv = self.get_object()
         if inv.canceled:
             raise ValidationError('The invoice has already been canceled.')
@@ -1805,7 +1861,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
             raise PermissionDenied('The invoice file is no longer stored on the server.')
         elif inv.sent_to_organizer:
             raise PermissionDenied('The invoice file has already been exported.')
-        elif now().astimezone(self.request.event.timezone).date() - inv.date > datetime.timedelta(days=1):
+        elif now().astimezone(inv.event.timezone).date() - inv.date > datetime.timedelta(days=1):
             raise PermissionDenied('The invoice file is too old to be regenerated.')
         else:
             inv = regenerate_invoice(inv)
@@ -1820,7 +1876,7 @@ class InvoiceViewSet(viewsets.ReadOnlyModelViewSet):
             return Response(status=204)
 
     @action(detail=True, methods=['POST'])
-    def reissue(self, request, **kwarts):
+    def reissue(self, request, **kwargs):
         inv = self.get_object()
         if inv.canceled:
             raise ValidationError('The invoice has already been canceled.')

src/pretix/api/webhooks.py

@@ -189,6 +189,34 @@ class ParametrizedOrderPositionWebhookEvent(ParametrizedOrderWebhookEvent):
         return d
 
 
+class ParametrizedWaitingListEntryWebhookEvent(ParametrizedWebhookEvent):
+
+    def build_payload(self, logentry: LogEntry):
+        # do not use content_object, this is also called in deletion
+        return {
+            'notification_id': logentry.pk,
+            'organizer': logentry.event.organizer.slug,
+            'event': logentry.event.slug,
+            'waitinglistentry': logentry.object_id,
+            'action': logentry.action_type,
+        }
+
+
+class ParametrizedCustomerWebhookEvent(ParametrizedWebhookEvent):
+
+    def build_payload(self, logentry: LogEntry):
+        customer = logentry.content_object
+        if not customer:
+            return None
+
+        return {
+            'notification_id': logentry.pk,
+            'organizer': customer.organizer.slug,
+            'customer': customer.identifier,
+            'action': logentry.action_type,
+        }
+
+
 @receiver(register_webhook_events, dispatch_uid="base_register_default_webhook_events")
 def register_default_webhook_events(sender, **kwargs):
     return (
@@ -321,6 +349,34 @@ def register_default_webhook_events(sender, **kwargs):
             'pretix.event.testmode.deactivated',
             _('Test-Mode of shop has been deactivated'),
         ),
+        ParametrizedWaitingListEntryWebhookEvent(
+            'pretix.event.orders.waitinglist.added',
+            _('Waiting list entry added'),
+        ),
+        ParametrizedWaitingListEntryWebhookEvent(
+            'pretix.event.orders.waitinglist.changed',
+            _('Waiting list entry changed'),
+        ),
+        ParametrizedWaitingListEntryWebhookEvent(
+            'pretix.event.orders.waitinglist.deleted',
+            _('Waiting list entry deleted'),
+        ),
+        ParametrizedWaitingListEntryWebhookEvent(
+            'pretix.event.orders.waitinglist.voucher_assigned',
+            _('Waiting list entry received voucher'),
+        ),
+        ParametrizedCustomerWebhookEvent(
+            'pretix.customer.created',
+            _('Customer account created'),
+        ),
+        ParametrizedCustomerWebhookEvent(
+            'pretix.customer.changed',
+            _('Customer account changed'),
+        ),
+        ParametrizedCustomerWebhookEvent(
+            'pretix.customer.anonymized',
+            _('Customer account anonymized'),
+        ),
     )
 
 

src/pretix/base/cache.py

@@ -62,27 +62,27 @@ class NamespacedCache:
             prefix = int(time.time())
             self.cache.set(self.prefixkey, prefix)
 
-    def set(self, key: str, value: str, timeout: int=300):
+    def set(self, key: str, value: any, timeout: int=300):
         return self.cache.set(self._prefix_key(key), value, timeout)
 
-    def get(self, key: str) -> str:
+    def get(self, key: str) -> any:
         return self.cache.get(self._prefix_key(key, known_prefix=self._last_prefix))
 
-    def get_or_set(self, key: str, default: Callable, timeout=300) -> str:
+    def get_or_set(self, key: str, default: Callable, timeout=300) -> any:
         return self.cache.get_or_set(
             self._prefix_key(key, known_prefix=self._last_prefix),
             default=default,
             timeout=timeout
         )
 
-    def get_many(self, keys: List[str]) -> Dict[str, str]:
+    def get_many(self, keys: List[str]) -> Dict[str, any]:
         values = self.cache.get_many([self._prefix_key(key) for key in keys])
         newvalues = {}
         for k, v in values.items():
             newvalues[self._strip_prefix(k)] = v
         return newvalues
 
-    def set_many(self, values: Dict[str, str], timeout=300):
+    def set_many(self, values: Dict[str, any], timeout=300):
         newvalues = {}
         for k, v in values.items():
             newvalues[self._prefix_key(k)] = v

src/pretix/base/email.py

@@ -134,8 +134,11 @@ class TemplateBasedMailRenderer(BaseHTMLMailRenderer):
     def template_name(self):
         raise NotImplementedError()
 
+    def compile_markdown(self, plaintext):
+        return markdown_compile_email(plaintext)
+
     def render(self, plain_body: str, plain_signature: str, subject: str, order, position) -> str:
-        body_md = markdown_compile_email(plain_body)
+        body_md = self.compile_markdown(plain_body)
         htmlctx = {
             'site': settings.PRETIX_INSTANCE_NAME,
             'site_url': settings.SITE_URL,
@@ -153,7 +156,7 @@ class TemplateBasedMailRenderer(BaseHTMLMailRenderer):
 
         if plain_signature:
             signature_md = plain_signature.replace('\n', '<br>\n')
-            signature_md = markdown_compile_email(signature_md)
+            signature_md = self.compile_markdown(signature_md)
             htmlctx['signature'] = signature_md
 
         if order:

src/pretix/base/exporter.py

@@ -140,7 +140,7 @@ class BaseExporter:
         """
         return {}
 
-    def render(self, form_data: dict) -> Tuple[str, str, bytes]:
+    def render(self, form_data: dict) -> Tuple[str, str, Optional[bytes]]:
         """
         Render the exported file and return a tuple consisting of a filename, a file type
         and file content.
@@ -157,6 +157,13 @@ class BaseExporter:
         """
         raise NotImplementedError()  # NOQA
 
+    def available_for_user(self, user) -> bool:
+        """
+        Allows to do additional checks whether an exporter is available based on the user who calls it. Note that
+        ``user`` may be ``None`` e.g. during API usage.
+        """
+        return True
+
 
 class OrganizerLevelExportMixin:
     @property

src/pretix/base/exporters/orderlist.py

@@ -549,7 +549,9 @@ class OrderListExporter(MultiSheetListExporter):
             headers.append(_('End date'))
         headers += [
             _('Product'),
+            _('Product ID'),
             _('Variation'),
+            _('Variation ID'),
             _('Price'),
             _('Tax rate'),
             _('Tax rule'),
@@ -656,7 +658,9 @@ class OrderListExporter(MultiSheetListExporter):
                         row.append('')
                 row += [
                     str(op.item),
+                    str(op.item_id),
                     str(op.variation) if op.variation else '',
+                    str(op.variation_id) if op.variation_id else '',
                     op.price,
                     op.tax_rate,
                     str(op.tax_rule) if op.tax_rule else '',
@@ -850,6 +854,8 @@ class TransactionListExporter(ListExporter):
             _('Tax rule ID'),
             _('Tax rule'),
             _('Tax value'),
+            _('Gross total'),
+            _('Tax total'),
         ]
 
         if form_data.get('_format') == 'xlsx':
@@ -901,6 +907,8 @@ class TransactionListExporter(ListExporter):
                 t.tax_rule_id or '',
                 str(t.tax_rule.internal_name or t.tax_rule.name) if t.tax_rule_id else '',
                 t.tax_value,
+                t.price * t.count,
+                t.tax_value * t.count,
             ]
 
             if form_data.get('_format') == 'xlsx':
@@ -1143,7 +1151,7 @@ class GiftcardTransactionListExporter(OrganizerLevelExportMixin, ListExporter):
     def iterate_list(self, form_data):
         qs = GiftCardTransaction.objects.filter(
             card__issuer=self.organizer,
-        ).order_by('datetime').select_related('card', 'order', 'order__event')
+        ).order_by('datetime').select_related('card', 'order', 'order__event', 'acceptor')
 
         if form_data.get('date_range'):
             dt_start, dt_end = resolve_timeframe_to_datetime_start_inclusive_end_exclusive(now(), form_data['date_range'], self.timezone)
@@ -1159,6 +1167,7 @@ class GiftcardTransactionListExporter(OrganizerLevelExportMixin, ListExporter):
             _('Amount'),
             _('Currency'),
             _('Order'),
+            _('Organizer'),
         ]
         yield headers
 
@@ -1170,6 +1179,7 @@ class GiftcardTransactionListExporter(OrganizerLevelExportMixin, ListExporter):
                 obj.value,
                 obj.card.currency,
                 obj.order.full_code if obj.order else None,
+                str(obj.acceptor or ""),
             ]
             yield row
 

src/pretix/base/forms/questions.py

@@ -61,6 +61,7 @@ from django.utils.timezone import get_current_timezone, now
 from django.utils.translation import gettext_lazy as _, pgettext_lazy
 from django_countries import countries
 from django_countries.fields import Country, CountryField
+from geoip2.errors import AddressNotFoundError
 from phonenumber_field.formfields import PhoneNumberField
 from phonenumber_field.phonenumber import PhoneNumber
 from phonenumber_field.widgets import PhoneNumberPrefixWidget
@@ -356,9 +357,12 @@ class WrappedPhoneNumberPrefixWidget(PhoneNumberPrefixWidget):
 def guess_country_from_request(request, event):
     if settings.HAS_GEOIP:
         g = GeoIP2()
-        res = g.country(get_client_ip(request))
-        if res['country_code'] and len(res['country_code']) == 2:
-            return Country(res['country_code'])
+        try:
+            res = g.country(get_client_ip(request))
+            if res['country_code'] and len(res['country_code']) == 2:
+                return Country(res['country_code'])
+        except AddressNotFoundError:
+            pass
     return guess_country(event)
 
 

src/pretix/base/media.py

@@ -49,6 +49,9 @@ class BaseMediaType:
     def handle_unknown(self, organizer, identifier, user, auth):
         pass
 
+    def handle_new(self, organizer, medium, user, auth):
+        pass
+
     def __str__(self):
         return str(self.verbose_name)
 
@@ -108,9 +111,43 @@ class NfcUidMediaType(BaseMediaType):
                 return m
 
 
+class NfcMf0aesMediaType(BaseMediaType):
+    identifier = 'nfc_mf0aes'
+    verbose_name = 'NFC Mifare Ultralight AES'
+    medium_created_by_server = False
+    supports_giftcard = True
+    supports_orderposition = False
+
+    def handle_new(self, organizer, medium, user, auth):
+        from pretix.base.models import GiftCard
+
+        if organizer.settings.get(f'reusable_media_type_{self.identifier}_autocreate_giftcard', as_type=bool):
+            with transaction.atomic():
+                gc = GiftCard.objects.create(
+                    issuer=organizer,
+                    expires=organizer.default_gift_card_expiry,
+                    currency=organizer.settings.get(f'reusable_media_type_{self.identifier}_autocreate_giftcard_currency'),
+                )
+                medium.linked_giftcard = gc
+                medium.save()
+                medium.log_action(
+                    'pretix.reusable_medium.linked_giftcard.changed',
+                    user=user, auth=auth,
+                    data={
+                        'linked_giftcard': gc.pk
+                    }
+                )
+                gc.log_action(
+                    'pretix.giftcards.created',
+                    user=user, auth=auth,
+                )
+                return medium
+
+
 MEDIA_TYPES = {
     m.identifier: m for m in [
         BarcodePlainMediaType(),
         NfcUidMediaType(),
+        NfcMf0aesMediaType(),
     ]
 }

src/pretix/base/middleware.py

@@ -26,7 +26,7 @@ from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
 from django.conf import settings
 from django.http import Http404, HttpRequest, HttpResponse
 from django.middleware.common import CommonMiddleware
-from django.urls import get_script_prefix
+from django.urls import get_script_prefix, resolve
 from django.utils import timezone, translation
 from django.utils.cache import patch_vary_headers
 from django.utils.deprecation import MiddlewareMixin
@@ -230,6 +230,8 @@ class SecurityMiddleware(MiddlewareMixin):
     )
 
     def process_response(self, request, resp):
+        url = resolve(request.path_info)
+
         if settings.DEBUG and resp.status_code >= 400:
             # Don't use CSP on debug error page as it breaks of Django's fancy error
             # pages
@@ -249,20 +251,28 @@ class SecurityMiddleware(MiddlewareMixin):
 
         h = {
             'default-src': ["{static}"],
-            'script-src': ['{static}', 'https://checkout.stripe.com', 'https://js.stripe.com'],
+            'script-src': ['{static}'],
             'object-src': ["'none'"],
-            'frame-src': ['{static}', 'https://checkout.stripe.com', 'https://js.stripe.com'],
+            'frame-src': ['{static}'],
             'style-src': ["{static}", "{media}"],
-            'connect-src': ["{dynamic}", "{media}", "https://checkout.stripe.com"],
-            'img-src': ["{static}", "{media}", "data:", "https://*.stripe.com"] + img_src,
+            'connect-src': ["{dynamic}", "{media}"],
+            'img-src': ["{static}", "{media}", "data:"] + img_src,
             'font-src': ["{static}"],
             'media-src': ["{static}", "data:"],
             # form-action is not only used to match on form actions, but also on URLs
             # form-actions redirect to. In the context of e.g. payment providers or
-            # single-sign-on this can be nearly anything so we cannot really restrict
+            # single-sign-on this can be nearly anything, so we cannot really restrict
             # this. However, we'll restrict it to HTTPS.
             'form-action': ["{dynamic}", "https:"] + (['http:'] if settings.SITE_URL.startswith('http://') else []),
         }
+        # Only include pay.google.com for wallet detection purposes on the Payment selection page
+        if (
+                url.url_name == "event.order.pay.change" or
+                (url.url_name == "event.checkout" and url.kwargs['step'] == "payment")
+        ):
+            h['script-src'].append('https://pay.google.com')
+            h['frame-src'].append('https://pay.google.com')
+            h['connect-src'].append('https://google.com/pay')
         if settings.LOG_CSP:
             h['report-uri'] = ["/csp_report/"]
         if 'Content-Security-Policy' in resp:

src/pretix/base/migrations/0242_auto_20230512_1008.py

@@ -0,0 +1,38 @@
+# Generated by Django 3.2.18 on 2023-05-12 10:08
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pretixbase', '0241_itemmetaproperties_required_values'),
+    ]
+
+    operations = [
+        migrations.RenameField(
+            model_name='giftcardacceptance',
+            old_name='collector',
+            new_name='acceptor',
+        ),
+        migrations.AddField(
+            model_name='giftcardacceptance',
+            name='active',
+            field=models.BooleanField(default=True),
+        ),
+        migrations.AddField(
+            model_name='giftcardacceptance',
+            name='reusable_media',
+            field=models.BooleanField(default=False),
+        ),
+        migrations.AlterField(
+            model_name='giftcardacceptance',
+            name='issuer',
+            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='gift_card_acceptor_acceptance', to='pretixbase.organizer'),
+        ),
+        migrations.AlterUniqueTogether(
+            name='giftcardacceptance',
+            unique_together={('issuer', 'acceptor')},
+        ),
+    ]

src/pretix/base/migrations/0243_device_os_name_and_os_version.py

@@ -0,0 +1,23 @@
+# Generated by Django 4.1.9 on 2023-06-26 10:59
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pretixbase', '0242_auto_20230512_1008'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='device',
+            name='os_name',
+            field=models.CharField(max_length=190, null=True),
+        ),
+        migrations.AddField(
+            model_name='device',
+            name='os_version',
+            field=models.CharField(max_length=190, null=True),
+        ),
+    ]

src/pretix/base/migrations/0244_mediumkeyset.py

@@ -0,0 +1,35 @@
+# Generated by Django 3.2.18 on 2023-05-17 11:32
+
+import django.db.models.deletion
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('pretixbase', '0243_device_os_name_and_os_version'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='device',
+            name='rsa_pubkey',
+            field=models.TextField(null=True),
+        ),
+        migrations.CreateModel(
+            name='MediumKeySet',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False)),
+                ('public_id', models.BigIntegerField(unique=True)),
+                ('media_type', models.CharField(max_length=100)),
+                ('active', models.BooleanField(default=True)),
+                ('uid_key', models.BinaryField()),
+                ('diversification_key', models.BinaryField()),
+                ('organizer', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='medium_key_sets', to='pretixbase.organizer')),
+            ],
+        ),
+        migrations.AddConstraint(
+            model_name='mediumkeyset',
+            constraint=models.UniqueConstraint(condition=models.Q(('active', True)), fields=('organizer', 'media_type'), name='keyset_unique_active'),
+        ),
+    ]

src/pretix/base/migrations/0245_discount_benefit_products.py

@@ -0,0 +1,34 @@
+# Generated by Django 4.2.4 on 2023-08-28 12:30
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("pretixbase", "0244_mediumkeyset"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_apply_to_addons",
+            field=models.BooleanField(default=True),
+        ),
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_ignore_voucher_discounted",
+            field=models.BooleanField(default=False),
+        ),
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_limit_products",
+            field=models.ManyToManyField(
+                related_name="benefit_discounts", to="pretixbase.item"
+            ),
+        ),
+        migrations.AddField(
+            model_name="discount",
+            name="benefit_same_products",
+            field=models.BooleanField(default=True),
+        ),
+    ]

src/pretix/base/models/_transactions.py

@@ -97,7 +97,7 @@ def _transactions_mark_order_dirty(order_id, using=None):
     if getattr(dirty_transactions, 'order_ids', None) is None:
         dirty_transactions.order_ids = set()
 
-    if _check_for_dirty_orders not in [func for savepoint_id, func in conn.run_on_commit]:
+    if _check_for_dirty_orders not in [func for (savepoint_id, func, *__) in conn.run_on_commit]:
         transaction.on_commit(_check_for_dirty_orders, using)
         dirty_transactions.order_ids.clear()  # This is necessary to clean up after old threads with rollbacked transactions
 

src/pretix/base/models/devices.py

@@ -143,6 +143,14 @@ class Device(LoggedModel):
         max_length=190,
         null=True, blank=True
     )
+    os_name = models.CharField(
+        max_length=190,
+        null=True, blank=True
+    )
+    os_version = models.CharField(
+        max_length=190,
+        null=True, blank=True
+    )
     software_brand = models.CharField(
         max_length=190,
         null=True, blank=True
@@ -158,6 +166,10 @@ class Device(LoggedModel):
         null=True,
         blank=False
     )
+    rsa_pubkey = models.TextField(
+        null=True,
+        blank=True,
+    )
     info = models.JSONField(
         null=True, blank=True,
     )

src/pretix/base/models/discount.py

@@ -99,7 +99,7 @@ class Discount(LoggedModel):
     )
     condition_apply_to_addons = models.BooleanField(
         default=True,
-        verbose_name=_("Apply to add-on products"),
+        verbose_name=_("Count add-on products"),
         help_text=_("Discounts never apply to bundled products"),
     )
     condition_ignore_voucher_discounted = models.BooleanField(
@@ -107,7 +107,7 @@ class Discount(LoggedModel):
         verbose_name=_("Ignore products discounted by a voucher"),
         help_text=_("If this option is checked, products that already received a discount through a voucher will not "
                     "be considered for this discount. However, products that use a voucher only to e.g. unlock a "
-                    "hidden product or gain access to sold-out quota will still receive the discount."),
+                    "hidden product or gain access to sold-out quota will still be considered."),
     )
     condition_min_count = models.PositiveIntegerField(
         verbose_name=_('Minimum number of matching products'),
@@ -120,6 +120,19 @@ class Discount(LoggedModel):
         default=Decimal('0.00'),
     )
 
+    benefit_same_products = models.BooleanField(
+        default=True,
+        verbose_name=_("Apply discount to same set of products"),
+        help_text=_("By default, the discount is applied across the same selection of products than the condition for "
+                    "the discount given above. If you want, you can however also select a different selection of "
+                    "products.")
+    )
+    benefit_limit_products = models.ManyToManyField(
+        'Item',
+        verbose_name=_("Apply discount to specific products"),
+        related_name='benefit_discounts',
+        blank=True
+    )
     benefit_discount_matching_percent = models.DecimalField(
         verbose_name=_('Percentual discount on matching products'),
         decimal_places=2,
@@ -139,6 +152,18 @@ class Discount(LoggedModel):
         blank=True,
         validators=[MinValueValidator(1)],
     )
+    benefit_apply_to_addons = models.BooleanField(
+        default=True,
+        verbose_name=_("Apply to add-on products"),
+        help_text=_("Discounts never apply to bundled products"),
+    )
+    benefit_ignore_voucher_discounted = models.BooleanField(
+        default=False,
+        verbose_name=_("Ignore products discounted by a voucher"),
+        help_text=_("If this option is checked, products that already received a discount through a voucher will not "
+                    "be discounted. However, products that use a voucher only to e.g. unlock a hidden product or gain "
+                    "access to sold-out quota will still receive the discount."),
+    )
 
     # more feature ideas:
     # - max_usages_per_order
@@ -187,6 +212,14 @@ class Discount(LoggedModel):
                   'on a minimum value.')
             )
 
+        if data.get('subevent_mode') == cls.SUBEVENT_MODE_DISTINCT and not data.get('benefit_same_products'):
+            raise ValidationError(
+                {'benefit_same_products': [
+                    _('You cannot apply the discount to a different set of products if the discount is only valid '
+                      'for bookings of different dates.')
+                ]}
+            )
+
     def allow_delete(self):
         return not self.orderposition_set.exists()
 
@@ -197,6 +230,7 @@ class Discount(LoggedModel):
             'condition_min_value': self.condition_min_value,
             'benefit_only_apply_to_cheapest_n_matches': self.benefit_only_apply_to_cheapest_n_matches,
             'subevent_mode': self.subevent_mode,
+            'benefit_same_products': self.benefit_same_products,
         })
 
     def is_available_by_time(self, now_dt=None) -> bool:
@@ -207,14 +241,14 @@ class Discount(LoggedModel):
             return False
         return True
 
-    def _apply_min_value(self, positions, idx_group, result):
-        if self.condition_min_value and sum(positions[idx][2] for idx in idx_group) < self.condition_min_value:
+    def _apply_min_value(self, positions, condition_idx_group, benefit_idx_group, result):
+        if self.condition_min_value and sum(positions[idx][2] for idx in condition_idx_group) < self.condition_min_value:
             return
 
         if self.condition_min_count or self.benefit_only_apply_to_cheapest_n_matches:
             raise ValueError('Validation invariant violated.')
 
-        for idx in idx_group:
+        for idx in benefit_idx_group:
             previous_price = positions[idx][2]
             new_price = round_decimal(
                 previous_price * (Decimal('100.00') - self.benefit_discount_matching_percent) / Decimal('100.00'),
@@ -222,8 +256,8 @@ class Discount(LoggedModel):
             )
             result[idx] = new_price
 
-    def _apply_min_count(self, positions, idx_group, result):
-        if len(idx_group) < self.condition_min_count:
+    def _apply_min_count(self, positions, condition_idx_group, benefit_idx_group, result):
+        if len(condition_idx_group) < self.condition_min_count:
             return
 
         if not self.condition_min_count or self.condition_min_value:
@@ -233,15 +267,17 @@ class Discount(LoggedModel):
             if not self.condition_min_count:
                 raise ValueError('Validation invariant violated.')
 
-            idx_group = sorted(idx_group, key=lambda idx: (positions[idx][2], -idx))  # sort by line_price
+            condition_idx_group = sorted(condition_idx_group, key=lambda idx: (positions[idx][2], -idx))  # sort by line_price
+            benefit_idx_group = sorted(benefit_idx_group, key=lambda idx: (positions[idx][2], -idx))  # sort by line_price
 
             # Prevent over-consuming of items, i.e. if our discount is "buy 2, get 1 free", we only
             # want to match multiples of 3
-            consume_idx = idx_group[:len(idx_group) // self.condition_min_count * self.condition_min_count]
-            benefit_idx = idx_group[:len(idx_group) // self.condition_min_count * self.benefit_only_apply_to_cheapest_n_matches]
+            n_groups = min(len(condition_idx_group) // self.condition_min_count, len(benefit_idx_group))
+            consume_idx = condition_idx_group[:n_groups * self.condition_min_count]
+            benefit_idx = benefit_idx_group[:n_groups * self.benefit_only_apply_to_cheapest_n_matches]
         else:
-            consume_idx = idx_group
-            benefit_idx = idx_group
+            consume_idx = condition_idx_group
+            benefit_idx = benefit_idx_group
 
         for idx in benefit_idx:
             previous_price = positions[idx][2]
@@ -276,7 +312,7 @@ class Discount(LoggedModel):
             limit_products = {p.pk for p in self.condition_limit_products.all()}
 
         # First, filter out everything not even covered by our product scope
-        initial_candidates = [
+        condition_candidates = [
             idx
             for idx, (item_id, subevent_id, line_price_gross, is_addon_to, voucher_discount) in positions.items()
             if (
@@ -286,11 +322,25 @@ class Discount(LoggedModel):
             )
         ]
 
+        if self.benefit_same_products:
+            benefit_candidates = list(condition_candidates)
+        else:
+            benefit_products = {p.pk for p in self.benefit_limit_products.all()}
+            benefit_candidates = [
+                idx
+                for idx, (item_id, subevent_id, line_price_gross, is_addon_to, voucher_discount) in positions.items()
+                if (
+                    item_id in benefit_products and
+                    (self.benefit_apply_to_addons or not is_addon_to) and
+                    (not self.benefit_ignore_voucher_discounted or voucher_discount is None or voucher_discount == Decimal('0.00'))
+                )
+            ]
+
         if self.subevent_mode == self.SUBEVENT_MODE_MIXED:  # also applies to non-series events
             if self.condition_min_count:
-                self._apply_min_count(positions, initial_candidates, result)
+                self._apply_min_count(positions, condition_candidates, benefit_candidates, result)
             else:
-                self._apply_min_value(positions, initial_candidates, result)
+                self._apply_min_value(positions, condition_candidates, benefit_candidates, result)
 
         elif self.subevent_mode == self.SUBEVENT_MODE_SAME:
             def key(idx):
@@ -299,17 +349,18 @@ class Discount(LoggedModel):
             # Build groups of candidates with the same subevent, then apply our regular algorithm
             # to each group
 
-            _groups = groupby(sorted(initial_candidates, key=key), key=key)
-            candidate_groups = [list(g) for k, g in _groups]
+            _groups = groupby(sorted(condition_candidates, key=key), key=key)
+            candidate_groups = [(k, list(g)) for k, g in _groups]
 
-            for g in candidate_groups:
+            for subevent_id, g in candidate_groups:
+                benefit_g = [idx for idx in benefit_candidates if positions[idx][1] == subevent_id]
                 if self.condition_min_count:
-                    self._apply_min_count(positions, g, result)
+                    self._apply_min_count(positions, g, benefit_g, result)
                 else:
-                    self._apply_min_value(positions, g, result)
+                    self._apply_min_value(positions, g, benefit_g, result)
 
         elif self.subevent_mode == self.SUBEVENT_MODE_DISTINCT:
-            if self.condition_min_value:
+            if self.condition_min_value or not self.benefit_same_products:
                 raise ValueError('Validation invariant violated.')
 
             # Build optimal groups of candidates with distinct subevents, then apply our regular algorithm
@@ -336,7 +387,7 @@ class Discount(LoggedModel):
                 candidates = []
                 cardinality = None
                 for se, l in subevent_to_idx.items():
-                    l = [ll for ll in l if ll in initial_candidates and ll not in current_group]
+                    l = [ll for ll in l if ll in condition_candidates and ll not in current_group]
                     if cardinality and len(l) != cardinality:
                         continue
                     if se not in {positions[idx][1] for idx in current_group}:
@@ -373,5 +424,5 @@ class Discount(LoggedModel):
                                 break
 
             for g in candidate_groups:
-                self._apply_min_count(positions, g, result)
+                self._apply_min_count(positions, g, g, result)
         return result

src/pretix/base/models/event.py

@@ -907,14 +907,18 @@ class Event(EventMixin, LoggedModel):
             self.items.filter(hidden_if_available_id=oldid).update(hidden_if_available=q)
 
         for d in Discount.objects.filter(event=other).prefetch_related('condition_limit_products'):
-            items = list(d.condition_limit_products.all())
+            c_items = list(d.condition_limit_products.all())
+            b_items = list(d.benefit_limit_products.all())
             d.pk = None
             d.event = self
             d.save(force_insert=True)
             d.log_action('pretix.object.cloned')
-            for i in items:
+            for i in c_items:
                 if i.pk in item_map:
                     d.condition_limit_products.add(item_map[i.pk])
+            for i in b_items:
+                if i.pk in item_map:
+                    d.benefit_limit_products.add(item_map[i.pk])
 
         question_map = {}
         for q in Question.objects.filter(event=other).prefetch_related('items', 'options'):
@@ -1277,6 +1281,9 @@ class Event(EventMixin, LoggedModel):
         return not self.orders.exists() and not self.invoices.exists()
 
     def delete_sub_objects(self):
+        from .checkin import Checkin
+
+        Checkin.all.filter(successful=False, list__event=self).delete()
         self.cartposition_set.filter(addon_to__isnull=False).delete()
         self.cartposition_set.all().delete()
         self.vouchers.all().delete()

src/pretix/base/models/giftcards.py

@@ -46,14 +46,19 @@ def gen_giftcard_secret(length=8):
 class GiftCardAcceptance(models.Model):
     issuer = models.ForeignKey(
         'Organizer',
-        related_name='gift_card_collector_acceptance',
+        related_name='gift_card_acceptor_acceptance',
         on_delete=models.CASCADE
     )
-    collector = models.ForeignKey(
+    acceptor = models.ForeignKey(
         'Organizer',
         related_name='gift_card_issuer_acceptance',
         on_delete=models.CASCADE
     )
+    active = models.BooleanField(default=True)
+    reusable_media = models.BooleanField(default=False)
+
+    class Meta:
+        unique_together = (('issuer', 'acceptor'),)
 
 
 class GiftCard(LoggedModel):
@@ -114,7 +119,7 @@ class GiftCard(LoggedModel):
         return self.transactions.aggregate(s=Sum('value'))['s'] or Decimal('0.00')
 
     def accepted_by(self, organizer):
-        return self.issuer == organizer or GiftCardAcceptance.objects.filter(issuer=self.issuer, collector=organizer).exists()
+        return self.issuer == organizer or GiftCardAcceptance.objects.filter(issuer=self.issuer, acceptor=organizer, active=True).exists()
 
     def save(self, *args, **kwargs):
         if not self.secret:

src/pretix/base/models/items.py

@@ -43,6 +43,7 @@ from typing import Optional, Tuple
 from zoneinfo import ZoneInfo
 
 import dateutil.parser
+import django_redis
 from dateutil.tz import datetime_exists
 from django.conf import settings
 from django.core.exceptions import ValidationError
@@ -57,7 +58,6 @@ from django.utils.functional import cached_property
 from django.utils.timezone import is_naive, make_aware, now
 from django.utils.translation import gettext_lazy as _, pgettext_lazy
 from django_countries.fields import Country
-from django_redis import get_redis_connection
 from django_scopes import ScopedManager
 from i18nfield.fields import I18nCharField, I18nTextField
 
@@ -1910,8 +1910,13 @@ class Quota(LoggedModel):
 
     def rebuild_cache(self, now_dt=None):
         if settings.HAS_REDIS:
-            rc = get_redis_connection("redis")
-            rc.hdel(f'quotas:{self.event_id}:availabilitycache', str(self.pk))
+            rc = django_redis.get_redis_connection("redis")
+            p = rc.pipeline()
+            p.hdel(f'quotas:{self.event_id}:availabilitycache', str(self.pk))
+            p.hdel(f'quotas:{self.event_id}:availabilitycache:nocw', str(self.pk))
+            p.hdel(f'quotas:{self.event_id}:availabilitycache:igcl', str(self.pk))
+            p.hdel(f'quotas:{self.event_id}:availabilitycache:nocw:igcl', str(self.pk))
+            p.execute()
             self.availability(now_dt=now_dt)
 
     def availability(

src/pretix/base/models/log.py

@@ -88,9 +88,7 @@ class LogEntry(models.Model):
 
     class Meta:
         ordering = ('-datetime', '-id')
-        index_together = [
-            ['datetime', 'id']
-        ]
+        indexes = [models.Index(fields=["datetime", "id"])]
 
     def display(self):
         from ..signals import logentry_display

src/pretix/base/models/media.py

@@ -121,5 +121,30 @@ class ReusableMedium(LoggedModel):
 
     class Meta:
         unique_together = (("identifier", "type", "organizer"),)
-        index_together = (("identifier", "type", "organizer"), ("updated", "id"))
+        indexes = [
+            models.Index(fields=("identifier", "type", "organizer")),
+            models.Index(fields=("updated", "id")),
+        ]
         ordering = "identifier", "type", "organizer"
+
+
+class MediumKeySet(models.Model):
+    organizer = models.ForeignKey('Organizer', on_delete=models.CASCADE, related_name='medium_key_sets')
+    public_id = models.BigIntegerField(
+        unique=True,
+    )
+    media_type = models.CharField(max_length=100)
+    active = models.BooleanField(default=True)
+    uid_key = models.BinaryField()
+    diversification_key = models.BinaryField()
+
+    objects = ScopedManager(organizer='organizer')
+
+    class Meta:
+        constraints = [
+            models.UniqueConstraint(
+                fields=["organizer", "media_type"],
+                condition=Q(active=True),
+                name="keyset_unique_active",
+            ),
+        ]

src/pretix/base/models/orders.py

@@ -270,9 +270,9 @@ class Order(LockModel, LoggedModel):
         verbose_name = _("Order")
         verbose_name_plural = _("Orders")
         ordering = ("-datetime", "-pk")
-        index_together = [
-            ["datetime", "id"],
-            ["last_modified", "id"],
+        indexes = [
+            models.Index(fields=["datetime", "id"]),
+            models.Index(fields=["last_modified", "id"]),
         ]
 
     def __str__(self):
@@ -896,6 +896,33 @@ class Order(LockModel, LoggedModel):
             ), tz)
         return term_last
 
+    @property
+    def payment_term_expire_date(self):
+        delay = self.event.settings.get('payment_term_expire_delay_days', as_type=int)
+        if not delay:  # performance saver + backwards compatibility
+            return self.expires
+
+        term_last = self.payment_term_last
+        if term_last and self.expires > term_last:  # backwards compatibility
+            return self.expires
+
+        expires = self.expires.date() + timedelta(days=delay)
+        if self.event.settings.get('payment_term_weekdays'):
+            if expires.weekday() == 5:
+                expires += timedelta(days=2)
+            elif expires.weekday() == 6:
+                expires += timedelta(days=1)
+
+        tz = ZoneInfo(self.event.settings.timezone)
+        expires = make_aware(datetime.combine(
+            expires,
+            time(hour=23, minute=59, second=59)
+        ), tz)
+        if term_last:
+            return min(expires, term_last)
+        else:
+            return expires
+
     def _can_be_paid(self, count_waitinglist=True, ignore_date=False, force=False) -> Union[bool, str]:
         error_messages = {
             'late_lastdate': _("The payment can not be accepted as the last date of payments configured in the "
@@ -1645,12 +1672,13 @@ class OrderPayment(models.Model):
         if status_change:
             self.order.create_transactions()
 
-    def fail(self, info=None, user=None, auth=None, log_data=None):
+    def fail(self, info=None, user=None, auth=None, log_data=None, send_mail=True):
         """
         Marks the order as failed and sets info to ``info``, but only if the order is in ``created`` or ``pending``
         state. This is equivalent to setting ``state`` to ``OrderPayment.PAYMENT_STATE_FAILED`` and logging a failure,
-        but it adds strong database logging since we do not want to report a failure for an order that has just
+        but it adds strong database locking since we do not want to report a failure for an order that has just
         been marked as paid.
+        :param send_mail: Whether an email should be sent to the user about this event (default: ``True``).
         """
         with transaction.atomic():
             locked_instance = OrderPayment.objects.select_for_update(of=OF_SELF).get(pk=self.pk)
@@ -1675,6 +1703,17 @@ class OrderPayment(models.Model):
             'info': info,
             'data': log_data,
         }, user=user, auth=auth)
+
+        if send_mail:
+            with language(self.order.locale, self.order.event.settings.region):
+                email_subject = self.order.event.settings.mail_subject_order_payment_failed
+                email_template = self.order.event.settings.mail_text_order_payment_failed
+                email_context = get_email_context(event=self.order.event, order=self.order)
+                self.order.send_mail(
+                    email_subject, email_template, email_context,
+                    'pretix.event.order.email.payment_failed', user=user, auth=auth,
+                )
+
         return True
 
     def confirm(self, count_waitinglist=True, send_mail=True, force=False, user=None, auth=None, mail_text='',
@@ -2717,8 +2756,8 @@ class Transaction(models.Model):
 
     class Meta:
         ordering = 'datetime', 'pk'
-        index_together = [
-            ['datetime', 'id']
+        indexes = [
+            models.Index(fields=['datetime', 'id'])
         ]
 
     def save(self, *args, **kwargs):

src/pretix/base/models/organizer.py

@@ -40,7 +40,7 @@ from django.conf import settings
 from django.core.mail import get_connection
 from django.core.validators import MinLengthValidator, RegexValidator
 from django.db import models
-from django.db.models import Exists, OuterRef, Q
+from django.db.models import Q
 from django.urls import reverse
 from django.utils.crypto import get_random_string
 from django.utils.functional import cached_property
@@ -157,17 +157,19 @@ class Organizer(LoggedModel):
         return self.cache.get_or_set(
             key='has_gift_cards',
             timeout=15,
-            default=lambda: self.issued_gift_cards.exists() or self.gift_card_issuer_acceptance.exists()
+            default=lambda: self.issued_gift_cards.exists() or self.gift_card_issuer_acceptance.filter(active=True).exists()
         )
 
     @property
     def accepted_gift_cards(self):
         from .giftcards import GiftCard, GiftCardAcceptance
 
-        return GiftCard.objects.annotate(
-            accepted=Exists(GiftCardAcceptance.objects.filter(issuer=OuterRef('issuer'), collector=self))
-        ).filter(
-            Q(issuer=self) | Q(accepted=True)
+        return GiftCard.objects.filter(
+            Q(issuer=self) |
+            Q(issuer__in=GiftCardAcceptance.objects.filter(
+                acceptor=self,
+                active=True,
+            ).values_list('issuer', flat=True))
         )
 
     @property

src/pretix/base/models/tax.py

@@ -22,9 +22,12 @@
 import json
 from decimal import Decimal
 
+import jsonschema
+from django.contrib.staticfiles import finders
 from django.core.exceptions import ValidationError
 from django.core.validators import MaxValueValidator, MinValueValidator
 from django.db import models
+from django.utils.deconstruct import deconstructible
 from django.utils.formats import localize
 from django.utils.translation import gettext_lazy as _, pgettext
 from i18nfield.fields import I18nCharField
@@ -135,6 +138,25 @@ def cc_to_vat_prefix(country_code):
     return country_code
 
 
+@deconstructible
+class CustomRulesValidator:
+    def __call__(self, value):
+        if not isinstance(value, dict):
+            try:
+                val = json.loads(value)
+            except ValueError:
+                raise ValidationError(_('Your layout file is not a valid JSON file.'))
+        else:
+            val = value
+        with open(finders.find('schema/tax-rules-custom.schema.json'), 'r') as f:
+            schema = json.loads(f.read())
+        try:
+            jsonschema.validate(val, schema)
+        except jsonschema.ValidationError as e:
+            e = str(e).replace('%', '%%')
+            raise ValidationError(_('Your set of rules is not valid. Error message: {}').format(e))
+
+
 class TaxRule(LoggedModel):
     event = models.ForeignKey('Event', related_name='tax_rules', on_delete=models.CASCADE)
     internal_name = models.CharField(
@@ -318,10 +340,17 @@ class TaxRule(LoggedModel):
         rules = self._custom_rules
         if invoice_address:
             for r in rules:
-                if r['country'] == 'EU' and not is_eu_country(invoice_address.country):
-                    continue
-                if r['country'] not in ('ZZ', 'EU') and r['country'] != str(invoice_address.country):
-                    continue
+                if r['country'] == 'ZZ':    # Rule: Any country
+                    pass
+                elif r['country'] == 'EU':  # Rule: Any EU country
+                    if not is_eu_country(invoice_address.country):
+                        continue
+                elif '-' in r['country']:   # Rule: Specific country and state
+                    if r['country'] != str(invoice_address.country) + '-' + str(invoice_address.state):
+                        continue
+                else:                       # Rule: Specific country
+                    if r['country'] != str(invoice_address.country):
+                        continue
                 if r['address_type'] == 'individual' and invoice_address.is_business:
                     continue
                 if r['address_type'] in ('business', 'business_vat_id') and not invoice_address.is_business:

src/pretix/base/models/waitinglist.py

@@ -218,7 +218,7 @@ class WaitingListEntry(LoggedModel):
                 'waitinglistentry': self.pk,
                 'subevent': self.subevent.pk if self.subevent else None,
             }, user=user, auth=auth)
-            self.log_action('pretix.waitinglist.voucher', user=user, auth=auth)
+            self.log_action('pretix.event.orders.waitinglist.voucher_assigned', user=user, auth=auth)
             self.voucher = v
             self.save()
 

src/pretix/base/orderimport.py

@@ -28,6 +28,7 @@ import pycountry
 from django.conf import settings
 from django.core.exceptions import ValidationError
 from django.core.validators import EmailValidator
+from django.db.models import Q
 from django.utils import formats
 from django.utils.functional import cached_property
 from django.utils.translation import (
@@ -42,8 +43,8 @@ from phonenumbers import SUPPORTED_REGIONS
 from pretix.base.channels import get_all_sales_channels
 from pretix.base.forms.questions import guess_country
 from pretix.base.models import (
-    ItemVariation, OrderPosition, Question, QuestionAnswer, QuestionOption,
-    Seat, SubEvent,
+    Customer, ItemVariation, OrderPosition, Question, QuestionAnswer,
+    QuestionOption, Seat, SubEvent,
 )
 from pretix.base.services.pricing import get_price
 from pretix.base.settings import (
@@ -804,7 +805,7 @@ class QuestionColumn(ImportColumn):
                 return self.q.clean_answer(value)
 
     def assign(self, value, order, position, invoice_address, **kwargs):
-        if value:
+        if value is not None:
             if not hasattr(order, '_answers'):
                 order._answers = []
             if isinstance(value, QuestionOption):
@@ -826,6 +827,28 @@ class QuestionColumn(ImportColumn):
                 a.options.add(*a._options)
 
 
+class CustomerColumn(ImportColumn):
+    identifier = 'customer'
+    verbose_name = gettext_lazy('Customer')
+
+    def clean(self, value, previous_values):
+        if value:
+            try:
+                value = self.event.organizer.customers.get(
+                    Q(identifier=value) | Q(email__iexact=value) | Q(external_identifier=value)
+                )
+            except Customer.MultipleObjectsReturned:
+                value = self.event.organizer.customers.get(
+                    Q(identifier=value)
+                )
+            except Customer.DoesNotExist:
+                raise ValidationError(_('No matching customer was found.'))
+        return value
+
+    def assign(self, value, order, position, invoice_address, **kwargs):
+        order.customer = value
+
+
 def get_all_columns(event):
     default = []
     if event.has_subevents:
@@ -837,6 +860,10 @@ def get_all_columns(event):
         Variation(event),
         InvoiceAddressCompany(event),
     ]
+    if event.settings.customer_accounts:
+        default += [
+            CustomerColumn(event),
+        ]
     scheme = PERSON_NAME_SCHEMES.get(event.settings.name_scheme)
     for n, l, w in scheme['fields']:
         default.append(InvoiceAddressNamePart(event, n, l))

src/pretix/base/payment.py

@@ -60,7 +60,7 @@ from pretix.base.channels import get_all_sales_channels
 from pretix.base.forms import PlaceholderValidator
 from pretix.base.models import (
     CartPosition, Event, GiftCard, InvoiceAddress, Order, OrderPayment,
-    OrderRefund, Quota,
+    OrderRefund, Quota, TaxRule,
 )
 from pretix.base.reldate import RelativeDateField, RelativeDateWrapper
 from pretix.base.settings import SettingsSandbox
@@ -78,6 +78,16 @@ from pretix.presale.views.cart import cart_session, get_or_create_cart_id
 logger = logging.getLogger(__name__)
 
 
+class WalletQueries:
+    APPLEPAY = 'applepay'
+    GOOGLEPAY = 'googlepay'
+
+    WALLETS = (
+        (APPLEPAY, pgettext_lazy('payment', 'Apple Pay')),
+        (GOOGLEPAY, pgettext_lazy('payment', 'Google Pay')),
+    )
+
+
 class PaymentProviderForm(Form):
     def clean(self):
         cleaned_data = super().clean()
@@ -436,6 +446,19 @@ class BasePaymentProvider:
         d['_restrict_to_sales_channels']._as_type = list
         return d
 
+    @property
+    def walletqueries(self):
+        """
+        .. warning:: This property is considered **experimental**. It might change or get removed at any time without
+                     prior notice.
+
+        A list of wallet payment methods that should be dynamically joined to the public name of the payment method,
+        if they are available to the user.
+        The detection is made on a best effort basis with no guarantees of correctness and actual availability.
+        Wallets that pretix can check for are exposed through ``pretix.base.payment.WalletQueries``.
+        """
+        return []
+
     def settings_form_clean(self, cleaned_data):
         """
         Overriding this method allows you to inject custom validation into the settings form.
@@ -1015,7 +1038,11 @@ class FreeOrderProvider(BasePaymentProvider):
 
         cart = get_cart(request)
         total = get_cart_total(request)
-        total += sum([f.value for f in get_fees(self.event, request, total, None, None, cart)])
+        try:
+            total += sum([f.value for f in get_fees(self.event, request, total, None, None, cart)])
+        except TaxRule.SaleNotAllowed:
+            # ignore for now, will fail on order creation
+            pass
         return total == 0
 
     def order_change_allowed(self, order: Order) -> bool:

src/pretix/base/pdf.py

@@ -43,16 +43,18 @@ import subprocess
 import tempfile
 import unicodedata
 import uuid
-from collections import OrderedDict
+from collections import OrderedDict, defaultdict
 from functools import partial
 from io import BytesIO
 
 import jsonschema
+import reportlab.rl_config
 from bidi.algorithm import get_display
 from django.conf import settings
 from django.contrib.staticfiles import finders
 from django.core.exceptions import ValidationError
 from django.db.models import Max, Min
+from django.db.models.fields.files import FieldFile
 from django.dispatch import receiver
 from django.utils.deconstruct import deconstructible
 from django.utils.formats import date_format
@@ -60,7 +62,8 @@ from django.utils.html import conditional_escape
 from django.utils.timezone import now
 from django.utils.translation import gettext_lazy as _, pgettext
 from i18nfield.strings import LazyI18nString
-from pypdf import PdfReader
+from pypdf import PdfReader, PdfWriter, Transformation
+from pypdf.generic import RectangleObject
 from reportlab.graphics import renderPDF
 from reportlab.graphics.barcode.qr import QrCodeWidget
 from reportlab.graphics.shapes import Drawing
@@ -85,6 +88,9 @@ from pretix.presale.style import get_fonts
 
 logger = logging.getLogger(__name__)
 
+if not settings.DEBUG:
+    reportlab.rl_config.shapeChecking = 0
+
 
 DEFAULT_VARIABLES = OrderedDict((
     ("secret", {
@@ -102,7 +108,10 @@ DEFAULT_VARIABLES = OrderedDict((
     ("positionid", {
         "label": _("Order position number"),
         "editor_sample": "1",
-        "evaluate": lambda orderposition, order, event: str(orderposition.positionid)
+        "evaluate": lambda orderposition, order, event: str(orderposition.positionid),
+        # There is no performance gain in using evaluate_bulk here, but we want to make sure it is used somewhere
+        # in core to make sure we notice if the implementation of the API breaks.
+        "evaluate_bulk": lambda orderpositions: [str(p.positionid) for p in orderpositions],
     }),
     ("order_positionid", {
         "label": _("Order code and position number"),
@@ -355,14 +364,9 @@ DEFAULT_VARIABLES = OrderedDict((
     }),
     ("addons", {
         "label": _("List of Add-Ons"),
-        "editor_sample": _("Add-on 1\nAdd-on 2"),
+        "editor_sample": _("Add-on 1\n2x Add-on 2"),
         "evaluate": lambda op, order, ev: "\n".join([
-            '{} - {}'.format(p.item.name, p.variation.value) if p.variation else str(p.item.name)
-            for p in (
-                op.addons.all() if 'addons' in getattr(op, '_prefetched_objects_cache', {})
-                else op.addons.select_related('item', 'variation')
-            )
-            if not p.canceled
+            str(p) for p in generate_compressed_addon_list(op, order, ev)
         ])
     }),
     ("organizer", {
@@ -696,6 +700,30 @@ def get_seat(op: OrderPosition):
     return None
 
 
+def generate_compressed_addon_list(op, order, event):
+    itemcount = defaultdict(int)
+    addons = [p for p in (
+        op.addons.all() if 'addons' in getattr(op, '_prefetched_objects_cache', {})
+        else op.addons.select_related('item', 'variation')
+    ) if not p.canceled]
+    for pos in addons:
+        itemcount[pos.item, pos.variation] += 1
+
+    addonlist = []
+    for (item, variation), count in itemcount.items():
+        if variation:
+            if count > 1:
+                addonlist.append('{}x {} - {}'.format(count, item.name, variation.value))
+            else:
+                addonlist.append('{} - {}'.format(item.name, variation.value))
+        else:
+            if count > 1:
+                addonlist.append('{}x {}'.format(count, item.name))
+            else:
+                addonlist.append(item.name)
+    return addonlist
+
+
 class Renderer:
 
     def __init__(self, event, layout, background_file):
@@ -860,22 +888,37 @@ class Renderer:
                 image_file = None
 
         if image_file:
-            ir = ThumbnailingImageReader(image_file)
             try:
+                ir = ThumbnailingImageReader(image_file)
                 ir.resize(float(o['width']) * mm, float(o['height']) * mm, 300)
+                canvas.drawImage(
+                    image=ir,
+                    x=float(o['left']) * mm,
+                    y=float(o['bottom']) * mm,
+                    width=float(o['width']) * mm,
+                    height=float(o['height']) * mm,
+                    preserveAspectRatio=True,
+                    anchor='c',  # centered in frame
+                    mask='auto'
+                )
+                if isinstance(image_file, FieldFile):
+                    # ThumbnailingImageReader "closes" the file, so it's no use to use the same file pointer
+                    # in case we need it again. For FieldFile, fortunately, there is an easy way to make the file
+                    # refresh itself when it is used next.
+                    del image_file.file
             except:
-                logger.exception("Can not resize image")
-                pass
-            canvas.drawImage(
-                image=ir,
-                x=float(o['left']) * mm,
-                y=float(o['bottom']) * mm,
-                width=float(o['width']) * mm,
-                height=float(o['height']) * mm,
-                preserveAspectRatio=True,
-                anchor='c',  # centered in frame
-                mask='auto'
-            )
+                logger.exception("Can not load or resize image")
+                canvas.saveState()
+                canvas.setFillColorRGB(.8, .8, .8, alpha=1)
+                canvas.rect(
+                    x=float(o['left']) * mm,
+                    y=float(o['bottom']) * mm,
+                    width=float(o['width']) * mm,
+                    height=float(o['height']) * mm,
+                    stroke=0,
+                    fill=1,
+                )
+                canvas.restoreState()
         else:
             canvas.saveState()
             canvas.setFillColorRGB(.8, .8, .8, alpha=1)
@@ -929,7 +972,7 @@ class Renderer:
 
         # reportlab does not support unicode combination characters
         # It's important we do this before we use ArabicReshaper
-        text = unicodedata.normalize("NFKC", text)
+        text = unicodedata.normalize("NFC", text)
 
         # reportlab does not support RTL, ligature-heavy scripts like Arabic. Therefore, we use ArabicReshaper
         # to resolve all ligatures and python-bidi to switch RTL texts.
@@ -982,7 +1025,10 @@ class Renderer:
                 elif o['type'] == "poweredby":
                     self._draw_poweredby(canvas, op, o)
                 if self.bg_pdf:
-                    page_size = (self.bg_pdf.pages[0].mediabox[2], self.bg_pdf.pages[0].mediabox[3])
+                    page_size = (
+                        self.bg_pdf.pages[0].mediabox[2] - self.bg_pdf.pages[0].mediabox[0],
+                        self.bg_pdf.pages[0].mediabox[3] - self.bg_pdf.pages[0].mediabox[1]
+                    )
                     if self.bg_pdf.pages[0].get('/Rotate') in (90, 270):
                         # swap dimensions due to pdf being rotated
                         page_size = page_size[::-1]
@@ -1010,14 +1056,12 @@ class Renderer:
                 with open(os.path.join(d, 'out.pdf'), 'rb') as f:
                     return BytesIO(f.read())
         else:
-            from pypdf import PdfReader, PdfWriter, Transformation
-            from pypdf.generic import RectangleObject
             buffer.seek(0)
             new_pdf = PdfReader(buffer)
             output = PdfWriter()
 
             for i, page in enumerate(new_pdf.pages):
-                bg_page = copy.copy(self.bg_pdf.pages[i])
+                bg_page = copy.deepcopy(self.bg_pdf.pages[i])
                 bg_rotation = bg_page.get('/Rotate')
                 if bg_rotation:
                     # /Rotate is clockwise, transformation.rotate is counter-clockwise
@@ -1054,6 +1098,56 @@ class Renderer:
             return outbuffer
 
 
+def merge_background(fg_pdf, bg_pdf, out_file, compress):
+    if settings.PDFTK:
+        with tempfile.TemporaryDirectory() as d:
+            fg_filename = os.path.join(d, 'fg.pdf')
+            bg_filename = os.path.join(d, 'bg.pdf')
+            fg_pdf.write(fg_filename)
+            bg_pdf.write(bg_filename)
+            pdftk_cmd = [
+                settings.PDFTK,
+                fg_filename,
+                'multibackground',
+                bg_filename,
+                'output',
+                '-',
+            ]
+            if compress:
+                pdftk_cmd.append('compress')
+            subprocess.run(pdftk_cmd, check=True, stdout=out_file)
+    else:
+        output = PdfWriter()
+        for i, page in enumerate(fg_pdf.pages):
+            bg_page = copy.deepcopy(bg_pdf.pages[i])
+            bg_rotation = bg_page.get('/Rotate')
+            if bg_rotation:
+                # /Rotate is clockwise, transformation.rotate is counter-clockwise
+                t = Transformation().rotate(bg_rotation)
+                w = float(page.mediabox.getWidth())
+                h = float(page.mediabox.getHeight())
+                if bg_rotation in (90, 270):
+                    # offset due to rotation base
+                    if bg_rotation == 90:
+                        t = t.translate(h, 0)
+                    else:
+                        t = t.translate(0, w)
+                    # rotate mediabox as well
+                    page.mediabox = RectangleObject((
+                        page.mediabox.left.as_numeric(),
+                        page.mediabox.bottom.as_numeric(),
+                        page.mediabox.top.as_numeric(),
+                        page.mediabox.right.as_numeric(),
+                    ))
+                    page.trimbox = page.mediabox
+                elif bg_rotation == 180:
+                    t = t.translate(w, h)
+                page.add_transformation(t)
+            bg_page.merge_page(page)
+            output.add_page(bg_page)
+        output.write(out_file)
+
+
 @deconstructible
 class PdfLayoutValidator:
     def __call__(self, value):

src/pretix/base/services/cart.py

@@ -141,9 +141,10 @@ error_messages = {
     'price_not_a_number': gettext_lazy('The entered price is not a number.'),
     'price_too_high': gettext_lazy('The entered price is to high.'),
     'voucher_invalid': gettext_lazy('This voucher code is not known in our database.'),
-    'voucher_min_usages': gettext_lazy(
-        'The voucher code "%(voucher)s" can only be used if you select at least %(number)s '
-        'matching products.'
+    'voucher_min_usages': ngettext_lazy(
+        'The voucher code "%(voucher)s" can only be used if you select at least %(number)s matching products.',
+        'The voucher code "%(voucher)s" can only be used if you select at least %(number)s matching products.',
+        'number'
     ),
     'voucher_min_usages_removed': ngettext_lazy(
         'The voucher code "%(voucher)s" can only be used if you select at least %(number)s matching products. '
@@ -317,6 +318,9 @@ class CartManager:
     def _delete_out_of_timeframe(self):
         err = None
         for cp in self.positions:
+            if not cp.pk:
+                continue
+
             if cp.subevent and cp.subevent.presale_start and self.now_dt < cp.subevent.presale_start:
                 err = error_messages['some_subevent_not_started']
                 cp.addons.all().delete()
@@ -1074,6 +1078,7 @@ class CartManager:
         quotas_ok = _get_quota_availability(self._quota_diff, self.now_dt)
         err = None
         new_cart_positions = []
+        deleted_positions = set()
 
         err = err or self._check_min_max_per_product()
 
@@ -1085,7 +1090,10 @@ class CartManager:
                 if op.position.expires > self.now_dt:
                     for q in op.position.quotas:
                         quotas_ok[q] += 1
-                op.position.addons.all().delete()
+                addons = op.position.addons.all()
+                deleted_positions |= {a.pk for a in addons}
+                addons.delete()
+                deleted_positions.add(op.position.pk)
                 op.position.delete()
 
             elif isinstance(op, (self.AddOperation, self.ExtendOperation)):
@@ -1235,20 +1243,28 @@ class CartManager:
                     if op.seat and not op.seat.is_available(ignore_cart=op.position, sales_channel=self._sales_channel,
                                                             ignore_voucher_id=op.position.voucher_id):
                         err = err or error_messages['seat_unavailable']
-                        op.position.addons.all().delete()
+
+                        addons = op.position.addons.all()
+                        deleted_positions |= {a.pk for a in addons}
+                        deleted_positions.add(op.position.pk)
+                        addons.delete()
                         op.position.delete()
                     elif available_count == 1:
                         op.position.expires = self._expiry
                         op.position.listed_price = op.listed_price
                         op.position.price_after_voucher = op.price_after_voucher
                         # op.position.price will be updated by recompute_final_prices_and_taxes()
-                        try:
-                            op.position.save(force_update=True, update_fields=['expires', 'listed_price', 'price_after_voucher'])
-                        except DatabaseError:
-                            # Best effort... The position might have been deleted in the meantime!
-                            pass
+                        if op.position.pk not in deleted_positions:
+                            try:
+                                op.position.save(force_update=True, update_fields=['expires', 'listed_price', 'price_after_voucher'])
+                            except DatabaseError:
+                                # Best effort... The position might have been deleted in the meantime!
+                                pass
                     elif available_count == 0:
-                        op.position.addons.all().delete()
+                        addons = op.position.addons.all()
+                        deleted_positions |= {a.pk for a in addons}
+                        deleted_positions.add(op.position.pk)
+                        addons.delete()
                         op.position.delete()
                     else:
                         raise AssertionError("ExtendOperation cannot affect more than one item")

src/pretix/base/services/checkin.py

@@ -53,7 +53,8 @@ from django.utils.translation import gettext as _
 from django_scopes import scope, scopes_disabled
 
 from pretix.base.models import (
-    Checkin, CheckinList, Device, Order, OrderPosition, QuestionOption,
+    Checkin, CheckinList, Device, Event, ItemVariation, Order, OrderPosition,
+    QuestionOption,
 )
 from pretix.base.signals import checkin_created, order_placed, periodic_task
 from pretix.helpers import OF_SELF
@@ -65,12 +66,13 @@ from pretix.helpers.jsonlogic_query import (
 )
 
 
-def _build_time(t=None, value=None, ev=None):
+def _build_time(t=None, value=None, ev=None, now_dt=None):
+    now_dt = now_dt or now()
     if t == "custom":
         return dateutil.parser.parse(value)
     elif t == "customtime":
         parsed = dateutil.parser.parse(value)
-        return now().astimezone(ev.timezone).replace(
+        return now_dt.astimezone(ev.timezone).replace(
             hour=parsed.hour,
             minute=parsed.minute,
             second=parsed.second,
@@ -84,7 +86,42 @@ def _build_time(t=None, value=None, ev=None):
         return ev.date_admission or ev.date_from
 
 
-def _logic_explain(rules, ev, rule_data):
+def _logic_annotate_for_graphic_explain(rules, ev, rule_data, now_dt):
+    logic_environment = _get_logic_environment(ev, now_dt)
+    event = ev if isinstance(ev, Event) else ev.event
+
+    def _evaluate_inners(r):
+        if not isinstance(r, dict):
+            return r
+        operator = list(r.keys())[0]
+        values = r[operator]
+        if operator in ("and", "or"):
+            return {operator: [_evaluate_inners(v) for v in values]}
+        result = logic_environment.apply(r, rule_data)
+        return {**r, '__result': result}
+
+    def _add_var_values(r):
+        if not isinstance(r, dict):
+            return r
+        operator = [k for k in r.keys() if not k.startswith("__")][0]
+        values = r[operator]
+        if operator == "var":
+            var = values[0] if isinstance(values, list) else values
+            val = rule_data[var]
+            if var == "product":
+                val = str(event.items.get(pk=val))
+            elif var == "variation":
+                val = str(ItemVariation.objects.get(item__event=event, pk=val))
+            elif isinstance(val, datetime):
+                val = date_format(val.astimezone(ev.timezone), "SHORT_DATETIME_FORMAT")
+            return {"var": var, "__result": val}
+        else:
+            return {**r, operator: [_add_var_values(v) for v in values]}
+
+    return _add_var_values(_evaluate_inners(rules))
+
+
+def _logic_explain(rules, ev, rule_data, now_dt=None):
     """
     Explains when the logic denied the check-in. Only works for a denied check-in.
 
@@ -114,7 +151,8 @@ def _logic_explain(rules, ev, rule_data):
     Additionally, we favor a "close failure". Therefore, in the above example, we'd show "You can only
     get in before 17:00". In the middle of the night it would switch to "You can only get in after 09:00".
     """
-    logic_environment = _get_logic_environment(ev)
+    now_dt = now_dt or now()
+    logic_environment = _get_logic_environment(ev, now_dt)
     _var_values = {'False': False, 'True': True}
     _var_explanations = {}
 
@@ -191,16 +229,16 @@ def _logic_explain(rules, ev, rule_data):
     for vname, data in _var_explanations.items():
         var, operator, rhs = data['var'], data['operator'], data['rhs']
         if var == 'now':
-            compare_to = _build_time(*rhs[0]['buildTime'], ev=ev).astimezone(ev.timezone)
+            compare_to = _build_time(*rhs[0]['buildTime'], ev=ev, now_dt=now_dt).astimezone(ev.timezone)
             tolerance = timedelta(minutes=float(rhs[1])) if len(rhs) > 1 and rhs[1] else timedelta(seconds=0)
             if operator == 'isBefore':
                 compare_to += tolerance
             else:
                 compare_to -= tolerance
 
-            var_weights[vname] = (200, abs(now() - compare_to).total_seconds())
+            var_weights[vname] = (200, abs(now_dt - compare_to).total_seconds())
 
-            if abs(now() - compare_to) < timedelta(hours=12):
+            if abs(now_dt - compare_to) < timedelta(hours=12):
                 compare_to_text = date_format(compare_to, 'TIME_FORMAT')
             else:
                 compare_to_text = date_format(compare_to, 'SHORT_DATETIME_FORMAT')
@@ -299,7 +337,7 @@ def _logic_explain(rules, ev, rule_data):
     return ', '.join(var_texts[v] for v in paths_with_min_weight[0] if not _var_values[v])
 
 
-def _get_logic_environment(ev):
+def _get_logic_environment(ev, now_dt):
     # Every change to our supported JSON logic must be done
     # * in pretix.base.services.checkin
     # * in pretix.base.models.checkin
@@ -316,7 +354,7 @@ def _get_logic_environment(ev):
     logic.add_operation('objectList', lambda *objs: list(objs))
     logic.add_operation('lookup', lambda model, pk, str: int(pk))
     logic.add_operation('inList', lambda a, b: a in b)
-    logic.add_operation('buildTime', partial(_build_time, ev=ev))
+    logic.add_operation('buildTime', partial(_build_time, ev=ev, now_dt=now_dt))
     logic.add_operation('isBefore', is_before)
     logic.add_operation('isAfter', lambda t1, t2, tol=None: is_before(t2, t1, tol))
     return logic
@@ -357,7 +395,7 @@ class LazyRuleVars:
     @cached_property
     def entries_today(self):
         tz = self._clist.event.timezone
-        midnight = now().astimezone(tz).replace(hour=0, minute=0, second=0, microsecond=0)
+        midnight = self._dt.astimezone(tz).replace(hour=0, minute=0, second=0, microsecond=0)
         return self._position.checkins.filter(type=Checkin.TYPE_ENTRY, list=self._clist, datetime__gte=midnight).count()
 
     @cached_property
@@ -378,7 +416,7 @@ class LazyRuleVars:
                 # between platforms (None<1 is true on some, but not all), we rather choose something that is at least
                 # consistent.
                 return -1
-            return (now() - last_entry.datetime).total_seconds() // 60
+            return (self._dt - last_entry.datetime).total_seconds() // 60
 
     @cached_property
     def minutes_since_first_entry(self):
@@ -390,7 +428,7 @@ class LazyRuleVars:
                 # between platforms (None<1 is true on some, but not all), we rather choose something that is at least
                 # consistent.
                 return -1
-            return (now() - last_entry.datetime).total_seconds() // 60
+            return (self._dt - last_entry.datetime).total_seconds() // 60
 
 
 class SQLLogic:
@@ -693,7 +731,7 @@ def _save_answers(op, answers, given_answers):
 def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict, force=False,
                     ignore_unpaid=False, nonce=None, datetime=None, questions_supported=True,
                     user=None, auth=None, canceled_supported=False, type=Checkin.TYPE_ENTRY,
-                    raw_barcode=None, raw_source_type=None, from_revoked_secret=False):
+                    raw_barcode=None, raw_source_type=None, from_revoked_secret=False, simulate=False):
     """
     Create a checkin for this particular order position and check-in list. Fails with CheckInError if the check in is
     not valid at this time.
@@ -707,6 +745,7 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
     :param questions_supported: When set to False, questions are ignored
     :param nonce: A random nonce to prevent race conditions.
     :param datetime: The datetime of the checkin, defaults to now.
+    :param simulate: If true, the check-in is not saved.
     """
 
     # !!!!!!!!!
@@ -734,7 +773,7 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
                 'blocked'
             )
 
-    if type != Checkin.TYPE_EXIT and op.valid_from and op.valid_from > now():
+    if type != Checkin.TYPE_EXIT and op.valid_from and op.valid_from > dt:
         if force:
             force_used = True
         else:
@@ -748,7 +787,7 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
                 ),
             )
 
-    if type != Checkin.TYPE_EXIT and op.valid_until and op.valid_until < now():
+    if type != Checkin.TYPE_EXIT and op.valid_until and op.valid_until < dt:
         if force:
             force_used = True
         else:
@@ -773,7 +812,8 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
             if q not in given_answers and q not in answers:
                 require_answers.append(q)
 
-        _save_answers(op, answers, given_answers)
+        if not simulate:
+            _save_answers(op, answers, given_answers)
 
     with transaction.atomic():
         # Lock order positions, if it is an entry. We don't need it for exits, as a race condition wouldn't be problematic
@@ -821,7 +861,7 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
 
         if type == Checkin.TYPE_ENTRY and clist.rules:
             rule_data = LazyRuleVars(op, clist, dt)
-            logic = _get_logic_environment(op.subevent or clist.event)
+            logic = _get_logic_environment(op.subevent or clist.event, now_dt=dt)
             if not logic.apply(clist.rules, rule_data):
                 if force:
                     force_used = True
@@ -846,7 +886,7 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
         if isinstance(auth, Device):
             device = auth
 
-        last_cis = list(op.checkins.order_by('-datetime').filter(list=clist).only('type', 'nonce'))
+        last_cis = list(op.checkins.order_by('-datetime').filter(list=clist).only('type', 'nonce', 'position_id'))
         entry_allowed = (
             type == Checkin.TYPE_EXIT or
             clist.allow_multiple_entries or
@@ -859,30 +899,33 @@ def perform_checkin(op: OrderPosition, clist: CheckinList, given_answers: dict,
             return
 
         if entry_allowed or force:
-            ci = Checkin.objects.create(
-                position=op,
-                type=type,
-                list=clist,
-                datetime=dt,
-                device=device,
-                gate=device.gate if device else None,
-                nonce=nonce,
-                forced=force and (not entry_allowed or from_revoked_secret or force_used),
-                force_sent=force,
-                raw_barcode=raw_barcode,
-                raw_source_type=raw_source_type,
-            )
-            op.order.log_action('pretix.event.checkin', data={
-                'position': op.id,
-                'positionid': op.positionid,
-                'first': True,
-                'forced': force or op.order.status != Order.STATUS_PAID,
-                'datetime': dt,
-                'type': type,
-                'answers': {k.pk: str(v) for k, v in given_answers.items()},
-                'list': clist.pk
-            }, user=user, auth=auth)
-            checkin_created.send(op.order.event, checkin=ci)
+            if simulate:
+                return True
+            else:
+                ci = Checkin.objects.create(
+                    position=op,
+                    type=type,
+                    list=clist,
+                    datetime=dt,
+                    device=device,
+                    gate=device.gate if device else None,
+                    nonce=nonce,
+                    forced=force and (not entry_allowed or from_revoked_secret or force_used),
+                    force_sent=force,
+                    raw_barcode=raw_barcode,
+                    raw_source_type=raw_source_type,
+                )
+                op.order.log_action('pretix.event.checkin', data={
+                    'position': op.id,
+                    'positionid': op.positionid,
+                    'first': True,
+                    'forced': force or op.order.status != Order.STATUS_PAID,
+                    'datetime': dt,
+                    'type': type,
+                    'answers': {k.pk: str(v) for k, v in given_answers.items()},
+                    'list': clist.pk
+                }, user=user, auth=auth)
+                checkin_created.send(op.order.event, checkin=ci)
         else:
             raise CheckInError(
                 _('This ticket has already been redeemed.'),

src/pretix/base/services/export.py

@@ -26,7 +26,7 @@ from typing import Any, Dict, Union
 from celery.exceptions import MaxRetriesExceededError
 from django.conf import settings
 from django.core.files.base import ContentFile
-from django.db import connection, transaction
+from django.db import close_old_connections, connection, transaction
 from django.dispatch import receiver
 from django.utils.timezone import now, override
 from django.utils.translation import gettext
@@ -86,9 +86,12 @@ def export(self, event: Event, fileid: str, provider: str, form_data: Dict[str,
                         gettext('Your export did not contain any data.')
                     )
                 file.filename, file.type, data = d
+
+                close_old_connections()  # This task can run very long, we might need a new DB connection
+
                 f = ContentFile(data)
                 file.file.save(cachedfile_name(file, file.filename), f)
-    return file.pk
+    return str(file.pk)
 
 
 @app.task(base=ProfiledOrganizerUserTask, throws=(ExportError,), bind=True)
@@ -154,9 +157,12 @@ def multiexport(self, organizer: Organizer, user: User, device: int, token: int,
                         gettext('Your export did not contain any data.')
                     )
                 file.filename, file.type, data = d
+
+                close_old_connections()  # This task can run very long, we might need a new DB connection
+
                 f = ContentFile(data)
                 file.file.save(cachedfile_name(file, file.filename), f)
-    return file.pk
+    return str(file.pk)
 
 
 def _run_scheduled_export(schedule, context: Union[Event, Organizer], exporter, config_url, retry_func, has_permission):
@@ -214,6 +220,11 @@ def _run_scheduled_export(schedule, context: Union[Event, Organizer], exporter,
                 raise ExportError(
                     gettext('Your exported data exceeded the size limit for scheduled exports.')
                 )
+
+            conn = transaction.get_connection()
+            if not conn.in_atomic_block:  # atomic execution only happens during tests or with celery always_eager on
+                close_old_connections()  # This task can run very long, we might need a new DB connection
+
             f = ContentFile(data)
             file.file.save(cachedfile_name(file, file.filename), f)
         except ExportEmptyError as e:
@@ -290,6 +301,8 @@ def scheduled_organizer_export(self, organizer: Organizer, schedule: int) -> Non
     if isinstance(exporter, OrganizerLevelExportMixin):
         if not schedule.owner.has_organizer_permission(organizer, exporter.organizer_required_permission):
             has_permission = False
+    if exporter and not exporter.available_for_user(schedule.owner):
+        has_permission = False
 
     _run_scheduled_export(
         schedule,

src/pretix/base/services/media.py

@@ -0,0 +1,72 @@
+#
+# This file is part of pretix (Community Edition).
+#
+# Copyright (C) 2014-2020 Raphael Michel and contributors
+# Copyright (C) 2020-2021 rami.io GmbH and contributors
+#
+# This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General
+# Public License as published by the Free Software Foundation in version 3 of the License.
+#
+# ADDITIONAL TERMS APPLY: Pursuant to Section 7 of the GNU Affero General Public License, additional terms are
+# applicable granting you additional permissions and placing additional restrictions on your usage of this software.
+# Please refer to the pretix LICENSE file to obtain the full terms applicable to this work. If you did not receive
+# this file, see <https://pretix.eu/about/en/license>.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for more
+# details.
+#
+# You should have received a copy of the GNU Affero General Public License along with this program.  If not, see
+# <https://www.gnu.org/licenses/>.
+#
+import secrets
+
+from django.db import IntegrityError
+from django.db.models import Q
+from django_scopes import scopes_disabled
+
+from pretix.base.models import GiftCardAcceptance
+from pretix.base.models.media import MediumKeySet
+
+
+def create_nfc_mf0aes_keyset(organizer):
+    for i in range(20):
+        public_id = secrets.randbelow(2 ** 32)
+        uid_key = secrets.token_bytes(16)
+        diversification_key = secrets.token_bytes(16)
+        try:
+            return MediumKeySet.objects.create(
+                organizer=organizer,
+                media_type="nfc_mf0aes",
+                public_id=public_id,
+                diversification_key=diversification_key,
+                uid_key=uid_key,
+                active=True,
+            )
+        except IntegrityError:  # either race condition with another thread or duplicate public ID
+            try:
+                return MediumKeySet.objects.get(
+                    organizer=organizer,
+                    media_type="nfc_mf0aes",
+                    active=True,
+                )
+            except MediumKeySet.DoesNotExist:
+                continue  # duplicate public ID, let's try again
+
+
+@scopes_disabled()
+def get_keysets_for_organizer(organizer):
+    sets = list(MediumKeySet.objects.filter(
+        Q(organizer=organizer) | Q(organizer__in=GiftCardAcceptance.objects.filter(
+            acceptor=organizer,
+            active=True,
+            reusable_media=True,
+        ).values_list("issuer_id", flat=True))
+    ))
+    if organizer.settings.reusable_media_type_nfc_mf0aes and not any(
+        ks.organizer == organizer and ks.media_type == "nfc_mf0aes" for ks in sets
+    ):
+        new_set = create_nfc_mf0aes_keyset(organizer)
+        if new_set:
+            sets.append(new_set)
+    return sets

src/pretix/base/services/orders.py

@@ -935,7 +935,10 @@ def _create_order(event: Event, email: str, positions: List[CartPosition], now_d
             raise OrderError(e.message)
 
         require_approval = any(p.requires_approval(invoice_address=address) for p in positions)
-        fees = _get_fees(positions, payment_requests, address, meta_info, event, require_approval=require_approval)
+        try:
+            fees = _get_fees(positions, payment_requests, address, meta_info, event, require_approval=require_approval)
+        except TaxRule.SaleNotAllowed:
+            raise OrderError(error_messages['country_blocked'])
         total = pending_sum = sum([c.price for c in positions]) + sum([c.value for c in fees])
 
         order = Order(
@@ -968,7 +971,10 @@ def _create_order(event: Event, email: str, positions: List[CartPosition], now_d
 
         for fee in fees:
             fee.order = order
-            fee._calculate_tax()
+            try:
+                fee._calculate_tax()
+            except TaxRule.SaleNotAllowed:
+                raise OrderError(error_messages['country_blocked'])
             if fee.tax_rule and not fee.tax_rule.pk:
                 fee.tax_rule = None  # TODO: deprecate
             fee.save()
@@ -1264,12 +1270,12 @@ def expire_orders(sender, **kwargs):
         Exists(
             OrderFee.objects.filter(order_id=OuterRef('pk'), fee_type=OrderFee.FEE_TYPE_CANCELLATION)
         )
-    ).select_related('event').order_by('event_id')
+    ).prefetch_related('event').order_by('event_id')
     for o in qs:
         if o.event_id != event_id:
             expire = o.event.settings.get('payment_term_expire_automatically', as_type=bool)
             event_id = o.event_id
-        if expire:
+        if expire and now() >= o.payment_term_expire_date:
             mark_order_expired(o)
 
 
@@ -2470,6 +2476,11 @@ class OrderChangeManager:
             split_order.status = Order.STATUS_PAID
         else:
             split_order.status = Order.STATUS_PENDING
+            if self.order.status == Order.STATUS_PAID:
+                split_order.set_expires(
+                    now(),
+                    list(set(p.subevent_id for p in split_positions))
+                )
         split_order.save()
 
         if offset_amount > Decimal('0.00'):

src/pretix/base/services/pricing.py

@@ -171,7 +171,7 @@ def apply_discounts(event: Event, sales_channel: str,
         Q(available_until__isnull=True) | Q(available_until__gte=now()),
         sales_channels__contains=sales_channel,
         active=True,
-    ).prefetch_related('condition_limit_products').order_by('position', 'pk')
+    ).prefetch_related('condition_limit_products', 'benefit_limit_products').order_by('position', 'pk')
     for discount in discount_qs:
         result = discount.apply({
             idx: (item_id, subevent_id, line_price_gross, is_addon_to, voucher_discount)

src/pretix/base/services/quotas.py

@@ -24,13 +24,13 @@ import time
 from collections import Counter, defaultdict
 from itertools import zip_longest
 
+import django_redis
 from django.conf import settings
 from django.db import models
 from django.db.models import (
     Case, Count, F, Func, Max, OuterRef, Q, Subquery, Sum, Value, When,
 )
 from django.utils.timezone import now
-from django_redis import get_redis_connection
 
 from pretix.base.models import (
     CartPosition, Checkin, Order, OrderPosition, Quota, Voucher,
@@ -102,6 +102,12 @@ class QuotaAvailability:
         self.count_waitinglist = defaultdict(int)
         self.count_cart = defaultdict(int)
 
+        self._cache_key_suffix = ""
+        if not self._count_waitinglist:
+            self._cache_key_suffix += ":nocw"
+        if self._ignore_closed:
+            self._cache_key_suffix += ":igcl"
+
         self.sizes = {}
 
     def queue(self, *quota):
@@ -121,17 +127,14 @@ class QuotaAvailability:
             if self._full_results:
                 raise ValueError("You cannot combine full_results and allow_cache.")
 
-            elif not self._count_waitinglist:
-                raise ValueError("If you set allow_cache, you need to set count_waitinglist.")
-
             elif settings.HAS_REDIS:
-                rc = get_redis_connection("redis")
+                rc = django_redis.get_redis_connection("redis")
                 quotas_by_event = defaultdict(list)
                 for q in [_q for _q in self._queue if _q.id in quota_ids_set]:
                     quotas_by_event[q.event_id].append(q)
 
                 for eventid, evquotas in quotas_by_event.items():
-                    d = rc.hmget(f'quotas:{eventid}:availabilitycache', [str(q.pk) for q in evquotas])
+                    d = rc.hmget(f'quotas:{eventid}:availabilitycache{self._cache_key_suffix}', [str(q.pk) for q in evquotas])
                     for redisval, q in zip(d, evquotas):
                         if redisval is not None:
                             data = [rv for rv in redisval.decode().split(',')]
@@ -164,12 +167,12 @@ class QuotaAvailability:
         if not settings.HAS_REDIS or not quotas:
             return
 
-        rc = get_redis_connection("redis")
+        rc = django_redis.get_redis_connection("redis")
         # We write the computed availability to redis in a per-event hash as
         #
         #   quota_id -> (availability_state, availability_number, timestamp).
         #
-        # We store this in a hash instead of inidividual values to avoid making two many redis requests
+        # We store this in a hash instead of individual values to avoid making too many redis requests
         # which would introduce latency.
 
         # The individual entries in the hash are "valid" for 120 seconds. This means in a typical peak scenario with
@@ -179,16 +182,16 @@ class QuotaAvailability:
         # these quotas. We choose 10 seconds since that should be well above the duration of a write.
 
         lock_name = '_'.join([str(p) for p in sorted([q.pk for q in quotas])])
-        if rc.exists(f'quotas:availabilitycachewrite:{lock_name}'):
+        if rc.exists(f'quotas:availabilitycachewrite:{lock_name}{self._cache_key_suffix}'):
             return
-        rc.setex(f'quotas:availabilitycachewrite:{lock_name}', '1', 10)
+        rc.setex(f'quotas:availabilitycachewrite:{lock_name}{self._cache_key_suffix}', '1', 10)
 
         update = defaultdict(list)
         for q in quotas:
             update[q.event_id].append(q)
 
         for eventid, quotas in update.items():
-            rc.hmset(f'quotas:{eventid}:availabilitycache', {
+            rc.hmset(f'quotas:{eventid}:availabilitycache{self._cache_key_suffix}', {
                 str(q.id): ",".join(
                     [str(i) for i in self.results[q]] +
                     [str(int(time.time()))]
@@ -197,7 +200,7 @@ class QuotaAvailability:
             # To make sure old events do not fill up our redis instance, we set an expiry on the cache. However, we set it
             # on 7 days even though we mostly ignore values older than 2 monites. The reasoning is that we have some places
             # where we set allow_cache_stale and use the old entries anyways to save on performance.
-            rc.expire(f'quotas:{eventid}:availabilitycache', 3600 * 24 * 7)
+            rc.expire(f'quotas:{eventid}:availabilitycache{self._cache_key_suffix}', 3600 * 24 * 7)
 
         # We used to also delete item_quota_cache:* from the event cache here, but as the cache
         # gets more complex, this does not seem worth it. The cache is only present for up to

src/pretix/base/services/shredder.py

@@ -20,6 +20,7 @@
 # <https://www.gnu.org/licenses/>.
 #
 
+
 # This file is based on an earlier version of pretix which was released under the Apache License 2.0. The full text of
 # the Apache License 2.0 can be obtained at <http://www.apache.org/licenses/LICENSE-2.0>.
 #
@@ -31,7 +32,7 @@
 # Unless required by applicable law or agreed to in writing, software distributed under the Apache License 2.0 is
 # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 # License for the specific language governing permissions and limitations under the License.
-
+import inspect
 import json
 from datetime import timedelta
 from tempfile import NamedTemporaryFile
@@ -41,10 +42,13 @@ from zipfile import ZipFile
 from dateutil.parser import parse
 from django.conf import settings
 from django.utils.crypto import get_random_string
+from django.utils.formats import date_format
 from django.utils.timezone import now
 from django.utils.translation import gettext_lazy as _
 
-from pretix.base.models import CachedFile, Event, cachedfile_name
+from pretix.base.i18n import language
+from pretix.base.models import CachedFile, Event, User, cachedfile_name
+from pretix.base.services.mail import SendMailException, mail
 from pretix.base.services.tasks import ProfiledEventTask
 from pretix.base.shredder import ShredError
 from pretix.celery_app import app
@@ -101,8 +105,20 @@ def export(event: Event, shredders: List[str], session_key=None, cfid=None) -> N
     return cf.pk
 
 
-@app.task(base=ProfiledEventTask, throws=(ShredError,))
-def shred(event: Event, fileid: str, confirm_code: str) -> None:
+@app.task(base=ProfiledEventTask, throws=(ShredError,), bind=True)
+def shred(self, event: Event, fileid: str, confirm_code: str, user: int=None, locale: str='en') -> None:
+    steps = []
+
+    if user:
+        user = User.objects.get(pk=user)
+
+    def set_progress(val):
+        if not self.request.called_directly:
+            self.update_state(
+                state='PROGRESS',
+                meta={'value': val, 'steps': steps}
+            )
+
     known_shredders = event.get_data_shredders()
     try:
         cf = CachedFile.objects.get(pk=fileid)
@@ -124,8 +140,52 @@ def shred(event: Event, fileid: str, confirm_code: str) -> None:
     if event.logentry_set.filter(datetime__gte=parse(indexdata['time'])):
         raise ShredError(_("Something happened in your event after the export, please try again."))
 
-    for shredder in shredders:
-        shredder.shred_data()
+    event.log_action(
+        'pretix.event.shredder.started', user=user, data={
+            'indexdata': indexdata
+        }
+    )
+
+    for i, shredder in enumerate(shredders):
+        with language(locale):
+            steps.append({'label': str(shredder.verbose_name), 'done': False})
+        set_progress(i * 100 / len(shredders))
+        if 'progress_callback' in inspect.signature(shredder.shred_data).parameters:
+            shredder.shred_data(
+                progress_callback=lambda y: set_progress(
+                    i * 100 / len(shredders) + min(max(y, 0), 100) / 100 * 100 / len(shredders)
+                )
+            )
+        else:
+            shredder.shred_data()
+        steps[-1]['done'] = True
 
     cf.file.delete(save=False)
     cf.delete()
+
+    event.log_action(
+        'pretix.event.shredder.completed', user=user, data={
+            'indexdata': indexdata
+        }
+    )
+
+    if user:
+        with language(user.locale):
+            try:
+                mail(
+                    user.email,
+                    _('Data shredding completed'),
+                    'pretixbase/email/shred_completed.txt',
+                    {
+                        'user': user,
+                        'organizer': event.organizer.name,
+                        'event': str(event.name),
+                        'start_time': date_format(parse(indexdata['time']).astimezone(event.timezone), 'SHORT_DATETIME_FORMAT'),
+                        'shredders': ', '.join([str(s.verbose_name) for s in shredders])
+                    },
+                    event=None,
+                    user=user,
+                    locale=user.locale,
+                )
+            except SendMailException:
+                pass  # Already logged

src/pretix/base/services/waitinglist.py

@@ -22,13 +22,15 @@
 import sys
 from datetime import timedelta
 
-from django.db.models import Exists, F, OuterRef, Q, Sum
+from django.db.models import (
+    Exists, F, OuterRef, Prefetch, Q, Sum, prefetch_related_objects,
+)
 from django.dispatch import receiver
 from django.utils.timezone import now
 from django_scopes import scopes_disabled
 
 from pretix.base.models import (
-    Event, SeatCategoryMapping, User, WaitingListEntry,
+    Event, EventMetaValue, SeatCategoryMapping, User, WaitingListEntry,
 )
 from pretix.base.models.waitinglist import WaitingListException
 from pretix.base.services.tasks import EventTask
@@ -59,8 +61,21 @@ def assign_automatically(event: Event, user_id: int=None, subevent_id: int=None)
         ).aggregate(free=Sum(F('max_usages') - F('redeemed')))['free'] or 0
         seats_available[(m.product_id, m.subevent_id)] = num_free_seets_for_product - num_valid_vouchers_for_product
 
-    qs = WaitingListEntry.objects.filter(
-        event=event, voucher__isnull=True
+    prefetch_related_objects(
+        [event.organizer],
+        'meta_properties'
+    )
+    prefetch_related_objects(
+        [event],
+        Prefetch(
+            'meta_values',
+            EventMetaValue.objects.select_related('property'),
+            to_attr='meta_values_cached'
+        )
+    )
+
+    qs = event.waitinglistentries.filter(
+        voucher__isnull=True
     ).select_related('item', 'variation', 'subevent').prefetch_related(
         'item__quotas', 'variation__quotas'
     ).order_by('-priority', 'created')

src/pretix/base/settings.py

@@ -259,6 +259,46 @@ DEFAULTS = {
             label=_("Gift card currency"),
         )
     },
+    'reusable_media_type_nfc_mf0aes': {
+        'default': 'False',
+        'type': bool,
+        'form_class': forms.BooleanField,
+        'serializer_class': serializers.BooleanField,
+        'form_kwargs': dict(
+            label=_("Active"),
+        )
+    },
+    'reusable_media_type_nfc_mf0aes_autocreate_giftcard': {
+        'default': 'False',
+        'type': bool,
+        'form_class': forms.BooleanField,
+        'serializer_class': serializers.BooleanField,
+        'form_kwargs': dict(
+            label=_("Automatically create a new gift card if a new chip is encoded"),
+        )
+    },
+    'reusable_media_type_nfc_mf0aes_autocreate_giftcard_currency': {
+        'default': 'EUR',
+        'type': str,
+        'form_class': forms.ChoiceField,
+        'serializer_class': serializers.ChoiceField,
+        'serializer_kwargs': dict(
+            choices=[(c.alpha_3, c.alpha_3 + " - " + c.name) for c in settings.CURRENCIES],
+        ),
+        'form_kwargs': dict(
+            choices=[(c.alpha_3, c.alpha_3 + " - " + c.name) for c in settings.CURRENCIES],
+            label=_("Gift card currency"),
+        )
+    },
+    'reusable_media_type_nfc_mf0aes_random_uid': {
+        'default': 'False',
+        'type': bool,
+        'form_class': forms.BooleanField,
+        'serializer_class': serializers.BooleanField,
+        'form_kwargs': dict(
+            label=_("Use UID protection feature of NFC chip"),
+        )
+    },
     'max_items_per_order': {
         'default': '10',
         'type': int,
@@ -893,6 +933,28 @@ DEFAULTS = {
                         "the pool and can be ordered by other people."),
         )
     },
+    'payment_term_expire_delay_days': {
+        'default': '0',
+        'type': int,
+        'form_class': forms.IntegerField,
+        'serializer_class': serializers.IntegerField,
+        'form_kwargs': dict(
+            label=_('Expiration delay'),
+            help_text=_("The order will only actually expire this many days after the expiration date communicated "
+                        "to the customer. If you select \"Only end payment terms on weekdays\" above, this will also "
+                        "be respected. However, this will not delay beyond the \"last date of payments\" "
+                        "configured above, which is always enforced."),
+            # Every order in between the official expiry date and the delayed expiry date has a performance penalty
+            # for the cron job, so we limit this feature to 30 days to prevent arbitrary numbers of orders needing
+            # to be checked.
+            min_value=0,
+            max_value=30,
+        ),
+        'serializer_kwargs': dict(
+            min_value=0,
+            max_value=30,
+        ),
+    },
     'payment_pending_hidden': {
         'default': 'False',
         'type': bool,
@@ -2018,8 +2080,8 @@ to your order for {event}.
 You can change your order details and view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_resend_all_links': {
         'type': LazyI18nString,
@@ -2034,8 +2096,8 @@ The list is as follows:
 
 {orders}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_free_attendee': {
         'type': LazyI18nString,
@@ -2050,8 +2112,8 @@ you have been registered for {event} successfully.
 You can view the details and status of your ticket here:
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_send_order_free_attendee': {
         'type': bool,
@@ -2071,8 +2133,8 @@ no payment is required.
 You can change your order details and view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_placed_require_approval': {
         'type': LazyI18nString,
@@ -2089,8 +2151,8 @@ be patient and wait for our next email.
 You can change your order details and view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_placed': {
         'type': LazyI18nString,
@@ -2108,8 +2170,8 @@ of {total_with_currency}. Please complete your payment before {expire_date}.
 You can change your order details and view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_attachment_new_order': {
         'default': None,
@@ -2119,11 +2181,14 @@ Your {event} team"""))
             label=_('Attachment for new orders'),
             ext_whitelist=(".pdf",),
             max_size=settings.FILE_UPLOAD_MAX_SIZE_EMAIL_AUTO_ATTACHMENT,
-            help_text=_('This file will be attached to the first email that we send for every new order. Therefore it will be '
-                        'combined with the "Placed order", "Free order", or "Received order" texts from above. It will be sent '
-                        'to both order contacts and attendees. You can use this e.g. to send your terms of service. Do not use '
-                        'it to send non-public information as this file might be sent before payment is confirmed or the order '
-                        'is approved. To avoid this vital email going to spam, you can only upload PDF files of up to {size} MB.').format(
+            help_text=format_lazy(
+                _(
+                    'This file will be attached to the first email that we send for every new order. Therefore it will be '
+                    'combined with the "Placed order", "Free order", or "Received order" texts from above. It will be sent '
+                    'to both order contacts and attendees. You can use this e.g. to send your terms of service. Do not use '
+                    'it to send non-public information as this file might be sent before payment is confirmed or the order '
+                    'is approved. To avoid this vital email going to spam, you can only upload PDF files of up to {size} MB.'
+                ),
                 size=settings.FILE_UPLOAD_MAX_SIZE_EMAIL_AUTO_ATTACHMENT // (1024 * 1024),
             )
         ),
@@ -2152,8 +2217,8 @@ a ticket for {event} has been ordered for you.
 You can view the details and status of your ticket here:
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_changed': {
         'type': LazyI18nString,
@@ -2168,8 +2233,8 @@ your order for {event} has been changed.
 You can view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_paid': {
         'type': LazyI18nString,
@@ -2186,8 +2251,8 @@ we successfully received your payment for {event}. Thank you!
 You can change your order details and view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_send_order_paid_attendee': {
         'type': bool,
@@ -2206,8 +2271,8 @@ a ticket for {event} that has been ordered for you is now paid.
 You can view the details and status of your ticket here:
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_days_order_expire_warning': {
         'form_class': forms.IntegerField,
@@ -2239,8 +2304,8 @@ your payment before {expire_date}.
 You can view the payment information and the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_pending_warning': {
         'type': LazyI18nString,
@@ -2256,8 +2321,8 @@ Please keep in mind that you are required to pay before {expire_date}.
 You can view the payment information and the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_incomplete_payment': {
         'type': LazyI18nString,
@@ -2276,8 +2341,26 @@ missing additional payment of **{pending_sum}**.
 You can view the payment information and the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
+    },
+    'mail_subject_order_payment_failed': {
+        'type': LazyI18nString,
+        'default': LazyI18nString.from_gettext(gettext_noop("Payment failed for your order: {code}")),
+    },
+    'mail_text_order_payment_failed': {
+        'type': LazyI18nString,
+        'default': LazyI18nString.from_gettext(gettext_noop("""Hello,
+
+your payment attempt for your order for {event} has failed.
+
+Your order is still valid and you can try to pay again using the same or a different payment method. Please complete your payment before {expire_date}.
+
+You can retry the payment and view the status of your order at
+{url}
+
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_waiting_list': {
         'type': LazyI18nString,
@@ -2309,8 +2392,8 @@ as possible to the next person on the waiting list:
 
 {url_remove}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_canceled': {
         'type': LazyI18nString,
@@ -2327,8 +2410,8 @@ your order {code} for {event} has been canceled.
 You can view the details of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_approved': {
         'type': LazyI18nString,
@@ -2347,8 +2430,8 @@ You can select a payment method and perform the payment here:
 
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_send_order_approved_attendee': {
         'type': bool,
@@ -2367,8 +2450,8 @@ we approved a ticket ordered for you for {event}.
 You can view the details and status of your ticket here:
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_approved_free': {
         'type': LazyI18nString,
@@ -2384,8 +2467,8 @@ at our event. As you only ordered free products, no payment is required.
 You can change your order details and view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_send_order_approved_free_attendee': {
         'type': bool,
@@ -2404,8 +2487,8 @@ we approved a ticket ordered for you for {event}.
 You can view the details and status of your ticket here:
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_order_denied': {
         'type': LazyI18nString,
@@ -2423,8 +2506,8 @@ You can view the details of your order here:
 
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_text_order_custom_mail': {
         'type': LazyI18nString,
@@ -2433,8 +2516,8 @@ Your {event} team"""))
 You can change your order details and view the status of your order at
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_days_download_reminder': {
         'type': int,
@@ -2452,13 +2535,13 @@ Your {event} team"""))
         'type': LazyI18nString,
         'default': LazyI18nString.from_gettext(gettext_noop("""Hello {attendee_name},
 
-    you are registered for {event}.
+you are registered for {event}.
 
-    If you did not do so already, you can download your ticket here:
-    {url}
+If you did not do so already, you can download your ticket here:
+{url}
 
-    Best regards,
-    Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_download_reminder': {
         'type': LazyI18nString,
@@ -2473,8 +2556,8 @@ you bought a ticket for {event}.
 If you did not do so already, you can download your ticket here:
 {url}
 
-Best regards,
-Your {event} team"""))
+Best regards,  
+Your {event} team"""))  # noqa: W291
     },
     'mail_subject_customer_registration': {
         'type': LazyI18nString,
@@ -2494,9 +2577,9 @@ This link is valid for one day.
 
 If you did not sign up yourself, please ignore this email.
 
-Best regards,
+Best regards,  
 
-Your {organizer} team"""))
+Your {organizer} team"""))  # noqa: W291
     },
     'mail_subject_customer_email_change': {
         'type': LazyI18nString,
@@ -2516,9 +2599,9 @@ This link is valid for one day.
 
 If you did not request this, please ignore this email.
 
-Best regards,
+Best regards,  
 
-Your {organizer} team"""))
+Your {organizer} team"""))  # noqa: W291
     },
     'mail_subject_customer_reset': {
         'type': LazyI18nString,
@@ -2538,9 +2621,9 @@ This link is valid for one day.
 
 If you did not request a new password, please ignore this email.
 
-Best regards,
+Best regards,  
 
-Your {organizer} team"""))
+Your {organizer} team"""))  # noqa: W291
     },
     'smtp_use_custom': {
         'default': 'False',
@@ -3583,8 +3666,10 @@ class SettingsSandbox:
     def __delattr__(self, key: str) -> None:
         del self._event.settings[self._convert_key(key)]
 
-    def get(self, key: str, default: Any = None, as_type: type = str):
-        return self._event.settings.get(self._convert_key(key), default=default, as_type=as_type)
+    def get(self, key: str, default: Any = None, as_type: type = str, binary_file: bool = False):
+        return self._event.settings.get(
+            self._convert_key(key), default=default, as_type=as_type, binary_file=binary_file
+        )
 
     def set(self, key: str, value: Any):
         self._event.settings.set(self._convert_key(key), value)
@@ -3630,14 +3715,10 @@ def validate_organizer_settings(organizer, settings_dict):
     # This is not doing anything for the time being.
     # But earlier we called validate_event_settings for the organizer, too - and that didn't do anything for
     # organizer-settings either.
-    #
-    # N.B.: When actually fleshing out this stub, adding it to the OrganizerUpdateForm should be considered.
-    """
-    if settings_dict.get('reusable_media_type_ntag_pretix1') and settings_dict.get('reusable_media_type_nfc_uid'):
+    if settings_dict.get('reusable_media_type_nfc_mf0aes') and settings_dict.get('reusable_media_type_nfc_uid'):
         raise ValidationError({
             'reusable_media_type_nfc_uid': _('This needs to be disabled if other NFC-based types are active.')
         })
-    """
 
 
 def global_settings_object(holder):

src/pretix/base/shredder.py

@@ -32,11 +32,12 @@
 # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 # License for the specific language governing permissions and limitations under the License.
 
+import copy
 import json
 import os
+import time
 from typing import List, Tuple
 
-from django.db import transaction
 from django.db.models import Max, Q
 from django.db.models.functions import Greatest
 from django.dispatch import receiver
@@ -99,11 +100,13 @@ class BaseDataShredder:
         """
         raise NotImplementedError()  # NOQA
 
-    def shred_data(self):
+    def shred_data(self, progress_callback=None):
         """
         This method is called to actually remove the data from the system. You should remove any database objects
         here.
 
+        You can call ``progress_callback`` with an integer value between 0 and 100 to communicate back your progress.
+
         You should never delete ``LogEntry`` objects, but you might modify them to remove personal data. In this
         case, set the ``LogEntry.shredded`` attribute to ``True`` to show that this is no longer original log data.
         """
@@ -151,6 +154,7 @@ class BaseDataShredder:
 
 def shred_log_fields(logentry, banlist=None, whitelist=None):
     d = logentry.parsed_data
+    initial_data = copy.copy(d)
     shredded = False
     if whitelist:
         for k, v in d.items():
@@ -162,9 +166,63 @@ def shred_log_fields(logentry, banlist=None, whitelist=None):
             if f in d:
                 d[f] = '█'
                 shredded = True
-    logentry.data = json.dumps(d)
-    logentry.shredded = logentry.shredded or shredded
-    logentry.save(update_fields=['data', 'shredded'])
+    if d != initial_data:
+        logentry.data = json.dumps(d)
+        logentry.shredded = logentry.shredded or shredded
+        logentry.save(update_fields=['data', 'shredded'])
+
+
+def slow_update(qs, batch_size=1000, sleep_time=.5, progress_callback=None, progress_offset=0, progress_total=None, **update):
+    """
+    Doing UPDATE queries on hundreds of thousands of rows can cause outages due to high write load on the database.
+    This provides a throttled way to update rows. The condition for this to work properly is that the queryset has a
+    filter condition that no longer applies after the update!
+    Otherwise, this will be an endless loop!
+    """
+    total_updated = 0
+    while True:
+        updated = qs.order_by().filter(
+            pk__in=qs.order_by().values_list('pk', flat=True)[:batch_size]
+        ).update(**update)
+        total_updated += updated
+        if not updated:
+            break
+        if total_updated >= 0.8 * batch_size:
+            time.sleep(sleep_time)
+        if progress_callback and progress_total:
+            progress_callback((progress_offset + total_updated) / progress_total)
+
+    return total_updated
+
+
+def slow_delete(qs, batch_size=1000, sleep_time=.5, progress_callback=None, progress_offset=0, progress_total=None):
+    """
+    Doing DELETE queries on hundreds of thousands of rows can cause outages due to high write load on the database.
+    This provides a throttled way to update rows.
+    """
+    total_deleted = 0
+    while True:
+        deleted = qs.order_by().filter(
+            pk__in=qs.order_by().values_list('pk', flat=True)[:batch_size]
+        ).delete()[0]
+        total_deleted += deleted
+        if not deleted:
+            break
+        if total_deleted >= 0.8 * batch_size:
+            time.sleep(sleep_time)
+        if progress_callback and progress_total:
+            progress_callback((progress_offset + total_deleted) / progress_total)
+    return total_deleted
+
+
+def _progress_helper(queryset, progress_callback, offset, total):
+    if not progress_callback:
+        yield from queryset
+    else:
+        for i, o in enumerate(queryset):
+            yield o
+            if i % 10 == 0:
+                progress_callback((i + offset) / total * 100)
 
 
 class PhoneNumberShredder(BaseDataShredder):
@@ -177,18 +235,26 @@ class PhoneNumberShredder(BaseDataShredder):
             o.code: o.phone for o in self.event.orders.filter(phone__isnull=False)
         }, cls=CustomJSONEncoder, indent=4)
 
-    @transaction.atomic
-    def shred_data(self):
-        for o in self.event.orders.all():
+    def shred_data(self, progress_callback=None):
+        qs_orders = self.event.orders.all()
+        qs_orders_cnt = qs_orders.count()
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.phone.changed")
+        qs_le_cnt = qs_le.count()
+        total = qs_le_cnt + qs_orders_cnt
+
+        for o in _progress_helper(qs_orders, progress_callback, 0, total):
+            changed = bool(o.phone)
             o.phone = None
             d = o.meta_info_data
             if d:
                 if 'contact_form_data' in d and 'phone' in d['contact_form_data']:
+                    changed = True
                     del d['contact_form_data']['phone']
-                o.meta_info = json.dumps(d)
-            o.save(update_fields=['meta_info', 'phone'])
+                    o.meta_info = json.dumps(d)
+            if changed:
+                o.save(update_fields=['meta_info', 'phone'])
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.phone.changed"):
+        for le in _progress_helper(qs_le, progress_callback, qs_orders_cnt, total):
             shred_log_fields(le, banlist=['old_phone', 'new_phone'])
 
 
@@ -207,37 +273,66 @@ class EmailAddressShredder(BaseDataShredder):
             for op in OrderPosition.all.filter(order__event=self.event, attendee_email__isnull=False)
         }, indent=4)
 
-    @transaction.atomic
-    def shred_data(self):
-        OrderPosition.all.filter(order__event=self.event, attendee_email__isnull=False).update(attendee_email=None)
+    def shred_data(self, progress_callback=None):
+        qs_op = OrderPosition.all.filter(order__event=self.event, attendee_email__isnull=False)
+        qs_op_cnt = qs_op.count()
 
-        for o in self.event.orders.all():
+        qs_orders = self.event.orders.all()
+        qs_orders_cnt = qs_orders.count()
+
+        qs_le = self.event.logentry_set.filter(
+            Q(action_type__contains="order.email") | Q(action_type__contains="position.email") |
+            Q(action_type="pretix.event.order.contact.changed") |
+            Q(action_type="pretix.event.order.modified")
+        ).exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_op_cnt + qs_orders_cnt + qs_le_cnt
+
+        slow_update(
+            qs_op,
+            attendee_email=None,
+            progress_callback=progress_callback,
+            progress_offset=0,
+            progress_total=total,
+            # Updates to order position table are slow, since PostgreSQL needs to update many indexes, so let's
+            # take them really slowly to not overwhelm the database.
+            batch_size=100,
+            sleep_time=2,
+        )
+
+        for o in _progress_helper(qs_orders, progress_callback, qs_op_cnt, total):
+            changed = bool(o.email) or bool(o.customer)
             o.email = None
             o.customer = None
             d = o.meta_info_data
             if d:
                 if 'contact_form_data' in d and 'email' in d['contact_form_data']:
                     del d['contact_form_data']['email']
-                o.meta_info = json.dumps(d)
-            o.save(update_fields=['meta_info', 'email', 'customer'])
+                    changed = True
+                    o.meta_info = json.dumps(d)
+                if 'contact_form_data' in d and 'email_repeat' in d['contact_form_data']:
+                    del d['contact_form_data']['email_repeat']
+                    changed = True
+            if changed:
+                if d:
+                    o.meta_info = json.dumps(d)
+                o.save(update_fields=['meta_info', 'email', 'customer'])
 
-        for le in self.event.logentry_set.filter(
-            Q(action_type__contains="order.email") | Q(action_type__contains="position.email"),
-        ):
-            shred_log_fields(le, banlist=['recipient', 'message', 'subject', 'full_mail'])
-
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.contact.changed"):
-            shred_log_fields(le, banlist=['old_email', 'new_email'])
-
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
-            d = le.parsed_data
-            if 'data' in d:
-                for row in d['data']:
-                    if 'attendee_email' in row:
-                        row['attendee_email'] = '█'
-                le.data = json.dumps(d)
-                le.shredded = True
-                le.save(update_fields=['data', 'shredded'])
+        for le in _progress_helper(qs_le, progress_callback, qs_op_cnt + qs_orders_cnt, total):
+            if le.action_type == "pretix.event.order.modified":
+                d = le.parsed_data
+                if 'data' in d:
+                    for row in d['data']:
+                        if 'attendee_email' in row:
+                            row['attendee_email'] = '█'
+                    le.data = json.dumps(d)
+                    le.shredded = True
+                    le.save(update_fields=['data', 'shredded'])
+            else:
+                shred_log_fields(le, banlist=[
+                    'recipient', 'message', 'subject', 'full_mail', 'old_email', 'new_email'
+                ])
 
 
 class WaitingListShredder(BaseDataShredder):
@@ -251,16 +346,35 @@ class WaitingListShredder(BaseDataShredder):
             for wle in self.event.waitinglistentries.all()
         ], indent=4)
 
-    @transaction.atomic
-    def shred_data(self):
-        self.event.waitinglistentries.update(name_cached=None, name_parts={'_shredded': True}, email='█', phone='█')
+    def shred_data(self, progress_callback=None):
+        qs_wle = self.event.waitinglistentries.exclude(email='█')
+        qs_wle_cnt = qs_wle.count()
 
-        for wle in self.event.waitinglistentries.select_related('voucher').filter(voucher__isnull=False):
+        qs_voucher = self.event.waitinglistentries.select_related('voucher').filter(voucher__isnull=False)
+        qs_voucher_cnt = qs_voucher.count()
+
+        qs_le = self.event.logentry_set.filter(action_type="pretix.voucher.added.waitinglist").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_voucher_cnt + qs_wle_cnt + qs_le_cnt
+
+        slow_update(
+            qs_wle,
+            name_cached=None,
+            name_parts={'_shredded': True},
+            email='█',
+            phone='█',
+            progress_callback=progress_callback,
+            progress_offset=0,
+            progress_total=total,
+        )
+
+        for wle in _progress_helper(qs_voucher, progress_callback, qs_wle_cnt, total):
             if '@' in wle.voucher.comment:
                 wle.voucher.comment = '█'
                 wle.voucher.save(update_fields=['comment'])
 
-        for le in self.event.logentry_set.filter(action_type="pretix.voucher.added.waitinglist").exclude(data=""):
+        for le in _progress_helper(qs_le, progress_callback, qs_wle_cnt + qs_voucher_cnt, total):
             d = le.parsed_data
             if 'name' in d:
                 d['name'] = '█'
@@ -298,17 +412,41 @@ class AttendeeInfoShredder(BaseDataShredder):
             )
         }, indent=4)
 
-    @transaction.atomic
-    def shred_data(self):
-        OrderPosition.all.filter(
+    def shred_data(self, progress_callback=None):
+        qs_op = OrderPosition.all.filter(
             order__event=self.event
         ).filter(
-            Q(attendee_name_cached__isnull=False) | Q(attendee_name_parts__isnull=False) |
-            Q(company__isnull=False) | Q(street__isnull=False) | Q(zipcode__isnull=False) | Q(city__isnull=False)
-        ).update(attendee_name_cached=None, attendee_name_parts={'_shredded': True}, company=None, street=None,
-                 zipcode=None, city=None)
+            Q(attendee_name_cached__isnull=False) |
+            Q(company__isnull=False) |
+            Q(street__isnull=False) |
+            Q(zipcode__isnull=False) |
+            Q(city__isnull=False)
+        )
+        qs_op_cnt = qs_op.count()
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_op_cnt + qs_le_cnt
+
+        slow_update(
+            qs_op,
+            attendee_name_cached=None,
+            attendee_name_parts={'_shredded': True},
+            company=None,
+            street=None,
+            zipcode=None,
+            city=None,
+            progress_callback=progress_callback,
+            progress_total=total,
+            progress_offset=0,
+            # Updates to order position table are slow, since PostgreSQL needs to update many indexes, so let's
+            # take them really slowly to not overwhelm the database.
+            batch_size=100,
+            sleep_time=2,
+        )
+
+        for le in _progress_helper(qs_le, progress_callback, qs_op_cnt, total):
             d = le.parsed_data
             if 'data' in d:
                 for i, row in enumerate(d['data']):
@@ -343,11 +481,18 @@ class InvoiceAddressShredder(BaseDataShredder):
             for ia in InvoiceAddress.objects.filter(order__event=self.event)
         }, indent=4)
 
-    @transaction.atomic
-    def shred_data(self):
-        InvoiceAddress.objects.filter(order__event=self.event).delete()
+    def shred_data(self, progress_callback=None):
+        qs_ia = InvoiceAddress.objects.filter(order__event=self.event)
+        qs_ia_cnt = qs_ia.count()
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_ia_cnt + qs_le_cnt
+
+        slow_delete(qs_ia, progress_callback=progress_callback, progress_total=total, progress_offset=0)
+
+        for le in _progress_helper(qs_le, progress_callback, qs_ia_cnt, total):
             d = le.parsed_data
             if 'invoice_data' in d and not isinstance(d['invoice_data'], bool):
                 for field in d['invoice_data']:
@@ -375,11 +520,18 @@ class QuestionAnswerShredder(BaseDataShredder):
             ).data
         yield 'question-answers.json', 'application/json', json.dumps(d, indent=4)
 
-    @transaction.atomic
-    def shred_data(self):
-        QuestionAnswer.objects.filter(orderposition__order__event=self.event).delete()
+    def shred_data(self, progress_callback=None):
+        qs_qa = QuestionAnswer.objects.filter(orderposition__order__event=self.event)
+        qs_qa_cnt = qs_qa.count()
 
-        for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data=""):
+        qs_le = self.event.logentry_set.filter(action_type="pretix.event.order.modified").exclude(data="")
+        qs_le_cnt = qs_le.count()
+
+        total = qs_qa_cnt + qs_le_cnt
+
+        slow_delete(qs_qa, progress_callback=progress_callback, progress_total=total, progress_offset=0)
+
+        for le in _progress_helper(qs_le, progress_callback, qs_qa_cnt, total):
             d = le.parsed_data
             if 'data' in d:
                 for i, row in enumerate(d['data']):
@@ -408,9 +560,11 @@ class InvoiceShredder(BaseDataShredder):
             yield 'invoices/{}.pdf'.format(i.number), 'application/pdf', i.file.read()
             i.file.close()
 
-    @transaction.atomic
-    def shred_data(self):
-        for i in self.event.invoices.filter(shredded=False):
+    def shred_data(self, progress_callback=None):
+        qs_i = self.event.invoices.filter(shredded=False)
+        total = qs_i.count()
+
+        for i in _progress_helper(qs_i, progress_callback, 0, total):
             if i.file:
                 i.file.delete()
                 i.shredded = True
@@ -430,10 +584,17 @@ class CachedTicketShredder(BaseDataShredder):
     def generate_files(self) -> List[Tuple[str, str, str]]:
         pass
 
-    @transaction.atomic
-    def shred_data(self):
-        CachedTicket.objects.filter(order_position__order__event=self.event).delete()
-        CachedCombinedTicket.objects.filter(order__event=self.event).delete()
+    def shred_data(self, progress_callback=None):
+        qs_1 = CachedTicket.objects.filter(order_position__order__event=self.event)
+        qs_1_cnt = qs_1.count()
+
+        qs_2 = CachedCombinedTicket.objects.filter(order__event=self.event)
+        qs_2_cnt = qs_2.count()
+
+        total = qs_1_cnt + qs_2_cnt
+
+        slow_delete(qs_1, progress_callback=progress_callback, progress_total=total, progress_offset=0)
+        slow_delete(qs_2, progress_callback=progress_callback, progress_total=total, progress_offset=qs_1_cnt)
 
 
 class PaymentInfoShredder(BaseDataShredder):
@@ -446,14 +607,21 @@ class PaymentInfoShredder(BaseDataShredder):
     def generate_files(self) -> List[Tuple[str, str, str]]:
         pass
 
-    @transaction.atomic
-    def shred_data(self):
+    def shred_data(self, progress_callback=None):
+        qs_p = OrderPayment.objects.filter(order__event=self.event)
+        qs_p_count = qs_p.count()
+        qs_r = OrderRefund.objects.filter(order__event=self.event)
+        qs_r_count = qs_r.count()
+
+        total = qs_p_count + qs_r_count
+
         provs = self.event.get_payment_providers()
-        for obj in OrderPayment.objects.filter(order__event=self.event):
+        for obj in _progress_helper(qs_p, progress_callback, 0, total):
             pprov = provs.get(obj.provider)
             if pprov:
                 pprov.shred_payment_info(obj)
-        for obj in OrderRefund.objects.filter(order__event=self.event):
+
+        for obj in _progress_helper(qs_r, progress_callback, qs_p_count, total):
             pprov = provs.get(obj.provider)
             if pprov:
                 pprov.shred_payment_info(obj)

src/pretix/base/signals.py

@@ -683,12 +683,16 @@ dictionaries as values that contain keys like in the following example::
         "product": {
             "label": _("Product name"),
             "editor_sample": _("Sample product"),
-            "evaluate": lambda orderposition, order, event: str(orderposition.item)
+            "evaluate": lambda orderposition, order, event: str(orderposition.item),
+            "evaluate_bulk": lambda orderpositions: [str(op.item) for op in orderpositions],
         }
     }
 
 The ``evaluate`` member will be called with the order position, order and event as arguments. The event might
 also be a subevent, if applicable.
+
+The ``evaluate_bulk`` member is optional but can significantly improve performance in some situations because you
+can perform database fetches in bulk instead of single queries for every position.
 """
 
 
@@ -787,3 +791,23 @@ return a dictionary mapping names of attributes in the settings store to DRF ser
 
 As with all event-plugin signals, the ``sender`` keyword argument will contain the event.
 """
+
+customer_created = GlobalSignal()
+"""
+Arguments: ``customer``
+
+This signal is sent out every time a customer account is created. The ``customer``
+object is given as the first argument.
+
+The ``sender`` keyword argument will contain the organizer.
+"""
+
+customer_signed_in = GlobalSignal()
+"""
+Arguments: ``customer``
+
+This signal is sent out every time a customer signs in. The ``customer`` object
+is given as the first argument.
+
+The ``sender`` keyword argument will contain the organizer.
+"""

src/pretix/base/templates/pretixbase/email/base.html

@@ -175,6 +175,10 @@
             font-size: 12px;
         }
 
+        pre, pre code {
+            white-space: pre-line;
+        }
+
         {% if rtl %}
             body {
                 direction: rtl;

src/pretix/base/templates/pretixbase/email/shred_completed.txt

@@ -0,0 +1,17 @@
+{% load i18n %}
+{% load i18n %}{% blocktrans with url=url|safe %}Hello,
+
+we hereby confirm that the following data shredding job has been completed:
+
+Organizer: {{ organizer }}
+
+Event: {{ event }}
+
+Data selection: {{ shredders }}
+
+Start time: {{ start_time }} (new data added after this time might not have been deleted)
+
+Best regards,
+
+Your pretix team
+{% endblocktrans %}

src/pretix/base/templates/pretixbase/email/simple_logo.html

@@ -91,12 +91,9 @@
             /* These are technically the same, but use both */
             overflow-wrap: break-word;
             word-wrap: break-word;
+            word-break: break-word;
 
             -ms-word-break: break-all;
-            /* This is the dangerous one in WebKit, as it breaks things wherever */
-            word-break: break-all;
-            /* Instead use this non-standard one: */
-            word-break: break-word;
 
             /* Adds a hyphen where the word breaks, if supported (No Blink) */
             -ms-hyphens: auto;
@@ -105,6 +102,15 @@
             hyphens: auto;
         }
 
+        p:last-child {
+            margin-bottom: 0;
+        }
+
+        p a {
+            word-wrap: anywhere;
+            word-break: break-all;
+        }
+
         .footer {
             padding: 10px;
             text-align: center;
@@ -177,6 +183,7 @@
         }
 
         .order {
+            border-top: 1px solid #ccc;
             font-size: 12px;
         }
 
@@ -194,6 +201,10 @@
             font-size: 12px;
         }
 
+        pre, pre code {
+            white-space: pre-line;
+        }
+
         {% if rtl %}
             body {
                 direction: rtl;

src/pretix/base/templatetags/rich_text.py

@@ -46,6 +46,10 @@ from django.urls import reverse
 from django.utils.functional import SimpleLazyObject
 from django.utils.http import url_has_allowed_host_and_scheme
 from django.utils.safestring import mark_safe
+from markdown import Extension
+from markdown.inlinepatterns import SubstituteTagInlineProcessor
+from markdown.postprocessors import Postprocessor
+from markdown.treeprocessors import UnescapeTreeprocessor
 from tlds import tld_set
 
 register = template.Library()
@@ -106,6 +110,8 @@ URL_RE = SimpleLazyObject(lambda: build_url_re(tlds=sorted(tld_set, key=len, rev
 
 EMAIL_RE = SimpleLazyObject(lambda: build_email_re(tlds=sorted(tld_set, key=len, reverse=True)))
 
+DOT_ESCAPE = "|escaped-dot-sGnY9LMK|"
+
 
 def safelink_callback(attrs, new=False):
     """
@@ -168,25 +174,145 @@ def abslink_callback(attrs, new=False):
     return attrs
 
 
-def markdown_compile_email(source):
+class EmailNl2BrExtension(Extension):
+    """
+    In emails (mostly for backwards-compatibility), we do not follow GitHub Flavored Markdown in preserving newlines.
+    Instead, we follow the CommonMark specification:
+
+    "A line ending (not in a code span or HTML tag) that is preceded by two or more spaces and does not occur at the
+    end of a block is parsed as a hard line break (rendered in HTML as a <br /> tag)"
+    """
+    BR_RE = r'  \n'
+
+    def extendMarkdown(self, md):
+        br_tag = SubstituteTagInlineProcessor(self.BR_RE, 'br')
+        md.inlinePatterns.register(br_tag, 'nl', 5)
+
+
+class LinkifyPostprocessor(Postprocessor):
+    def __init__(self, linker):
+        self.linker = linker
+        super().__init__()
+
+    def run(self, text):
+        return self.linker.linkify(text)
+
+
+class CleanPostprocessor(Postprocessor):
+    def __init__(self, tags, attributes, protocols, strip):
+        self.tags = tags
+        self.attributes = attributes
+        self.protocols = protocols
+        self.strip = strip
+        super().__init__()
+
+    def run(self, text):
+        return bleach.clean(
+            text,
+            tags=self.tags,
+            attributes=self.attributes,
+            protocols=self.protocols,
+            strip=self.strip
+        )
+
+
+class CustomUnescapeTreeprocessor(UnescapeTreeprocessor):
+    """
+    This un-escapes everything except \\.
+    """
+
+    def _unescape(self, m):
+        if m.group(1) == "46":  # 46 is the ASCII position of .
+            return DOT_ESCAPE
+        return chr(int(m.group(1)))
+
+
+class CustomUnescapePostprocessor(Postprocessor):
+    """
+    Restore escaped .
+    """
+
+    def run(self, text):
+        return text.replace(DOT_ESCAPE, ".")
+
+
+class LinkifyAndCleanExtension(Extension):
+    r"""
+    We want to do:
+
+    input --> markdown --> bleach clean --> linkify --> output
+
+    Internally, the markdown library does:
+
+    source --> parse --> (tree|inline)processors --> serializing --> postprocessors
+
+    All escaped characters such as \. will be turned to something like <STX>46<ETX> in the processors
+    step and then will be converted to . back again in the last tree processor, before serialization.
+    Therefore, linkify does not see the escaped character anymore. This is annoying for the one case
+    where you want to type "rich_text.py" and *not* have it turned into a link, since you can't type
+    "rich_text\.py" either.
+
+    A simple solution would be to run linkify before markdown, but that may cause other issues when
+    linkify messes with the markdown syntax and it makes handling our attributes etc. harder.
+
+    So we do a weird hack where we modify the unescape processor to unescape everything EXCEPT for the
+    dot and then unescape that one manually after linkify. However, to make things even harder, the bleach
+    clean step removes any invisible characters, so we need to cheat a bit more.
+    """
+
+    def __init__(self, linker, tags, attributes, protocols, strip):
+        self.linker = linker
+        self.tags = tags
+        self.attributes = attributes
+        self.protocols = protocols
+        self.strip = strip
+        super().__init__()
+
+    def extendMarkdown(self, md):
+        md.treeprocessors.deregister('unescape')
+        md.treeprocessors.register(
+            CustomUnescapeTreeprocessor(md),
+            'unescape',
+            0
+        )
+        md.postprocessors.register(
+            CleanPostprocessor(self.tags, self.attributes, self.protocols, self.strip),
+            'clean',
+            2
+        )
+        md.postprocessors.register(
+            LinkifyPostprocessor(self.linker),
+            'linkify',
+            1
+        )
+        md.postprocessors.register(
+            CustomUnescapePostprocessor(self.linker),
+            'unescape_dot',
+            0
+        )
+
+
+def markdown_compile_email(source, allowed_tags=ALLOWED_TAGS, allowed_attributes=ALLOWED_ATTRIBUTES):
     linker = bleach.Linker(
         url_re=URL_RE,
         email_re=EMAIL_RE,
         callbacks=DEFAULT_CALLBACKS + [truelink_callback, abslink_callback],
         parse_email=True
     )
-    return linker.linkify(bleach.clean(
-        markdown.markdown(
-            source,
-            extensions=[
-                'markdown.extensions.sane_lists',
-                #  'markdown.extensions.nl2br' # disabled for backwards-compatibility
-            ]
-        ),
-        tags=ALLOWED_TAGS,
-        attributes=ALLOWED_ATTRIBUTES,
-        protocols=ALLOWED_PROTOCOLS,
-    ))
+    return markdown.markdown(
+        source,
+        extensions=[
+            'markdown.extensions.sane_lists',
+            EmailNl2BrExtension(),
+            LinkifyAndCleanExtension(
+                linker,
+                tags=allowed_tags,
+                attributes=allowed_attributes,
+                protocols=ALLOWED_PROTOCOLS,
+                strip=False,
+            )
+        ]
+    )
 
 
 class SnippetExtension(markdown.extensions.Extension):
@@ -196,23 +322,24 @@ class SnippetExtension(markdown.extensions.Extension):
         md.parser.blockprocessors.deregister('quote')
 
 
-def markdown_compile(source, snippet=False):
+def markdown_compile(source, linker, snippet=False):
     tags = ALLOWED_TAGS_SNIPPET if snippet else ALLOWED_TAGS
     exts = [
         'markdown.extensions.sane_lists',
-        'markdown.extensions.nl2br'
+        'markdown.extensions.nl2br',
+        LinkifyAndCleanExtension(
+            linker,
+            tags=tags,
+            attributes=ALLOWED_ATTRIBUTES,
+            protocols=ALLOWED_PROTOCOLS,
+            strip=snippet,
+        )
     ]
     if snippet:
         exts.append(SnippetExtension())
-    return bleach.clean(
-        markdown.markdown(
-            source,
-            extensions=exts
-        ),
-        strip=snippet,
-        tags=tags,
-        attributes=ALLOWED_ATTRIBUTES,
-        protocols=ALLOWED_PROTOCOLS,
+    return markdown.markdown(
+        source,
+        extensions=exts
     )
 
 
@@ -228,7 +355,7 @@ def rich_text(text: str, **kwargs):
         callbacks=DEFAULT_CALLBACKS + ([truelink_callback, safelink_callback] if kwargs.get('safelinks', True) else [truelink_callback, abslink_callback]),
         parse_email=True
     )
-    body_md = linker.linkify(markdown_compile(text))
+    body_md = markdown_compile(text, linker)
     return mark_safe(body_md)
 
 
@@ -244,5 +371,5 @@ def rich_text_snippet(text: str, **kwargs):
         callbacks=DEFAULT_CALLBACKS + ([truelink_callback, safelink_callback] if kwargs.get('safelinks', True) else [truelink_callback, abslink_callback]),
         parse_email=True
     )
-    body_md = linker.linkify(markdown_compile(text, snippet=True))
+    body_md = markdown_compile(text, linker, snippet=True)
     return mark_safe(body_md)

src/pretix/base/views/tasks.py

@@ -115,7 +115,8 @@ class AsyncMixin:
         elif state == 'PROGRESS':
             data.update({
                 'started': True,
-                'percentage': info.get('value', 0) if isinstance(info, dict) else 0
+                'percentage': info.get('value', 0) if isinstance(info, dict) else 0,
+                'steps': info.get('steps', []) if isinstance(info, dict) else None,
             })
         elif state == 'STARTED':
             data.update({

src/pretix/control/forms/checkin.py

@@ -31,7 +31,8 @@ from django_scopes.forms import (
 )
 
 from pretix.base.channels import get_all_sales_channels
-from pretix.base.models.checkin import CheckinList
+from pretix.base.forms.widgets import SplitDateTimePickerWidget
+from pretix.base.models.checkin import Checkin, CheckinList
 from pretix.control.forms import ItemMultipleChoiceField
 from pretix.control.forms.widgets import Select2
 
@@ -177,3 +178,26 @@ class SimpleCheckinListForm(forms.ModelForm):
             'subevent': SafeModelChoiceField,
             'gates': SafeModelMultipleChoiceField,
         }
+
+
+class CheckinListSimulatorForm(forms.Form):
+    raw_barcode = forms.CharField(
+        label=_("Barcode"),
+    )
+    datetime = forms.SplitDateTimeField(
+        label=_("Check-in time"),
+        widget=SplitDateTimePickerWidget(),
+    )
+    checkin_type = forms.ChoiceField(
+        label=_("Check-in type"),
+        choices=Checkin.CHECKIN_TYPES,
+    )
+    ignore_unpaid = forms.BooleanField(
+        label=_("Allow check-in of unpaid order (if check-in list permits it)"),
+        required=False,
+    )
+    questions_supported = forms.BooleanField(
+        label=_("Support for check-in questions"),
+        initial=True,
+        required=False,
+    )

src/pretix/control/forms/discounts.py

@@ -50,11 +50,16 @@ class DiscountForm(I18nModelForm):
             'condition_ignore_voucher_discounted',
             'benefit_discount_matching_percent',
             'benefit_only_apply_to_cheapest_n_matches',
+            'benefit_same_products',
+            'benefit_limit_products',
+            'benefit_apply_to_addons',
+            'benefit_ignore_voucher_discounted',
         ]
         field_classes = {
             'available_from': SplitDateTimeField,
             'available_until': SplitDateTimeField,
             'condition_limit_products': ItemMultipleChoiceField,
+            'benefit_limit_products': ItemMultipleChoiceField,
         }
         widgets = {
             'subevent_mode': forms.RadioSelect,
@@ -64,11 +69,14 @@ class DiscountForm(I18nModelForm):
                 'data-inverse-dependency': '<[name$=all_products]',
                 'class': 'scrolling-multiple-choice',
             }),
+            'benefit_limit_products': forms.CheckboxSelectMultiple(attrs={
+                'class': 'scrolling-multiple-choice',
+            }),
             'benefit_only_apply_to_cheapest_n_matches': forms.NumberInput(
                 attrs={
                     'data-display-dependency': '#id_condition_min_count',
                 }
-            )
+            ),
         }
 
     def __init__(self, *args, **kwargs):
@@ -85,6 +93,7 @@ class DiscountForm(I18nModelForm):
             widget=forms.CheckboxSelectMultiple,
         )
         self.fields['condition_limit_products'].queryset = self.event.items.all()
+        self.fields['benefit_limit_products'].queryset = self.event.items.all()
         self.fields['condition_min_count'].required = False
         self.fields['condition_min_count'].widget.is_required = False
         self.fields['condition_min_value'].required = False

src/pretix/control/forms/event.py

@@ -38,6 +38,7 @@ from decimal import Decimal
 from urllib.parse import urlencode, urlparse
 from zoneinfo import ZoneInfo
 
+import pycountry
 from django import forms
 from django.conf import settings
 from django.core.exceptions import NON_FIELD_ERRORS, ValidationError
@@ -65,7 +66,8 @@ from pretix.base.models import Event, Organizer, TaxRule, Team
 from pretix.base.models.event import EventFooterLink, EventMetaValue, SubEvent
 from pretix.base.reldate import RelativeDateField, RelativeDateTimeField
 from pretix.base.settings import (
-    PERSON_NAME_SCHEMES, PERSON_NAME_TITLE_GROUPS, validate_event_settings,
+    COUNTRIES_WITH_STATE_IN_ADDRESS, PERSON_NAME_SCHEMES,
+    PERSON_NAME_TITLE_GROUPS, validate_event_settings,
 )
 from pretix.base.validators import multimail_validate
 from pretix.control.forms import (
@@ -749,6 +751,7 @@ class PaymentSettingsForm(EventSettingsValidationMixin, SettingsForm):
         'payment_term_minutes',
         'payment_term_last',
         'payment_term_expire_automatically',
+        'payment_term_expire_delay_days',
         'payment_term_accept_late',
         'payment_pending_hidden',
         'payment_explanation',
@@ -1115,6 +1118,16 @@ class MailSettingsForm(SettingsForm):
         help_text=_("This email only applies to payment methods that can receive incomplete payments, "
                     "such as bank transfer."),
     )
+    mail_subject_order_payment_failed = I18nFormField(
+        label=_("Subject"),
+        required=False,
+        widget=I18nTextInput,
+    )
+    mail_text_order_payment_failed = I18nFormField(
+        label=_("Text"),
+        required=False,
+        widget=I18nTextarea,
+    )
     mail_subject_waiting_list = I18nFormField(
         label=_("Subject"),
         required=False,
@@ -1262,8 +1275,12 @@ class MailSettingsForm(SettingsForm):
         'mail_subject_order_placed_require_approval': ['event', 'order'],
         'mail_text_order_approved': ['event', 'order'],
         'mail_subject_order_approved': ['event', 'order'],
+        'mail_text_order_approved_attendee': ['event', 'order'],
+        'mail_subject_order_approved_attendee': ['event', 'order'],
         'mail_text_order_approved_free': ['event', 'order'],
         'mail_subject_order_approved_free': ['event', 'order'],
+        'mail_text_order_approved_free_attendee': ['event', 'order'],
+        'mail_subject_order_approved_free_attendee': ['event', 'order'],
         'mail_text_order_denied': ['event', 'order', 'comment'],
         'mail_subject_order_denied': ['event', 'order', 'comment'],
         'mail_text_order_paid': ['event', 'order', 'payment_info'],
@@ -1284,6 +1301,8 @@ class MailSettingsForm(SettingsForm):
         'mail_subject_order_pending_warning': ['event', 'order'],
         'mail_text_order_incomplete_payment': ['event', 'order', 'pending_sum'],
         'mail_subject_order_incomplete_payment': ['event', 'order'],
+        'mail_text_order_payment_failed': ['event', 'order'],
+        'mail_subject_order_payment_failed': ['event', 'order'],
         'mail_text_order_custom_mail': ['event', 'order'],
         'mail_text_download_reminder': ['event', 'order'],
         'mail_subject_download_reminder': ['event', 'order'],
@@ -1396,7 +1415,7 @@ class CommentForm(I18nModelForm):
         fields = ['comment']
         widgets = {
             'comment': forms.Textarea(attrs={
-                'rows': 3,
+                'rows': 6,
                 'class': 'helper-width-100',
             }),
         }
@@ -1411,9 +1430,20 @@ class CountriesAndEU(CachedCountries):
     cache_subkey = 'with_any_or_eu'
 
 
+class CountriesAndEUAndStates(CountriesAndEU):
+    def __iter__(self):
+        for country_code, country_name in super().__iter__():
+            yield country_code, country_name
+            if country_code in COUNTRIES_WITH_STATE_IN_ADDRESS:
+                types, form = COUNTRIES_WITH_STATE_IN_ADDRESS[country_code]
+                yield from sorted(((state.code, country_name + " - " + state.name)
+                                   for state in pycountry.subdivisions.get(country_code=country_code)
+                                   if state.type in types), key=lambda s: s[1])
+
+
 class TaxRuleLineForm(I18nForm):
     country = LazyTypedChoiceField(
-        choices=CountriesAndEU(),
+        choices=CountriesAndEUAndStates(),
         required=False
     )
     address_type = forms.ChoiceField(

src/pretix/control/forms/filter.py

@@ -573,6 +573,11 @@ class EventOrderExpertFilterForm(EventOrderFilterForm):
         label=_('Sales channel'),
         required=False,
     )
+    has_checkin = forms.NullBooleanField(
+        required=False,
+        widget=FilterNullBooleanSelect,
+        label=_('At least one ticket with check-in'),
+    )
     checkin_attention = forms.NullBooleanField(
         required=False,
         widget=FilterNullBooleanSelect,
@@ -745,6 +750,12 @@ class EventOrderExpertFilterForm(EventOrderFilterForm):
             qs = qs.filter(
                 all_positions__country=fdata.get('attendee_address_country')
             ).distinct()
+        if fdata.get('has_checkin') is not None:
+            qs = qs.annotate(
+                has_checkin=Exists(
+                    Checkin.all.filter(position__order_id=OuterRef('pk'))
+                )
+            ).filter(has_checkin=fdata['has_checkin'])
         if fdata.get('ticket_secret'):
             qs = qs.filter(
                 all_positions__secret__icontains=fdata.get('ticket_secret')
@@ -1721,8 +1732,8 @@ class CheckinListAttendeeFilterForm(FilterForm):
         '-timestamp': (OrderBy(F('last_entry'), nulls_last=True, descending=True), '-order__code'),
         'item': ('item__name', 'variation__value', 'order__code'),
         '-item': ('-item__name', '-variation__value', '-order__code'),
-        'seat': ('seat__sorting_rank', 'seat__guid'),
-        '-seat': ('-seat__sorting_rank', '-seat__guid'),
+        'seat': ('seat__sorting_rank', 'seat__seat_guid'),
+        '-seat': ('-seat__sorting_rank', '-seat__seat_guid'),
         'date': ('subevent__date_from', 'subevent__id', 'order__code'),
         '-date': ('-subevent__date_from', 'subevent__id', '-order__code'),
         'name': {'_order': F('display_name').asc(nulls_first=True),
@@ -1743,9 +1754,9 @@ class CheckinListAttendeeFilterForm(FilterForm):
         label=_('Check-in status'),
         choices=(
             ('', _('All attendees')),
-            ('3', pgettext_lazy('checkin state', 'Checked in but left')),
-            ('2', pgettext_lazy('checkin state', 'Present')),
             ('1', _('Checked in')),
+            ('2', pgettext_lazy('checkin state', 'Present')),
+            ('3', pgettext_lazy('checkin state', 'Checked in but left')),
             ('0', _('Not checked in')),
         ),
         required=False,
@@ -1929,7 +1940,7 @@ class VoucherFilterForm(FilterForm):
             'item__category__position',
             'item__category',
             'item__position',
-            'item__variation__position',
+            'variation__position',
             'quota__name',
         ),
         'subevent': 'subevent__date_from',
@@ -1939,7 +1950,7 @@ class VoucherFilterForm(FilterForm):
             '-item__category__position',
             '-item__category',
             '-item__position',
-            '-item__variation__position',
+            '-variation__position',
             '-quota__name',
         )
     }