Fix #678 -- Data shredders for personally identifiable information (#817)

* Add data shredders for PII

* First working shredder

* Add more shredders

* Add new shredders and download confirmation

* tmp

* PayPal, Stripe, banktransfer

* Add icon to logs

* Untested payment log shredders

* Add waiting list shredder

* First tests

* Add tests for shredders

* Improve templats, link to shredder

* Test payment info shredders

* More tests

* Documentation

* Fix enabled flag in payment provider overview

* Fix minor issues

doc/development/api/index.rst

@@ -11,5 +11,7 @@ Contents:
    ticketoutput
    payment
    invoice
+   shredder
    customview
    general
+   quality

doc/development/api/payment.rst

@@ -104,6 +104,8 @@ The provider class
 
    .. automethod:: is_implicit
 
+   .. automethod:: shred_payment_info
+
 
 Additional views
 ----------------

doc/development/api/quality.rst

@@ -0,0 +1,125 @@
+.. highlight:: python
+   :linenothreshold: 5
+
+.. _`pluginquality`:
+
+Plugin quality checklist
+========================
+
+If you want to write a high-quality pretix plugin, this is a list of things you should check before
+you publish it. This is also a list of things that we check, if we consider installing an externally
+developed plugin on our hosted infrastructure.
+
+A. Meta
+-------
+
+#. The plugin is clearly licensed under an appropriate license.
+
+#. The plugin has an unambiguous name, description, and author metadata.
+
+#. The plugin has a clear versioning scheme and the latest version of the plugin is kept compatible to the latest
+   stable version of pretix.
+
+#. The plugin is properly packaged using standard Python packaging tools.
+
+#. The plugin correctly declares its external dependencies.
+
+#. A contact address is provided in case of security issues.
+
+B. Isolation
+------------
+
+#. If any signal receivers use the `dispatch_uid`_ feature, the UIDs are prefixed by the plugin's name and do not
+   clash with other plugins.
+
+#. If any templates or static files are shipped, they are located in subdirectories with the name of the plugin and do
+   not clash with other plugins or core files.
+
+#. Any keys stored to the settings store are prefixed with the plugin's name and do not clash with other plugins or
+   core.
+
+#. Any keys stored to the user session are prefixed with the plugin's name and do not clash with other plugins or
+   core.
+
+#. Any registered URLs are unlikely to clash with other plugins or future core URLs.
+
+C. Security
+-----------
+
+#. All important actions are logged to the :ref:`shared log storage <logging>` and a signal receiver is registered to
+   provide a human-readable representation of the log entry.
+
+#. All views require appropriate permissions and use the ``event_urls`` mechanism if appropriate.
+   :ref:`Read more <customview>`
+
+#. Any session data for customers is stored in the cart session system if appropriate.
+
+#. If the plugin is a payment provider:
+
+  #. No credit card numbers may be stored within pretix.
+
+  #. A notification/webhook system is implemented to notify pretix of any refunds.
+
+  #. If such a webhook system is implemented, contents of incoming webhooks are either verified using a cryptographic
+     signature or are not being trusted and all data is fetched from an API instead.
+
+D. Privacy
+----------
+
+#. No personal data is stored that is not required for the plugin's functionality.
+
+#. For any personal data that is saved to the database, an appropriate :ref:`data shredder <shredder>` is provided
+   that offers the data for download and then removes it from the database (including log entries).
+
+E. Internationalization
+-----------------------
+
+#. All user-facing strings in templates, Python code, and templates are wrapped in `gettext calls`_.
+
+#. No languages, time zones, date formats, or time formats are hardcoded.
+
+#. Installing the plugin automatically compiles ``.po`` files to ``.mo`` files. This is fulfilled automatically if
+   you use the ``setup.py`` file form our plugin cookiecutter.
+
+F. Functionality
+----------------
+
+#. If the plugin adds any database models or relationships from the settings storage to database models, it registers
+   a receiver to the :py:attr:`pretix.base.signals.event_copy_data` or :py:attr:`pretix.base.signals.item_copy_data`
+   signals.
+
+#. If the plugin is a payment provider:
+
+    #. A webhook-like system is implemented if payment confirmations are not sent instantly.
+
+    #. Refunds are implemented, if possible.
+
+    #. In case of overpayment or external refunds, a "required action" is created to notify the event organizer.
+
+#. If the plugin adds steps to the checkout process, it has been tested in combination with the pretix widget.
+
+G. Code quality
+---------------
+
+#. `isort`_ and `flake8`_ are used to ensure consistent code styling.
+
+#. Unit tests are provided for important pieces of business logic.
+
+#. Functional tests are provided for important interface parts.
+
+#. Tests are provided to check that permission checks are working.
+
+#. Continuous Integration is set up to check that tests are passing and styling is consistent.
+
+H. Specific to pretix.eu
+------------------------
+
+#. pretix.eu integrates the data stored by this plugin with its data report features.
+
+#. pretix.eu integrates this plugin in its generated privacy statements, if necessary.
+
+
+.. _isort: https://www.google.de/search?q=isort&oq=isort&aqs=chrome..69i57j0j69i59j69i60l2j69i59.599j0j4&sourceid=chrome&ie=UTF-8
+.. _flake8: http://flake8.pycqa.org/en/latest/
+.. _gettext calls: https://docs.djangoproject.com/en/2.0/topics/i18n/translation/
+.. _dispatch_uid: https://docs.djangoproject.com/en/2.0/topics/signals/#django.dispatch.Signal.connect

doc/development/api/shredder.rst

@@ -0,0 +1,94 @@
+.. highlight:: python
+   :linenothreshold: 5
+
+.. _`shredder`:
+
+Writing a data shredder
+=======================
+
+If your plugin adds the ability to store personal data within pretix, you should also implement a "data shredder"
+to anonymize or pseudonymize the data later.
+
+Shredder registration
+---------------------
+
+The data shredder API does not make a lot of usage from signals, however, it
+does use a signal to get a list of all available data shredders. Your plugin
+should listen for this signal and return the subclass of ``pretix.base.shredder.BaseDataShredder``
+that we'll provide in this plugin:
+
+.. sourcecode:: python
+
+    from django.dispatch import receiver
+
+    from pretix.base.signals import register_data_shredders
+
+
+    @receiver(register_data_shredders, dispatch_uid="custom_data_shredders")
+    def register_shredder(sender, **kwargs):
+        return [
+            PluginDataShredder,
+        ]
+
+The shredder class
+------------------
+
+.. class:: pretix.base.shredder.BaseDataShredder
+
+   The central object of each invoice renderer is the subclass of ``BaseInvoiceRenderer``.
+
+   .. py:attribute:: BaseInvoiceRenderer.event
+
+      The default constructor sets this property to the event we are currently
+      working for.
+
+   .. autoattribute:: identifier
+
+      This is an abstract attribute, you **must** override this!
+
+   .. autoattribute:: verbose_name
+
+      This is an abstract attribute, you **must** override this!
+
+   .. autoattribute:: description
+
+      This is an abstract attribute, you **must** override this!
+
+   .. automethod:: generate_files
+
+   .. automethod:: shred_data
+
+Example
+-------
+
+For example, the core data shredder responsible for removing invoice address information including their history
+looks like this:
+
+.. sourcecode:: python
+
+    class InvoiceAddressShredder(BaseDataShredder):
+        verbose_name = _('Invoice addresses')
+        identifier = 'invoice_addresses'
+        description = _('This will remove all invoice addresses from orders, '
+                        'as well as logged changes to them.')
+
+        def generate_files(self) -> List[Tuple[str, str, str]]:
+            yield 'invoice-addresses.json', 'application/json', json.dumps({
+                ia.order.code: InvoiceAdddressSerializer(ia).data
+                for ia in InvoiceAddress.objects.filter(order__event=self.event)
+            }, indent=4)
+
+        @transaction.atomic
+        def shred_data(self):
+            InvoiceAddress.objects.filter(order__event=self.event).delete()
+
+            for le in self.event.logentry_set.filter(action_type="pretix.event.order.modified"):
+                d = le.parsed_data
+                if 'invoice_data' in d and not isinstance(d['invoice_data'], bool):
+                    for field in d['invoice_data']:
+                        if d['invoice_data'][field]:
+                            d['invoice_data'][field] = '█'
+                    le.data = json.dumps(d)
+                    le.shredded = True
+                    le.save(update_fields=['data', 'shredded'])
+

doc/development/implementation/logging.rst

@@ -4,6 +4,8 @@ Logging and notifications
 As pretix is handling monetary transactions, we are very careful to make it possible to review all changes
 in the system that lead to the current state.
 
+.. _`logging`:
+
 Logging changes
 ---------------