Improved and documented i18n and background tasks

doc/development/api/general.rst

@@ -29,6 +29,9 @@ on the type of navigation. You should also return an ``active`` key with a boole
 set to ``True``, when this item should be marked as active. The ``request`` object
 will have an attribute ``event``.
 
+If you use this, you should read the documentation on :ref:`how to deal with URLs <urlconf>`
+in pretix.
+
 ``pretix.control.signals.nav_event``
     The sidebar navigation when the admin has selected an event.
 

doc/development/api/index.rst

@@ -1,5 +1,5 @@
-Plugin API
+Plugin hooks
-==========
+============
 
 Contents:
 

doc/development/implementation/background.rst

@@ -0,0 +1,96 @@
+Background tasks
+================
+
+pretix provides the ability to run all longer-running tasks like generating ticket files or sending emails
+in a background thread instead of the webserver process. We use the well-established `Celery`_ project to
+implement this. However, as celery requires running a task queue like RabbitMQ and a result storage such as
+Redis to work efficiently, we don't like to *depend* on celery being available to make small-scale installations
+of pretix more straightforward. For this reason, the "background" in "background task" is always optional.
+
+The Django settings variable ``settings.HAS_CELERY`` provides information on whether celery is configured
+in the current installation.
+
+Implementing a task
+-------------------
+
+A common pattern for implementing "optionally-asynchronous" tasks that can be seen a lot in ``pretix.base.services``
+looks like this::
+
+    def my_task(argument1, argument2):
+        # Important: All arguments and return values need to be serializable into JSON.
+        # Do not use model instances, use their primary keys instead!
+        pass  # do your work here
+
+
+    if settings.HAS_CELERY:
+        # Transform this into a background task
+        from pretix.celery import app  # Important: Do not import this unconditionally!
+
+        my_task_async = app.task(export)
+
+        def my_task(*args, **kwargs):
+            my_task_async.apply_async(args=args, kwargs=kwargs)
+
+This explicit declaration method also allows you to place some custom retry logic etc. in the asynchronous version.
+
+Tasks in the request-response flow
+----------------------------------
+
+If your user needs to wait for the response of the asynchronous task, there are helpers available in ``pretix.presale``
+that will probably move to ``pretix.base`` at some point. They consist of the view mixin ``AsyncAction`` that allows
+you to easily write a view that kicks off and waits for an asynchronous task. ``AsyncAction`` will determine whether
+to run the task asynchronously or not and will do some magic to look nice for users with and without JavaScript support.
+A usage example taken directly from the code is::
+
+    class OrderCancelDo(EventViewMixin, OrderDetailMixin, AsyncAction, View):
+        """
+        A view that executes a task asynchronously. A POST request will kick of the
+        task into the background or run it in the foreground, if celery is not installed.
+        In the former case, subsequent GET calls can be used to determinine the current
+        status of the task.
+        """
+
+        task = cancel_order  # The task to be used, defined like above
+
+        def get_success_url(self, value):
+            """
+            Returns the URL the user will be redirected to if the task succeeded.
+            """
+            return self.get_order_url()
+
+        def get_error_url(self):
+            """
+            Returns the URL the user will be redirected to if the task failed.
+            """
+            return self.get_order_url()
+
+        def post(self, request, *args, **kwargs):
+            """
+            Will be called while handling a POST request. This should process the
+            request arguments in some way and call ``self.do`` with the task arguments
+            to kick of the task.
+            """
+            if not self.order:
+                raise Http404(_('Unknown order code or not authorized to access this order.'))
+            return self.do(self.order.pk)
+
+        def get_error_message(self, exception):
+            """
+            Returns the message that will be shown to the user if the task has failed.
+            """
+            if isinstance(exception, dict) and exception['exc_type'] == 'OrderError':
+                return gettext(exception['exc_message'])
+            elif isinstance(exception, OrderError):
+                return str(exception)
+            return super().get_error_message(exception)
+
+On the client side, this can be used by simply adding a ``data-asynctask`` attribute to an HTML form. This will enable
+AJAX sending of the form and display a loading indicator::
+
+    <form method="post" data-asynctask
+          action="{% eventurl request.event "presale:event.order.cancel.do" … %}">
+        {% csrf_token %}
+        ...
+    </form>
+
+.. _Celery: http://www.celeryproject.org/

doc/development/implementation/i18n.rst

@@ -0,0 +1,72 @@
+Internationalization
+====================
+
+One of pretix' major selling points is it's multi-language capability. We make heavy use of Django's
+`translation features`_ that are built upon `GNU gettext`_. However, Django does not provide a standard
+way to translate *user-generated content*. In our case, we need to translate strings like product names
+or event descriptions, so we need event organizers to be able to fill in all fields in multiple languages
+at the same time.
+
+.. note:: Implementing object-level translation in a relational database is a task that requires some difficult
+          trade-off. We decided for a design that is not elegant on the database level (as it violates the `1NF`_) and
+          makes searching in the respective database fields very hard, but allows for a simple design on the ORM level
+          and adds only minimal performance overhead.
+
+All classes and functions introduced in this document are located in ``pretix.base.i18n`` if not stated otherwise.
+
+Database storage
+----------------
+
+pretix provides two custom model field types that allow you to work with localized strings: ``I18nCharField`` and
+``I18nTextField``. Both of them are stored in the database as a ``TextField`` internally, they only differ in the
+default form widget that is used by ``ModelForm``.
+
+Yes, we know that this has negative impact on performance when indexing or searching them, but as mentioned above,
+within pretix this is not used in places that need to be searched. Lookups are currently not even implemented on these
+fields. In the database, the strings will be stored as a JSON-encoded mapping of language codes to strings.
+
+Whenever you interact with those fields, you will either provide or receive an instance of the following class:
+
+.. autoclass:: pretix.base.i18n.LazyI18nString
+   :members: __init__, localize, __str__
+
+
+Forms
+-----
+
+We provide i18n-aware versions of the respective form fields and widgets: ``I18nFormField`` with the ``I18nTextInput``
+and ``I18nTextarea`` widgets. They transparently allow you to use ``LazyI18nString`` values in forms and render text
+inputs for multiple languages.
+
+.. autoclass:: pretix.base.i18n.I18nFormField
+
+To easily limit the displayed languages to the languages relevant to an event, there is a custom ``ModelForm`` subclass
+that deals with this for you:
+
+.. autoclass:: pretix.base.forms.I18nModelForm
+
+There are equivalents for ``BaseModelFormSet`` and ``BaseInlineFormSet``:
+
+.. autoclass:: pretix.base.forms.I18nFormSet
+
+.. autoclass:: pretix.base.forms.I18nInlineFormSet
+
+Useful utilities
+----------------
+
+The ``i18n`` module contains a few more useful utilities, starting with simple lazy-evaluation wrappers for formatted
+numbers and dates, ``LazyDate`` and ``LazyNumber``. There also is a ``LazyLocaleException`` base class that provides
+exceptions with gettext-localized exception messages.
+
+Last, but definitely not least, we have the ``language`` context manager that allows you to execute a piece of code with
+a different locale::
+
+    with language('de'):
+        render_mail_template()
+
+This is very useful e.g. when sending an email to a user that has a different language than the user performing the
+action that causes the mail to be sent.
+
+.. _translation features: https://docs.djangoproject.com/en/1.9/topics/i18n/translation/
+.. _GNU gettext: https://www.gnu.org/software/gettext/
+.. _1NF: https://en.wikipedia.org/wiki/First_normal_form

doc/development/implementation/index.rst

@@ -0,0 +1,16 @@
+Implementation and Utilities
+============================
+
+This chapter describes the various inner workings that power pretix, most of them living in ``pretix.base``.
+If you want to develop around pretix' core or advanced plugins, this aims to describe everything you absolutely
+need to know.
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   models
+   background
+   urlconfig
+   i18n

doc/development/implementation/models.rst

@@ -1,10 +1,10 @@
 .. highlight:: python
    :linenothreshold: 5
 
-Data models
+Data model
-===========
+==========
 
-Pretix provides the following data(base) models. Every model and every model method or field that is not
+pretix provides the following data(base) models. Every model and every model method or field that is not
 documented here is considered private and should not be used by third-party plugins, as it may change
 without advance notice.
 

doc/development/index.rst

@@ -10,9 +10,8 @@ Contents:
    setup
    structure
    contribution/index
-   models
+   implementation/index
    api/index
-   urlconfig
 
 .. TODO::
    Document settings objects, ItemVariation objects, form fields.

src/pretix/base/forms/__init__.py

@@ -1,12 +1,12 @@
-import copy
 import logging
-import os
 
 from django import forms
 from django.core.files import File
 from django.core.files.storage import default_storage
 from django.core.files.uploadedfile import UploadedFile
-from django.forms.models import BaseModelForm, ModelFormMetaclass
+from django.forms.models import (
+    BaseInlineFormSet, BaseModelForm, BaseModelFormSet, ModelFormMetaclass,
+)
 from django.utils import six
 from django.utils.translation import ugettext_lazy as _
 
@@ -18,7 +18,7 @@ logger = logging.getLogger('pretix.plugins.ticketoutputpdf')
 
 class BaseI18nModelForm(BaseModelForm):
     """
-    This is a helperclass to construct I18nModelForm
+    This is a helperclass to construct an I18nModelForm.
     """
     def __init__(self, *args, **kwargs):
         event = kwargs.pop('event', None)
@@ -33,13 +33,54 @@ class I18nModelForm(six.with_metaclass(ModelFormMetaclass, BaseI18nModelForm)):
     """
     This is a modified version of Django's ModelForm which differs from ModelForm in
     only one way: The constructor takes one additional optional argument ``event``
-    which may be given an :pyclass:`pretix.base.models.Event` instance. If given, this
+    which may be given an `Event` instance. If given, this instance is used to select
-    instance is used to select the visible languages in all I18nFormFields of the form. If
+    the visible languages in all I18nFormFields of the form. If not given, all languages
-    not given, all languages will be displayed.
+    will be displayed.
     """
     pass
 
 
+class I18nFormSet(BaseModelFormSet):
+    """
+    This is equivalent to a normal BaseModelFormset, but cares for the special needs
+    of I18nForms (see there for more information).
+    """
+
+    def __init__(self, *args, **kwargs):
+        self.event = kwargs.pop('event', None)
+        super().__init__(*args, **kwargs)
+
+    def _construct_form(self, i, **kwargs):
+        kwargs['event'] = self.event
+        return super()._construct_form(i, **kwargs)
+
+    @property
+    def empty_form(self):
+        form = self.form(
+            auto_id=self.auto_id,
+            prefix=self.add_prefix('__prefix__'),
+            empty_permitted=True,
+            event=self.event
+        )
+        self.add_fields(form, None)
+        return form
+
+
+class I18nInlineFormSet(BaseInlineFormSet):
+    """
+    This is equivalent to a normal BaseInlineFormset, but cares for the special needs
+    of I18nForms (see there for more information).
+    """
+
+    def __init__(self, *args, **kwargs):
+        self.event = kwargs.pop('event', None)
+        super().__init__(*args, **kwargs)
+
+    def _construct_form(self, i, **kwargs):
+        kwargs['event'] = self.event
+        return super()._construct_form(i, **kwargs)
+
+
 class SettingsForm(forms.Form):
     """
     This form is meant to be used for modifying Event- or OrganizerSettings

src/pretix/base/i18n.py

@@ -6,7 +6,6 @@ from django import forms
 from django.conf import settings
 from django.core.serializers.json import DjangoJSONEncoder
 from django.db.models import Model, QuerySet, TextField
-from django.forms import BaseModelFormSet
 from django.utils import translation
 from django.utils.formats import date_format, number_format
 from django.utils.safestring import mark_safe
@@ -21,7 +20,11 @@ class LazyI18nString:
 
     def __init__(self, data: Optional[Union[str, Dict[str, str]]]):
         """
-        Input data should be a dictionary which maps language codes to content.
+        Creates a new i18n-aware string.
+
+        :param data: If this is a dictionary, it is expected to map language codes to translations.
+            If this is a string that can be parsed as JSON, it will be parsed and used as such a dictionary.
+            If this is anything else, it will be casted to a string and used for all languages.
         """
         self.data = data
         if isinstance(self.data, str) and self.data is not None:
@@ -35,8 +38,10 @@ class LazyI18nString:
     def __str__(self) -> str:
         """
         Evaluate the given string with respect to the currently active locale.
-        This will rather return you a string in a wrong language than give you an
+
-        empty value.
+        If no string is available in the currently active language, this will give you
+        the string in the system's default language. If this is unavailable as well, it
+        will give you the string in the first language available.
         """
         return self.localize(translation.get_language())
 
@@ -47,13 +52,24 @@ class LazyI18nString:
             return any(self.data.values())
         return True
 
-    def localize(self, lng):
+    def localize(self, lng: str) -> str:
+        """
+        Evaluate the given string with respect to the locale defined by ``lng``.
+
+        If no string is available in the currently active language, this will give you
+        the string in the system's default language. If this is unavailable as well, it
+        will give you the string in the first language available.
+
+        :param lng: A locale code, e.g. ``de``. If you specify a code including a country
+            or region like ``de-AT``, exact matches will be used preferably, but if only
+            a ``de`` or ``de-AT`` translation exists, this might be returned as well.
+        """
         if self.data is None:
             return ""
 
         if isinstance(self.data, dict):
             firstpart = lng.split('-')[0]
-            similar = [l for l in self.data.keys() if l.startswith(firstpart + "-")]
+            similar = [l for l in self.data.keys() if l.startswith(firstpart + "-") or firstpart == l]
             if lng in self.data and self.data[lng]:
                 return self.data[lng]
             elif firstpart in self.data:
@@ -181,6 +197,14 @@ class I18nFormField(forms.MultiValueField):
     The form field that is used by I18nCharField and I18nTextField. It makes use
     of Django's MultiValueField mechanism to create one sub-field per available
     language.
+
+    It contains special treatment to make sure that a field marked as "required" is validated
+    as "filled out correctly" if *at least one* translation is filled it. It is never required
+    to fill in all of them. This has the drawback that the HTML property ``required`` is set on
+    none of the fields as this would lead to irritating behaviour.
+
+    :param langcodes: An iterable of locale codes that the widget should render a field for. If
+        omitted, fields will be rendered for all languages supported by pretix.
     """
 
     def compress(self, data_list):
@@ -299,32 +323,6 @@ class I18nJSONEncoder(DjangoJSONEncoder):
             return super().default(obj)
 
 
-class I18nFormSet(BaseModelFormSet):
-    """
-    This is equivalent to a normal BaseModelFormset, but cares for the special needs
-    of I18nForms (see there for more information).
-    """
-
-    def __init__(self, *args, **kwargs):
-        self.event = kwargs.pop('event', None)
-        super().__init__(*args, **kwargs)
-
-    def _construct_form(self, i, **kwargs):
-        kwargs['event'] = self.event
-        return super()._construct_form(i, **kwargs)
-
-    @property
-    def empty_form(self):
-        form = self.form(
-            auto_id=self.auto_id,
-            prefix=self.add_prefix('__prefix__'),
-            empty_permitted=True,
-            event=self.event
-        )
-        self.add_fields(form, None)
-        return form
-
-
 class LazyDate:
     def __init__(self, value):
         self.value = value

src/pretix/control/forms/__init__.py

@@ -1,62 +1,9 @@
 import os
-from functools import partial
-from itertools import product
 
 from django import forms
-from django.core.exceptions import ValidationError
-from django.db import transaction
-from django.forms import (
-    BaseInlineFormSet, BaseModelFormSet, ModelForm, modelformset_factory,
-)
-from django.forms.widgets import flatatt
-from django.utils.encoding import force_text
-from django.utils.html import format_html
-from django.utils.safestring import mark_safe
 from django.utils.translation import ugettext_lazy as _
 
-from pretix.base.forms import I18nModelForm
+from ...base.forms import I18nModelForm
-from pretix.base.models import Item, ItemVariation
-
-
-class I18nInlineFormSet(BaseInlineFormSet):
-    """
-    This is equivalent to a normal BaseInlineFormset, but cares for the special needs
-    of I18nForms (see there for more information).
-    """
-
-    def __init__(self, *args, **kwargs):
-        self.event = kwargs.pop('event', None)
-        super().__init__(*args, **kwargs)
-
-    def _construct_form(self, i, **kwargs):
-        kwargs['event'] = self.event
-        return super()._construct_form(i, **kwargs)
-
-
-class I18nFormSet(BaseModelFormSet):
-    """
-    This is equivalent to a normal BaseModelFormset, but cares for the special needs
-    of I18nForms (see there for more information).
-    """
-
-    def __init__(self, *args, **kwargs):
-        self.event = kwargs.pop('event', None)
-        super().__init__(*args, **kwargs)
-
-    def _construct_form(self, i, **kwargs):
-        kwargs['event'] = self.event
-        return super()._construct_form(i, **kwargs)
-
-    @property
-    def empty_form(self):
-        form = self.form(
-            auto_id=self.auto_id,
-            prefix=self.add_prefix('__prefix__'),
-            empty_permitted=True,
-            event=self.event
-        )
-        self.add_fields(form, None)
-        return form
 
 
 class TolerantFormsetModelForm(I18nModelForm):

src/pretix/control/views/item.py

@@ -12,7 +12,7 @@ from django.views.generic.base import TemplateView
 from django.views.generic.detail import SingleObjectMixin
 from django.views.generic.edit import DeleteView
 
-from pretix.base.i18n import I18nFormSet
+from pretix.base.forms import I18nFormSet
 from pretix.base.models import (
     Item, ItemCategory, ItemVariation, Question, QuestionOption, Quota,
 )