Deprecate MySQL support (#3017)

doc/admin/config.rst

@@ -141,7 +141,7 @@ Database settings
 Example::
 
     [database]
-    backend=mysql
+    backend=postgresql
     name=pretix
     user=pretix
     password=abcd
@@ -149,7 +149,7 @@ Example::
     port=3306
 
 ``backend``
-    One of ``mysql``, ``sqlite3``, ``oracle`` and ``postgresql``.
+    One of ``mysql`` (deprecated), ``sqlite3`` and ``postgresql``.
     Default: ``sqlite3``.
 
     If you use MySQL, be sure to create your database using
@@ -163,7 +163,7 @@ Example::
     Connection details for the database connection. Empty by default.
 
 ``galera``
-    Indicates if the database backend is a MySQL/MariaDB Galera cluster and
+    (Deprecated) Indicates if the database backend is a MySQL/MariaDB Galera cluster and
     turns on some optimizations/special case handlers. Default: ``False``
 
 .. _`config-replica`:

doc/admin/index.rst

@@ -14,4 +14,5 @@ This documentation is for everyone who wants to install pretix on a server.
    maintainance
    scaling
    errors
+   mysql2postgres
    indexes

doc/admin/installation/docker_smallscale.rst

@@ -14,7 +14,7 @@ This has some trade-offs in terms of performance and isolation but allows a rath
              get it right. If you're not feeling comfortable managing a Linux server, check out our hosting and service
              offers at `pretix.eu`_.
 
-We tested this guide on the Linux distribution **Debian 8.0** but it should work very similar on other
+We tested this guide on the Linux distribution **Debian 11.0** but it should work very similar on other
 modern distributions, especially on all systemd-based ones.
 
 Requirements
@@ -26,7 +26,7 @@ installation guides):
 * `Docker`_
 * A SMTP server to send out mails, e.g. `Postfix`_ on your machine or some third-party server you have credentials for
 * A HTTP reverse proxy, e.g. `nginx`_ or Apache to allow HTTPS connections
-* A `PostgreSQL`_ 9.6+, `MySQL`_ 5.7+, or MariaDB 10.2.7+ database server
+* A `PostgreSQL`_ 9.6+ database server
 * A `redis`_ server
 
 We also recommend that you use a firewall, although this is not a pretix-specific recommendation. If you're new to
@@ -58,9 +58,6 @@ directory writable to the user that runs pretix inside the docker container::
 Database
 --------
 
-.. warning:: **Please use PostgreSQL for all new installations**. If you need to go for MySQL, make sure you run
-             **MySQL 5.7 or newer** or **MariaDB 10.2.7 or newer**.
-
 Next, we need a database and a database user. We can create these with any kind of database managing tool or directly on
 our database's shell. Please make sure that UTF8 is used as encoding for the best compatibility. You can check this with
 the following command::
@@ -86,13 +83,6 @@ Restart PostgreSQL after you changed these files::
 
 If you have a firewall running, you should also make sure that port 5432 is reachable from the ``172.17.0.1/16`` subnet.
 
-For MySQL, you can either also use network-based connections or mount the ``/var/run/mysqld/mysqld.sock`` socket into the docker container.
-When using MySQL, make sure you set the character set of the database to ``utf8mb4``, e.g. like this::
-
-    mysql > CREATE DATABASE pretix DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;
-
-You will also need to make sure that ``sql_mode`` in your ``my.cnf`` file does **not** include ``ONLY_FULL_GROUP_BY``.
-
 Redis
 -----
 
@@ -152,15 +142,13 @@ Fill the configuration file ``/etc/pretix/pretix.cfg`` with the following conten
     trust_x_forwarded_proto=on
 
     [database]
-    ; Replace postgresql with mysql for MySQL
     backend=postgresql
     name=pretix
     user=pretix
     ; Replace with the password you chose above
     password=*********
     ; In most docker setups, 172.17.0.1 is the address of the docker host. Adjust
-    ; this to wherever your database is running, e.g. the name of a linked container
+    ; this to wherever your database is running, e.g. the name of a linked container.
-    ; or of a mounted MySQL socket.
     host=172.17.0.1
 
     [mail]
@@ -212,8 +200,6 @@ named ``/etc/systemd/system/pretix.service`` with the following content::
     [Install]
     WantedBy=multi-user.target
 
-When using MySQL and socket mounting, you'll need the additional flag ``-v /var/run/mysqld:/var/run/mysqld`` in the command.
-
 You can now run the following commands
 to enable and start the service::
 
@@ -339,7 +325,6 @@ workers, e.g. ``docker run … taskworker -Q notifications --concurrency 32``.
 .. _nginx: https://botleg.com/stories/https-with-lets-encrypt-and-nginx/
 .. _Let's Encrypt: https://letsencrypt.org/
 .. _pretix.eu: https://pretix.eu/
-.. _MySQL: https://dev.mysql.com/doc/refman/5.7/en/linux-installation-apt-repo.html
 .. _PostgreSQL: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-20-04
 .. _redis: https://blog.programster.org/debian-8-install-redis-server/
 .. _ufw: https://en.wikipedia.org/wiki/Uncomplicated_Firewall

doc/admin/installation/manual_smallscale.rst

@@ -23,7 +23,7 @@ installation guides):
 
 * A SMTP server to send out mails, e.g. `Postfix`_ on your machine or some third-party server you have credentials for
 * A HTTP reverse proxy, e.g. `nginx`_ or Apache to allow HTTPS connections
-* A `PostgreSQL`_ 9.6+, `MySQL`_ 5.7+, or MariaDB 10.2.7+ database server
+* A `PostgreSQL`_ 9.6+ database server
 * A `redis`_ server
 * A `nodejs`_ installation
 
@@ -47,9 +47,6 @@ In this guide, all code lines prepended with a ``#`` symbol are commands that yo
 Database
 --------
 
-.. warning:: **Please use PostgreSQL for all new installations**. If you need to go for MySQL, make sure you run
-             **MySQL 5.7 or newer** or **MariaDB 10.2.7 or newer**.
-
 Having the database server installed, we still need a database and a database user. We can create these with any kind
 of database managing tool or directly on our database's shell. Please make sure that UTF8 is used as encoding for the
 best compatibility. You can check this with the following command::
@@ -61,12 +58,6 @@ For PostgreSQL database creation, we would do::
     # sudo -u postgres createuser pretix
     # sudo -u postgres createdb -O pretix pretix
 
-When using MySQL, make sure you set the character set of the database to ``utf8mb4``, e.g. like this::
-
-    mysql > CREATE DATABASE pretix DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci;
-
-You will also need to make sure that ``sql_mode`` in your ``my.cnf`` file does **not** include ``ONLY_FULL_GROUP_BY``.
-
 Package dependencies
 --------------------
 
@@ -74,7 +65,7 @@ To build and run pretix, you will need the following debian packages::
 
     # apt-get install git build-essential python-dev python3-venv python3 python3-pip \
                       python3-dev libxml2-dev libxslt1-dev libffi-dev zlib1g-dev libssl-dev \
-                      gettext libpq-dev libmariadb-dev libjpeg-dev libopenjp2-7-dev
+                      gettext libpq-dev libjpeg-dev libopenjp2-7-dev
 
 Config file
 -----------
@@ -97,16 +88,12 @@ Fill the configuration file ``/etc/pretix/pretix.cfg`` with the following conten
     trust_x_forwarded_proto=on
 
     [database]
-    ; For MySQL, replace with "mysql"
     backend=postgresql
     name=pretix
     user=pretix
-    ; For MySQL, enter the user password. For PostgreSQL on the same host,
+    ; For PostgreSQL on the same host, we don't need a password because we can use
-    ; we don't need one because we can use peer authentification if our
+    ; peer authentication if our PostgreSQL user matches our unix user.
-    ; PostgreSQL user matches our unix user.
     password=
-    ; For MySQL, use local socket, e.g. /var/run/mysqld/mysqld.sock
-    ; For a remote host, supply an IP address
     ; For local postgres authentication, you can leave it empty
     host=
 
@@ -140,10 +127,6 @@ We now install pretix, its direct dependencies and gunicorn::
 
     (venv)$ pip3 install pretix gunicorn
 
-If you're running MySQL, also install the client library::
-
-    (venv)$ pip3 install mysqlclient
-
 Note that you need Python 3.7 or newer. You can find out your Python version using ``python -V``.
 
 We also need to create a data directory::
@@ -344,7 +327,6 @@ Then, proceed like after any plugin installation::
 .. _nginx: https://botleg.com/stories/https-with-lets-encrypt-and-nginx/
 .. _Let's Encrypt: https://letsencrypt.org/
 .. _pretix.eu: https://pretix.eu/
-.. _MySQL: https://dev.mysql.com/doc/refman/5.7/en/linux-installation-apt-repo.html
 .. _PostgreSQL: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-20-04
 .. _redis: https://blog.programster.org/debian-8-install-redis-server/
 .. _ufw: https://en.wikipedia.org/wiki/Uncomplicated_Firewall

doc/admin/maintainance.rst

@@ -17,11 +17,11 @@ Backups
 There are essentially two things which you should create backups of:
 
 Database
-    Your SQL database (MySQL or PostgreSQL). This is critical and you should **absolutely
+    Your SQL database. This is critical and you should **absolutely always create automatic
-    always create automatic backups of your database**. There are tons of tutorials on the
+    backups of your database**. There are tons of tutorials on the internet on how to do this,
-    internet on how to do this, and the exact process depends on the choice of your database.
+    and the exact process depends on the choice of your database. For PostgreSQL, see the
-    For MySQL, see ``mysqldump`` and for PostgreSQL, see the ``pg_dump`` tool. You probably
+    ``pg_dump`` tool. You probably want to create a cronjob that does the backups for you on a
-    want to create a cronjob that does the backups for you on a regular schedule.
+    regular schedule.
 
 Data directory
     The data directory of your pretix configuration might contain some things that you should

doc/admin/mysql2postgres.rst

@@ -0,0 +1,148 @@
+.. highlight:: none
+
+Migrating from MySQL/MariaDB to PostgreSQL
+==========================================
+
+Our recommended database for all production installations is PostgreSQL. Support for MySQL/MariaDB will be removed in
+pretix 5.0.
+
+In order to follow this guide, your pretix installation needs to be a version that fully supports MySQL/MariaDB. If you
+already upgraded to pretix 5.0, downgrade back to the last 4.x release using ``pip``.
+
+.. note:: We have tested this guide carefully, but we can't assume any liability for its correctness. The data loss
+          risk should be low as long as pretix is not running while you do the migration. If you are a pretix Enterprise
+          customer, feel free to reach out in advance if you want us to support you along the way.
+
+Update database schema
+----------------------
+
+Before you start, make sure your database schema is up to date::
+
+    # sudo -u pretix -s
+    $ source /var/pretix/venv/bin/activate
+    (venv)$ python -m pretix migrate
+
+Install PostgreSQL
+------------------
+
+Now, install and set up a PostgreSQL server. For a local installation on Debian or Ubuntu, use::
+
+    # apt install postgresql
+
+Having the database server installed, we still need a database and a database user. We can create these with any kind
+of database managing tool or directly on our database's shell. Please make sure that UTF8 is used as encoding for the
+best compatibility. You can check this with the following command::
+
+    # sudo -u postgres psql -c 'SHOW SERVER_ENCODING'
+
+Without Docker
+""""""""""""""
+
+For our standard manual installation, create the database and user like this::
+
+    # sudo -u postgres createuser pretix
+    # sudo -u postgres createdb -O pretix pretix
+
+With Docker
+"""""""""""
+
+For our standard docker installation, create the database and user like this::
+
+    # sudo -u postgres createuser -P pretix
+    # sudo -u postgres createdb -O pretix pretix
+
+Make sure that your database listens on the network. If PostgreSQL on the same same host as docker, but not inside a docker container, we recommend that you just listen on the Docker interface by changing the following line in ``/etc/postgresql/<version>/main/postgresql.conf``::
+
+    listen_addresses = 'localhost,172.17.0.1'
+
+You also need to add a new line to ``/etc/postgresql/<version>/main/pg_hba.conf`` to allow network connections to this user and database::
+
+    host    pretix          pretix          172.17.0.1/16           md5
+
+Restart PostgreSQL after you changed these files::
+
+    # systemctl restart postgresql
+
+If you have a firewall running, you should also make sure that port 5432 is reachable from the ``172.17.0.1/16`` subnet.
+
+Of course, instead of all this you can also run a PostgreSQL docker container and link it to the pretix container.
+
+Stop pretix
+-----------
+
+To prevent any more changes to your data, stop pretix from running::
+
+    # systemctl stop pretix-web pretix-worker
+
+Change configuration
+--------------------
+
+Change the database configuration in your ``/etc/pretix/pretix.cfg`` file::
+
+    [database]
+    backend=postgresql
+    name=pretix
+    user=pretix
+    password=  ; only required for docker or remote database, can be kept empty for local auth
+    host=      ; set to 172.17.0.1 in docker setup, keep empty for local auth
+
+
+Create database schema
+-----------------------
+
+To create the schema in your new PostgreSQL database, use the following commands::
+
+    # sudo -u pretix -s
+    $ source /var/pretix/venv/bin/activate
+    (venv)$ python -m pretix migrate
+
+
+Migrate your data
+-----------------
+
+Install ``pgloader``::
+
+    # apt install pgloader
+
+Create a new file ``/tmp/pretix.load``, replacing the MySQL and PostgreSQL connection strings with the correct user names, passwords, and/or database names::
+
+    LOAD DATABASE
+        FROM mysql://pretix:password@localhost/pretix  -- replace with mysql://username:password@hostname/dbname
+        INTO postgresql:///pretix                      -- replace with dbname
+
+    WITH data only, include no drop, truncate, disable triggers,
+         create no indexes, drop indexes, reset sequences
+
+    ALTER SCHEMA 'pretix' RENAME TO 'public'           -- replace pretix with the name of the MySQL database
+
+    ALTER TABLE NAMES MATCHING ~/.*/
+        SET SCHEMA 'public'
+
+    SET timezone TO '+00:00'
+
+    SET PostgreSQL PARAMETERS
+         maintenance_work_mem to '128MB',
+         work_mem to '12MB';
+
+Then, run::
+
+    # sudo -u postgres pgloader /tmp/pretix.load
+
+The output should end with a table summarizing the results for every table. You can ignore warnings about type casts
+and missing constraints.
+
+Afterwards, delete the file again::
+
+    # rm -rf /tmp/pretix.load
+
+Start pretix
+------------
+
+Now, restart pretix. Maybe stop your MySQL server as a verification step that you are no longer using it::
+
+    # systemctl stop mariadb
+    # systemctl start pretix-web pretix-worker
+
+And you're done! After you've verified everything has been copied correctly, you can delete the old MySQL database.
+
+.. note:: Don't forget to update your backup process to back up your PostgreSQL database instead of your MySQL database now.

doc/admin/scaling.rst

@@ -42,7 +42,7 @@ A pretix installation usually consists of the following components which run per
 
 * ``pretix-worker`` is a Celery-based application that processes tasks that should be run asynchronously outside of the web application process.
 
-* A **SQL database** keeps all the important data and processes the actual transactions. We recommend using PostgreSQL, but MySQL/MariaDB works as well.
+* A **PostgreSQL database** keeps all the important data and processes the actual transactions.
 
 * A **web server** that terminates TLS and HTTP connections and forwards them to ``pretix-web``. In some cases, e.g. when serving static files, the web servers might return a response directly. We recommend using ``nginx``.
 

src/pretix/control/templates/pretixcontrol/base.html

@@ -423,6 +423,17 @@
                         </div>
                     {% endif %}
 
+                    {% if "mysql" in settings.DATABASES.default.ENGINE and not request.organizer %}
+                        <div class="alert alert-warning">
+                            {% blocktrans trimmed %}
+                                You are using MySQL or MariaDB as your database backend for pretix.
+                                Starting in pretix 5.0, these will no longer be supported and you will need to migrate to PostgreSQL.
+                                Please see the pretix administrator documentation for a migration guide, and the pretix 4.16
+                                release notes for more information.
+                            {% endblocktrans %}
+                        </div>
+                    {% endif %}
+
                     {% if debug_warning %}
                         <div class="alert alert-danger">
                             {% trans "pretix is running in debug mode. For security reasons, please never run debug mode on a production instance." %}