Merge pull request #19 from tbicr/dj30

django 3.0 support

CHANGES.md

@@ -1,5 +1,14 @@
 # django-pg-zero-downtime-migrations changelog
 
+## 0.8
+  - added django 3.0 support
+  - added concurrently index creation and removal operations
+  - added exclude constraint support as unsafe operation
+  - drop postgres 9.4 support
+  - drop django 2.0 support
+  - drop django 2.1 support
+  - drop deprecated `django_zero_downtime_migrations_postgres_backend` module
+
 ## 0.7
   - added python 3.8 support
   - added postgres specific indexes support

README.md

@@ -1,7 +1,7 @@
 [![PyPI](https://img.shields.io/pypi/v/django-pg-zero-downtime-migrations.svg)](https://pypi.org/project/django-pg-zero-downtime-migrations/)
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/django-pg-zero-downtime-migrations.svg)
 ![PyPI - Django Version](https://img.shields.io/pypi/djversions/django-pg-zero-downtime-migrations.svg?label=django)
-![Postgres Version](https://img.shields.io/badge/postgres-9.4%20|%209.5%20|%209.6%20|%2010%20|%2011%20|%2012%20-blue.svg)
+![Postgres Version](https://img.shields.io/badge/postgres-9.5%20|%209.6%20|%2010%20|%2011%20|%2012%20-blue.svg)
 [![PyPI - License](https://img.shields.io/pypi/l/django-pg-zero-downtime-migrations.svg)](https://raw.githubusercontent.com/tbicr/django-pg-zero-downtime-migrations/master/LICENSE)
 
 [![PyPI - Downloads](https://img.shields.io/pypi/dm/django-pg-zero-downtime-migrations.svg)](https://pypistats.org/packages/django-pg-zero-downtime-migrations)
@@ -14,8 +14,6 @@ Django postgresql backend that apply migrations with respect to database locks.
 ## Installation
 
     pip install django-pg-zero-downtime-migrations
-
-> *NOTE:* this package works with django 2.0+.
 
 ## Usage
 
@@ -112,20 +110,6 @@ Allowed values:
 
 > *NOTE:* For postgres 12 and newest `NOT NULL` constraint creation has migration replacement that provide same state as default django backend, so this option deprecated and doesn't used this postgres version. If you use `CHECK NOT NULL` compatible constraint before you can migrate it to `NOT NULL` constraints with `manage.py migrate_isnotnull_check_constraints` management command (add `INSTALLED_APPS += ['django_zero_downtime_migrations']` to `settings.py` to use management command).
 
-### Dealing with partial indexes
-
-> *NOTE:* django 2.2 support native partial index mechanism: https://docs.djangoproject.com/en/2.2/ref/models/indexes/#condition and https://docs.djangoproject.com/en/2.2/ref/models/constraints/#condition.
-
-If you using https://github.com/mattiaslinnap/django-partial-index package for partial indexes in postgres, then you can easily make this package also safe for migrations:
-
-    from django_zero_downtime_migrations_postgres_backend.schema import PGShareUpdateExclusive
-    from partial_index import PartialIndex
-
-    PartialIndex.sql_create_index['postgresql'] = PGShareUpdateExclusive(
-        'CREATE%(unique)s INDEX CONCURRENTLY %(name)s ON %(table)s%(using)s (%(columns)s)%(extra)s WHERE %(where)s',
-        disable_statement_timeout=True
-    )
-
 ## How it works
 
 ### Postgres table level locks
@@ -152,19 +136,17 @@ Lets split this lock to migration and business logic operations.
 
 #### Migration locks
 
-| lock                     | operations                                                                                                |
-|--------------------------|-----------------------------------------------------------------------------------------------------------|
-| `ACCESS EXCLUSIVE`       | `CREATE SEQUENCE`, `DROP SEQUENCE`, `CREATE TABLE`, `DROP TABLE` \*, `ALTER TABLE` \*\*, `DROP INDEX`     |
-| `SHARE`                  | `CREATE INDEX`                                                                                            |
-| `SHARE UPDATE EXCLUSIVE` | `CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY` \*\*\*, `ALTER TABLE VALIDATE CONSTRAINT` \*\*\*\* |
+| lock                     | operations                                                                                            |
+|--------------------------|-------------------------------------------------------------------------------------------------------|
+| `ACCESS EXCLUSIVE`       | `CREATE SEQUENCE`, `DROP SEQUENCE`, `CREATE TABLE`, `DROP TABLE` \*, `ALTER TABLE` \*\*, `DROP INDEX` |
+| `SHARE`                  | `CREATE INDEX`                                                                                        |
+| `SHARE UPDATE EXCLUSIVE` | `CREATE INDEX CONCURRENTLY`, `DROP INDEX CONCURRENTLY`, `ALTER TABLE VALIDATE CONSTRAINT` \*\*\*      |
 
 \*: `CREATE SEQUENCE`, `DROP SEQUENCE`, `CREATE TABLE`, `DROP TABLE` shouldn't have conflicts, because your business logic shouldn't yet operate with created tables and shouldn't already operate with deleted tables.
 
 \*\*: Not all `ALTER TABLE` operations take `ACCESS EXCLUSIVE` lock, but all current django's migrations take it https://github.com/django/django/blob/master/django/db/backends/base/schema.py, https://github.com/django/django/blob/master/django/db/backends/postgresql/schema.py and https://www.postgresql.org/docs/current/static/sql-altertable.html.
 
-\*\*\*: Bare django currently doesn't support `CONCURRENTLY` operations.
-
-\*\*\*\*: Django doesn't have `VALIDATE CONSTRAINT` logic, but we will use it for some cases.
+\*\*\*: Django doesn't have `VALIDATE CONSTRAINT` logic, but we will use it for some cases.
 
 #### Business logic locks
 
@@ -256,8 +238,12 @@ Any schema changes can be processed with creation of new table and copy data to
 | 24 | `ALTER TABLE DROP CONSTRAINT` (`PRIMARY KEY`) | X    |                               | safe operation \*\*\*
 | 25 | `ALTER TABLE ADD CONSTRAINT UNIQUE`           |      | add index and add constraint  | **unsafe operation**, because you spend time in migration to create index \*\*\*
 | 26 | `ALTER TABLE DROP CONSTRAINT` (`UNIQUE`)      | X    |                               | safe operation \*\*\*
-| 27 | `CREATE INDEX`                                |      | `CREATE INDEX CONCURRENTLY`   | **unsafe operation**, because you spend time in migration to create index
-| 28 | `DROP INDEX`                                  | X    | `DROP INDEX CONCURRENTLY`     | safe operation  \*\*\*
+| 27 | `ALTER TABLE ADD CONSTRAINT EXCLUDE`          |      | add new table and copy data   |
+| 28 | `ALTER TABLE DROP CONSTRAINT (EXCLUDE)`       | X    |                               |
+| 29 | `CREATE INDEX`                                |      | `CREATE INDEX CONCURRENTLY`   | **unsafe operation**, because you spend time in migration to create index
+| 30 | `DROP INDEX`                                  | X    | `DROP INDEX CONCURRENTLY`     | safe operation  \*\*\*
+| 31 | `CREATE INDEX CONCURRENTLY`                   | X    |                               | safe operation
+| 32 | `DROP INDEX CONCURRENTLY`                     | X    |                               | safe operation  \*\*\*
 
 \*: main point with migration on production without downtime that your code should correctly work before and after migration, lets look this point closely in [Dealing with logic that should work before and after migration](#dealing-with-logic-that-should-work-before-and-after-migration) section.
 
@@ -279,7 +265,7 @@ This migrations are pretty safe, because your logic doesn't work with this data
 
 ##### Changes for working logic
 
-Migrations: `ALTER TABLE RENAME TO`, `ALTER TABLE SET TABLESPACE`, `ALTER TABLE RENAME COLUMN`.
+Migrations: `ALTER TABLE RENAME TO`, `ALTER TABLE SET TABLESPACE`, `ALTER TABLE RENAME COLUMN`, `ALTER TABLE ADD CONSTRAINT EXCLUDE`.
 
 For this migration too hard implement logic that will work correctly for all instances, so there are two ways to dealing with it:
 

django_zero_downtime_migrations/__init__.py

@@ -1 +1 @@
-__version__ = '0.7'
+__version__ = '0.8'

django_zero_downtime_migrations/backends/postgres/schema.py

@@ -32,6 +32,11 @@ class Unsafe:
         "See details for safe alternative "
         "https://github.com/tbicr/django-pg-zero-downtime-migrations#dealing-with-alter-table-alter-column-type"
     )
+    ADD_CONSTRAINT_EXCLUDE = (
+        "ADD CONSTRAINT EXCLUDE is unsafe operation\n"
+        "See details for safe alternative "
+        "https://github.com/tbicr/django-pg-zero-downtime-migrations#changes-for-working-logic"
+    )
     ALTER_TABLE_RENAME = (
         "ALTER TABLE RENAME is unsafe operation\n"
         "See details for save alternative "
@@ -153,6 +158,7 @@ class DatabaseSchemaEditorMixin:
     sql_rename_table = PGAccessExclusive(PostgresDatabaseSchemaEditor.sql_rename_table)
     sql_retablespace_table = PGAccessExclusive(PostgresDatabaseSchemaEditor.sql_retablespace_table)
 
+    sql_create_column_inline_fk = None
     sql_create_column = PGAccessExclusive(PostgresDatabaseSchemaEditor.sql_create_column)
     sql_alter_column = PGAccessExclusive(PostgresDatabaseSchemaEditor.sql_alter_column)
     sql_delete_column = PGAccessExclusive(PostgresDatabaseSchemaEditor.sql_delete_column)
@@ -187,29 +193,24 @@ class DatabaseSchemaEditorMixin:
     )
     sql_delete_pk = PGAccessExclusive(PostgresDatabaseSchemaEditor.sql_delete_pk)
 
-    if django.VERSION[:2] < (2, 2):
-        sql_create_index = PGShareUpdateExclusive(
-            "CREATE INDEX CONCURRENTLY %(name)s ON %(table)s%(using)s (%(columns)s)%(extra)s",
-            disable_statement_timeout=True
-        )
-        sql_create_varchar_index = PGShareUpdateExclusive(
-            "CREATE INDEX CONCURRENTLY %(name)s ON %(table)s (%(columns)s varchar_pattern_ops)%(extra)s",
-            disable_statement_timeout=True
-        )
-        sql_create_text_index = PGShareUpdateExclusive(
-            "CREATE INDEX CONCURRENTLY %(name)s ON %(table)s (%(columns)s text_pattern_ops)%(extra)s",
-            disable_statement_timeout=True
-        )
-    else:
-        sql_create_index = PGShareUpdateExclusive(
-            "CREATE INDEX CONCURRENTLY %(name)s ON %(table)s%(using)s (%(columns)s)%(extra)s%(condition)s",
+    sql_create_index = PGShareUpdateExclusive(
+        "CREATE INDEX CONCURRENTLY %(name)s ON %(table)s%(using)s (%(columns)s)%(extra)s%(condition)s",
+        disable_statement_timeout=True
+    )
+    if django.VERSION[:2] >= (3, 0):
+        sql_create_index_concurrently = PGShareUpdateExclusive(
+            PostgresDatabaseSchemaEditor.sql_create_index_concurrently,
             disable_statement_timeout=True
         )
     sql_create_unique_index = PGShareUpdateExclusive(
         "CREATE UNIQUE INDEX CONCURRENTLY %(name)s ON %(table)s (%(columns)s)%(condition)s",
         disable_statement_timeout=True
     )
     sql_delete_index = PGShareUpdateExclusive("DROP INDEX CONCURRENTLY IF EXISTS %(name)s")
+    if django.VERSION[:2] >= (3, 0):
+        sql_delete_index_concurrently = PGShareUpdateExclusive(
+            PostgresDatabaseSchemaEditor.sql_delete_index_concurrently
+        )
 
     _sql_table_count = "SELECT reltuples FROM pg_class WHERE oid = '%(table)s'::regclass"
     _sql_check_notnull_constraint = (
@@ -346,15 +347,28 @@ def alter_unique_together(self, model, old_unique_together, new_unique_together)
         super().alter_unique_together(model, old_unique_together, new_unique_together)
         self._flush_deferred_sql()
 
-    def add_index(self, model, index):
-        super().add_index(model, index)
+    def add_index(self, model, index, concurrently=False):
+        if django.VERSION[:2] >= (3, 0):
+            super().add_index(model, index, concurrently=concurrently)
+        else:
+            super().add_index(model, index)
         self._flush_deferred_sql()
 
-    def remove_index(self, model, index):
-        super().remove_index(model, index)
+    def remove_index(self, model, index, concurrently=False):
+        if django.VERSION[:2] >= (3, 0):
+            super().remove_index(model, index, concurrently=concurrently)
+        else:
+            super().remove_index(model, index)
         self._flush_deferred_sql()
 
     def add_constraint(self, model, constraint):
+        if django.VERSION[:2] >= (3, 0):
+            from django.contrib.postgres.constraints import ExclusionConstraint
+            if isinstance(constraint, ExclusionConstraint):
+                if self.RAISE_FOR_UNSAFE:
+                    raise UnsafeOperationException(Unsafe.ADD_CONSTRAINT_EXCLUDE)
+                else:
+                    warnings.warn(UnsafeOperationWarning(Unsafe.ADD_CONSTRAINT_EXCLUDE))
         super().add_constraint(model, constraint)
         self._flush_deferred_sql()
 

django_zero_downtime_migrations/management/commands/migrate_isnotnull_check_constraints.py

@@ -11,8 +11,8 @@ class Command(BaseCommand):
     def _is_postgres_12(self):
         return connection.pg_version >= 120000
 
-    def _is_postgres_94(self):
-        return connection.pg_version <= 90500
+    def _is_postgres_95(self):
+        return connection.pg_version >= 90500
 
     def _can_update_pg_attribute(self):
         sql = (
@@ -28,7 +28,7 @@ def _can_update_pg_attribute(self):
             return result is not None
 
     def _can_migrate(self):
-        return self._is_postgres_12() or (not self._is_postgres_94() and self._can_update_pg_attribute())
+        return self._is_postgres_12() or (self._is_postgres_95() and self._can_update_pg_attribute())
 
     def _migrate_for_postgres_12(self, ignore, only):
         with connection.temporary_connection() as cursor:

docker-compose.yml

@@ -40,14 +40,6 @@ services:
     volumes:
       - ./docker_postgres_init.sql:/docker-entrypoint-initdb.d/docker_postgres_init.sql
 
-  pg94:
-    image: postgres:9.4-alpine
-    environment:
-      POSTGRES_USER: root
-      POSTGRES_PASSWORD: root
-    volumes:
-      - ./docker_postgres_init.sql:/docker-entrypoint-initdb.d/docker_postgres_init.sql
-
   postgis11:
     image: mdillon/postgis:11-alpine
     environment:
@@ -64,7 +56,6 @@ services:
       - pg10
       - pg96
       - pg95
-      - pg94
       - postgis11
     volumes:
       - .:/app

setup.py

@@ -40,14 +40,12 @@ def _get_long_description():
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
         'Framework :: Django',
-        'Framework :: Django :: 2.0',
-        'Framework :: Django :: 2.1',
         'Framework :: Django :: 2.2',
     ],
     keywords='django postgres postgresql migrations',
     packages=find_packages(exclude=['manage*', 'tests*']),
     python_requires='>=3.5',
     install_requires=[
-        'django>=2.0',
+        'django>=2.2',
     ]
 )

tests/apps/good_flow_app_concurrently/migrations/0001_initial.py

@@ -0,0 +1,37 @@
+# Generated by Django 3.1 on 2019-09-21 20:09
+
+import django.contrib.postgres.search
+from django.db import migrations, models
+
+
+def insert_objects(apps, schema_editor):
+    db_alias = schema_editor.connection.alias
+    TestTable = apps.get_model('good_flow_app_concurrently', 'TestTable')
+    TestTable.objects.using(db_alias).create(test_field_int=1)
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='RelatedTestTable',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+            ],
+        ),
+        migrations.CreateModel(
+            name='TestTable',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('test_field_int', models.IntegerField()),
+                ('test_field_str', models.CharField(max_length=10)),
+                ('test_field_tsv', django.contrib.postgres.search.SearchVectorField()),
+            ],
+        ),
+        migrations.RunPython(insert_objects, migrations.RunPython.noop),
+    ]

tests/apps/good_flow_app_concurrently/migrations/0002_auto_20191210_2147.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.0a1 on 2019-12-10 21:47
+from django.contrib.postgres.operations import AddIndexConcurrently
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    atomic = False
+    dependencies = [
+        ('good_flow_app_concurrently', '0001_initial'),
+    ]
+
+    operations = [
+        AddIndexConcurrently(
+            model_name='testtable',
+            index=models.Index(fields=['test_field_int'], name='good_flow_a_test_fi_0b7e6f_idx'),
+        ),
+    ]

tests/apps/good_flow_app_concurrently/migrations/0003_auto_20191210_2148.py

@@ -0,0 +1,18 @@
+# Generated by Django 3.0a1 on 2019-12-10 21:48
+from django.contrib.postgres.operations import RemoveIndexConcurrently
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    atomic = False
+    dependencies = [
+        ('good_flow_app_concurrently', '0002_auto_20191210_2147'),
+    ]
+
+    operations = [
+        RemoveIndexConcurrently(
+            model_name='testtable',
+            name='good_flow_a_test_fi_0b7e6f_idx',
+        ),
+    ]

tests/apps/good_flow_app_concurrently/models.py

@@ -0,0 +1,12 @@
+from django.contrib.postgres.search import SearchVectorField
+from django.db import models
+
+
+class TestTable(models.Model):
+    test_field_int = models.IntegerField()
+    test_field_str = models.CharField(max_length=10)
+    test_field_tsv = SearchVectorField()
+
+
+class RelatedTestTable(models.Model):
+    pass

tests/integration/test_management_command_notnull_migration.py

@@ -40,9 +40,7 @@ def assert_constraints(null, check):
 
 
 @pytest.mark.skipif(
-    settings.DATABASES['default']['HOST'] not in ['pg12']
-    and settings.DATABASES['default']['USER'] != 'root'
-    or settings.DATABASES['default']['HOST'] in ['pg94'],
+    settings.DATABASES['default']['HOST'] not in ['pg12'] and settings.DATABASES['default']['USER'] != 'root',
     reason='superuser permissions required',
 )
 @pytest.mark.django_db(transaction=True)