Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

django-spanner's autofield generation does not work for non-default spanner DBs #783

Open
untitaker opened this issue Aug 9, 2022 · 3 comments · May be fixed by #784
Open

django-spanner's autofield generation does not work for non-default spanner DBs #783

untitaker opened this issue Aug 9, 2022 · 3 comments · May be fixed by #784
Labels
api: spanner Issues related to the googleapis/python-spanner-django API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@untitaker
Copy link

#780 introduced a check as to whether the default database's engine is spanner or not, and depending on that decides to override id generation with clientside uuid or not.

This fix is insufficient. It does improve the situation, as django-spanner will disable its monkeypatches if the default db is not spanner.

But the original issue #742 was talking about using multiple databases, where default db is mysql. This is what django.db.connection.settings_dict refers to. The correct key to check would be django.db.connections[???].settings_dict.

Steps to reproduce

  1. set up spanner as a non-default database
  2. see that even running basic migrations against that non-default db fails, as the inserts into the django_migrations table already fail.

I'll investigate how to fix. It's likely something else has to be patched entirely.

Environment details

  • Programming language: Python
  • OS: Doesn't matter
  • Language runtime version: 3.8
  • Package version: master
@untitaker untitaker added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Aug 9, 2022
@product-auto-label product-auto-label bot added the api: spanner Issues related to the googleapis/python-spanner-django API. label Aug 9, 2022
@untitaker untitaker changed the title django-spanner master broke autofield generation entirely django-spanner's autofield generation does not work for non-default spanner DBs Aug 9, 2022
untitaker added a commit to untitaker/python-spanner-django that referenced this issue Aug 9, 2022
Fix googleapis#783

Also refactored code a bit such that version detection is no longer used
to determine whether BigAutoField exists. BigAutoField exists in Django
versions older than 3.0, and the way this is currently monkeypatched
prevents us from using any BigAutoField in Spanner with Django 2.2.
@untitaker untitaker linked a pull request Aug 9, 2022 that will close this issue
untitaker added a commit to untitaker/python-spanner-django that referenced this issue Aug 9, 2022
Fix googleapis#783

Also refactored code a bit such that version detection is no longer used
to determine whether BigAutoField exists. BigAutoField exists in Django
versions older than 3.0, and the way this is currently monkeypatched
prevents us from using any BigAutoField in Spanner with Django 2.2.
@IlyaFaer
Copy link
Contributor

About the next spring a new auto incrementing primary keys features is expected to be released in Spanner. It should make autofield patch unnecessary at all.

@mitsuo0114
Copy link

mitsuo0114 commented May 26, 2024

+1. what is current status?
I've found the auto incrementing primary keys feature was introduced.
https://cloud.google.com/blog/products/databases/announcing-support-for-auto-generated-keys-in-spanner?hl=en

But still I'm facing error when migration info is inserted to django_migrations.

"id must not be NULL in table django_migrations."

On Cloud Spanner, following table is created automatically.
The table definition of django_migrations does not include configs for auto-generate-keys.

CREATE TABLE django_migrations (
  id INT64 NOT NULL,
  app STRING(255) NOT NULL,
  name STRING(255) NOT NULL,
  applied TIMESTAMP NOT NULL,
) PRIMARY KEY(id);;

I manually configured django_migrations with sequence like below and migration succeeded.

CREATE SEQUENCE serial OPTIONS(sequence_kind='bit_reversed_positive');

CREATE TABLE django_migrations (
  id INT64 DEFAULT (GET_NEXT_SEQUENCE_VALUE(SEQUENCE serial)),
  app STRING(255) NOT NULL,
  name STRING(255) NOT NULL,
  applied TIMESTAMP NOT NULL,
) PRIMARY KEY(id);;

but I'm concerning this is right way for work around. It would be helpful if someone give advice to me and for this workaround.

@asthamohta asthamohta removed their assignment Jun 27, 2024
@sarafraghav
Copy link

Facing this same issue - using spanner as a non-default DB. Whenever I am inserting records to another db - bulk_create is failing with thw following strace

File "/Users/saraf/Library/Caches/pypoetry/virtualenvs/workflow-platform-HlqtOrhH-py3.12/lib/python3.12/site-packages/django/contrib/contenttypes/management/__init__.py", line 152, in create_contenttypes ContentType.objects.using(using).bulk_create(cts) File "/Users/saraf/Library/Caches/pypoetry/virtualenvs/workflow-platform-HlqtOrhH-py3.12/lib/python3.12/site-packages/django/db/models/query.py", line 816, in bulk_create assert len(returned_columns) == len(objs_without_pk)

If I remove django_spanner from installed_apps before running this command - it works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner Issues related to the googleapis/python-spanner-django API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants