Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrations: Update migrator to run faster by delegating document iteration to MongoDB #2265

Conversation

eecavanna
Copy link
Collaborator

@eecavanna eecavanna commented Nov 12, 2024

In this branch, I did one new thing, plus I merged in two other branches that haven't yet been merged into main (so, if this PR branch gets merged into main, I'll close the PRs associated with those other two branches).

The new thing I did in this branch is: I updated the migrator_from_11_0_3_to_11_1_0_part_2 so it would run faster. Specifically, I updated it to delegate an intensive task to MongoDB (which is optimized for that task), instead of doing it "manually" in Python.

In order to accomplish that, I merged (into this branch) a couple branches, each of which introduced some necessary adapter method upon which the above migrator change depends. That's why this PR branch "includes" two other branches.

This PR supersedes #2262 and #2263.

I hope it works.

Copy link

PR Preview Action v1.4.8
🚀 Deployed preview to https://microbiomedata.github.io/nmdc-schema/pr-preview/pr-2265/
on branch gh-pages at 2024-11-12 23:35 UTC

@eecavanna
Copy link
Collaborator Author

Hi @kheal and @brynnz22, this PR contains the changes I made to the migrator_from_11_0_3_to_11_1_0_part_2.py migrator (using the newly-implemented adapter methods). When reviewing this PR, I'm OK with you reviewing only that file (in case you're short on time), and ignoring the others.

@eecavanna eecavanna merged commit c594645 into main Nov 13, 2024
3 checks passed
@eecavanna eecavanna deleted the 2264-migrations-update-migrators-to-use-newly-implemented-adapter-methods-for-copying-and-deleting branch November 13, 2024 00:09
@eecavanna eecavanna mentioned this pull request Nov 13, 2024
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrations: Update migrators to use newly-implemented adapter methods for copying and deleting
2 participants