Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support doing a full sync without deleting the existing index #81

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from

Conversation

matinone
Copy link

@matinone matinone commented Jan 31, 2024

This PR adds a new command line argument (--keep-index) to the refresh command, to support doing a full sync without deleting the existing index.
Currently, when doing a full sync the index is recreated and all the existing data from the database table is added to the index. In this context, it makes perfect sense to delete and create again the index, because otherwise the deleted items in the database table would remain in the index. It also takes advantage of the swap index feature to perform the update without downtime.

But if you want to sync multiple tables to the same index (so you can search everything you need in a single index afterwards), then that process doesn't work if you want to do a full sync of all the tables to that same index, because each table sync would delete the index and overwrite the previous content, so at the end only one table would have been synced to the index. This is why having this additional option to preserve the index would be useful.
To sync all the tables, you run the refresh command once without the --keep-index option for one table (it would work in exactly the same way as it works now):

meilisync refresh -t table_1

And then you run it with the --keep-index flag for the rest of the tables you want to sync to the same index:

meilisync refresh --keep-index -t table_2
meilisync refresh --keep-index -t table_3

The PR also fixes a small bug where the table name wouldn't be available in the plugins when events are generated as a result of a full sync (see the change in the async def add_data(self, sync: Sync, data: list) method).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant