Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Better index cleanup #3808

Open
thommyhh opened this issue Sep 28, 2023 · 2 comments · May be fixed by #3809
Open

[FEATURE] Better index cleanup #3808

thommyhh opened this issue Sep 28, 2023 · 2 comments · May be fixed by #3809

Comments

@thommyhh
Copy link

thommyhh commented Sep 28, 2023 •
edited
Loading

Is your feature request related to a problem? Please describe.
The ReIndex task deletes all documents from the index, leaving it empty until the IndexQueueWorker has processed the queue. It removes only documents from the types, scheduled to be indexed, leaving files in the index forever.

Describe the solution you'd like
A proper cleanup task, that accepts a time threshold to delete old documents, leaving the newer ones in the index.

Target versions
The newest release, but it may also be backported to 11.x, as this could really be helpful.
@thommyhh thommyhh linked a pull request Sep 28, 2023 that will close this issue
Open
@dkd-kaehm
Copy link
Collaborator

dkd-kaehm commented Oct 6, 2023 •
edited
Loading

@thommyhh
I forget about that, but EXT:solr has a nice feature to avoid the troubles of this issue.
READ and WRITE connections introduced with EXT:solr 9: https://docs.typo3.org/p/apache-solr-for-typo3/solr/main/en-us/Releases/solr-release-9-0.html#support-to-differ-between-read-and-write-connections
The docs show old legacy mode settings via TypoScript, but now you can configure it in site-config: https://docs.typo3.org/p/apache-solr-for-typo3/solr/main/en-us/Backend/ConnectionManager.html#typo3-site-mode

Beside of that the ReIndex-Task uses already the write connection:

ext-solr/Classes/Task/ReIndexTask.php

Lines 86 to 93 in 1ed1c4a

$solrServer->getWriteService()->deleteByQuery($deleteQuery);
if (!$enableCommitsSetting) {
// Do not commit
continue;
}
$response = $solrServer->getWriteService()->commit(false, false);

But, there is currently no automation to switch the READ and WRITE connections automatically(solr_use_write_connection true|false).
Currently my idea is to provide a set of events to signal that all Index-Queues are indexed to 100% and the switch can be done and the switch is happened and so on.
The events set should be compatible with EXT:solrfal's index queue, multiple sites and our new external data indexer feature.

What do you think, can it help you?

@christophlehmann
Copy link
Contributor

@thommyhh I wrote a Cleanup Command for that. Wanna give it a try? With it the strategy for "Re-indexing" is to re-initialize the index queue and then execute the command. So no documents are missing at any time in the index. I'm would appreciate, when the command finds its way into EXT:solr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[FEATURE] Better index cleanup webcoast-dk/ext-solr
3 participants
@christophlehmann @thommyhh @dkd-kaehm