Immediate ingest

[stranak]

It is possible (and we should) not to rely on the scheduler to run daemon that ingests submissions periodically, but instead trigger the ingestion directly by the action of submission.

It is obviously better solution, since it makes new submissions immediately visible and accessible. It is also how CERN library, Zenodo and other Invenio users do it. With a bit of luck the config will be in their repository: https://github.com/zenodo

[emanueldima]

I have been working on this. The ingest process is not slow at all. I tested it twice, with 2 files, and I get the correct result page after about 5 seconds after I click the "Deposit" button. This is the behavior of the testing server, I can't test what is going on in the production environment.

From the code paths, I can speculate that delays happen when multiple uploads are submitted at the same time. All the queued bibXXX tasks/actions are queued (using the database, see schTASK) and they are executed sequentially. Apparently, bad things happen when tasks are run concurrently -- it seems that there are no fine grained locks for avoiding race conditions.

So, if several uploads are submitted at once, the first will get ingested immediately and the last will wait for all the others. Or, if the upload is done while any indexing/ranking/sorting service is running, it will have to wait.

The only improvement I see now is to increase the priority of the bibupload tasks, to be executed before indexing/sorting/ranking tasks. Other than that, the UI must change to tell the user that the record is being processed.