Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize resource retrieval when asserting paths by maling cache #57

Open
cedricdcc opened this issue Jun 4, 2024 · 0 comments
Open
Assignees
Labels
enhancement New feature or request

Comments

@cedricdcc
Copy link
Member

Currently travahrv retrieves a resource every time it needs to;
It doesn't look if the resource (URI) was already retrieved in the past.
This results in the same resource being retrieved multiple times which results in long waitng tiles for some tasks that have a lot of assertion paths that need to be traversal harvested.

A solution for this can be looking at the execution report and retrieving all resources that were harvested already together with their date of harvest and mimetype to assure that all diff mimetypes were harvested.

With this a cache can be made that travharv can use.

@cedricdcc cedricdcc added the enhancement New feature or request label Jun 4, 2024
@cedricdcc cedricdcc self-assigned this Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant