Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Sigil process #462

Open
1 of 2 tasks
TobiasNx opened this issue Jul 11, 2023 · 3 comments
Open
1 of 2 tasks

Update Sigil process #462

TobiasNx opened this issue Jul 11, 2023 · 3 comments

Comments

@TobiasNx
Copy link
Contributor

TobiasNx commented Jul 11, 2023

We are using an old dump from 2013. This makes the transformation pretty pretty slow, since it harvests all updates since then via OAI-PMH.
We have a new one bit it is binary pica+. We need to remodel it.

  • Change the process. 462 sigel process new #472
  • Add cron job that looks for new updated batch file. That downloads the file to app/transformation/input, renames the file to sigel.dat and cjanges the date in transformation.updates.start="2023-06-01" to the creation date of the sourcefile in conf/application.conf
TobiasNx added a commit that referenced this issue Jul 18, 2023
TobiasNx added a commit that referenced this issue Aug 7, 2023
TobiasNx added a commit that referenced this issue Aug 10, 2023
TobiasNx added a commit that referenced this issue Aug 17, 2023
TobiasNx added a commit that referenced this issue Aug 22, 2023
TobiasNx added a commit that referenced this issue Sep 18, 2023
@TobiasNx
Copy link
Contributor Author

Deployed with: #450

@TobiasNx TobiasNx reopened this Sep 21, 2023
@TobiasNx
Copy link
Contributor Author

TobiasNx commented Sep 21, 2023

Reopen this ticket:

Since the process has still some old part and is complicated to comprehend:
Question, do we still need the intervals for OAI, I suggest not, since we only fetch data for a year or so.
We should simplify the process.

Do we need the OAI PMH Data as XML saved at the server?
Do we need this XML Data as fallback?

Tried to simplify the process here:
#489

@fsteeg
Copy link
Member

fsteeg commented Sep 22, 2023

Do we need the OAI PMH Data as XML saved at the server?
Do we need this XML Data as fallback?

If I remember correctly I added this because it was useful to (re-)run the transformation only (e.g. after making some changes), without having to get the data via OAI-PMH again.

TobiasNx added a commit that referenced this issue Oct 12, 2023
Get rid of xml splitting process and only use a single OAI-PMH inerval steps due to newer pica bulks.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants