Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Last day of the month is not processed in bin/cron4dbs_condor #104

Closed
mrceyhun opened this issue Aug 19, 2022 · 4 comments · May be fixed by #100
Closed

Last day of the month is not processed in bin/cron4dbs_condor #104

mrceyhun opened this issue Aug 19, 2022 · 4 comments · May be fixed by #100
Assignees
Labels

Comments

@mrceyhun
Copy link
Contributor

bin/cron4dbs_condor does not process last day of months. It can be seen in /cms/dbs_condor/release. Since it is processing previous day's data and in each months starting to process current month, it is skipping last day of each month.

Probably a easy fix, opening this issue to not forget next week.

@mrceyhun mrceyhun added the bug label Aug 19, 2022
@mrceyhun mrceyhun self-assigned this Aug 19, 2022
@mrceyhun
Copy link
Contributor Author

catched by @kyrylogy

@kyrylogy
Copy link
Contributor

kyrylogy commented Aug 22, 2022

I believe the issue lies here and here

tmpDirs=hadoop fs -ls ${hdir}/$year/$month | grep tmp$ | awk '{print $8}' | sed -e "s,\.tmp,,g" -e "s,${hdir},,g"
This gives us the date of .tmp file, which is the file of the day - on 1st of the month it would give /2022/08/01

lastSnapshot=hadoop fs -ls ${hdir}/$year/$month | egrep -v ${pat} | tail -1 | awk '{print $8}'
Here we get the same list of files, but without todays date. Then we take the path of last file - on 1st of the month it would pop the only available file. Additionally, we only apply it to current month's list of data, so there's no file behind the first file of current month.

I will fix it

@mrceyhun
Copy link
Contributor Author

Thanks @kyrylogy , it would be great!

kyrylogy added a commit to kyrylogy/CMSSpark that referenced this issue Aug 23, 2022
kyrylogy added a commit to kyrylogy/CMSSpark that referenced this issue Aug 23, 2022
kyrylogy added a commit to kyrylogy/CMSSpark that referenced this issue Sep 14, 2022
@mrceyhun
Copy link
Contributor Author

Fixed by #109

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants