Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

better backup solution for DESI_SPECTRO_DARK #2436

Open
sybenzvi opened this issue Jan 28, 2025 · 0 comments
Open

better backup solution for DESI_SPECTRO_DARK #2436

sybenzvi opened this issue Jan 28, 2025 · 0 comments

Comments

@sybenzvi
Copy link
Contributor

Since October 2024, we've been using a daily wget to synchronize the DESI_SPECTRO_DARK folder at NERSC with the DESI cluster at KPNO. This replaced a previous rsync from perlmutter. I'm told that support for data transfers via wget will not continue indefinitely, but access to SVN repos will be allowed.

I recall that DESI_SPECTRO_DARK was split off from the DESI_SPECTRO_CALIB repo because the dataset was getting too large. The current set of bias and dark files in DESI_SPECTRO_DARK uses about 120G on disk, with ~40G in the bias frames and ~80G in the dark frames, compared to about 60G in DESI_SPECTRO_CALIB.

If we need to keep the repo below a certain size, we might sort them into separate repos by spectrograph or by year. For the convenience of maintainers (@Waelthus and @julienguy) we could probably still approximate the appearance of a single DESI_SPECTRO_DARK folder using svn:externals. To handle the changes to the folder structure, some work would need to be done to the CalibFinder code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant