You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since October 2024, we've been using a daily wget to synchronize the DESI_SPECTRO_DARK folder at NERSC with the DESI cluster at KPNO. This replaced a previous rsync from perlmutter. I'm told that support for data transfers via wget will not continue indefinitely, but access to SVN repos will be allowed.
I recall that DESI_SPECTRO_DARK was split off from the DESI_SPECTRO_CALIB repo because the dataset was getting too large. The current set of bias and dark files in DESI_SPECTRO_DARK uses about 120G on disk, with ~40G in the bias frames and ~80G in the dark frames, compared to about 60G in DESI_SPECTRO_CALIB.
If we need to keep the repo below a certain size, we might sort them into separate repos by spectrograph or by year. For the convenience of maintainers (@Waelthus and @julienguy) we could probably still approximate the appearance of a single DESI_SPECTRO_DARK folder using svn:externals. To handle the changes to the folder structure, some work would need to be done to the CalibFinder code.
The text was updated successfully, but these errors were encountered:
Since October 2024, we've been using a daily
wget
to synchronize theDESI_SPECTRO_DARK
folder at NERSC with the DESI cluster at KPNO. This replaced a previous rsync from perlmutter. I'm told that support for data transfers viawget
will not continue indefinitely, but access to SVN repos will be allowed.I recall that
DESI_SPECTRO_DARK
was split off from theDESI_SPECTRO_CALIB
repo because the dataset was getting too large. The current set of bias and dark files inDESI_SPECTRO_DARK
uses about 120G on disk, with ~40G in the bias frames and ~80G in the dark frames, compared to about 60G inDESI_SPECTRO_CALIB
.If we need to keep the repo below a certain size, we might sort them into separate repos by spectrograph or by year. For the convenience of maintainers (@Waelthus and @julienguy) we could probably still approximate the appearance of a single DESI_SPECTRO_DARK folder using svn:externals. To handle the changes to the folder structure, some work would need to be done to the
CalibFinder
code.The text was updated successfully, but these errors were encountered: