-
Notifications
You must be signed in to change notification settings - Fork 4
Set up HSC Gen2 and Gen3 CI Testing Areas
Following Dino's recipe posted on Slack.
Using the descdm collaboration account to setup git-lfs access which is now completed and does not need to be repeated for this shared account. Here are the steps that were used:
module load git-lfs/2.8.0
git lfs install
Then edited $HOME/.gitconfig and $HOME/.git-credentials as explained in the LSST developer documentation.
HSC test data is 8 GB.
Installation Directory: /global/cscratch1/sd/descdm/hsc
collabsu descdm
module load git-lfs/2.8.0
cd $CSCRATCH/hsc
git clone https://github.com/lsst/testdata_ci_hsc
Installation Directory: /global/cscratch1/sd/descdm/hsc
Running time for full CI Gen3 tests:
~1hr 50 min on Cori Haswell
Over 4 hrs on Cori KNL
DMstack installations are available on CVMFS.
Gen2: DMstack v19
Gen3: DMstack w_2020_05
collabsu descdm
source /global/cscratch1/sd/descdm/hsc/setup_cvmfs_w_2020_05.sh (or setup_cvmfs_v19.sh)
cd /global/cscratch1/sd/descdm/hsc/testdata_ci_hsc
setup -k -r .
cd ..
git clone https://github.com/lsst/ci_hsc_gen3 (or ci_hsc_gen2)
cd ci_hsc_gen3 (or ci_hsc_gen2)
setup -k -r .
scons (or check pipetask --help for alternative ways to run examples)
Output will appear in the ci_hsc_gen3/DATA subdirectory.
When ready to reset the CI directory and start over, run: scons --clean, will empty the ci_hsc_gen*/DATA directory.
For initial testing, each of us will likely want our own copies of ci_hsc_gen3 and ci_hsc_gen2 to play around with, but we could share the testdata_ci_hsc area.