NERSC SFAPI Reconstruction Flow

[davramov]

Setting this up as a draft PR for now while I iron out implementation details to match the ALCF workflow.

These two steps are currently working via sfapi and the container Raja built (registry.nersc.gov/als/tomorecon_nersc_mpi_hdf5):

• Reconstruction
• Tiff to Zarr

There are still at least a few things that need to be addressed before it is production ready:

• Pass into reconstruction.py the same file_path that new_832_file_flow and alcf_recon_flow expect (still using a legacy version of reconstruction.py that expects an input.txt file)
• File transfers back to data832
• Schedule Pruning
• Pytest cases

[dylanmcreynolds]

Nice work.

I added a number of review comments. This is a really nice start, and makes the nersc-specific code easy to see and follow. If my comments were a little picky, it's because the code was easy to follow.

tomocagrahpy_hpt.py should be split into sepearate modules, something like: tomography_hpc.py, nersc.py and alcf.py so that no more code than necessary is imported (e.g. when running an ALCF job, you don't need to import NerscClient.)

Also, what about job_controller.py rather than tomography_hpc.py? Since this is under flows/bl832, tomography is implied. And we might some day write the job controllers for local (non-HPC) jobs.

[dylanmcreynolds]

This

[davramov]

Several queue things I have figured out:

• debug has no minimum time requirement, but using a different queue such as preempt requires allocating 2 hours minimum for a job (which I think is a bit overkill in our case, since recon should only take 5-10 min).
• Based on testing debug vs preempt, it seems like the the debug jobs get picked up much faster (perhaps due to smaller time constraint, as they are both medium priority).
• There is also the interactive queue, which has a high priority, but only 4 nodes x 2 jobs can be active simultaneously. We would have to modify the job submission script to use salloc

Just to add some numbers to this after some testing today:

• jobs in the debug queue have been consistently picked up after about 10 minutes
• 2 jobs I scheduled this morning in the preempt queue have yet to be picked up (approaching 4 hours...)

[davramov]

Bjoern let us know that we (under the 'als' project) have access to the realtime queue at NERSC, which should reduce latency. I had an issue using that queue with the following error:
Error during job submission or completion: sbatch: error: Batch job submission failed: Invalid qos specification

I opened a ticket and we are still waiting to hear from the help desk about the correct job parameters to use the queue:
https://nersc.servicenowservices.com/sp?sys_id=03d747c187a216101c6cea470cbb35e9&view=sp&id=ticket&table=incident

[dylanmcreynolds]

Now that you have a nice clean NERSCTomographyHPCController, you have something you can easy write unit tests for. I'd recommend mocking NerscClient and test that the calls to it are as expected. Also, are we still sure that we need NerscClient?

[davramov]

Now that you have a nice clean NERSCTomographyHPCController, you have something you can easy write unit tests for. I'd recommend mocking NerscClient and test that the calls to it are as expected. Also, are we still sure that we need NerscClient?

Sounds good to me, I will work on adding unit tests. I suspect we can replace references to NerscClient with the official sfapi_client Client class, and capture that logic in create_nersc_client() in NERSCTomographyHPCController. That could be a bit cleaner.

[dylanmcreynolds]

Let's do the following after discussion:

• add a big comment to the NerscClient saying that it's deprecated and will be removed when we refactor the ptychography code

• Remove initialization of Config832 and sfapi Client from NERSCTomogrpahyHPCController, passing them down from flows to tasks as parameters

• Create unit tests for NERSCTomogrpahyHPCController methods

[davramov]

• Added comments in orchestration/nersc.py indicating that NerscClient is deprecated and will be removed.
• Removed init of Config832 and sfapi_client from NERSCTomographyHPCController, and instead inject them (no longer optional).
• Updated unit tests in test_sfapi_flow.py

[davramov]

I met with Raja and we came up with a few next steps to wrap this up:

• Clean up home/scratch directories to make it production ready (in recon and multires implementations)
  • Copy data from /global/cfs/cdirs/als/data_mover/8.3.2/raw to pscratch for processing
• Reconstructions need to be saved in the appropriate scratch/ location on NERSC
  • Working through a "permission denied" issue copying reconstruction from pscratch to /global/cfs/cdirs/als/data_mover/8.3.2/scratch
• Transfer reconstructions back to data832
• Scheduling pruning
  • Code exists, just waiting to test once data transfers are sorted out.
• Register Prefect flows/deployments for NERSC tomo reconstruction
  • Ready to deploy using create_deployments_832_nersc.py once the prior steps are sorted out
• Create a new agent container on flow-prd to handle NERSC tomo reconstruction
• Come up with a game plan for sfapi credential reauthorization (can we request an SFAPI Client that will be valid for the whole cycle?)