feature request: ability to generate a mini, custom, complete dataset / specprod from existing specprod files

[moustakas]

io.spectra.read_spectra_parallel is terrific, but it would be really useful if it had a redrock option like io.spectra.read_tile_spectra, with additional keywords to also generate row-matched catalogs of all the after-burner files (quasarnet, MgII, emfit, etc.).

With this capability, it becomes trivial to generate a mini "specprod" of arbritrary size across the DESI footprint.

In fact, what I really want isn't just the "read" functionality, but the ability to gather all the data and then write out the corresponding files (with all the appropriate headers). For example, redrock.external.desi.write_zbest requires not just the Redrock table, but also metadata regarding the templates, the spectroscopic header, as well as the row-matched fibermap, exp_fibermap, and tsnr2 tables in order for the output file to be used downstream.

Perhaps we need some new functionality (beyond read_spectra_parallel) to implement this request, but I think it would be really useful to have.

Related: I wrote some code to do exactly what I've requested as part of the template work, so in principle this could be generalized, documented, and moved over to desispec:
https://github.com/desihub/rrevaluator/blob/main/py/rrevaluator/io.py#L302-L416

@sbailey, thoughts?

[sbailey]

I really like this idea. Our spectra format was specifically designed to support read+filter+write, and it would be great to expand that concept to filtering all of the matching redrock+afterburner files.

Caveat: I would not want to get into the business of using this tool to make N>>1 companion prods for every major prod, e.g. separate prods for each of ELG, LRG, QSO, BGS, MWS. But it could be great for making small test datasets and for combining data from tertiary programs or individual analyses into a more digestible form.

IIUC you are only suggesting this for the end-product tiles/ and healpix/ based files, not the intermediate cframe etc. files in the exposures. That's fine with me, but I highlight that since this kind of request could get bogged down in "but it doesn't do everything!" when doing a subset of the possibilities would still be really useful.