util_normflat missing one of the orders with latest ESO cr2res pipeline

[avigan]

The latest version of the cr2res pipeline misses the top order in the util_normflat recipe with the default parameters:

I ran again the pipeline on old calibrations (November 2023) and the same problem occurs, which is why I think it comes from the pipeline. I guess the problem can be easily corrected with some changes in the default parameters of the recipe, but I have not yet had the time to look into it. I create this ticket to keep track of the problem.

[avigan]

It seems the problem is at the level of util_extract() for the flat. One of the orders cannot be extracted:

In [9]: pipeline.util_extract(calib_type='flat', verbose=False, create_sof=True)

---------------------
Extract FLAT spectrum
---------------------

EsoRex recipe: cr2res_util_extract

Calibration type: flat
Verbose: False
Create SOF: True

Creating SOF file:
   - calib/util_calib_flat/cr2res_util_calib_calibrated_collapsed.fits UTIL_CALIB
   - calib/util_slit_curv/cr2res_util_calib_calibrated_collapsed_tw_tw.fits UTIL_SLIT_CURV_TW

Creating configuration file: config/util_extract_flat.rc [DONE]

Running EsoRex...[ ERROR ] cr2res_util_extract:       Extraction outside detector edges impossible
[ ERROR ] cr2res_util_extract:       Cannot extract order 9, trace 1
[ ERROR ] cr2res_util_extract:       Extraction outside detector edges impossible
[ ERROR ] cr2res_util_extract:       Cannot extract order 9, trace 1
[ ERROR ] cr2res_util_extract:       Extraction outside detector edges impossible
[ ERROR ] cr2res_util_extract:       Cannot extract order 9, trace 1
 [DONE]

Output files:
   - calib/util_extract_flat/cr2res_util_calib_calibrated_collapsed_extr1D.fits UTIL_EXTRACT_1D
   - calib/util_extract_flat/cr2res_util_calib_calibrated_collapsed_extrSlitFu.fits UTIL_SLIT_FUNC
   - calib/util_extract_flat/cr2res_util_calib_calibrated_collapsed_extrModel.fits UTIL_SLIT_MODEL

[avigan]

I managed to make it work by changing the slit_frac parameter of the cr2res_util_extract recipe:

# --slit_frac
# Wished slit fraction.
cr2res.cr2res_util_extract.slit_frac=0.0, 0.7

I think the current behaviour of the ESO pipeline is not appropriate for the top order on the CRIRES detectors, which is always cropped. For that order the extraction should not fail with the default slit_frac=-1.0, -1.0 values. Anyway, until they correct this, I see two possible solutions to implement a fix in pycrires:

1. Either add systematically the option --slit_frac=0.0, 0.7 when calling cr2res_util_extract, but this could potentially have side effects that I do not see right now.
2. Or add a slit_fraction: list = [-1.0, -1.0] in pipeline.util_extract(), which is similar to what already exists in pipeline.util_extract_science(). At least it keeps the flexibility of changing how the function is called.

@tomasstolker tell me what you prefer. I can make the implementation and create a pull request.

[tomasstolker]

Thanks for finding a solution! The slit_frac values can be adjusted in the config/util_extract_flat.rc file. The updated config file will be used when rerunning the recipe with the file already present.

This seems indeed a bug in the EsoRex recipe that was introduced recently since it was not an issue in earlier versions. Maybe mentioning this solution in the docstring and/or documentation might be sufficient until it is fixed in EsoRex.

[avigan]

I have created a ticket on the ESO helpdesk. Let's hope they implement a fix rapidly.