Add median value quality checks when reading flats #284
Comments
mgalloy
added
enhancement
mgalloy
changed the title
|
@bberkeyU Are the times listed in Julian date? Why is 12:00:00 so frequent? i.e. is the "2022-11-18 12:00:00.000" listed actually "2022-11-18T12:00:00Z"? |
|
I think there is a missing epoch for 706. I think that the next epoch after: is: So 2021-11-11 12:00:00.000 to 2022-02-23 19:12:00.000 is missing. |
|
@bberkeyU Are there example files that you know will fail the quality check? I would like some good days to test on. |
|
@mgalloy I think they are in Julian Dates: To get the epochs, I pulled the DATE-BEG from all of the header files and converted it to Julian data with the I then plotted the flat medians vs this Julien data and tweaked my X and Y limits until I found good start/stop times and intensity limits. (Note: when doing the start/stop finding, I subtracted 2459000 from all of the Julien dates so I could think about smaller numbers. I converted the values back from Julien date to date time strings with the following command This process gave a 12:00 reference instead of a 00:00 reference time. In most but not all cases, the epochs change at the beginning or end of an observing day, so the exact timestamp isn't going to be important. |
|
@mgalloy I am looking into finding bad days that are both clearly and marginally bad. |
|
I created a workbook with Flat epoch plots. It is a fairly long document with the interesting part most of the way down. In this example plot from the 637 section There is a similar plot for each of the six wave regions (530, 637, 706, 789, 1074,1079) for each epoch region I identified. The upper and lower limits were empirically selected; and others might pick different values. |
Currently, we do not perform image data quality checks on the flats, which has allowed some marginal or deficient flats to be used in the image correction.
Between spot-checking good and bad flats and looking at metrics across the UCoMP mission, I find that for a given epoch, good flats have a range of meidan values vary by wave region, epoch, camera, and onband tuning.
I have computed a table of high and low flat values for each of the 637,706,1074, and 1079 regions. I also looked at 530, but all flats were equally good (or bad?), so the pipeline should pass all the 530 data. If the pipeline needs values for 530 and the other wavelengths, we should start with min=0 and max =4095 and allow everything to pass.
For the analysis, the median value of the raw L0 flat pixels within an annulus centered in the middle of the detector with an inner radius of 375 and an outer radius of 600 pixels was computed; the computation also excluded post values between x=550 ->700 in the raw detector coordinate systems. These median values were then divided by NUMSUM, and the mean was computed across the 4 modulation states.
The table below contains the high and low flat limits for the following states:
Date-beg and date-end define the start and stop of the epochs.
@mgalloy Please let me know if this table should be in a different format; I can easily transform and recreate it.
Possible epoch value definitions, which are all 2-element arrays of min/max values:
Tasks
The text was updated successfully, but these errors were encountered: