GBU not finding bad images

[detoma]

On 2022 02 17 there are some 1079 images that appear to have no signal and are labeled good:

• 20220217.203601.ucomp.1079.l1.p3.fts
• 20220217.203833.ucomp.1079.l1.p3.fts
• 20220217.204105.ucomp.1079.l1.p3.fts

[mgalloy]

Here is the 1079 nm GBU log for the day:

Filename                               Reason     Bkg         V      RCAM      TCAM   Sigma
20220217.193203.ucomp.1079.l1.p3.fts        0     7.6      0.14       0.0       0.0    0.64
20220217.193435.ucomp.1079.l1.p3.fts        0     7.6      0.14       0.0       0.0    0.64
20220217.193707.ucomp.1079.l1.p3.fts        0     7.6      0.14       0.0       0.0    0.64
20220217.193940.ucomp.1079.l1.p3.fts        0     7.6      0.14       0.0       0.0    0.64
20220217.203601.ucomp.1079.l1.p3.fts        0    12.2      0.14       0.0       0.0    1.47
20220217.203833.ucomp.1079.l1.p3.fts        0    12.2      0.14       0.0       0.0    1.47
20220217.204105.ucomp.1079.l1.p3.fts        0    12.3      0.14       0.0       0.0    1.48

GBU bitmask codes
Code    Description
    1   spar guide control loop is below threshold (0.99)
    2   spar guider intensity below threshold (0.9 * 9.256 * exp(-0.05 * secz))
    4   median background is above threshold (35.0)
    8   median background is below threshold (5.0)
   16   spurious Stokes V signal is above threshold (8.0)
   32   the chi-squared of the occulter fit is above threshold (80.0)
   64   the difference of the image with the median is above threshold (1.5)

The sigma was just under the threshold, e.g., 1.48 < 1.5, so it just passed the condition 64. This is partly because there were only 7 files that passed quality. So the 3 files with no signal are averaged into a mean of 7 files and then those 3 bad files are not that far from the mean.

There is no condition for signal, but we could add one. I need thresholds and what exactly to test. Just intensity in some annulus at center line?

[detoma]

The sigma criteria is usually too restrictive and we often eliminate good 1079 images. I think we need to add an intensity level criteria to the level0 when we do the quality.

The bad files are not all zeros, there is noise and this is why the average level is not too off. I see the instrument was paused for clouds but it was later in the day

It seems in this time interval there was a polarization calibration in 656 and 789. Not sure why the images were labelled 1079. We need Ben to look into it. This seems a problem at the telescope.

[bberkeyU]

The L0 images have significant signals around the occulter, so they are not darks. However, I don't see any coronal signatures in these files like I did earlier in the day. The diffuser may be in the beam or some other transparent blocker.

By eye, the most significant difference between the good 1079 earlier in the day and these bad files later in the day is the lack of a bright ring around the occulter. Perhaps we need a metric to account for a steep gradient in the image as we approach the occulter.

[jburkepile]

I checked headers and there is nothing to indicate there is anything in the beam other than the occulter. The observer log does report pausing for clouds:
'Thu Feb 17 20:26:29 GMT 2022 UCoMP Paused for clouds'
Then UCoMP was restarted:
'Thu Feb 17 20:34:51 GMT 2022 UCoMP Restarted from pause'
The only good image I see after 20:26 is a 637 nm image at 20:45. There are no 789, 1074 or 706 images after 20:26. There are only bad 1079 images after 20:26.

Could clouds have still been a problem but just not mentioned in the log?

[bberkeyU]

Looking at the webcam images, the 20:26:29 pause was related to a visual inspection of the Kcor Foreopitcs that involved opening covers, etc.

Then, at about 20:31:00, the spar was rotated horizontally to remove the kcor O1 for cleaning.

Skies remained clear (maybe not wholly coronal) until about 20:47:00 when clouds started building, and the dome was not closed for these clouds until about 21:58:00

The webcam images show no obvious external changes to UCoMP, I dont see anything getting placed in front of the O1 like an ND filter or something else. It remains a mystery.

[detoma]

Ben: According to the observer log these were not science image and not 1079 image either. It seems they were mislabelled as 1079 science images but were not.

[bberkeyU]

In ucomp l0 sci just means the dark shutter, diffuser or calib optics are not in the beam. The observing code does know anything other than the reported stages of these optics there is no place for the observer to add input (beyond moving a stage into the beam). kcor has this idea of sci/engineering but for ucomp we didn't build that feature.

To catch whatever this was every time it happens I think we need to look at the structure of the image valid science images will see the intensity increase toward the occulter the late 1079 don't.

[detoma]

I do not think there is an easy way to test the profile. Other simple criteria failed to find these bad images because there was some light coming in but no corona. Mike will mark these images as bad in the epoch file. We discussed it and this is the easiest way to get this particular instance taken care of.

About the GBU criteria, the sigma criteria does not work well on short days or on days when all images are bad. On short days with a mix of good and bad images, often good images are eliminated. On days when all images are bad, they all pass the sigma criteria. Unfortunately for most wavelengths, we do not have a lot of statistics. With Mike we discuss the following approach:

• increase the sigma threshold to 1.8 to avoid label bad good images (this will pass some bad images, hopefully they will be caught by the other criteria)
• do not apply the sigma criteria if there are fewer than 8 images

An engineering product that can be useful to find outliers and monitor long term changes is to store in the database the median value of the center line intensity inside some predefined annuli. We used this to look at long-term trend in COMP. We selected these heights for UCOMP:

• 1.10solar radii +/- 0.02
• 1.15solar radii +/-0.02
• 1.20 solar radii +/-0.02
  This can also be done with the peak intensity.

[mgalloy]

Adding the new database values is now #287.