All-zero, phantom frame w/unqiue ID -1 seen by HDF plugin on acquire

[jwlodek]

We are working on moving our xs3 systems from CentOS7 to RH8 given EOL of CentOS7.

The XS3 connects just fine, and collects data that looks correct, however, all the plugins seem to get an extra phantom frame. This frame has a unique id of -1, and is compromised of all zeros. This frame appears immediately, regardless of deadtime. The next frame has a unique id of 1 and is correct w/ the expected peak shape, and takes the correct amount of deadtime. In ADBase I only see one frame was captured which is what I expect. Where could this phantom frame be coming from?

[newville]

@jwlodek I don't think I've seen a frame with unique id of -1. Can you send more information about what mode you're running in (like, internal mode, hardware triggered, etc) and what the order of operations is (like, do you first erase, then start acquire, etc).

I don't have a fully connected system at the moment, but we were running with Centos 9 Stream and will be running with RedHat 9.4, and I hope to have something up and running in a few weeks.

Do

[jwlodek]

We are running in internal mode, although we see the same w/ TTL Veto Only mode (with the difference being that the single phantom frame is acquired, and then the detector waits for TTL pulses.

I then set NumCapture in the HDF plugin to 0, and start capturing. Then, I start acquisition on the XS3 screen. The detector reports collecting a single frame, but the HDF plugin shows two frames. First frame is always Unique ID -1 w/ 0s for all points, while the second frame is valid, and what we expect to see (and is consistent with data produced with the same calibration on the old CentOS 7 server).

I checked out the same commit hash of this module's code that we were running on CentOS 7 and am using that, but the behavior is the same. See the camonitor/caputs I can use to reproduce below

[jwlodek@xf06bm-ws3 ~]$ caput XF:06BM-ES{Xsp:1}:HDF1:NumCapture 0│
Old : XF:06BM-ES{Xsp:1}:HDF1:NumCapture 0                        │
New : XF:06BM-ES{Xsp:1}:HDF1:NumCapture 0                        │
[jwlodek@xf06bm-ws3 ~]$ caput XF:06BM-ES{Xsp:1}:HDF1:Capture 1   │
Old : XF:06BM-ES{Xsp:1}:HDF1:Capture Done                        │
New : XF:06BM-ES{Xsp:1}:HDF1:Capture Capture                     │[jwlodek@xf06bm-ws3 ~]$ camonitor XF:06BM-ES{Xsp:1}:HDF1:UniqueId_RBV        
[jwlodek@xf06bm-ws3 ~]$ caput XF:06BM-ES{Xsp:1}:det1:Acquire 1   │XF:06BM-ES{Xsp:1}:HDF1:UniqueId_RBV 2024-05-21 14:25:10.757567 1             
Old : XF:06BM-ES{Xsp:1}:det1:Acquire Done                        │XF:06BM-ES{Xsp:1}:HDF1:UniqueId_RBV 2024-05-21 14:41:55.741129 -1            
New : XF:06BM-ES{Xsp:1}:det1:Acquire Acquire                     │XF:06BM-ES{Xsp:1}:HDF1:UniqueId_RBV 2024-05-21 14:41:56.815208 1             
[jwlodek@xf06bm-ws3 ~]$                                          │

[dmgav]

This issue seems to be related to #46 (@jklynch did not fully describe the details). We still observe this issue and it is not limited to TTL Veto Only mode.

The issue is observed if XS3 is switched to TTL Veto Only mode and capture is enabled in HDF5 plugin. Once acquisition is enabled, one 'empty' frame (all zeros) is saved to HDF5 file before any trigger pulses are applied. The Unique ID of the frame is -1. Additional frames are saved to the file if trigger pulses are applied, but the file always contains one additional empty frame.

[jwlodek]

Here is the first frame in Nexpy, all zeros.

And the second image w/ real data.

Here are the Unique IDs saved as attributes in the HDF file.

[jwlodek]

xspress3/xspress3App/src/xspress3Epics.cpp

Line 984 in a50c2b0

this->setNDArrayAttributes(pMCA, -1);

I believe this line is where the UID of -1 is coming from

[newville]

@jwlodek Hm, I'm not sure what could be happening, and sorry I don't have a setup I can use for testing at the moment -- the APS upgrade has been a bit disruptive in getting detectors set up ;).

FWIW the change you point to was made 8 years ago, so it seems strange that it would only show up now, and only when switching OS versions.

One more technical question: are you running with "circular buffers" enabled (last argument of xspress3Config()? If so, I might suggest trying setting that to 0.

[jwlodek]

I was originally running with the newest released version (3.2.8), but downgraded to the version we had running on CentOS 7 (2.6), which removed the circular buffer option, it didn't seem to make a difference.

I was able to find the cause of the phantom frame - basically, it seems that the above line of code is called when you erase, it creates an NDArray with all zeros, with uniqueID of -1, and feeds it through the plugins to zero them out. When I turn off EraseOnStart the phantom frame goes away - and on the other hand, if I start the HDF5 plugin in capture mode w/ auto-saving, and repeatedly press the Erase button, each press generates one of these all-zero frames with ID -1.

I wonder if there would be a way to do callbacks on the zero frame only with the ROI/Stat plugins to zero them out without doing a callback to the HDF plugin so that the phantom frame is not saved?

[newville]

@jwlodek Hm, EraseOnStart should not be needed. And you should not need to do an ERASE before doing an Acquire.

But also, turning off circular buffer mode does not require rolling back to an earlier version, just setting that to 0 for xspress3Config will be enough.

If you are seeing a frame with ID -1 saved to the HDF5 plugin, what are the dwell time and other stats for that frame?