Unexpected "tail" from previous time frame showing in points layer in multi-animal datasets

[lochhh]

Steps:
1- napari -w movement
2- Load the sample data file SLEAP_two-mice_octagon.analysis.h5
3- The frame slider is at frame 256, but I can see keypoints for both frame 255 and frame 256. Notice there are more keypoints per individual than expected (the corresponding movement dataset says 7 keypoints)

I also get this error message:

The issue is sometimes fixed when loading the video. It is so far always fixed if I load the video and then flick back and forth a few frames.

Note that in the screenshots above, the total number of frames does not correspond to the video length - seems like an issue with the order of the axes when the video is not loaded?

Originally posted by @sfmig in #393 (comment)

---

As far as I can tell, this only happens with SLEAP multi-animal datasets. Interestingly, it happens with the "mixed-labels" version of the Aeon dataset, but not with the "proofread" one. I wonder whether this is somehow related to mixed identities/tracks in SLEAP files. Intriguingly, whether we are using the analysis.h5 or the .slp file doesn't matter.

Originally posted by @niksirbi in #393 (comment)

---

To me it happens with all the multi-animal datasets, not only SLEAP ones (the Sherlock bboxes dataset - which is not in GIN but can share- , and the DLC two mice dataset). I think maybe that is a stronger hint.

Another hint is that the frame slider end is set to 512 in all cases - seems like a default? It makes me think napari is interpreting the axes wrong... Just to be extra clear I am not loading any videos, just data.

Originally posted by @sfmig in #393 (comment)

---

Yay, you make a strong case. If it's all multi-animal datasets, and the problem is solved by pre-loading a video first, it is indeed highly likely that napari does something funky with the axes there. I guess if the video is added first and the points later, napari aligns the axes of the points layer to the image layer? Do you know if pre-loading the image only (still frame) also solves this or do we always need the video?

Originally posted by @niksirbi in #393 (comment)

---

I just tried with the two mice octagon data - the problem still exists if you load a still frame. The number of frames in the slider does not match the number of frames in the data.

Originally posted by @sfmig in #393 (comment)

[willGraham01]

FYI These warnings that were thrown to the terminal

are because the data being loaded in contains NaN values, so the _extent_data property of the Points layer that is created returns

>>> self._extent_data
array([[   0.,   nan,   nan],
       [8999.,   nan,   nan]])

From what I can tell trawling the napari API reference and the docstrings, this property is used by napari to determine whether the given Points layer can be seen in the current view, and thus needs to be drawn. I don't think they're actually related to the double-plotted bug (at least, not causally).

[willGraham01]

I also did some playing around today whilst looking at #433, and noticed the below which might be relevant here:

The ViewerModel is not being updated to reflect the loaded data (or the loaded data is not being mapped correctly to the pre-existing view model) when data is loaded via the movement widget. We see 512 frames in the example workflow given because napari -m movement loads an empty viewer, and the default view model seems to take the default range of 0 -> 512 frames.

Pre-loading a video might be "fixing" the issue because loading a video is updating the range (in the frames dimension) of the ViewModel correctly, and so when the poses data is loaded on top, it fits nicely to the ViewModel range created by the video (since it has the same number of frames as the video). This hypothesis is supported by the fact that you can replicate this effect by "pre-loading" a Points layer that has the same number of "frames" as the poses data, and then loading your poses data:

import napari
import napari.layers
import numpy as np

viewer = napari.Viewer(ndisplay=2)

# A point at the origin in all 9000 frames.
# SLEAP_two-mice_octagon.analysis.h5 has 9000 frames, for reference
points = np.zeros((9000, 3))
points[:, 0] = np.arange(9000)
pl = napari.layers.Points(points)

viewer.add_layer(pl)

dock_widget, plugin_widget = viewer.window.add_plugin_dock_widget("movement")

napari.run()

# Now loading the SLEAP_two-mice_octagon.analysis.h5 dataset via the widget works,
# and there is no "double-plotting" bug. And you can slide to the end of the dataset (frame 8999).

So I think the place to start is to try and figure out why the "load data" button in the widget is not emitting the appropriate signal to the ViewerModel, telling it to update its dims property to fit the dataset that is loaded (in particular, to fit the number of frames). I don't think the issue is napari doing someting funky with the axes per-se, I think we just might not be telling napari that it needs to update the extent of its current VeiwerModel dimensions, or not conforming to its existing ones.

[willGraham01]

Further to the above, manually setting the .scale of the Points layer we are creating bypasses the issue, which further leads me to believe there's an issue with how the layer we are adding is talking to the ViewerModel.

Place a breakpoint at line 151 of loader_widgets.py. Then follow the instructions below in a debugger:

• napari -w movement. NOTE: Default ViewerModel is created, with 512 frames.
• Load SLEAP_two-mice_octagon.analysis.h5. This will hit the breakpoint you placed. Before passing over the breakpoint, notice that the layer we just added to the viewer (self.viewer.layers) has a scale of [1., 1., 1.]. This has not been scaled to the current limits of the ViewerModel - the appropriate scale for the frames dimension should be 512 / 9000 in this case.
• Before passing over the breakpoint, identify the index i in self.viewer.layers that corresponds to the Points layer we are currently adding. Manually set self.viewer.layers[i].scale = np.array([512 / 9000, 1, 1]).
• Allow the debugger to continue. The breakpoint will be passed over, and you will be back in the napari viewer. However you'll notice that now all 9000 frames are visible, and the double-render bug is gone.

[niksirbi]

Thanks @willGraham01 for this excellent piece of deep investigative debugging. I sincerely appreciate it. This gives us a concrete thread to pick up.