propagate_in_video can not Propagate the input points in entire video

[RCLaplace]

when I was tracking a video, I noticed some errors in the tracking results. Then, I followed the instruction in the ipynb to add some points, but the changes were not reflected except for the specific frames where I added the points.

What should I do?

[heyoeyo]

One simple/quick thing that might help is to limit the number of 'prompted' frames the system uses to 1 (it uses them all by default). It may be that your first/oldest prompt is influencing the result, preventing the later correction prompt from working properly. Limiting the model to use only 1 prompt should allow the newer prompt to 'take over' later in the video.

You can change this either by directly editing the default value in the code itself, or otherwise it should be possible to add a new line (for example, just after the image_size entry should work) to the config files:

# Only allow one prompt to affect video segmentation
max_cond_frames_in_attn: 1

Strangely, I think you also need to comment out the assert line in the select_closest_cond_frames function (which is where this setting gets used), so that it doesn't trigger an error. There doesn't seem to be any reason for the assertion, but as long as it's there, you can't run with a setting of 1 if more than one is available.

[RCLaplace]

One simple/quick thing that might help is to limit the number of 'prompted' frames the system uses to 1 (it uses them all by default). It may be that your first/oldest prompt is influencing the result, preventing the later correction prompt from working properly. Limiting the model to use only 1 prompt should allow the newer prompt to 'take over' later in the video.

You can change this either by directly editing the default value in the code itself, or otherwise it should be possible to add a new line (for example, just after the image_size entry should work) to the config files:

# Only allow one prompt to affect video segmentation
max_cond_frames_in_attn: 1

Strangely, I think you also need to comment out the assert line in the select_closest_cond_frames function (which is where this setting gets used), so that it doesn't trigger an error. There doesn't seem to be any reason for the assertion, but as long as it's there, you can't run with a setting of 1 if more than one is available.

oh! Thank you very much! The problem regarding prompt propagation has been solved. But then I found another one, when I add new points to the same object, the mask of the object before is not inherited on the frame after the ponit is added, but only the mask recoginzed by the added points appears

[heyoeyo]

when I add new points to the same object, the mask of the object before is not inherited on the frame after the ponit is added, but only the mask recoginzed by the added points appears

I think this is normal, especially if using the change to the max_cond_frames_in_attn setting, which will result in only 1 prompt being considered when masking. If the mask is incomplete after adding a new point, I think the best option would be to use a multi-point prompt (like the video notebook example shows in cell [12] under the section: Step 2: Add a second click to refine the prediction) to expand the mask to include any areas that are missing.

Alternatively, you can try increasing that max_cond_frames_in_attn setting so that more than 1 prompt influences the masking result, but that might go back to causing problems where the older prompts interfere with newer prompts, so it's a bit of a tradeoff.