What exactly is object pointer?

[jeezrick]

Hi, I would appreciate it if someone could shed some light on the object pointer feature introduced in SAM2.

As mentioned in the paper:
In addition to the spatial memory, we store a list of object pointers as lightweight vectors for high-level semantic information of the object to segment, based on mask decoder output tokens of each frame.

However, in the (inference) code, I just don't see the benefit of this particular token. Moreover, the tracking still seems to work if you disable it. I wonder if someone could share their insights on this.

[jeezrick]

and of course I see the ablation experiment.

[13512263278]

the object pointer vector for the output mask, extracted based on the output token from the SAM mask decoder.

[Lr-2002]

I'm now trying to extract the object pointer from the sam2 model and return it.
while I see the num of the object pointer always is 3 while i input 5 points into the sam with multimask_output = True
Have anyone faced this problem before

I've already notice the pointer num was hard coded in the _build_sam_heads, Does it has no realtionship with the decoded mask ?