Thanks for releasing the code and checkpoint. We reproduced most RoboTwin-MeM tasks close to the paper, but
pick_objects_in_order remains substantially lower.
Using the released checkpoint and the reported evaluation setup:
- checkpoint:
RoboTwin-MeM/final_model/pytorch_model.pt
task_config=demo_clean
instruction_type=unseen
ACTION_MODE=abs
unnorm_key=new_embodiment
camera=LargeView (FOV 50)
- seed starts at
100000
- step limit:
1500
- one policy server for this task
we obtain 28/50 = 56.0% on seeds 100000–100049, while Table 2 reports 90% for Pick Objects in Order.
The rollout logs show normal execution and no crashes. Other tasks are broadly consistent with the paper, so this
appears task-specific.
We also noticed that a previous reproduction issue reported the authors' local result as 5/9 for this task, which
is close to our 56%, but it was not further addressed.
Could you please clarify:
- Does Table 2's 90% use exactly
pick_objects_in_order with the released steps_100000 / final_model
checkpoint?
- Can you share the exact evaluation command/config used to obtain the reported 90%?
We can provide rollout videos and full logs if useful.
Thanks for releasing the code and checkpoint. We reproduced most RoboTwin-MeM tasks close to the paper, but
pick_objects_in_orderremains substantially lower.Using the released checkpoint and the reported evaluation setup:
RoboTwin-MeM/final_model/pytorch_model.pttask_config=demo_cleaninstruction_type=unseenACTION_MODE=absunnorm_key=new_embodimentcamera=LargeView(FOV 50)1000001500we obtain 28/50 = 56.0% on seeds
100000–100049, while Table 2 reports 90% forPick Objects in Order.The rollout logs show normal execution and no crashes. Other tasks are broadly consistent with the paper, so this
appears task-specific.
We also noticed that a previous reproduction issue reported the authors' local result as
5/9for this task, whichis close to our 56%, but it was not further addressed.
Could you please clarify:
pick_objects_in_orderwith the releasedsteps_100000/final_modelcheckpoint?
We can provide rollout videos and full logs if useful.