Skip to content

Pick Objects in Order cannot reproduce Table 2 result (56% vs 90%) #4

Description

@yunhang8658

Thanks for releasing the code and checkpoint. We reproduced most RoboTwin-MeM tasks close to the paper, but
pick_objects_in_order remains substantially lower.

Using the released checkpoint and the reported evaluation setup:

  • checkpoint: RoboTwin-MeM/final_model/pytorch_model.pt
  • task_config=demo_clean
  • instruction_type=unseen
  • ACTION_MODE=abs
  • unnorm_key=new_embodiment
  • camera=LargeView (FOV 50)
  • seed starts at 100000
  • step limit: 1500
  • one policy server for this task

we obtain 28/50 = 56.0% on seeds 100000–100049, while Table 2 reports 90% for Pick Objects in Order.

The rollout logs show normal execution and no crashes. Other tasks are broadly consistent with the paper, so this
appears task-specific.

We also noticed that a previous reproduction issue reported the authors' local result as 5/9 for this task, which
is close to our 56%, but it was not further addressed.

Could you please clarify:

  1. Does Table 2's 90% use exactly pick_objects_in_order with the released steps_100000 / final_model
    checkpoint?
  2. Can you share the exact evaluation command/config used to obtain the reported 90%?

We can provide rollout videos and full logs if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions