Sim2Real Support and Roadmap for Franka Panda Robot

[mikelasa]

Hi ManiSkill team,

First off, thanks for the amazing work on ManiSkill and ManiSkill2 — the environments and documentation are super helpful for sim-based manipulation research. I'm kind of new at this but I'm interested in the sim2real part. So far:

1.     I trained a diffusion policy using RGB inputs in your simulation environments (with a Franka robot).
2.     I’m now setting up a physical Franka Panda to test this model in the real world.
3.     I saw the [Digital Twins page](https://maniskill.readthedocs.io/en/latest/tasks/digital_twins/index.html) and noticed that sim2real training support is still under development.

Could you share:

1.     What’s your roadmap or timeline for supporting sim2real transfer (especially using real hardware)?
2.     Will there be official support for the Franka Panda robot (e.g., ROS integration, observation/action format alignment)?
3.     Are there plans to include tools or wrappers to help connect trained ManiSkill policies to real robot platforms?

Thank you!

[StoneT2000]

For ROS integration at least our side won't work on it soon. There seems to be some community members investigating this on our discord. One reason why I won't work on it is because I am not experienced with ROS and even then I think it is not clear where ROS fits in maniskill.

For sim2real we have a demo sim2real project for visual zero shot sim2real of a RL policy. It's essentially just a lot of domain randomization.

For panda specifically that arm is fairly good for sim2real since dynamics wise that arm is very simple. You may want to tune the PD parameters of the robot controller for your own use cases however, we don't have a real Panda and thus can't do system ID for that; we only confirm the robot simulates correctly

For your diffusion policy training if you train on visual data it will most likely not transfer to the real world because sim rendering has a domain gap vs real world images. State based will probably transfer fine if you build a state estimator of eg object position.

Timeline wise probably in next month we will release a sim2real visual RL demo and tutorial

For connecting ManiSkill to real hardware deployments the only integration we support now is via LeRobot. If your robot works with LeRobot, it can connect with ManiSkill so that you can control the real robot with a gym interface like ManiSkill and use the same sim controllers in the real world (tutorial for how to set that up also next month)