This project provides a PyBullet-based simulation environment for the Franka Emika Panda robot, integrated with ROS functionality. It allows users to simulate the Panda robot in both physics and visual modes, offering various control and interaction features.
The core of this project lies in its teleoperation assistance functionality. We utilize GraspNet for grasp pose estimation, generating potential grasp poses. These poses are then incorporated into a 6D Gaussian field, which generates grasp guidance for every point in space. This guidance is superimposed on the Quest-controlled Franka end-effector, providing assisted robot grasping.
Experimental results demonstrate that this assistance feature significantly improves grasp success rates and reduces the cognitive load on operators.
Key benefits:
- Enhanced grasp success rates
- Reduced operator mental workload
- Intuitive and seamless integration of AI-assisted grasping with human control
This innovative approach bridges the gap between autonomous robotic grasping and human-controlled teleoperation, offering a more efficient and user-friendly solution for remote manipulation tasks.
- PyBullet-based Franka Panda robot simulation
- ROS integration with support for publishing and subscribing to relevant topics
- Both physics and visual simulation modes
- Support for position, velocity, and torque control
- Gripper control functionality
- Camera image acquisition and publishing
- Object manipulation and grasping capabilities
- VR control interface
Base simulation environment class, defined in env.py. It initializes the PyBullet environment, loads the Panda robot model, and provides basic simulation stepping methods.
Physics simulation environment class, defined in env_ros.py. It inherits from FrankaPandaEnv and adds ROS-related functionality, such as publishing joint states and subscribing to control commands.
Visual simulation environment class, also defined in env_ros.py. It provides functionality for capturing camera images and publishing image data.
Robot control class, defined in panda_robot.py. It encapsulates low-level control methods for the robot, such as setting joint positions, velocities, and torques.
Grasping node class, defined in grasp_node.py. It implements point cloud-based grasp pose estimation functionality.
VR control node class, defined in vr_control_node.py. It allows control of the robot through VR devices.
-
Launch physics simulation
-
Launch visual simulation
-
Run VR control
-
Run grasp estimation
- ROS
- PyBullet
- NumPy
- OpenCV
- roboticstoolbox
- Open3D
- The simulation environment uses a default sampling rate of 1000Hz.
- Both complete Panda models and simplified models without hands are provided.
- Inverse kinematics calculations may be slow; optimization should be considered.
- VR control and grasp estimation features require additional hardware or sensor support.
Issue reports and pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
MIT