Skip to content

POC: add runtime teleoperation policy#158

Draft
MarkRedeman wants to merge 3 commits into
openvinotoolkit:mainfrom
MarkRedeman:mark/runtime-teleoperation-policy
Draft

POC: add runtime teleoperation policy#158
MarkRedeman wants to merge 3 commits into
openvinotoolkit:mainfrom
MarkRedeman:mark/runtime-teleoperation-policy

Conversation

@MarkRedeman

Copy link
Copy Markdown

Reopening #131 on main branch.

TL/DR: implement a model policy whose predicted actions are the observed state from a leader robot.

class TeleoperatorPolicy:
    def predict_action_chunk(self, observation: dict[str, np.ndarray]) -> np.ndarray:  # noqa: ARG002
        """Read the leader and return one action for the follower."""
        leader_obs = self.leader_robot.get_observation()
        action = np.asarray(leader_obs.joint_positions, dtype=np.float32)
        return action[None, :]

To do this we change the InferenceModel in our runtime and execution classes to be a RuntimePolicy, which is a protocol that's implemented by both InferenceModel and TeleoperatorPolicy.

I'm reopening this mainly to show how teleoperation could be done, though it might be better to spend a bit more time on the interface design for this. Another open that we would need to look into is see if we can add a RecordingCallback so that we can record a teleoperation session.

Adds teleoperation support to PolicyRuntime by introducing a generic
RuntimePolicy protocol and TeleoperatorPolicy, allowing a leader robot’s
joint state to be used as follower actions without a trained model.

Includes a new examples/runtime/teleoperation.py script for SO101,
WidowXAI, and bimanual WidowXAI leader-follower setups, with
Rerun/camera support and separate SO101 leader/follower calibration
files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant