-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Add APIs to Isaac Lab Mimic for supporting loco-manipulation datagen #3992
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greptile Overview
Greptile Summary
This PR adds support for loco-manipulation data generation in Isaac Lab Mimic by introducing three key features:
- Body End Effector Processing: Treats the robot's base as an end effector (named "body") to enable lower-body movement control during data generation
- Navigation P-Controller Integration: Adds
use_navigation_p_controllerflag andget_navigation_state()API to manage mobile base navigation, with logic to repeat/skip trajectory steps based on navigation state - Custom Recorder Configuration: Allows users to provide optional custom recorders via
mimic_recorder_configfor recording additional data beyond default action/state
Key Issues Found:
- Critical timing bug in
data_generator.pylines 911-918: The step index is incremented at line 898 before checking if it's at the last step at line 912. This means the "repeat last action" logic triggers one step early (atlen-2instead oflen-1), causing incorrect synchronization between navigation state and trajectory execution. - Performance inefficiency where "body" validation runs inside nested loops instead of once at initialization.
Confidence Score: 2/5
- This PR has a critical logic bug that will cause incorrect behavior during navigation-controlled data generation
- The step index increment timing bug in the navigation controller logic will cause the trajectory synchronization to be off by one step, potentially leading to premature or delayed navigation transitions. While the feature additions are well-structured (clean API design, proper config additions, good documentation), the core navigation logic flaw makes this unsafe to merge without correction.
- Pay close attention to
source/isaaclab_mimic/isaaclab_mimic/datagen/data_generator.py- the navigation controller logic needs the step index check timing fixed
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| source/isaaclab_mimic/isaaclab_mimic/datagen/data_generator.py | 2/5 | Added navigation p-controller support with body end effector processing, but has critical timing bug where step index check happens after increment |
| source/isaaclab_mimic/isaaclab_mimic/datagen/generation.py | 5/5 | Added exception handling with traceback and optional custom recorder config parameter - clean implementation |
| source/isaaclab/isaaclab/envs/manager_based_rl_mimic_env.py | 5/5 | Added get_navigation_state API method stub with clear documentation - implementation left to subclasses as intended |
Sequence Diagram
sequenceDiagram
participant Env as ManagerBasedRLMimicEnv
participant DG as DataGenerator
participant Nav as Navigation P-Controller
participant Body as Body End Effector
Note over DG: use_navigation_p_controller enabled
DG->>DG: Initialize was_navigating = False
loop Each Timestep
DG->>Env: get_navigation_state(env_id)
Env-->>DG: {is_navigating, navigation_goal_reached}
loop For Each End Effector
DG->>DG: Increment current_eef_subtask_step_indices[eef_name]
alt Body at last step AND navigating AND not goal reached
DG->>DG: Decrement all step indices (repeat last action)
Note over DG,Body: Keep robot moving until waypoint reached
else Was navigating AND stopped AND not processed
DG->>DG: Skip to end of nav subtask
Note over DG,Body: Goal reached early, skip remaining trajectory
end
DG->>DG: Check if subtask complete
end
DG->>DG: Update was_navigating = is_navigating
end
8 files reviewed, 3 comments
| """ | ||
| return dict(env_name=self.spec.id, type=2, env_kwargs=dict()) | ||
|
|
||
| def get_navigation_state(self, env_ids: Sequence[int] | None = None) -> dict[str, torch.Tensor]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you make it as optional, ideally both in doc and code? It will confuse users about why navigation appears in this mimic_env.py. At least at code level, it shall specify it's not needed for tabletop manip. You can define a decorator @optional_method.
def optional_method(func):
"""
Decorator to mark a method as optional for subclasses.
"""
func.__optional__ = True
return func
class Base:
@optional_method
def get_navigation_state(self):
print("Base implementation")
| Added | ||
| ^^^^^ | ||
|
|
||
| * Add body end effector to Mimic data generation to enable loco-manipulation data generation when a navigation p-controller is provided. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If possible, considering what else needed to go with any feedback controller(i.e. P vs PD vs PID). It sounds to me this Mimic eef interface shall be agnostic to what controller it is.
| is_navigating = navigation_state["is_navigating"] | ||
| navigation_goal_reached = navigation_state["navigation_goal_reached"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add assertion (navigation_state is not None && keys in the dict) before accessing the keys here?
Description
This PR adds new APIs to Isaac Lab Mimic to expand support for loco-manipulation data generation.
It adds:
Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there