[Question] Is there support/potential bug for multiple cameras per parallel environment and RGB+Depth together? #535
-
QuestionI just saw isaac lab get released and was trying to test out the new tiled rendering features. I was wondering what is the state of this feature now, will there be planned support for RGB and/or Depth and/or Segmentation? Currently I was successfully able to run tiled rendering with one of RGB or Depth. I am also wondering is it possible to put more than one camera into the scene? Happy to share example code that I tried writing to get multiple cameras but the error I get so far is internal that can't be debugged from my side: The relevant code snippets are these: In the Cartpole direct env class: def __init__(...):
# ...
self.tiled_rgb_camera_cfgs = []
self.tiled_depth_camera_cfgs = []
for i in range(num_cameras):
if self.has_rgb:
tiled_camera_cfg = TiledCameraCfg(
prim_path=f"/World/envs/env_.*/Camera_RGB_{i}",
offset=TiledCameraCfg.OffsetCfg(pos=(-4.0, 0.0, 3.0), rot=(0.9945, 0.0, 0.1045, 0.0), convention="world"),
data_types=["rgb"],
spawn=sim_utils.PinholeCameraCfg(
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20.0)
),
width=camera_width,
height=camera_height,
)
self.tiled_rgb_camera_cfgs.append(tiled_camera_cfg)
if self.has_depth:
tiled_camera_cfg = TiledCameraCfg(
prim_path=f"/World/envs/env_.*/Camera_Depth_{i}",
offset=TiledCameraCfg.OffsetCfg(pos=(-4.0, 0.0, 3.0), rot=(0.9945, 0.0, 0.1045, 0.0), convention="world"),
data_types=["depth"],
spawn=sim_utils.PinholeCameraCfg(
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20.0)
),
width=camera_width,
height=camera_height,
)
self.tiled_depth_camera_cfgs.append(tiled_camera_cfg)
# ...
def _setup_scene(self):
"""Setup the scene with the cartpole and camera."""
self._cartpole = Articulation(self.cfg.robot_cfg)
if self.has_rgb:
self.tiled_rgb_cameras = [TiledCamera(cfg) for cfg in self.tiled_rgb_camera_cfgs]
if self.has_depth:
self.tiled_depth_cameras = [TiledCamera(cfg) for cfg in self.tiled_depth_camera_cfgs]
# add ground plane
spawn_ground_plane(prim_path="/World/ground", cfg=GroundPlaneCfg(size=(500, 500)))
# clone, filter, and replicate
self.scene.clone_environments(copy_from_source=False)
self.scene.filter_collisions(global_prim_paths=[])
# add articultion and sensors to scene
self.scene.articulations["cartpole"] = self._cartpole
if self.has_rgb:
for i in range(self.num_cameras):
self.scene.sensors[f"tiled_rgb_camera_{i}"] = self.tiled_rgb_cameras[i]
if self.has_depth:
for i in range(self.num_cameras):
self.scene.sensors[f"tiled_depth_camera_{i}"] = self.tiled_depth_cameras[i]
# add lights
light_cfg = sim_utils.DomeLightCfg(intensity=2000.0, color=(0.75, 0.75, 0.75))
light_cfg.func("/World/Light", light_cfg) |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments
-
|
Seconded. This would be a very useful feature. |
Beta Was this translation helpful? Give feedback.
-
|
Shameless plug: I made a camera benchmark tool that within demonstrates how to do more than one camera per environment: If you clone my fork you can try more than one camera per env with ./isaaclab.sh -p source/standalone/tutorials/04_sensors/benchmark_cameras.py --task Isaac-Cartpole-v0 --num_tiled_cameras 100 --task_num_cameras_per_env 2 --tiled_camera_data_types rgb --experiment_length 500 --tiled_camera_data_types rgb depth |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I have similarly done the benchmarking as well on v1.2.0 ablating on camera sizes, number of cameras, and modalities, although code is not super clean at the moment: https://github.com/haosulab/ManiSkill/tree/main/mani_skill/examples/benchmarking, figures: https://maniskill.readthedocs.io/en/latest/user_guide/additional_resources/performance_benchmarking.html Anyway this issue is closed, it's great to see there is now support for these setups in v1.2.0 |
Beta Was this translation helpful? Give feedback.
Shameless plug: I made a camera benchmark tool that within demonstrates how to do more than one camera per environment:
#976
If you clone my fork you can try more than one camera per env with
./isaaclab.sh -p source/standalone/tutorials/04_sensors/benchmark_cameras.py --task Isaac-Cartpole-v0 --num_tiled_cameras 100 --task_num_cameras_per_env 2 --tiled_camera_data_types rgb --experiment_length 500 --tiled_camera_data_types rgb depth