use Intel RealSense camera D435i as a YARP device

[jimingda]

Hi,

I'm currently integrating the D435i camera with YARP, and I'm following the guidelines outlined in the "Configure YARP" section. However, I'm encountering a step where I need to enable specific YARP CMake flags, and I'm seeking clarification on how to enable the following YARP CMake flags:

ENABLE_yarpmod_RGBDSensorWrapper  
ENABLE_yarpmod_RGBDSensorClient  
ENABLE_yarpcar_mjpeg  
ENABLE_yarppm_depthimage_to_rgb

My current YARP version is 3.7.0. I appreciate any guidance you can provide on how to enable these flags correctly.

[pattacini]

Hi @jimingda

Did you build YARP via robotology-superbuild or as an isolated project?

[jimingda]

Thank you @pattacini

I didn't build YARP as a isolated project. I think it's built within robotology-superbuild

[traversaro]

Those options are already enabled by default if you build YARP in the robotology-superbuild, see https://github.com/robotology/robotology-superbuild/blob/da4b2d20b7ac08db956f9e8f9ec9f683b67eacb9/cmake/BuildYARP.cmake#L74 .

[jimingda]

Thanks @traversaro

Then i got error when command yarpdev --from sensors/RealSense_conf.ini

icub@icubsrv:~$ yarpdev --from sensors/RealSense_conf.ini
[ERROR] |yarp.os.Property| cannot read from sensors/RealSense_conf.ini
[INFO] |yarp.dev.Drivers| Welcome to yarpdev, a program to create YARP devices
[INFO] |yarp.dev.Drivers| To see the devices available, try:
[INFO] |yarp.dev.Drivers|    yarpdev --list
[INFO] |yarp.dev.Drivers| To create a device whose name you know, call yarpdev like this:
[INFO] |yarp.dev.Drivers|    yarpdev --device DEVICENAME --OPTION VALUE ...
[INFO] |yarp.dev.Drivers|    For example:
[INFO] |yarp.dev.Drivers|    yarpdev --device fakeFrameGrabber --width 32 --height 16 --name /grabber
[INFO] |yarp.dev.Drivers| You can always move options to a configuration file:
[INFO] |yarp.dev.Drivers|    yarpdev [--device DEVICENAME] --from CONFIG_FILENAME
[ERROR] |yarp.dev.Drivers| Unable to find --device option in file sensors/RealSense_conf.ini. Closing.

Shall i initial a RealSense_conf.ini file like below for D435i?

device       RGBDSensorWrapper
subdevice    realsense2withIMU
name         /depthCamera

[SETTINGS]
depthResolution (640 480)    #Note the parentesys
rgbResolution   (640 480)
framerate       30
enableEmitter   true
alignmentFrame  RGB

[HW_DESCRIPTION]
clipPlanes (0.2 10.0)

[pattacini]

Hi @jimingda

The failure is reported at this line:

[ERROR] |yarp.os.Property| cannot read from sensors/RealSense_conf.ini

This means that the configuration file is not found. The instructions do refer to robots that are shipped with that file integrated in the robot folder, like it happens for e.g., R1SN001, ergoCubSN001.

Therefore, as a quick test, you could:

1. Create the configuration file Realsense_conf.ini anew in the icub-head.
2. From the same directory, launch: yarpdev --from Realsense_conf.ini.

[jimingda]

Thanks @pattacini

After logging into icub-head, I added a RealSense_conf.ini file with the following configuration in the home directory:

device       RGBDSensorWrapper
subdevice    realsense2
name         /depthCamera

[SETTINGS]
depthResolution (640 480)    #Note the parentesys
rgbResolution   (640 480)
framerate       30
enableEmitter   true
alignmentFrame  RGB

[HW_DESCRIPTION]
clipPlanes (0.2 10.0)

However, upon attempting to run the command yarpdev --from RealSense_conf.ini, an error occurred indicating that the 'realsense2' plugin could not be found. The error log shows that the system failed to find the "realsense2" plugin and encountered difficulties opening the subdevice. As a result, the RGBDSensorWrapper driver was found but could not be opened. The specific error messages and warnings are as follows:

icub@icub-head:~$ yarpdev --from RealSense_conf.ini
[DEBUG] |yarp.dev.PolyDriver|RGBDSensorWrapper| Parameters are (HW_DESCRIPTION (clipPlanes (0.200000000000000011102 10.0))) (SETTINGS (depthResolution (640 480)) (rgbResolution (640 480)) (framerate 30) (enableEmitter true) (alignmentFrame RGB)) (device RGBDSensorWrapper) (from RealSense_conf.ini) (name "/depthCamera") (single_threaded 1) (subdevice realsense2)
[WARNING] |yarp.devices.RGBDSensorWrapper| The 'RGBDSensorWrapper' device is deprecated in favour of 'rgbdSensor_nws_yarp'.
[WARNING] |yarp.devices.RGBDSensorWrapper| The old device is no longer supported, and it will be deprecated in YARP 3.7 and removed in YARP 4.
[WARNING] |yarp.devices.RGBDSensorWrapper| Please update your scripts.
[INFO] |yarp.devices.RGBDSensorWrapper| Using default 'period' parameter of  0.03 s
[INFO] |yarp.devices.RGBDSensorWrapper| ROS configuration parameters are not set, skipping ROS topic initialization.
[INFO] |yarp.os.Port|/depthCamera/rpc:i| Port /depthCamera/rpc:i active at tcp://10.0.0.2:10042/
[INFO] |yarp.os.Port|/depthCamera/rgbImage:o| Port /depthCamera/rgbImage:o active at tcp://10.0.0.2:10043/
[INFO] |yarp.os.Port|/depthCamera/depthImage:o| Port /depthCamera/depthImage:o active at tcp://10.0.0.2:10044/
[DEBUG] |yarp.devices.RGBDSensorWrapper| Opening IRGBDSensor subdevice
[ERROR] |yarp.os.YarpPluginSettings| Cannot find "realsense2" plugin (not built in, and no .ini file found for it)Check that YARP_DATA_DIRS leads to at least one directory with plugins/realsense2.ini or share/yarp/plugins/realsense2.ini in it
[ERROR] |yarp.dev.PolyDriver|realsense2| Could not find device <realsense2>
[ERROR] |yarp.devices.RGBDSensorWrapper| Opening IRGBDSensor subdevice... FAILED
[ERROR] |yarp.devices.RGBDSensorWrapper| Error while opening subdevice
[ERROR] |yarp.dev.PolyDriver|RGBDSensorWrapper| Driver <RGBDSensorWrapper> was found but could not open
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ===============================================================
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| == Options checked by device:
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ==
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| device=RGBDSensorWrapper
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| id [RGBDSensorWrapper]
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper|     Id assigned to this device
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ==
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ===============================================================
[ERROR] |yarp.dev.Drivers|RGBDSensorWrapper| yarpdev: ***ERROR*** device not available.
[INFO] |yarp.dev.Drivers|RGBDSensorWrapper| Suggestions:
[INFO] |yarp.dev.Drivers|RGBDSensorWrapper| + Do "yarpdev --list" to see list of supported devices.

And the yarpdev --list also got error as:

icub@icub-head:~$ yarpdev --list
[INFO] |yarp.dev.Drivers| Here are devices listed for your system:
[ERROR] |yarp.os.YarpPluginSettings| Error while opening /usr/local/src/robot/robotology-superbuild/build/install/lib/yarp/yarp_fakeFrameGrabber.so:
  /usr/local/src/robot/robotology-superbuild/build/install/lib/yarp/yarp_fakeFrameGrabber.so: undefined symbol: test_grabber
[ERROR] |yarp.os.YarpPluginSettings| Error while opening /usr/local/src/robot/robotology-superbuild/build/install/lib/yarp/yarp_fakeFrameGrabber.so:
  /usr/local/src/robot/robotology-superbuild/build/install/lib/yarp/yarp_fakeFrameGrabber.so: undefined symbol: test_grabber
[WARNING] |yarp.dev.Drivers| Wrong library name for plugin test_grabber

yarpdev.txt

Shall i install the yarp-device-realsense2 on icub-head?

[pattacini]

Hi @jimingda

Sorry, I was probably mistaken. The Realsense will be connected to a PC, I imagine, not to icub-head. So, please proceed with those steps on the PC where you were using the camera.

[jimingda]

Thank you @pattacini

I have added a RealSense_conf.ini file on my laptop, and now it seems to be working fine.
By the way, my camera is D435i. Should I use the parameter subdevice realsense2withIMU instead of subdevice realsense2 in the RealSense_conf.ini file?

icub@icubsrv:~/workspace$ yarpdev --from realsense_conf.ini 
[DEBUG] |yarp.dev.PolyDriver|RGBDSensorWrapper| Parameters are (HW_DESCRIPTION (clipPlanes (0.200000000000000011102 10.0))) (SETTINGS (depthResolution (640 480)) (rgbResolution (640 480)) (framerate 30) (enableEmitter true) (alignmentFrame RGB)) (device RGBDSensorWrapper) (from realsense_conf.ini) (name "/depthCamera") (single_threaded 1) (subdevice realsense2)
[WARNING] |yarp.devices.RGBDSensorWrapper| The 'RGBDSensorWrapper' device is deprecated in favour of 'rgbdSensor_nws_yarp'.
[WARNING] |yarp.devices.RGBDSensorWrapper| The old device is no longer supported, and it will be deprecated in YARP 3.7 and removed in YARP 4.
[WARNING] |yarp.devices.RGBDSensorWrapper| Please update your scripts.
[INFO] |yarp.devices.RGBDSensorWrapper| Using default 'period' parameter of  0.03 s
[INFO] |yarp.devices.RGBDSensorWrapper| ROS configuration parameters are not set, skipping ROS topic initialization.
[INFO] |yarp.os.Port|/depthCamera/rpc:i| Port /depthCamera/rpc:i active at tcp://10.0.0.1:10042/
[INFO] |yarp.os.Port|/depthCamera/rgbImage:o| Port /depthCamera/rgbImage:o active at tcp://10.0.0.1:10043/
[INFO] |yarp.os.Port|/depthCamera/depthImage:o| Port /depthCamera/depthImage:o active at tcp://10.0.0.1:10044/
[DEBUG] |yarp.devices.RGBDSensorWrapper| Opening IRGBDSensor subdevice
[DEBUG] |yarp.dev.PolyDriver|realsense2| Parameters are (HW_DESCRIPTION (clipPlanes (0.200000000000000011102 10.0))) (SETTINGS (depthResolution (640 480)) (rgbResolution (640 480)) (framerate 30) (enableEmitter true) (alignmentFrame RGB)) (device realsense2) (from realsense_conf.ini) (name "/depthCamera") (single_threaded 1) (subdevice realsense2)
[INFO] |yarp.device.realsense2| Sensor warm-up...
[INFO] |yarp.device.realsense2| Device ready!
[INFO] |yarp.device.realsense2| Device consists of 3 sensors. More infos using --verbose option
[INFO] |yarp.dev.PolyDriver|realsense2| Created device <realsense2>. See C++ class realsense2Driver for documentation.
[INFO] |yarp.dev.PolyDriver|RGBDSensorWrapper| Created wrapper <RGBDSensorWrapper>. See C++ class RGBDSensorWrapper for documentation.
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ===============================================================
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| == Options checked by device:
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ==
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| device=RGBDSensorWrapper
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| id [RGBDSensorWrapper]
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper|     Id assigned to this device
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ==
[DEBUG] |yarp.dev.Drivers|RGBDSensorWrapper| ===============================================================
[INFO] |yarp.os.Port|/depthCamera/quit| Port /depthCamera/quit active at tcp://10.0.0.1:10068/
[INFO] |yarp.dev.Drivers|RGBDSensorWrapper| device active in background...
[INFO] |yarp.dev.Drivers|RGBDSensorWrapper| device active in background...
[INFO] |yarp.dev.Drivers|RGBDSensorWrapper| device active in background...
[INFO] |yarp.dev.Drivers|RGBDSensorWrapper| device active in background...
[INFO] |yarp.dev.Drivers|RGBDSensorWrapper| device active in background...

[pattacini]

I have added a RealSense_conf.ini file on my laptop, and now it seems to be working fine.

Great!

By the way, my camera is D435i. Should I use the parameter subdevice realsense2withIMU instead of subdevice realsense2 in the RealSense_conf.ini file?

Yep, indeed 👍🏻