Depth measurement offset (possibly due to hardcoded xtable/ztable)

[kohrt]

While calibrating the Kinect2 to our robots, I saw that the depth values do not match the robots model.

I investigated this and wrote a program that compared the measured distances to the calculated distances from a chess board I used for the calibration. From each image I used the region of the board and compared each pixels (valid) depth measurement to the calculated distance by ray intersection to the board plane. Before doing this I undistorted the images.

It seems like the Kinect2 has a static offset of around 24 mm. The depth measurement is always 24 mm further away than it should be. Here are some plots of the results. This offset seems to be not related to image coordinates or distance. The error grows towards the image corners, but there the distortion is much stronger and the values get more noisy.

I only used the OpenCL depth packet processor. So I can't tell right now, if it is just related to that specific implementation, all implementation or the device itself. Tomorrow I will try out the OpenGL and CPU depth packet processor as well. Did someone experienced a similar behavior until now?

@christiankerl: Do you think it could be a problem of the packet processor implementation?

I added the code for the depth calibration to the calibration tool of my ROS Kinect2 bridge. If the depth values are decreased by the calculated offset the resulting depth cloud perfectly matches the model of our robots.

[christiankerl]

I guess the problem is that the xtable and ztable are used in the depth calculations, but they are only valid for the device I used during the memory dump. Actually they should be computed from some calibration values stored on each device. However, the formula is unknown

[sh0]

I heard that Kinect 2 has slight distance offset depending on what material you are observing. I'm not a specialist on ToF cameras, but I guess that when the material is highly reflective then the reflected light intensity is stronger and will activate the photosensitive element faster. Reportedly there can be 1-2 cm difference when comparing specular metal to a material with very diffuse reflection. I haven't tested for this effect myself so I really can't say how true all of this actually is.

[kohrt]

@christiankerl: And the memory dump came from using the sensor on Windows, with the official SDK, right? And we don't know the formula nor the sensor data from which it is calculated. If it is related to the xtable and ztable, then there should be no offset on your sensor, but if your sensor has an offset too, then it's something else.

@sh0: Yes, I have seen this on shiny metal, opaque and translucent surfaces. But in this case the offset appeared on all surfaces. For example the PR2 has plastic and textile covers on its arms, and KUKA arms have shiny and non shiny pained parts. And the distance measurements are from a chess board printed on paper and glued to a thick and flat acrylic glass.

[kohrt]

I tested it with the OpenGL depth packet processor and it is the same offset.

[djgaspa]

How could we dump the proper xtable/ztable of our devices? Is there a procedure?
I've seen that somebody reckon those tables might depend on a couple of tables retrieved together with color intrinsec params. Is there any preliminary analysis of those data?

[floe]

The blob with the intrinsics seems to contain two "subtables" of floats, one of size 56x46 and one of size 28x23. This is almost the same aspect ratio as the xTable/zTable, which are 512x424, and I'm suspecting that they may be just upsampled versions. Some discussion related to this is also going on in #41.

[sh0]

I have been working on the xTable/zTable problem.

The packets with suspected tables are deserialized in KinectService.exe. There seem to be actually three not two tables. The reason is that the tables are copied as three separate blocks. Interestingly there seem to be different packet formats, possibly from older development firmware. The structure with tables is then copied around and I can't really pinpoint where it is going due to the massive amount of vcalls going on. The tables are arranged as follows:

0x0068: float table1[28 * 23 * 2]; // size 0x1420 // from 0x0054
0x1488: float table2[28 * 23 * 2]; // size 0x1420 // from 0x1474
0x28a8: float table3[28 * 23 * 1]; // size 0x0a10 // from 0x2894

I've found at least 20 functions with 512x424 loops in them and I'm sure there are more. This means that I don't really know what loop calculates what. Anyhow, eventually the computed tables end up in vvtechs.dll where they are fed into the shaders. Depth map calculation related shaders take various textures as input, including ones named zTable, xTable and irTable, particulariliy in that order. One guess would be that the table1 and table2 are related to z/xTable and table3 is related to irTable. The irTable is used to look up IR image intensity coefficients. Or at least it looked like it from one shader bytecode.

I have also found couple IR related hardcoded 512x424 tables in KinectService.exe. Basically only one of the tables is used based on Kinect version info from device. The values range from around 4.0 to 4000.0. It has some banding artifacts just like z/xTable but does not match the value range. In this sense I'm not sure what those tables are.

[Doc-Ok]

The subtables in the color camera intrinsics reply appear to have the following structure:

• One 28x23 table with 2 floats per entry, defining a 2D position in depth camera space.

The black outline is the 512x424 depth camera space.

• One 28x23 table with 2 floats per entry, defining a 2D position in color camera space.

The black outline is the 1920x1080 color camera space. I do not know why the grid is offset and oversized. This may be a way to express the color camera pixel grid in depth camera space, which would lead to some horizontal offset, but that's just a guess. The aspect ratio of the grid sort-of matches the depth camera; this might be a depth-to-color mapping.

• One 28x23 table with 1 float per entry, which in my camera all hover around 870, and the last entry in each row is 0. No idea what these mean

[floe]

@sh0 about those hardcoded tables - are they actually used anywhere? The current libfreenect2 code only uses xTable & zTable from the original memory dump.
@Doc-Ok Is it possible that this is just the original raw calibration grid data, included for reference, and has already been used to calculate the coefficients sent in the same packet?