How to extract depth data in real time?

[Ahad19931]

| Required Info
|---------------------------------|------------------------------------------- |
| Camera Model | {D435i } |
| Firmware Version | (05.11.06.200) |
| Operating System & Version | Linux (Ubuntu 16.04) |
| Kernel Version (Linux Only) | (4.15.0-50-generic) |
| Platform | PC |
| SDK Version | { 2.21.0 } |
| Language | {C++ } |
| Segment | {Robot} |

How can I extract the depth data from the camera during the live streaming. The main purpose of this is to perform obstacle avoidance using this depth data when the camera is mounted on a mobile robot.
I was able to extract the depth data from a recorded file using the rs-convert tool by converting the data into csv and raw files but have no idea about how to do it in real time.
Any assistance will be highly appreciated.

[scizors]

yes I also have same doubt in python. How can I extract it

[Ahad19931]

Is there anyone from the Intel RealSense team or anyone else in the community who can help? Have been waiting for a long time now.

[ev-mp]

@Ahad19931 hello,
Realsense devices stream depth frames with the depth data being stored in pixels and encoded as uint16 type. This format is known as Z16 (Link)

It is not clear though what do you mean exactly by extracting depth data, is it converting depth pixel to metric units, like in rs-hello-realsense example?

I would recommend checking the examples chapter which briefly describes the features each example demonstrates and also categorizes them according to experience level.
This may provide initial orientation and make the request more specific.

[Ahad19931]

Hi @ev-mp ,
Thanks for your response. What I meant to say is, I need to do obstacle detection for which I need the depth values of each frame in real time. These depth values needs to be some number or some readable format so that I can build the logic using these values.
For the rs-hello-realsense example, It prints the distance from the center of the camera field of view. I need to get the value of every pixel individually rather than just showing the distance from the center which is the case in rs-hello-realsense example.
I hope you got a better idea of what I exactly want. Looking forwatd to your reply.

[ev-mp]

@Ahad19931 hello,
The core flow can be summarized in the following three lines:

auto frames = pipe.wait_for_frames();
if (auto depth_frame = frames.get_depth_frame())
{
    auto pixels_buffer = depth_frame.get_data();`
    // Write your code here
}

The resulted pixels_buffer handle provides a direct access to the raw depth data ready for manipulations and its content is nxm matrix of pixels in Z16 format specified above.
In case the algorithm you're building required Z values only I would argue that it is sufficiently readable as converting Z16 into distances is a multiplication by const depth_units scale factor.

It is possible to convert the frame first from pixels to meters to make it human-friendly - this can be done manually or by employing newly-introduced units_transform processing block - #3982.

But as a general rule I would recommend working with the raw depth data, whenever possible, instead of using converted metric or any other arbitrary units.

• In many algorithms using 16bit integer values cannot be less efficient than using 32bit floats, especially given that there is no extra data/precision gained from converting 16bit integers to 32bit values.
• Each flow that requires metric units can be adjusted in O(1) to use Z16 units.
• There is actual penalty both in performance and memory footprint in transfer the raw data into human-readable units (O(n) order). And given that in many cases the run-time environment will utilize low-level HW platform may have a measurable impact on overall performance.

Last but not least there are additional cases such as point-cloud/ surface reconstruction and/or depth alignment/registration use-cases that can be facilitated via SDK-provided tools/APIs.

[Ahad19931]

Hi @ev-mp ,
Addressing the first solution that you provided where pixels_ buffer will provide the raw data in matrix of pixels, I tried to add your provided snippet of the code into the rs-hello-realsense example while making some amendments to the code and trying to first get the value of the pixels_buffer but couldn't succeed. Can you please provide me the code to use this snippet where I can just get the value coming from the pixels_buffer?

Regarding the units_transform block, isn't it the same as what we are getting after running the rs-hello-realsense example, because with that example, we get the distance in meters too?

As you recommended that working with raw depth data is a better option, can you please explain about what you exactly meant by raw depth data? Is it the depth stream that we are getting using the realsense-viewer?

Looking forward to your reply, Thanks

[ev-mp]

@Ahad19931 hello,

1. Please check the rs-align-advanced example here that shows how the depth frame raw data is read and converted on-the-fly to apply min/max range filter.

2. Using units_transform block is equivalent to doing it manually, indeed. But it provide advantages:

• Performance.
• the result is encapsulated into standard frame object which releaves the users from dealing with memory management.
• can be chained into Librealsense processing blocks pipe.

3. Correct - I refer to the depth frame with Z values encoded as uint16_t. Note that actually in realsense-viewer the presented depth is transformed and rendered as RGB to make it user-friendly.

[Ahad19931]

Hi @ev-mp ,
Can you please provide me with the steps on how to run the units transform block in the correct manner. Since there are are a number of files and headers so I am kind of lost and constantly making error.
Secondly, if I directly use the raw depth data for using obstacle avoidance as you mentioned previously, do you point to the method of using the image processing technique to process the depth frames?

[ev-mp]

@Ahad19931 , there is no need for header files other than #include <librealsense2/rs.hpp>
The code is not incorporated into demos - only unit-tests.
But you can try the following:

    ...
    rs2::units_transform ut;
    rs2::pipeline pipe;
    pipe.start();
    while (...)
    {
        rs2::frameset data = pipe.wait_for_frames();
        if (auto depth_raw_frame = data.get_depth_frame())
        {
            auto depth_metric_frame = ut.process(depth_raw_frame);
            // Add your code here
        }
    }

[RealSenseCustomerSupport]

---

@Ahad19931 Any update on this?

[Ahad19931]

Hi,
I was able to extract live stream from the camera using the matlab wrapper i.e. depth_example.m. So thanks for the assistance.