Send and Receive Serialized Data

[Karsten1987]

In order to be able to efficiently record and replay DDS data, I am looking for a way to bypass de-serialization and receive the raw CDR data stream from the subscriber. Similarly, I'd like to then publish this stored CDR data stream as is with a publisher.
We need that capability to implement rosbags for ROS2 with OpenSplice. The open ticket for it can be found here: ros2/demos#185

To give you an idea on how this API can look like, the referenced PR also shows a demo called talker_raw and 'listener_raw`.
The listener's callback can simply print the hex representation of the CDR stream:
https://github.com/ros2/demos/blob/9a64c8e1eeac647e1d1b4d10d54e9452e1f1dde5/demo_nodes_cpp/src/topics/listener_raw.cpp#L48-L50
The talker can send plain CDR data in a message:
https://github.com/ros2/demos/blob/9a64c8e1eeac647e1d1b4d10d54e9452e1f1dde5/demo_nodes_cpp/src/topics/talker_raw.cpp#L53-L55

We have that functionality currently available in all ROS2 supported DDS vendors, however I haven't found a way of doing this with OpenSplice.

Does anybody have an idea on how to accomplish something like this?

[dirk-thomas]

@michielb-adlink Any feedback on this?

[hansvanthag]

In the air-traffic control area we have some (Java-based) customers that also want to do their own deserialization where we provide them with 'raw CDR' as a 'special' but its currently not a supported feature unfortunately.

We'll look into this to see how feasible it would be to support something similar also for C++ (lot of people currently on holidays though) ..

[hansvanthag]

Current plan is to add this to our upcoming community-edition-refresh from 6.7 to 6.9
Exact timeline to be decided, when would you require this feature 'at the latest' ?

[dirk-thomas]

rosbag will require the raw interface and should be available in a couple of months. So the Crystal release in December will certainly need the functionality in order to support all tools.

[dirk-thomas]

@hansvanthag Do you have any update on the timeline of a 6.9 release?

[hansvanthag]

We want to extend the 6.9 release with the CDR support before opensourcing it (an an update to current 6.7 release) we've started the design of that functionality and (still) expect it to be timely available for your Crystal release.
Would be useful to know that how far 'in advance' you'd (ideally and/or worst-case) have that functionality to 'validate' that its good-enough for the Crystal release

[hansvanthag]

PS> just checked: looks like we'd be able to provide a pre-release (for testing) early October

[dirk-thomas]

we'd be able to provide a pre-release (for testing) early October

That sounds good. 👍 There should be enough time for the Crystal release I guess. Are you planning to also update the rmw_opensplice repository to use the new API and implement the updated RMW interface?

[dirk-thomas]

Any news on this?

[hansvanthag]

The 1-week ago released 6.9 version includes the support to read/write CDR-based samples, but we're struggling somewhat to find the time to create the RMW-interface on top of that new API .. I"ll get back with an update estimate if/when we could do that .. is there a chance that you guys could build that yourselves (as risk mitigation) ?

[dirk-thomas]

The 1-week ago released 6.9 version ...

Great to hear.

is there a chance that you guys could build that yourselves?

We will likely not have time to work on this part and rely on you (or anyone else) to contribute this feature to fully support the upcoming ROS 2 release. We are happy to work on rolling out the new version of OpenSplice to our build infrastructure in the coming month so that Crystal can use version 6.9 - has either of you tried to build the current state ROS 2 against the new version?

[b3em]

I've been able to successfully run the ROS2 master testsuite vs our 6.9 CE build

[hansvanthag]

Update: We've started the work on the rmw_opensplice layer and expect to have something working within the next 2 weeks. It will likely result also in a small update on the 'underlying' new CDR-API's in the 6.9 OpenSplice release.

[dirk-thomas]

We've started the work on the rmw_opensplice layer and expect to have something working within the next 2 weeks. It will likely result also in a small update on the 'underlying' new CDR-API's in the 6.9 OpenSplice release.

Any update on this?

[ThijsSassen]

@dirk-thomas we hope to push our work to the repository later this week

[MarcelJordense]

@dirk-thomas The changes to support send and receive serialized data by opensplice are implemented. The changes to the opensplice repository are currently being reviewed. I prematurely made a pull request for associated changes to the ros2 rmw_opensplice repository. Please ignore this until the changes to the opensplice repository become available. I will notify you when these changes become available.

[dirk-thomas]

I have commented on the PRs with some warnings / problems to be addressed. Please move quickly on these so that the changes can be merged in time for the upcoming release.

[dirk-thomas]

I have commented on the PRs with some warnings / problems to be addressed. Please move quickly on these so that the changes can be merged in time for the upcoming release.

We worked around the warnings in the code using OpenSplice even though suppressing them in all the files is not pretty: ros2/rmw_opensplice#250.

@Karsten1987 Can you please go ahead and close this?