in_someip: SOME/IP input plugin #9570

anthonypayne-GM · 2024-11-08T14:44:50Z

This change introduces a new input plugin, in_someip. This plugin is used to subscribe to events and capture notifications from a SOME/IP network. It also can be used in inject SOME/IP requests and capture the response to the data pipeline.

The input plugin uses an open source implementation of the SOME/IP stack, vsomeip. This library is used unchanged, but a wrapper library, libsomeip-c, exposing C-language APIs is part of the PR.

For reference to SOME/IP: protocol specification.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change
Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

[N/A] Run local packaging test showing all targets (including any new ones) build.
[N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

[N/A ] Documentation required for this feature

Backporting

[N/A] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

anthonypayne-GM · 2024-11-08T14:46:24Z

Example config file is included in the PR, conf/in_someip.conf:

[INPUT]
    Name someip
    Tag  in.someip

    # Events to subscribe to.
    # Each event should have form:
    #    Event <service id>,<instance id>,<event id>,<event group 1>,...
    #
    # Each event must have at least one event group
    Event 4,1,32768,1
    Event 4,1,32769,2

    # RPC to send on startup
    # Each RPC entry should have form:
    #    RPC <service id>,<instance id>,<method id>,<Request Payload>
    #
    # Request payload should be base64 encoded
    RPC   4,1,1,CgAQAw==

anthonypayne-GM · 2024-11-08T14:50:20Z

Debug log from runtime tests is attached.
rt_test.log

anthonypayne-GM · 2024-11-08T14:50:59Z

Output from valgrind:

[ OK ]
SUCCESS: All unit tests have passed.
==7910== 
==7910== HEAP SUMMARY:
==7910==     in use at exit: 5,741 bytes in 13 blocks
==7910==   total heap usage: 17,360 allocs, 17,347 frees, 5,723,214 bytes allocated
==7910== 
==7910== LEAK SUMMARY:
==7910==    definitely lost: 0 bytes in 0 blocks
==7910==    indirectly lost: 0 bytes in 0 blocks
==7910==      possibly lost: 0 bytes in 0 blocks
==7910==    still reachable: 5,741 bytes in 13 blocks
==7910==         suppressed: 0 bytes in 0 blocks
==7910== Rerun with --leak-check=full to see details of leaked memory
==7910== 
==7910== For lists of detected and suppressed errors, rerun with: -s
==7910== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

patrick-stephens

Thanks for the contribution, I think we need to look at those runtime dependencies though at the very least - also do we need to include them in the target builds as well under ./packaging/distros?

I don't think we can realistically globally enable this for every supported target. It also seems to use C++ compilation and dependencies.

patrick-stephens · 2024-11-12T14:06:03Z

dockerfiles/Dockerfile

+    libboost-system-dev \
+    libboost-thread-dev \
+    libboost-filesystem-dev \


This is including the dev libraries in the runtime image - we should just have the minimal set of libraries needed for running fluent bit, not those for building it.

Hi @patrick-stephens, I tried to take these out, leaving only the entries above (line 57). But when I did, I was not able to build the SOME/IP input plugin and its dependent libraries from the dev container.

Likewise when I tried to replace these in this section with the "non-dev" version (libboost-filesystem1.74.0) the header files are not installed in the container and the build fails as well.

Please advise if there is something I am doing wrong here.

Yeah I think you need to look at why, the dev libraries should only be in the build stage which is fine. At runtime we need the minimal set of dependencies required to run with, otherwise we end up opening a big bag of security worms with extra dependencies.

Header files are not needed at runtime, they should not be in the runtime image.

So I think you need to ensure the builder stage has the dev libraries to compile everything but then the dep-extractor stage only includes the runtime libraries.

patrick-stephens · 2024-11-12T14:06:18Z

dockerfiles/Dockerfile

@@ -221,6 +227,9 @@ RUN echo "deb http://deb.debian.org/debian bookworm-backports main" >> /etc/apt/
    libatomic1 \
    libgcrypt20 \
    libyaml-0-2 \
+    libboost-system-dev \


As above, do we really need the development libraries here or can we just use the runtime ones?

Same as above, I tried to change this section to the use the non-dev libs and was not able to build the SOME/IP plugin/libs from the dev container.

This section is unrelated to building so changing it here should have no impact on compilation.

Yes but this is not compiling anything, this is the debug image so again should just have the minimum set of packages per the deb-extractor stage.

dockerfiles/Dockerfile

patrick-stephens · 2024-11-12T14:07:58Z

lib/libsomeip-c/CMakeLists.txt

+FetchContent_Declare(
+    vsomeip3
+    GIT_REPOSITORY https://github.com/COVESA/vsomeip
+    GIT_TAG        0b83e24d16e1611958194e9b727136522f46556b # 3.5.1
+)
+FetchContent_MakeAvailable(vsomeip3)


Dependencies are generally vendored into the /lib directory and not fetched remotely.

Thanks. I added the source for vsomeip 3.5.1 as a sub-directory here and updated CMake file to build it before the C wrapper code.

patrick-stephens · 2024-11-12T14:08:38Z

CMakeLists.txt

@@ -227,6 +227,7 @@ if(FLB_ALL)
  set(FLB_IN_NGINX_STATUS 1)
  set(FLB_IN_EXEC         1)
  set(FLB_IN_UNIX_SOCKET  1)
+  set(FLB_IN_SOMEIP       1)


I don't think you can enable this globally without considering all platforms that are supported including Windows and macOS configuration.

Thanks @patrick-stephens , will remove this here.

patrick-stephens · 2024-11-12T14:10:32Z

lib/libsomeip-c/example/test_service.cc

@@ -0,0 +1,138 @@
+/*  Fluent Bit


This seems to be a C++ file, is that just a standard example and/or can it be C only? It seems to require a C++ compiler by looking at the example cmake linkage.

Yes, this is a C++ file.
I used it to execute against fluent-bit with the SOME/IP input plugin enabled.

I will re-write it in C.

patrick-stephens · 2024-11-12T14:14:40Z

For new plugins we must ensure it builds for all targets and CI so I've triggered that. It looks like we have a lot of failures around missing dependencies and the like in various places/platforms.

anthonypayne-GM

Tried to address some of the received comments. Will push modifications to the PR shortly.

anthonypayne-GM · 2024-12-09T17:03:47Z

Hi @patrick-stephens ,

In order to develop this plugin, I needed an update to the Docker image used by the devcontainer (.devcontainer/devcontainer.json) to include the boost dev libraries.

	"name": "devcontainer-fluent-bit",
	"image": "fluent/fluent-bit:latest-debug",

To get an update to this image, I modified ./dockerfiles/Dockerfile as per the PR and locally installed the image with the boost dev libraries ($ docker build -t fluent/fluent-bit:latest-debug -f dockerfiles/Dockerfile .).

The dev-container uses the docker image to build fluent bit so that is why I need the dev versions of the boost library in the container image.

It appears from the comments though, that the Docker file I updated in the PR is not the correct one for updating the docker image used by the dev container?

Can you please provide some guidance on how I should proceed?

Also, I am not sure about the Docker files that are under packaging/distros. Should those be updated with the plugin dependencies as well?

anthonypayne-GM · 2024-12-09T23:00:13Z

For new plugins we must ensure it builds for all targets and CI so I've triggered that. It looks like we have a lot of failures around missing dependencies and the like in various places/platforms.

This plugin should only be built for linux environments (no MacOS or Windows). Since this plugin requires boost to be installed on Ubuntu do I need to update the .github/workflows/unit-tests.yaml file to add in the apt install of the required boost libraries?

For Centos 7, the available boost from yum repo is to old to support this plugin (vsomeip requires boost 1.66 or newer; yum repo has boost 1.53 I think).

ammolitor · 2025-01-15T22:48:19Z

@patrick-stephens I'd like to help get this over the line. Is there something you need to start for CI builds so we can get feedback and address any issues? The only changes at this point is to get the PR up-to-date with current the master branch.

Thanks!

patrick-stephens · 2025-01-16T10:24:19Z

@patrick-stephens I'd like to help get this over the line. Is there something you need to start for CI builds so we can get feedback and address any issues? The only changes at this point is to get the PR up-to-date with current the master branch.

Thanks!

Apologies, missed some updates but I think all my previous comments still stand.
We cannot be adding -dev libraries to runtime images and we need to minimise any dependencies added to only those required.

Development libraries may be needed during the builder stage yes but the runtime libraries should be much more limited. The container definition is multi-stage which a build stage and then the runtime stage so the appropriate libraries need to be used in each location and minimal ones for the production or debug stages. We need a good understanding of what libraries are required in each stage and they need to be only the minimum required.

Builder stage:

fluent-bit/dockerfiles/Dockerfile

Lines 38 to 58 in e9a2f87

    
           RUN echo "deb http://deb.debian.org/debian bookworm-backports main" >> /etc/apt/sources.list && \ 
        
               apt-get update && \ 
        
               apt-get install -y --no-install-recommends \ 
        
               build-essential \ 
        
               curl \ 
        
               ca-certificates \ 
        
               cmake \ 
        
               git \ 
        
               make \ 
        
               tar \ 
        
               libssl-dev \ 
        
               libsasl2-dev \ 
        
               pkg-config \ 
        
               libsystemd-dev/bookworm-backports \ 
        
               zlib1g-dev \ 
        
               libpq-dev \ 
        
               postgresql-server-dev-all \ 
        
               flex \ 
        
               bison \ 
        
               libyaml-dev \ 
        
               && apt-get clean \

Production libraries:

fluent-bit/dockerfiles/Dockerfile

Lines 113 to 143 in e9a2f87

    
           libssl3 \ 
        
           libsasl2-2 \ 
        
           pkg-config \ 
        
           libpq5 \ 
        
           libsystemd0/bookworm-backports \ 
        
           zlib1g \ 
        
           ca-certificates \ 
        
           libatomic1 \ 
        
           libgcrypt20 \ 
        
           libzstd1 \ 
        
           liblz4-1 \ 
        
           libgssapi-krb5-2 \ 
        
           libldap-2.5 \ 
        
           libgpg-error0 \ 
        
           libkrb5-3 \ 
        
           libk5crypto3 \ 
        
           libcom-err2 \ 
        
           libkrb5support0 \ 
        
           libgnutls30 \ 
        
           libkeyutils1 \ 
        
           libp11-kit0 \ 
        
           libidn2-0 \ 
        
           libunistring2 \ 
        
           libtasn1-6 \ 
        
           libnettle8 \ 
        
           libhogweed6 \ 
        
           libgmp10 \ 
        
           libffi8 \ 
        
           liblzma5 \ 
        
           libyaml-0-2 \ 
        
           libcap2 \

Debug libraries:

fluent-bit/dockerfiles/Dockerfile

Lines 214 to 223 in e9a2f87

    
           libssl3 \ 
        
           libsasl2-2 \ 
        
           pkg-config \ 
        
           libpq5 \ 
        
           libsystemd0/bookworm-backports \ 
        
           zlib1g \ 
        
           ca-certificates \ 
        
           libatomic1 \ 
        
           libgcrypt20 \ 
        
           libyaml-0-2 \

The packaging/distros container definitions need updating to allow this plugin to be built for relevant targets or disabled appropriately if not possible - similarly for the macOS and Windows builds.

Is the only alternative to wrap a C++ library? I do not believe we have any C++ libraries in there currently as the preference has always been to keep it in C.

ammolitor · 2025-01-16T15:58:29Z

Thanks @patrick-stephens we'll work through these, and update.

patrick-stephens · 2025-02-25T09:56:21Z

As per my previous comment let's add YAML config as that is the new standard now.

We need to actually have this in the builds too - at the moment it is default disabled so is never built. Can we add it to the Dockerfile and any relevant distros under packaging so we can show it building?

anthonypayne-GM · 2025-02-25T18:56:04Z

@patrick-stephens,

I updated the docs PR to include both legacy and yaml configuration example.

Do I also need to add a yaml conf file under fluent-bit/conf as well? I didn't see any yaml conf files in this repo.

The SOME/IP C library is a wrapper around vsomeip. Signed-off-by: Anthony Payne <anthony.payne@gm.com>

Signed-off-by: Anthony Payne <anthony.payne@gm.com>

patrick-stephens · 2025-03-05T10:12:30Z

@patrick-stephens,

I updated the docs PR to include both legacy and yaml configuration example.

Do I also need to add a yaml conf file under fluent-bit/conf as well? I didn't see any yaml conf files in this repo.

If you're adding a legacy config then yes a YAML config too.

Signed-off-by: Aaron Molitor <ammolitor@gmail.com>

anthonypayne-GM requested review from niedbalski, patrick-stephens, celalettin1286, fujimotos, edsiper, leonardo-albertovich and koleini as code owners November 8, 2024 14:44

github-actions bot added the docs-required label Nov 8, 2024

patrick-stephens requested changes Nov 12, 2024

View reviewed changes

patrick-stephens added the ok-package-test Run PR packaging tests label Nov 12, 2024

anthonypayne-GM temporarily deployed to pr November 12, 2024 14:12 — with GitHub Actions Inactive

anthonypayne-GM temporarily deployed to pr November 12, 2024 14:36 — with GitHub Actions Inactive

anthonypayne-GM temporarily deployed to pr November 12, 2024 14:37 — with GitHub Actions Inactive

anthonypayne-GM commented Nov 18, 2024

View reviewed changes

anthonypayne-GM changed the title ~~SOME/IP input plugin~~ in_someip: SOME/IP input plugin Dec 9, 2024

ammolitor force-pushed the someip-input-plugin branch from 239b46b to 7981fa5 Compare January 15, 2025 22:43

ammolitor force-pushed the someip-input-plugin branch 2 times, most recently from 13d4610 to 5841f67 Compare January 16, 2025 20:48

anthonypayne-GM mentioned this pull request Feb 6, 2025

in_someip: Initial input plugin documentation. fluent/fluent-bit-docs#1569

Open

ammolitor force-pushed the someip-input-plugin branch 2 times, most recently from 693ce9e to 34bfa1e Compare February 25, 2025 01:59

ammolitor temporarily deployed to pr February 25, 2025 09:55 — with GitHub Actions Inactive

ammolitor temporarily deployed to pr February 25, 2025 10:19 — with GitHub Actions Inactive

ammolitor temporarily deployed to pr February 25, 2025 10:25 — with GitHub Actions Inactive

ammolitor force-pushed the someip-input-plugin branch from 34bfa1e to dc694bd Compare March 3, 2025 20:09

ammolitor temporarily deployed to pr March 4, 2025 09:58 — with GitHub Actions Inactive

ammolitor temporarily deployed to pr March 4, 2025 10:21 — with GitHub Actions Inactive

anthonypayne-GM added 2 commits March 4, 2025 21:18

lib: someip-c: Added SOME/IP C library

817744c

The SOME/IP C library is a wrapper around vsomeip. Signed-off-by: Anthony Payne <anthony.payne@gm.com>

in_someip: Added SOME/IP input plugin

f0d864c

Signed-off-by: Anthony Payne <anthony.payne@gm.com>

ammolitor force-pushed the someip-input-plugin branch from dc694bd to bdc72bb Compare March 5, 2025 03:18

ammolitor temporarily deployed to pr March 5, 2025 10:13 — with GitHub Actions Inactive

ammolitor temporarily deployed to pr March 5, 2025 10:35 — with GitHub Actions Inactive

ammolitor temporarily deployed to pr March 5, 2025 10:36 — with GitHub Actions Inactive

add someip libs to Dockerfile

affe8e1

Signed-off-by: Aaron Molitor <ammolitor@gmail.com>

ammolitor force-pushed the someip-input-plugin branch from bdc72bb to affe8e1 Compare March 5, 2025 15:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

in_someip: SOME/IP input plugin #9570

in_someip: SOME/IP input plugin #9570

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

patrick-stephens left a comment

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 18, 2024 •

edited

Loading

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 18, 2024

patrick-stephens Nov 18, 2024

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens Nov 12, 2024

anthonypayne-GM Nov 18, 2024

patrick-stephens commented Nov 12, 2024

anthonypayne-GM left a comment

anthonypayne-GM commented Dec 9, 2024

anthonypayne-GM commented Dec 9, 2024

ammolitor commented Jan 15, 2025

patrick-stephens commented Jan 16, 2025 •

edited

Loading

ammolitor commented Jan 16, 2025

patrick-stephens commented Feb 25, 2025

anthonypayne-GM commented Feb 25, 2025

patrick-stephens commented Mar 5, 2025

in_someip: SOME/IP input plugin #9570

Are you sure you want to change the base?

in_someip: SOME/IP input plugin #9570

Conversation

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

anthonypayne-GM commented Nov 8, 2024

patrick-stephens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrick-stephens Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrick-stephens commented Nov 12, 2024

anthonypayne-GM left a comment

Choose a reason for hiding this comment

anthonypayne-GM commented Dec 9, 2024

anthonypayne-GM commented Dec 9, 2024

ammolitor commented Jan 15, 2025

patrick-stephens commented Jan 16, 2025 • edited Loading

ammolitor commented Jan 16, 2025

patrick-stephens commented Feb 25, 2025

anthonypayne-GM commented Feb 25, 2025

patrick-stephens commented Mar 5, 2025

patrick-stephens Nov 18, 2024 •

edited

Loading

patrick-stephens commented Jan 16, 2025 •

edited

Loading