[receiver/podman] Add functionalities to Podman based on Docker stats receiver

[rogercoll]

The current implementation of the docker stats receiver provides more configurable options and robustness structure than the Podman receiver. For example, the Podman receiver does not include any filtering option, like excluding containers by its image name.
The goal of this issue is to note and keep track of different improvements that could be implemented in the Podman receiver in order to have as much functionalities as the Docker one.

Podman receiver configuration: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/podmanreceiver
Docker receiver configuration: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/dockerstatsreceiver

Proposal

To add in Podman:

• Timeout option for API requests: timeout (default to 5s)
• Exclude images option (all containers matching the pattern will be excluded): excluded_images (no default) See new proposal below.

Nice to have

• container_labels_to_metric_labels

Already aligned

• Endpoint
• API version

The current Podman receiver retrieves all the containers stats, and it cannot filter by an image name as it is not provided in the container stats response. To solve this issue, I would migrate it to the Docker structure, based on events:
1- Populate the containers list on start, launch a go routine that listens for events (new containers, removed containers)
2- Provide the stats of the defined containers

A long term goal would be to have a Container Engine interface to define Podman, Docker an any other engine entry points.

[rogercoll]

Fix stats error: #9411

[mcdoker18]

@rogercoll @codeboten

I'd like to complete this task, but I have several questions:

• Should I add env_vars_to_metric_labels to podman as well?
• What do you think about moving podman client to the separated internal package like docker client?
• I want to reuse the image matcher for image excluding in the podman client. I'd like to move it in internal/common/docker package, do you mind?

[rogercoll]

Definitely, anything that approximates the features of both receivers is welcomed. The env_vars_to_metrics_labels could be a good starting point.

For now, I would not move the podman client to the internal package as it is not used in any other component. What do you think?

Regarding the image matcher I was thinking to move it into a new package common for container components: internal/common/container

[rogercoll]

Exclude images option:

The actual docker stats receiver allows to filter the monitored containers by their image name, to do that it uses the excluded_images option. In order to extend the available options to exclude a container from being monitored, we are considering to use a more generic configuration that will enable us to easily add more filtering options:

exclude:
  container:
    images:
      - *nginx*
    labels:
      env: dev

Any of the containers matching an exclude rule (image, label, etc) won't be monitored.

Credits for the proposal to @mcdoker18

Any thought on this? Do you think we could easily extend the configuration in the docker stats receiver once implemented/stable in the podman receiver?

@dmitryax @rmfitzpatrick

[dmitryax]

I don't understand why do we need to keep container grouping?

[dmitryax]

Another concern is that we have different configuration interfaces for filtering across collector's components. Here are ones that I'm aware of (there are may be others):

1. docker stats receiver has only one option excluded_images with list of globs or regexps.
2. filter processor has include and exclude options with a bit different syntax having the match_type toggle.

I don't like that we promote one of the approaches further without discussing this inconsistency. I would like to have a single consistent config interface for filtering across all collector's components.

@rogercoll, do you have a chance to review all the different filtering configuration and start an issue where we can decide what can be a consistent approach that we can use going forward in all the components?

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/podman: @rogercoll

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[rogercoll]

Another functionality that is missing to align Podman and the Docker receiver is the Podman observer, at the moment only the docker one is available.

The structure and logic of both observers and receivers are very similar, to prevent duplicating code, we should move the Podman's client code to the internal package (like docker):

• Move Podman client's code from the receiver package to the internal one (e.g internal/podman).
• Implement Podman observer.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/podman: @rogercoll

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[cforce]

still important topic
@rogercoll

[rogercoll]

Good point @cforce , I am currently working on defining the exported metrics in the metadata file: #30232

This will help to have a more clear documentation on the generated metrics.

[cforce]

@rogercoll any news here?