feature-request: add Pod labels

[opvizordz]

It would be great to enhance process-exporter to include pod names as metric labels.
Perhaps process-exporter could be enhanced:

• process-exporter collects all the processes running on the node
• process-exporter collects all the linux pid namespaces the processes are in
• process-expoter looks up containers and corresponding pods correspond to the linux pid namespaces
• finally we have process-to-pod-name mapping that can be used for labelling...

currently the best output I could get is the following:
namedprocess_namegroup_num_procs{app="prometheus-process-exporter",chart="prometheus-process-exporter-0.2.2",groupname="promtail",heritage="Tiller",instance="10.1.83.224:9100",job="kubernetes-service-endpoints",kubernetes_name="process-exporter-prometheus-process-exporter",kubernetes_namespace="default",kubernetes_node="juju-e1b2fa-2",release="process-exporter"}

Great would be to have the POD Label (if its a POD process) additionally, like in that node exporter output:
kube_pod_container_resource_requests_cpu_cores{app="prometheus",chart="prometheus-8.9.0",component="kube-state-metrics",container="wordpress",heritage="Tiller",instance="10.1.83.164:8080",job="kubernetes-service-endpoints",kubernetes_name="eerie-snake-prometheus-kube-state-metrics",kubernetes_namespace="default",kubernetes_node="juju-e1b2fa-2",namespace="default",node="juju-e1b2fa-2",pod="my-wordpress-wordpress-6854b86fbc-7w9kc",release="eerie-snake"}

[ncabatoff]

Hi @opvizordz ,

I don't think I want to make process-exporter k8s-aware, though I guess I'd have to see what it entailed: if the dependencies and impact were small enough I'd consider it. Exposing the process namespace shouldn't be a problem though. Maybe then someone could write an exporter that wraps this one and does the k8s integration to produce the output you're hoping for.

[opvizordz]

Hi @ncabatoff ,
we currently look into it and will create a PR when we made the K8s-aware changes.
Thanks for your help!

[tusharbhasme]

Has anyone found any other solution for this yet? We need this same detail in k8s.

[Alex-wwei]

Has anyone found any other solution for this yet? We need this same detail in k8s.

[MagicalSheep]

For anyone who needs this feature, I found a possible solution. K8S (and container runtimes) use cgroups on Linux to limit the use of system resources. The directory paths created by K8S in the cgroups vfs are named with the Pod UID and container ID, which is exactly what we need.

Specifically, process-exporter reads the cgroup paths for each system resource from /proc/{pid}/cgroup. These paths are named like 1:cpu:/kubepods/burstable/pod{uid}/{container_id}. We can put the cgroup path with the Pod UID in the label.

I completely agree with @ncabatoff that process-exporter should not depend on K8S or other frameworks. Therefore, parsing cgroup paths and relabeling Pod UIDs to Pod names should be done in a separate module or program.

Unfortunately, the current K8S api-server does not support the API to get Pod resources by Pod UID (see: kubernetes/kubernetes#20572). But we can still maintain a mapping of UID to Pod resources in the local cache through the List-Watch mechanism. Through such a mapping, we can label the process metrics with any information related to K8S resources, including but not limited to Pod and Container.

In fact, perhaps we need a "cloud-native process-exporter" that is specifically integrated with K8S. I am working on related work and hope that this information can help people who see this issue.

Related code: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/helpers_linux.go#L317

[emzxcv]

@MagicalSheep I also need this same feature. I need the pod name though. What is the best way to get pod UID -> pod name? Can you explain more about the list-watch mechanism you mentioned?