use recording rule metric for cluster cpu utilization #745
Conversation
| @@ -140,7 +140,7 @@ local template = grafana.template; | |||
| }) | |||
| .addPanel( | |||
| g.panel('CPU Utilisation') + | |||
| g.statPanel('1 - sum(avg by (mode) (rate(node_cpu_seconds_total{%(nodeExporterSelector)s, mode=~"idle|iowait|steal", %(clusterLabel)s="$cluster"}[%(grafanaIntervalVar)s])))' % $._config) | |||
| g.statPanel('cluster:node_cpu:ratio{%(clusterLabel)s="$cluster"}' % $._config) | |||
There was a problem hiding this comment.
It is not a good pattern to refer a recording rule(cluster:node_cpu:ratio ) which is not defined in the dependency of kubernetes-mixin. kube-prometheus dependes on k8s-mixin, not the otherway!
There was a problem hiding this comment.
Yes, agreed. This mixin doesn't depend on node exporter as far as I remember. :)
There was a problem hiding this comment.
Ok. Is moving those recording rules from kube-prometheus to kubernetes-mixin an option ? who can make that call here ?
there are node-exporter related rules here at https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/rules/node.libsonnet
Can we move some rules from https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/components/mixin/rules/node-rules.libsonnet to above ? Or copying those rules here with different metric names works ?
There was a problem hiding this comment.
Mixin itself depends on the existence of node_exporter data for some queries (hence nodeExporterSelector option in config.jsonnet). So I would say it is fine to move the recording rule which speeds things up from kube-prometheus to this project. One caveat though, those recording rules need to be used by alerts or dashboards shipped by kubernetes-mixin project.
There was a problem hiding this comment.
Ok makes sense then. Agreed @paulfantom.
There was a problem hiding this comment.
alright. Updated to include the recording rule here. Also, renamed the metric name to be different from the one in kube-prometheus. PTAL.
ravilr
force-pushed
the
cluster_cpu_util
branch
from
733e189 to
16a46ed
Compare
|
Looks great! |
ravilr
force-pushed
the
cluster_cpu_util
branch
from
16a46ed to
49569b4
Compare
the recording rule in kube-prometheus was already broken(fixed recently at prometheus-operator/kube-prometheus#1628). |
cluster:node_cpu:ratio_rate5mmetric. the current query is slower to evaluate on clusters having many nodes with multi-core cpus ( cores >= 64 ).cc @povilasv @paulfantom