Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix prometheus URL in prometheus-adapter #1463

Merged
merged 1 commit into from
Oct 26, 2021

Conversation

andrein
Copy link
Contributor

@andrein andrein commented Oct 25, 2021

Description

If the kube-prometheus stack on a cluster with a cluster domain that is different than the default .cluster.local,
prometheus adapter fails to connect to the prometheus server with the following error:

E1025 04:54:59.638367       1 provider.go:163] unable to fetch metrics for pods in namespace "istio-system", skipping: unable to fetch node CPU metrics: unable to execute query: Get "http://prometheus-k8s.monitoring.svc.cluster.local:9090/api/v1/query?query=sum+by+%28pod%2Ccontainer%29+%28%0A++irate+%28%0A++++++container_cpu_usage_seconds_total%7Bnamespace%3D%22istio-system%22%2Cpod%3D%22istio-ingressgateway-78d9c46fcb-zv5bs%22%2Ccontainer%21%3D%22%22%2Cpod%21%3D%22%22%7D%5B120s%5D%0A++%29%0A%29%0A&time=1635137699.631": dial tcp: lookup prometheus-k8s.monitoring.svc.cluster.local on 169.254.25.10:53: no such host

This would also fail if prometheus was deployed in a different namespace than the namespace specified in $.values.common.namespace

Type of change

  • CHANGE (fix or feature that would cause existing functionality to not work as expected)
  • FEATURE (non-breaking change which adds functionality)
  • BUGFIX (non-breaking change which fixes an issue)
  • ENHANCEMENT (non-breaking change which improves existing functionality)
  • NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Fix prometheus URL in prometheus-adapter.

@andrein
Copy link
Contributor Author

andrein commented Oct 25, 2021

That check is running successfully on my laptop, how can I trigger another CI run?

@andrein andrein force-pushed the prometheus-adapter branch from 291738a to 53e0a47 Compare October 25, 2021 17:04
@@ -33,7 +33,7 @@ spec:
- --config=/etc/adapter/config.yaml
- --logtostderr=true
- --metrics-relist-interval=1m
- --prometheus-url=http://prometheus-k8s.monitoring.svc.cluster.local:9090/
- --prometheus-url=http://prometheus-k8s.monitoring:9090/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this actually work without svc in url? I would expect http://prometheus-k8s.monitoring.svc:9090/ to work correctly, but I am not sure about http://prometheus-k8s.monitoring:9090/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does. The kubelet sets the following in the pods resolv.conf

nameserver 10.32.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#namespaces-of-services

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

http://prometheus-k8s.monitoring.svc:9090/ also works, I can update the PR if you want it that way.

@andrein andrein requested a review from paulfantom October 25, 2021 18:26
@andrein andrein force-pushed the prometheus-adapter branch from 53e0a47 to a6442df Compare October 26, 2021 13:35
Copy link
Member

@paulfantom paulfantom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@paulfantom paulfantom merged commit ba8ec69 into prometheus-operator:main Oct 26, 2021
@andrein andrein deleted the prometheus-adapter branch October 26, 2021 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants