Skip to content

Handle OpenMetrics prometheus metrics from applications #7093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lobkovilya opened this issue Jun 22, 2023 · 2 comments · Fixed by #7572
Closed

Handle OpenMetrics prometheus metrics from applications #7093

lobkovilya opened this issue Jun 22, 2023 · 2 comments · Fixed by #7572
Labels
kind/feature New feature triage/accepted The issue was reviewed and is complete enough to start working on it

Comments

@lobkovilya
Copy link
Contributor

lobkovilya commented Jun 22, 2023

What happened?

After #6783 prometheus can't scrape targets and returns the following error, so it was reverted. This issue covers this feature

image

@lobkovilya lobkovilya added triage/pending This issue will be looked at on the next triage meeting kind/bug A bug labels Jun 22, 2023
@AyushSenapati
Copy link
Contributor

AyushSenapati commented Jun 23, 2023

expfmt.NegotiateIncludingOpenMetrics() only gets called when the hijacker fails to find a valid ContentType (CT) returned by the target applications. Upon debugging, it was found that Envoy returns Content-Type": "text/plain; charset=UTF-8" with out the version param. In result the responseFormat() is returning expfmt.FmtUnknown as we also validate the version param, which is why expfmt.NegotiateIncludingOpenMetrics() is called and it would return highest CT supported by the Prometheus Scraper (In our case this is expfmt.FmtOpenMetrics_1_0_0).

The actual metrics content of Envoy was text/plain but the Hijacker sets the CT header to Openmetrics, which is why prom scraper is failing to process.

I will update the hijacker to normalise the metrics content.

@jakubdyszkiewicz jakubdyszkiewicz added triage/accepted The issue was reviewed and is complete enough to start working on it and removed triage/pending This issue will be looked at on the next triage meeting labels Jun 26, 2023
@michaelbeaumont michaelbeaumont added kind/feature New feature and removed kind/bug A bug labels Jun 26, 2023
@michaelbeaumont michaelbeaumont changed the title Prometheus can't scrape metrics after introducing NegotiateIncludingOpenMetrics Handle OpenMetrics prometheus metrics from applications Jun 26, 2023
@jakubdyszkiewicz
Copy link
Contributor

was it solved? Because I see

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature triage/accepted The issue was reviewed and is complete enough to start working on it
Projects
None yet
4 participants