MeterRegistry IllegalArgumentException - Prometheus requires that all meters with the same name have the same set of tag keys

[harshitdx29]

I am using MeterReigstry to push metrics to be polled by Prometheus. However when I add dynamic tags, I get the following exception:

java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'execution_time_seconds' containing tag keys [application, errorCode, execution_identifier, execution_source, goods_code_enum]. The meter you are attempting to register has keys [application, execution_identifier, execution_source, goods_code_enum, status].

What is causing this? How can it be resolved? I am using

<dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-spring-legacy</artifactId> <version>1.3.2</version> </dependency>

[harshitdx29]

Hi @jkschneider I read other threads on the same issue, seems like there is no workaround for it yet? Specifically: #877

[checketts]

@harshitdx29 The recommended workaround is to provide an empty value for the tag you don't have a value for. In your case make sure to provide a status and errorCode tag in all calls.

[harshitdx29]

@checketts this was just one use case, we may have use cases where we need to send more tags, so what you recommend is to use the subset of tags which we have?

[harshitdx29]

Also, @checketts any particular reason why we have this limitation?

[checketts]

@harshitdx29 I seem to recall it was a limitation from the Prometheus client and Micrometer wrapped it so it would provide a more useful error message (displaying the tags for example)

It is possible that there were historical reasons. Prometheus 1.x had a different storage mechanism that was less efficient. If you discover that Prometheus itself no longer struggles with this then that would be good to know.

Aside from technical limitations, dynamic tags may be indicating 2 potential usage problems: 1) unlimited cardinality (which could cause a metric count/memory explosion) or 2) incorrect counts/promql query issues.

Could you take a step back and describe your code/usecase that makes it hard to account for the tags that are being applied? (I'm not trying to push back on your usecase, just trying to understand what is making things so dynamic)

[harshitdx29]

For now, the use case is only errorCode and status but dynamic tags by nature provide us the flexibility to push any number of metrics. One use case I can think of is when we monitor external clients in our system, in that case, we may need to add another tag url and more tags for responses by 3rd party.

I seem to recall it was a limitation from the Prometheus client
Does this mean it is a limitation of Prometheus in general?
What if I use TICK stack i.e. push the metrics to statsd registry which can be collected by an agent like telegraf and from there Prometheus can pull the metrics? Will it still give the same error?

[checketts]

What if I use TICK stack i.e. push the metrics to statsd registry which can be collected by an agent like telegraf and from there Prometheus can pull the metrics? Will it still give the same error?

I'm sorry I don't have any experience in that regard. Perhaps the Prometheus mailing list would be able to answer that specific question.

I'm not able to find any doc supporting this either, aside from https://www.robustperception.io/target-labels-are-for-life-not-just-for-christmas But even that is speaking of the goal to limit target labels, not labels in general.

[harshitdx29]

Yes I can limit the target labels but somehow not make them fixed set. But seems like we will have to do that. Anyways will do a POC on statsd flow and see if we get same error.

[checketts]

You also could try pulling out the underlying Prometheus registry that Micrometer is writing to and using it directly. If that works then that would uncover Prometheus client limitations.

[harshitdx29]

Do you have a demo code for that?

[harshitdx29]

I can see someone suggesting a change here which seems to work: #877 (comment)

[checketts]

Do you have a demo code for that?

Call getPrometheusRegistry() on your PrometheusMeterRegistry to get the underlying Prometheus client directly.

[harshitdx29]

ok let me try.

[harshitdx29]

Currently my code looks like

            meterRegistry
                .timer(MetricsIdentifier.keyForExecutionTime(), monitorDataObject.getTags())
                .record(Duration.ofMillis(monitorDataObject.getExecutionTime()));```
meterRegistry refers to PrometheusMeterRegistry
How do you want me to change it? 

[checketts]

Try something like:

 <Create a summary>
 ((PrometheusMeterRegistry)meterRegistry).getPrometheusRegistry()
 <Register summary>

Summary example is here: https://github.com/prometheus/client_java#summary

I apologize for not being able to help with the code more. I'll check back on this thread this evening after work.

[harshitdx29]

I am not sure how this is working. I will probably try out statsd flow and if we face the same flow, will probably stick to fixed set of tags.

[cforce]

@harshitdx29 Did you successfully implement that work around using Prometheus Registry directly ? Some working code example would be appreciated.
However currently dynamic tags are not usable with Prometheus, what is a required feature of course.

[harshitdx29]

Hi @cforce I have not tried this. For now restricted the tags to a fixed set.

[shakuzen]

Duplicate of #877