[BUG] Prometheus requires that all meters with the same name have the same set of tag keys.

[fieldju]

java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. 

There is already an existing meter named 

‘kubernetes_api_seconds’ 

containing tag keys 

[account, action, applicationName, armoryAppVersion, customerEnvName, customerName, hostname, kinds, lib, libVersion, namespace, ossAppVersion, spinnakerRelease, success, version]. 

The meter you are attempting to register has keys 

[account, action, applicationName, armoryAppVersion, customerEnvName, customerName, hostname, kinds, lib, libVersion, namespace, ossAppVersion, reason, spinnakerRelease, success, version].

[fieldju]

☝️ Added unit test that reproduces the issue for some TDD.

The way I see it there are a couple ways to tackle this.

• Maybe a Micrometer Transforming Filter can be used.
  • Not sure where filters are executed, might be too early to use this without maintaining manual brittle lists.
• Patch the registry so that the method is protected and extend it and add some sort of logic that makes tags the same here
• Port / Copy / Rewrite the Prometheus registry from scratch so that we can fix this and make sure that we can copy the Spinnaker Monitoring Daemons naming convention.

[fieldju]

I went with a combo of the last 2 bullet points and have a unit/functional test that works, need to point an actual Prometheus scrapper to it and see if it is happy.

Prometheus itself might get mad at the scrape, I'm not sure... ¯\_(ツ)_/¯.

[fieldju]

According to micrometer-metrics/micrometer#877 (comment), the IAE error that is being thrown is to protect against an error that I am no longer seeing on the Prometheus Java client.

That's not to say the author didn't mean that its to protect against an error on the actual scrapper instead.

Or maybe this limitation is fixed in Prometheus now?

[fieldju]

This comment/block of code makes me think that the scrapper might not handle the fix I have implemented in the plugin.

https://github.com/spinnaker/spinnaker-monitoring/blob/5599d1f708b23c7cd0df7b47f4b82cfcd425d095/spinnaker-monitoring-daemon/spinnaker-monitoring/prometheus_service.py#L100-L118

I am still not sure if this is for older scrappers or current.

• Fire up Prometheus and UAT that this fix works or doesn't and report results.

[fieldju]

Wrote some functional and integration tests that seem to prove that this fix is fine.