Timer did not reset after expire time.

[LatsykNS]

Hi, I have implemented a very simple configuration to use Prometheus Timer :

Config:

@Bean
  public PrometheusMeterRegistry meterRegistry () {
    return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
 }

Start-Stop Timer code with 1 hour expire time:

Timer.start(meterRegistry);
// some code
sample.stop(Timer.builder("my_endpoint")
       .publishPercentiles(0.95)
       .sla(ofMillis(1), ofMillis(500), ofMillis(1000),
           ofMillis(1500), ofMillis(2000),
           ofMinutes(2500), ofMillis(3000),
           ofMillis(3500), ofMillis(4000), ofMillis(5000))
       .distributionStatisticExpiry(Duration.ofHours(1))
       .register(meterRegistry)); 

However, during 1 hour I did not make any requests, but the current max time was still there:

TYPE my_endpoint_seconds_max gauge
my_endpoint_seconds_max 0.814347704

It expired after 2+ hours.

Dependencies:
io.micrometer:micrometer-spring-legacy:1.3.20
io.micrometer:micrometer-registry-prometheus:1.7.2

Could you please answer what is wrong? Maybe my understanding of Timer expiry time is incorrect.

[shakuzen]

Firstly, you are using different versions of micrometer dependencies, which is likely to cause issues. Also, micrometer-spring-legacy is no longer supported.

To answer your question, expiration of distribution statistics happens via a ring buffer. The expiration behavior is controlled by the ring buffer length and the time between rotation. The expiry duration set in your example controls the time between rotation. By default, the ring buffer has a length of 3. So it will take the buffer length * the rotation duration for a value to completely expire.

Admittedly, this could be explained better in the reference documentation and is not intuitive from the method name. It is explained in the JavaDocs for related methods already, though. https://www.javadoc.io/doc/io.micrometer/micrometer-core/latest/io/micrometer/core/instrument/AbstractTimerBuilder.html

Statistics emanating from a timer like max, percentiles, and histogram counts decay over time to give greater weight to recent samples (exception: histogram counts are cumulative for those systems that expect cumulative histogram buckets). Samples are accumulated to such statistics in ring buffers which rotate after this expiry, with a buffer length of distributionStatisticBufferLength(Integer).

Looking forward, we could migrate the API to make the method names more intuitive. Perhaps something like expiryRotationTime? Suggestions welcome.

[LatsykNS]

Hi @shakuzen , can you please provide a compatibility table between micrometer and prometheus libs ?

[LatsykNS]

@shakuzen, from the method description it is not fully clear what is complete expire time. Is it possible to update the documentation with the info like: completely expire time = buffer length (distributionStatisticBufferLength) * the rotation duration (distributionStatisticExpiry)?

[marcingrzejszczak]

Is this still a problem with the latest versions?

[github-actions]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[github-actions]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.