[prometheus-metrics-instrumentation-caffeine] Cannot register multiple Caffeine cache

[kaustubhDsarathi]

Context:
I'm using prometheus-metrics-instrumentation-caffeine to instrument my caffeine cache.

public class UserCache {

    public static final String KEY_REMOVED_FROM_CACHE = "key={} removed from cache";
    @Getter
    private LoadingCache<String, Optional<UserDetailDto>> cache1;
    private LoadingCache<String, Optional<UserDetailDto>> cache2;
    @Getter
    private LoadingCache<String, Optional<UserDetailDto>> cache3;
    private final UserRepository userRepository;
    private final CacheConfig cacheConfig;
    private final PrometheusMeterRegistry prometheusMeterRegistry;

    @PostConstruct
    void init() {
        if(cacheConfig.isEnabled()) {
            ExecutorService threadPool = Executors.newFixedThreadPool(4);
            cache1 = Caffeine.newBuilder()
                    .maximumSize(100_000) // 100_000 test-user data will be cached.
                    .removalListener(TestCache::onRemoval)
                    .refreshAfterWrite(2, TimeUnit.HOURS)
                    .executor(threadPool)
                    .recordStats()
                   .build(userRepository::loadCache1);

            cache2 = Caffeine.newBuilder()
                    .maximumSize(5_000) // 5000 test-user data will be cached.
                    .removalListener(TestCache::onRemoval)
                    .refreshAfterWrite(5, TimeUnit.HOURS) // refresh cache after 5 hours write
                    .executor(threadPool)
                    .build(userRepository::loadCache2);


            cache3 = Caffeine.newBuilder()
                    .maximumSize(100_000) // 100_000 test-user data will be cached.
                    .removalListener(TestCache::onRemoval)
                    .refreshAfterWrite(1, TimeUnit.HOURS)
                    .executor(threadPool)
                    .build(userRepository::loadCache3);

            CacheMetricsCollector collector = new CacheMetricsCollector();
            collector.addCache("cache1", cache1);
            collector.addCache("cache2", cache2);
            collector.addCache("cache3", cache3);

            prometheusMeterRegistry.getPrometheusRegistry().register(collector);
        }

    private static void onRemoval(@Nullable Object key, @Nullable Object value, RemovalCause cause) {
        log.atDebug().log(KEY_REMOVED_FROM_CACHE, key);
    }
}

After running the application and accessing the metrics, I'm getting the following error
caffeine_cache_estimated_size: duplicate metric name.

There is no mention in the Javadoc whether we can provide different names to the collector also the register() method seems to be missing in the actual code

As a workaround, I'm currently manually registering the Gauge objects in PrometheusMeterRegistry

public void registerCacheMetrics(String cacheName, PrometheusMeterRegistry prometheusMeterRegistry, LoadingCache<?, ?> loadingCache) {

        // Register custom gauges for cache metrics
        Gauge.builder(cacheName + ".hit.count", loadingCache, cache -> cache.stats().hitCount())
                .description("Number of cache hits")
                .register(prometheusMeterRegistry);

        Gauge.builder(cacheName + ".miss.count", loadingCache, cache -> cache.stats().missCount())
                .description("Number of cache misses")
                .register(prometheusMeterRegistry);

        Gauge.builder(cacheName + ".requests.count", loadingCache, cache -> cache.stats().requestCount())
                .description("Number of times cache requested")
                .register(prometheusMeterRegistry);

        Gauge.builder(cacheName + ".eviction.count", loadingCache, cache -> cache.stats().evictionCount())
                .description("Number of cache evictions")
                .register(prometheusMeterRegistry);

        Gauge.builder(cacheName + ".load.success.count", loadingCache, cache -> cache.stats().loadSuccessCount())
                .description("Number of successful cache loads")
                .register(prometheusMeterRegistry);

        Gauge.builder(cacheName + ".load.failure.count", loadingCache, cache -> cache.stats().loadFailureCount())
                .description("Number of cache load failures")
                .register(prometheusMeterRegistry);

        Gauge.builder(cacheName + ".load.total", loadingCache, cache -> cache.stats().totalLoadTime())
                .description("Time taken to load cache")
                .register(prometheusMeterRegistry);
    }

Is there a way we can instrument multiple Cache objects using CacheMetricsCollector?

[ChrisMcD1]

I am having a similar issue, although I am able to recreate it without even creating a cache:

  private val cacheMetrics = new CacheMetricsCollector()
  PrometheusRegistry.defaultRegistry.register(cacheMetrics)

is sufficient to reproduce the error:

An Exception occurred while scraping metrics: java.lang.IllegalStateException: caffeine_cache_estimated_size: duplicate metric name.

[pheyken]

Hey @kaustubhDsarathi & @ChrisMcD1, could you check if the old, simpleclient_caffeine CacheMetricCollector is registered via the simpleclient-bridge? This could indeed cause this error, as the metric names are the same for both.

Regarding the documentation: correct, this was a mistake on my side when migrating the CacheMetricCollector. The register method does not exist anymore

[ChrisMcD1]

I have identified that my problem was user error!

I had my cache being created and registered inside of a class that our dependency injection framework was instantiating twice. Fixing that and making it instantiated once fixed it for me, and I am happily getting metrics!

[ChrisMcD1]

I guess I should ask the question though, should having two CacheMetricsCollectors result in a crash?

[pheyken]

I guess I should ask the question though, should having two CacheMetricsCollectors result in a crash?

This is the intended behavior of the prometheus client in general. E.g. if you register any other metric twice, an error would be thrown as well (e.g. you cannot have two counters with the same name).

In the end some other library / user could create a metric with the same name and not the CacheMetricsCollector and this is a case where it should error

[kaustubhDsarathi]

@pheyken any reason why io.prometheus:prometheus-metrics-instrumentation-caffeine is not part of io.prometheus:prometheus-metrics-bom, if there isn't any then can we make it a part of it?

[zeitlinger]

@pheyken any reason why io.prometheus:prometheus-metrics-instrumentation-caffeine is not part of io.prometheus:prometheus-metrics-bom, if there isn't any then can we make it a part of it?

this was fixed in #1175

[kaustubhDsarathi]

Tested again using the 1.3.3 version with the same code above I'm able to register multiple CaffeineCache objects.