Observability: Gateway client requests clash with WebClient request metrics

[kkondratov]

Describe the bug
After upgrading to spring boot 3 with gateway 4.0.6 we've noticed that our metrics for the webclient are not available anymore in prometheus

With the use of micrometer observability an introduction was done to provide the http.client.request metrics for the gateway client.
This clashes with the default http.client.request provided by the DefaultClientRequestObservationConvention from spring-web.
This was introduced in: https://github.com/spring-cloud/spring-cloud-gateway/pull/2715/files

The prometheus lowCardinalityValues defined in GatewayDocumentedObservation differ from the ones defined in the ClientHttpObservationDocumentation. GatewayDocumentedObservation has for example http.method, http.status_code and the ClientHttpObservationDocumentation has method, uri, status

This causes the metric http.client.requests be reported with different labels, ones for the Gateway WebClient metrics and one for the Spring Web configured one.

Prometheus unfortunately ignores metrics if a metric is encountered with different labels than the initially scraped ones, for context:

• Same metric name, different set of label names prometheus/client_java#696
• Support meters with the same name but different set of tag keys in PrometheusMeterRegistry micrometer-metrics/micrometer#877

Ideally spring cloud client request metrics should have the prefix spring.cloud.gateway for the DefaultGatewayObservationConvention.

Its not easy to override this property either since the values are reference by the singleton instance. Only two approaches currently are: shadow the ObservedRequestHttpHeadersFilter and the DefaultGatewayObservationConvention or disable observability which is not desirable.

EDIT

Looks like overriding the bean DefaultGatewayObservationConvention and overriding the name works as a workaround:

@Component
public class CloudGatewayPrefixedGatewayObservationConvention extends DefaultGatewayObservationConvention {

    @Override
    @NonNull
    public String getName() {
        return "spring.cloud.gateway.http.client.requests";
    }
}

[spencergibb]

@jonatan-ivanov @shakuzen, should we make this workaround permanent?

[jonatan-ivanov]

@kkondratov

Prometheus unfortunately ignores metrics if a metric is encountered with different labels than the initially scraped ones, for context:

The Prometheus Java client does, the Prometheus Server must be ok with this since multiple apps can ship metrics with the same name but different labels, the missing labels are handled as label_name="".

Do you use WebClient in Gateway so is this a problem on the client side? If so, what is your use-case?

Its not easy to override this property either since the values are reference by the singleton instance. Only two approaches currently are...

This should be easy:

1. You can extend DefaultGatewayObservationConvention override the name and inject it as you have already found out.
2. You can create an ObservationFilter that you can use to modify your Observations (i.e.: rename them).

@spencergibb Permanently fixing this should be easy, it's just modifying this hardcoded value:

spring-cloud-gateway/spring-cloud-gateway-server/src/main/java/org/springframework/cloud/gateway/filter/headers/observation/DefaultGatewayObservationConvention.java

Line 75 in 43eb8ce

return "http.client.requests";

But there are some consequences: it is a breaking change that can break people's dashboards and automation since the name of the Observation and everything that is created from that Observation will have a different name. So I would like to understand the use case behind using WebClient in Gateway first. If it seems to impact lot of people maybe changing this in the next release is something we should consider, otherwise I would document it.

[kkondratov]

@jonatan-ivanov thank you for the explanation and detailed feedback.

You can extend DefaultGatewayObservationConvention override the name and inject it as you have already found out.

You're correct, that is the way we're currently solving the issue

Do you use WebClient in Gateway so is this a problem on the client side? If so, what is your use-case?

Indeed we use the WebClient in the Gateway. In our case it is a necessity because we're not in control of certain authentication mechanisms, we're forced to validate certain values with an external server. This leads to the usage of WebClient in Gateway.

But there are some consequences: it is a breaking change that can break people's dashboards and automation since the name of the Observation and everything that is created from that Observation will have a different name.

This is correct for all Gateway set-ups based on version 4.x or higher. This was a breaking change in upgrade to 4.x as well though coming from 3.x Gateway versions.

[kzander91]

We just upgraded to Boot 3.4.0 and Cloud 2024.0.0 and became aware of this issue due to the following warning now being logged:

The meter (MeterId{name='http.client.requests.active', tags=[tag(application-version=REDACTED),tag(http.method=GET),tag(http.status_code=UNKNOWN),tag(spring.cloud.gateway.route.id=REDACTED),tag(spring.cloud.gateway.route.uri=REDACTED)]}) registration has failed: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'http.client.requests.active' containing tag keys [application_version, client.name, exception, method, outcome, status, uri]. The meter you are attempting to register has keys [application_version, http.method, http.status_code, spring.cloud.gateway.route.id, spring.cloud.gateway.route.uri]. Note that subsequent logs will be logged at debug level.

We also use WebClient in gateway, in our case for audit logging purposes, where a custom filter intercepts all requests and sends metadata to a separate service.
Also, Spring Security uses WebClient to fetch the JWK set from the authorization server to validate bearer tokens.

[spencergibb]

We should make it configurable

[kzander91]

I noticed that there also is a spring.cloud.gateway.requests metric, recorded by GatewayMetricsFilter, which almost looks like a duplicate of http.client.requests.
The former directly records against MeterRegistry, while the latter is recorded implicitly through the Observation. Maybe it would make sense to deprecate and eventually remove the former.

[felix-ebert]

We're also facing this issue after upgrading to Spring Boot 3.4.1 and Spring Cloud 2024.0. The workaround mentioned by @kkondratov worked for us too. Has there been any update or progress on a permanent fix?