[confighttp] otelhttp-generated span names may have high cardinality

[axw]

Component(s)

No response

What happened?

Describe the bug

The otelhttp instrumentation in confighttp uses the URL path for span names:

opentelemetry-collector/config/confighttp/confighttp.go

Lines 468 to 470 in 2b5fa0e

	otelhttp.WithSpanNameFormatter(func(_ string, r *http.Request) string {
	return r.URL.Path
	}),

According to https://opentelemetry.io/docs/specs/semconv/http/http-spans/#name:

HTTP span names SHOULD be {method} {target} if there is a (low-cardinality) target available. If there is no (low-cardinality) {target} available, HTTP span names SHOULD be {method}.

I realise it's a SHOULD and not a MUST, but I would expect OpenTelemetry Collector to be setting an example and following documented recommendations. Producing high cardinality span names can be problematic for certain pipelines, e.g. when aggregating spans by name.

Steps to reproduce

1. Add trace telemetry to examples/local/otel-config.yaml:

$ git diff
diff --git a/examples/local/otel-config.yaml b/examples/local/otel-config.yaml
index 168a1631a..3e6266e70 100644
--- a/examples/local/otel-config.yaml
+++ b/examples/local/otel-config.yaml
@@ -39,3 +39,10 @@ service:
       exporters: [debug]
 
   extensions: [zpages]
+  telemetry:
+    traces:
+      level: basic
+      processors:
+        - batch:
+            exporter:
+              console: {}

2. make run
3. curl http://localhost:4318/foo/bar

What did you expect to see?

I would expect to see a span logged with the name GET, or something low-cardinality but more descriptive like GET unknown route.

What did you see instead?

A span is logged with the name /foo/bar:

{                                                                                                                                                                                                                                                       
        "Name": "/foo/bar",                                                                                                                                                                                                                             
        "SpanContext": {                                                                                                                                                                                                                                
                "TraceID": "9a023a975a32440d01b47c6f8f7a60c7",                                                                                                                                                                                          
                "SpanID": "aa890cf63d51904a",                                                                                                                                                                                                           
                "TraceFlags": "01",                                                                                                                                                                                                                     
                "TraceState": "",                                                                                                                                                                                                                       
                "Remote": false                                                                                                                                                                                                                         
        },
...

Collector version

2b5fa0e

Environment information

Environment

OS: Ubuntu 24.04
Compiler: go 1.24.0

OpenTelemetry Collector configuration

extensions:
  zpages:
    endpoint: localhost:55679

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318

processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 2G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

  extensions: [zpages]
  telemetry:
    traces:
      level: basic
      processors:
        - batch:
            exporter:
              console: {}

Log output

{
	"Name": "/foo/bar",
	"SpanContext": {
		"TraceID": "ca846f88686dc369615aa615b163a9cb",
		"SpanID": "23a71a43a538f482",
		"TraceFlags": "01",
		"TraceState": "",
		"Remote": false
	},
	"Parent": {
		"TraceID": "00000000000000000000000000000000",
		"SpanID": "0000000000000000",
		"TraceFlags": "00",
		"TraceState": "",
		"Remote": false
	},
	"SpanKind": 2,
	"StartTime": "2025-02-25T11:15:43.641507607+08:00",
	"EndTime": "2025-02-25T11:15:43.641723584+08:00",
	"Attributes": [
		{
			"Key": "http.method",
			"Value": {
				"Type": "STRING",
				"Value": "GET"
			}
		},
		{
			"Key": "http.scheme",
			"Value": {
				"Type": "STRING",
				"Value": "http"
			}
		},
		{
			"Key": "net.host.name",
			"Value": {
				"Type": "STRING",
				"Value": "localhost"
			}
		},
		{
			"Key": "net.host.port",
			"Value": {
				"Type": "INT64",
				"Value": 4318
			}
		},
		{
			"Key": "net.sock.peer.addr",
			"Value": {
				"Type": "STRING",
				"Value": "127.0.0.1"
			}
		},
		{
			"Key": "net.sock.peer.port",
			"Value": {
				"Type": "INT64",
				"Value": 47346
			}
		},
		{
			"Key": "user_agent.original",
			"Value": {
				"Type": "STRING",
				"Value": "curl/8.5.0"
			}
		},
		{
			"Key": "http.target",
			"Value": {
				"Type": "STRING",
				"Value": "/foo/bar"
			}
		},
		{
			"Key": "net.protocol.version",
			"Value": {
				"Type": "STRING",
				"Value": "1.1"
			}
		},
		{
			"Key": "http.response_content_length",
			"Value": {
				"Type": "INT64",
				"Value": 19
			}
		},
		{
			"Key": "http.status_code",
			"Value": {
				"Type": "INT64",
				"Value": 404
			}
		}
	],
	"Events": null,
	"Links": null,
	"Status": {
		"Code": "Unset",
		"Description": ""
	},
	"DroppedAttributes": 0,
	"DroppedEvents": 0,
	"DroppedLinks": 0,
	"ChildSpanCount": 0,
	"Resource": [
		{
			"Key": "service.instance.id",
			"Value": {
				"Type": "STRING",
				"Value": "1dd05082-b413-4990-af89-04751084d903"
			}
		},
		{
			"Key": "service.name",
			"Value": {
				"Type": "STRING",
				"Value": "otelcorecol"
			}
		},
		{
			"Key": "service.version",
			"Value": {
				"Type": "STRING",
				"Value": "0.120.0-dev"
			}
		}
	],
	"InstrumentationScope": {
		"Name": "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",
		"Version": "0.59.0",
		"SchemaURL": "",
		"Attributes": null
	},
	"InstrumentationLibrary": {
		"Name": "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",
		"Version": "0.59.0",
		"SchemaURL": "",
		"Attributes": null
	}
}

Additional context

No response

[axw]

I have a possible solution at #12593. There's an alternative described in the PR description.

[douglascamata]

@axw I think the path should be okay here for the span name because it will be low cardinality unless your Collector is under some sort of 404 spam attack. In this case, maybe we could have a middleware running at the end of the request lifecycle, when there might already be a 404 status code sent and rename the span at this moment?

[axw]

@douglascamata agreed, this is only likely to be a problem if a security scanner or some malicious client is sending spurious requests to your collector. That's not uncommon.

In this case, maybe we could have a middleware running at the end of the request lifecycle, when there might already be a 404 status code sent and rename the span at this moment?

How would you know that the 404 is due to there being no missing route vs. a matching route whose handler returns 404? Probably not likely, but also not impossible.

Another option, that we have implemented, is to put an ingress in front that allows only a strictly controlled set of paths through to the collector - so anything that's not a registered route will be filtered out before it hits the collector.

So it's not terrible as it is, but seems like we could improve things and follow the instrumentation recommendations.

[douglascamata]

How would you know that the 404 is due to there being no missing route vs. a matching route whose handler returns 404? Probably not likely, but also not impossible.

True, that would be a downside of my idea. I don't think though it would affect the Collector (please correct me if I'm wrong), because it's not using 404 as widely as a an API that follows the REST patterns very strictly.

So it's not terrible as it is, but seems like we could improve things and follow the instrumentation recommendations.

100% agreed. The Collector already partially follows some of the recommendation, i.e. it does not include the query parameters in the span name.

Personally I don't recall seeing a lot of tracing code out there that defends itself well against 404 attacks. It's good to think about it though and maybe create a good solution for this that could help many out there.

[axw]

open-telemetry/opentelemetry-go-contrib#6193 (comment) is relevant