added fix in entity logic to fallback to workload name when unknown service is set by OTEL SDK

[nathalapooja]

Description of the issue

Describe the problem or feature in addition to a link to the issues.

If an application instrumented with an OTEL SDK doesn't specify the service.name as a resource attribute, it will return unknown_service:<process.executable.name> or unknown_service for its Service name: https://opentelemetry.io/docs/specs/semconv/attributes-registry/service/.

[2] service.name: MUST be the same for all instances of horizontally scaled services. If the value was not specified, SDKs MUST fallback to unknown_service: concatenated with process.executable.name, e.g. unknown_service:bash. If process.executable.name is not available, the value MUST be set to unknown_service.

In the context of a service entity, this would populate the Service name to be an unknown service value.
com.amazonaws.cloudwatch.entity.internal.service.name: Str(unknown_service:java)

But for entity, we want the agent to fallback to workload when the service name is unknown.

Description of changes

How does this change address the problem?

Added logic in entity processor to check if the service name is prefixed with unknown_service, and fallback to workload.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Describe what tests you have done.
Unit tests
Manual tests

1. Created an EKS cluster and deployed the Amazon CloudWatch Observability EKS add-on.
2. Set up sample application by following https://aws-otel.github.io/docs/getting-started/adot-eks-add-on/sample-app.
   Removed resource attributes.
   Changed OTEL_EXPORTER_OTLP_ENDPOINT to http://cloudwatch-agent.amazon-cloudwatch:4317.
3. cloned amazon-cloudwatch-agent Add k8sattributesprocessor to otlp pipeline with workload type detection #1524 k8s-attr branch to replicate the issue
4. Built the agent image by running make docker-build-amd64 and changed the image in the AmazonCloudWatchAgent custom resource.
5. Added debug exporter to OTLP pipeline for testing.

Before code change

"com.amazonaws.cloudwatch.entity.internal.k8s.workload.name":"sample-app","com.amazonaws.cloudwatch.entity.internal.platform.type":"AWS::EKS","com.a │ │ mazonaws.cloudwatch.entity.internal.service.name":"unknown_service:java"
"service.name":"unknown_service:java"

After code change

"com.amazonaws.cloudwatch.entity.internal.k8s.workload.name":"sample-app","com.amazonaws.cloudwatch.entity.internal.platform.type":"AWS::EKS","com.am │
│ azonaws.cloudwatch.entity.internal.service.name":"sample-app","com.amazonaws.cloudwatch.entity.internal.service.name.source":"K8sWorkload"
"service.name":"unknown_service:java"

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

[musa-asad]

nit: wonder if we could use regex to match on ^unknown_service(?::.+)?$ to minimize edge cases?

EDIT: Nevermind, it uses HasPrefix, which is fine.

[lisguo]

nit: move "unknown_service" to a const

[lisguo]

nit: move "unknown_service" to a const

[lisguo]

is there a test case where we are unable to retrieve the workload but the service name is "unknown_service" ?