fix: Adding configs for Otel namespace retention (#9734)

# Change log

- **OTEL resource attribute to metric label conversion**

In addition to work for adding OTEL resource attributes in PR #9642 , a
second configuration is needed for OTEL to convert these into metrics
labels as outlined in opentelemetry-collector-contrib PR
[#15349](open-telemetry/opentelemetry-collector-contrib#15349).

If the resource attribute is node converted into a metics label, they
arrive to Prometheus with the namespace set as `metrics`, which is where
the OTEL collector is installed and not the application itself.

This change has ben tested by reinstalling the metrics charts into the
spartan cluster and confirming namespace was an available label on
metrics arriving to prometheus. (I manually connected to prometheus to
query, rather than downstream Grafana).

- **Added kube-state-metrics to prometheus**

Kube state metrics exposes kubernetes cluster stats to prometheus. This
is needed currently to show CPU usage in Grafana dashboards (since this
is not an application metric).

More info: https://github.com/kubernetes/kube-state-metrics

spartan/metrics/values.yaml

@@ -43,10 +43,16 @@ opentelemetry-collector:
       prometheus:
         endpoint: ${env:MY_POD_IP}:8889
         metric_expiration: 5m
+        resource_to_telemetry_conversion:
+          enabled: true
     extensions:
       health_check:
         endpoint: ${env:MY_POD_IP}:13133
     processors:
+      resource:
+        attributes:
+        - action: preserve
+          key: k8s.namespace.name
       batch: {}
     receivers:
       otlp:
@@ -153,6 +159,9 @@ prometheus:
         - job_name: aztec
           static_configs:
             - targets: ["metrics-opentelemetry-collector.metrics:8889"]
+        - job_name: 'kube-state-metrics'
+          static_configs:
+            - targets: ['metrics-kube-state-metrics.metrics.svc.cluster.local:8080']
 
 # Enable and configure Grafana
 # https://artifacthub.io/packages/helm/grafana/grafana