The OpenTelemetry Astronomy Shop is a demo app to illustrate the implementation of OpenTelemetry in a near real-world environment. It is a microservice-based distributed system which can be easily deployed using Docker or Kubernetes, where we will focus here on deploying it using Kubernetes/Helm.
Our aim is to guide you through the needed steps to deploy the OpenTelemetry Demo in a K8s cluster and send its data to OCI Observability and Management Services.
The following prerequisites are needed:
- Ensure to have an OCI account.
- In Application Performance Monitoring (APM) service, create an APM domain.
- Have the Data Upload Endpoint URL and the private data key of that domain available.
- Have a Kubernetes Cluster available, either locally, in OCI or at other cloud vendors, and provide access to it using kubectl.
We will start by creating a dedicated namespace for the demo app:
kubectl create namespace otel-demo-appNext, we will create a secret named oci-apm-secret:
kubectl create secret generic oci-apm-secret -n otel-demo-app --from-literal="OCI_APM_ENDPOINT=<Data Upload Endpoint>" --from-literal="OCI_APM_DATAKEY=<Private Data Key>"Add the OpenTelemetry Helm charts repo to your helm configuration to allow deploying the OpenTelemetry Demo:
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-chartsCreate file my-values.yaml with the following content:
opentelemetry-collector:
extraEnvsFrom:
- secretRef:
name: oci-apm-secret
config:
exporters:
otlphttp/oci_spans:
endpoint: "${OCI_APM_ENDPOINT}/20200101/opentelemetry/"
headers:
authorization: "dataKey ${OCI_APM_DATAKEY}"
tls:
insecure: false
otlphttp/oci_metrics:
endpoint: "${OCI_APM_ENDPOINT}/20200101/observations/metric?dataFormat=otlp-metric&dataFormatVersion=1"
headers:
authorization: "dataKey ${OCI_APM_DATAKEY}"
tls:
insecure: false
service:
pipelines:
traces:
exporters: [otlp, debug, spanmetrics, otlphttp/oci_spans]
metrics:
exporters: [otlphttp/prometheus, debug, otlphttp/oci_metrics]
Run the following helm command to deploy the OpenTelemetry Demo App:
helm install otel-demo-app open-telemetry/opentelemetry-demo --values my-values.yaml -n otel-demo-appHere is the expected output:
...
██████╗ ████████╗███████╗██╗ ██████╗ ███████╗███╗ ███╗ ██████╗
██╔═══██╗╚══██╔══╝██╔════╝██║ ██╔══██╗██╔════╝████╗ ████║██╔═══██╗
██║ ██║ ██║ █████╗ ██║ ██║ ██║█████╗ ██╔████╔██║██║ ██║
██║ ██║ ██║ ██╔══╝ ██║ ██║ ██║██╔══╝ ██║╚██╔╝██║██║ ██║
╚██████╔╝ ██║ ███████╗███████╗ ██████╔╝███████╗██║ ╚═╝ ██║╚██████╔╝
╚═════╝ ╚═╝ ╚══════╝╚══════╝ ╚═════╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝
- All services are available via the Frontend proxy: http://localhost:8080
by running these commands:
kubectl --namespace default port-forward svc/otel-demo-app-frontendproxy 8080:8080
The following services are available at these paths after the frontendproxy service is exposed with port forwarding:
Webstore http://localhost:8080/
Jaeger UI http://localhost:8080/jaeger/ui/
Grafana http://localhost:8080/grafana/
Load Generator UI http://localhost:8080/loadgen/
Feature Flags UI http://localhost:8080/feature/ With the OpenTelemetry Demo App installed, you are now ready to explore the capabilities of OCI APM (Application Performance Monitoring) and gain deep insights into your application's performance. Lets start by showing you the process of visualising spans and traces using OCI APM, showcasing its powerful features and benefits.
Log in to your Oracle Cloud Infrastructure console and navigate to the Application Performance Monitoring. Here, you will find a wealth of information about your application's performance
Click on the "Traces" tab in the APM console / Trace Explorer. You will see a list of traces generated by the OTel demo application. Each trace represents a user request or an operation within your application:
Select a specific trace to view its details, including the topology of the trace and its spans:
Click on any of the spans to see the span details, which include kubernetes data, SpanId, TraceID, etc.:
Services Topology view allows to visualize associations between services, several parameters can be used for the arrow width:
Using OCI Logging Analytics service to collect and analyze Logs from Kubernetes Infrastructure and Pods
OCI Logging Analytics (LA) provides a complete solution for monitoring Kubernetes (K8s) cluster deployed in OCI, third party public clouds, private clouds, or on-premises including managed Kubernetes deployments. We will start with discovering the K8s cluster running the OpenTelemetry Demo App.
Log in to your Oracle Cloud Infrastructure console and navigate to Logging Analytics Administration. Select Solutions -> Kubernetes -> Connect Clusters -> Monitor Kubernetes -> Oracle OKE, here we are assuming that the K8s cluster is running in OCI. Select the Cluster, press Next and select the right compartment to be used for telemetry data and related monitoring resources, usually it will be the same like the one used for collecting the logs. Click on Configure log collection, this will create all neded dynamic groups and policies to allow collecting logs, metrics, and object information from related Kubernetes components, compute nodes, subnets, and load balancers. The deployed solution will create these statefulsets, daemonsets and cronjobs in namespace oci-onm:
$ kubectl get pods -n oci-onm
NAME READY STATUS RESTARTS AGE
oci-onm-discovery-28982290-zdkxx 0/1 Completed 0 14m
oci-onm-discovery-28982295-bztr2 0/1 Completed 0 9m12s
oci-onm-discovery-28982300-zrxxq 0/1 Completed 0 4m12s
oci-onm-logan-krcmb 1/1 Running 3 80d
oci-onm-logan-npltp 1/1 Running 2 80d
oci-onm-logan-tthvm 1/1 Running 2 80d
oci-onm-mgmt-agent-0 1/1 Running 2 80dCluster overview:
Workload overview:
Node overview:
Pod overview:
List of TOP Log Sources:
Log Records by Pod name:
Compare Cluster results:
Metrics from the OpenTelemetry Demo App are routed via OpenTelemetry Collector to OCI APM service which in turn will forward them to OCI Monitoring service. In parallel, infrastructure metrics from the Kubernetes cluster itself are getting collected by the OCI Management Agent which is running as statefulset in the cluster.
The collected metrics can be inspected using the Metrics Explorer from the OCI Monitoring service.
Metrics collected by the Management Agent can be found in metric namespace mgmtagent_kubernetes_metrics:
Metrics sent by the OpenTelemetry Collector can be found in metric namespace oracle_apm_monitoring:
Finally, out-of-the-box or custom dashboards can be used to visualize the collected metrics:
Further resources for Logging Analytics can be found in folders log-sources, logan-lookups and dashboards. We are providing there an enhanced log source definition to allow queries used in a dashboard with Business Analytics charts.
To maximize the usage value of APM, you can find in folder apm-configs all needed info to enrich the OTel spans with attributes matching the expected APM naming schema.