Are you prepared to start out your journey on the street to amassing telemetry knowledge out of your purposes? Nice observability begins with nice instrumentation!
On this sequence, you will discover learn how to undertake OpenTelemetry (OTel) and learn how to instrument an software to gather tracing telemetry. You may learn to leverage out-of-the-box computerized instrumentation instruments and perceive when it’s a necessity to discover extra superior handbook instrumentation on your purposes. By the top of this sequence, you will have an understanding of how telemetry travels out of your purposes to the OpenTelemetry Collector, and be able to deliver OpenTelemetry to your future tasks. Every little thing mentioned right here is supported by a hands-on, self-paced workshop authored by Paige Cruz.
The earlier article explored how builders are in a position to manually instrument their purposes to generate particular metadata for our enterprise to derive higher insights sooner. On this article, we’ll take a look at the primary a part of learn how to hyperlink metrics to hint knowledge utilizing exemplars.
It’s assumed that you just adopted the earlier articles in organising each OpenTelemetry and the instance Python software challenge, but when not, return and see the earlier articles as it is not lined right here.
Earlier than we dive into linking our metrics to traces with exemplars, let us take a look at what precisely an exemplar is.
Exemplar Fundamentals
As outlined by Google Cloud Observability:
“Exemplars are a way to associate arbitrary data with metric data. You can use them to attach non-metric data to measurements. One use of exemplars is to associate trace data with metric data.”
Bear in mind, we use hint knowledge to offer an in depth view of a single request by means of our programs and metrics are used to supply an aggregated programs view. Exemplars are a method to mix the 2, such that when you slender your troubleshooting efforts to a single hint, you’ll be able to then discover the related metrics that the exemplar gives. This additionally works from metrics to traces, for instance, permitting you to leap from, “Hey, what’s that weird spike?” on a metric chart on to a hint related to that context in a single click on.
Our objective is tying metrics to traces by way of exemplars, so utilizing the CNCF open-source tasks, we’ll instrument metrics with Prometheus, which has secure APIs and an intensive ecosystem of instrumented libraries. Whereas OpenTelemetry additionally gives a metrics SDK, it is marked as combined and below lively improvement. The Python SDK doesn’t but assist exemplars, so let’s maintain our traces instrumented with OpenTelemetry.
The plan we’ll comply with in Half One is to first configure a Prometheus occasion to assemble metrics from our instance software, second to instrument our instance software to gather metrics, and verifying that that is amassing metrics.
In Half Two, we’ll then implement an exemplar connecting these metrics to present hint knowledge we have been amassing from our instance software, and at last, confirm all this work in Jaeger tooling.
Prometheus Configuration
Inside the instance Python software we have beforehand put in, we discover a configuration file for Prometheus in metrics/prometheus/prometheus.yml that appears like this:
international:
scrape_interval: 5s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "hello-otel"
static_configs:
- targets: ["localhost:5000"]
The default use of Prometheus is to observe targets over HTTP and scrape their endpoints. Our preliminary configuration is scraping Prometheus itself because the goal with the endpoint localhost:9090 and in search of hello-intel as a goal on the endpoint localhost:5000.
Utilizing this configuration we will now construct and begin our Prometheus occasion with:
$ podman construct -t workshop-prometheus:v2.54.1 -f ./metrics/prometheus/Buildfile-prom $ podman run -p 9090:9090 workshop-prometheus:v2.54.1 --config.file=/and so on/prometheus/prometheus.yml
We are able to confirm our targets are configured appropriately by opening the Prometheus console Targets web page at http://localhost:9090/targets, noting that Prometheus is scraping itself and ready for the hello-otel software to return on-line:
For now, cease the Prometheus occasion by utilizing CTRL-C as we’ll later be operating the applying and Prometheus occasion collectively in a single pod.
Gathering Metrics
The subsequent step is to start out amassing metrics from our Python instance software. To do that we will discover the prometheus-flask-exporter that gives a simple getting began expertise.
We begin by opening the construct file metrics/Buildfile-metrics and including the command to put in the prometheus_flask_exporter as proven right here in daring:
FROM python:3.12-bullseye WORKDIR /app COPY necessities.txt necessities.txt RUN pip set up -r necessities.txt RUN pip set up opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask opentelemetry-instrumentation-jinja2 opentelemetry-instrumentation-requests prometheus-flask-exporter COPY . . CMD [ "flask", "run", "--host=0.0.0.0"]
Now to start out utilizing it in our instance software we open the file metrics/app.py and add the prometheus-flask-exporter and specify import particularly PrometheusMetrics as proven in daring beneath:
... from opentelemetry.hint import set_tracer_provider from opentelemetry.sdk.hint import TracerProvider from opentelemetry.sdk.hint.export import SimpleSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.instrumentation.flask import FlaskInstrumentor from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor from opentelemetry.instrumentation.requests import RequestsInstrumentor from prometheus_flask_exporter import PrometheusMetrics ...
Search additional down this identical file, metrics/app.py, and confirm that the programmatic Flask metric instrumentation has been added as proven in daring right here:
...
supplier = TracerProvider()
processor = SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
supplier.add_span_processor(processor)
set_tracer_provider(supplier)
app = Flask("hello-otel")
FlaskInstrumentor().instrument_app(app)
Jinja2Instrumentor().instrument()
RequestsInstrumentor().instrument()
metrics = PrometheusMetrics(app)
...
Now that we now have all of it configured, let’s strive it out.
Verifying Metrics to Traces
We have to begin verification by (re)constructing the applying picture as follows:
$ podman construct -t hello-otel:metrics -f metrics/Buildfile-metrics ... Efficiently tagged localhost/hello-otel:metrics 81039de9e73baf0c2ee04d75f7c4ed0361cd97cf927f46020e295a30ec34af8f
Now to run this, we’re utilizing a pod configuration with our instance software, a Prometheus occasion, and the Jaeger tooling to visualise our telemetry knowledge as follows:
$ podman play kube metrics/app_pod.yaml --down
As soon as this has began, we will open a browser and make a number of requests to every of the pages listed right here beneath to generate metrics and tracing knowledge:
- http://localhost:8001
- http://localhost:8001/rolldice
- http://localhost:8001/doggo
I discussed beforehand that Prometheus makes use of a pull-model the place a Prometheus server scrapes a /metric endpoint. Let’s examine our instance software’s metric endpoint to confirm that the prometheus-flask-exporter is working by opening http://localhost:8001/metrics in a browser and confirming we’re seeing metrics prefixed with flask_* and python_* as follows:
Now after we examine if the hello-otel software is obtainable to be scrapped by Prometheus by opening the targets web page in Prometheus at http://localhost:9090/targets, we see that each Prometheus and the instance software are within the state of UP (it was DOWN, beforehand):
This concludes Half One. These examples use code from a Python software you could discover within the offered hands-on workshop.
What’s Subsequent?
This text, half one in all two, began the journey to linking metrics to our hint knowledge with exemplars utilizing Prometheus and OpenTelemetry with our instance software.
In our subsequent article, half two, we’ll end our journey by implementing an exemplar to hyperlink these collected metrics to their related traces.