Linking Metrics to Traces With Exemplars, Half 2 – DZone – Uplaza

Are you prepared to start out your journey on the highway to accumulating telemetry information out of your purposes? Nice observability begins with nice instrumentation! 

On this collection, you may discover the best way to undertake OpenTelemetry (OTel) and the best way to instrument an software to gather tracing telemetry. You will discover ways to leverage out-of-the-box computerized instrumentation instruments and perceive when it’s a necessity to discover extra superior guide instrumentation to your purposes. By the tip of this collection, you may have an understanding of how telemetry travels out of your purposes to the OpenTelemetry Collector, and be able to carry OpenTelemetry to your future initiatives. Every thing mentioned right here is supported by a hands-on, self-paced workshop authored by Paige Cruz. 

The earlier article explored the primary a part of the best way to hyperlink metrics to hint information utilizing exemplars, the place we configured our software to reveal metrics and a Prometheus occasion to gather these metrics. On this article, we’ll take a look at the second half centered on implementing the exemplars and tying collectively metrics with our hint information.

It’s assumed that you simply adopted the earlier articles in organising each OpenTelemetry and the instance Python software mission, but when not, return and see the earlier articles as it isn’t lined right here.

Prometheus Exemplar Configuration

Let’s begin by including exemplars to our Prometheus metrics which is tying in our traces from OpenTelemetry. Open up the metrics/prometheus/prometheus.yml file, add or confirm there’s a scrape job outlined for the hello-otel software as proven, and save the file:

international:
  scrape_interval: 5s

scrape_configs:

  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "hello-otel"
    static_configs:
      - targets: ["localhost:5000"]

Open up the metrics/prometheus/Buildfile-prom and make sure the CMD is utilizing the exemplar storage characteristic by including the flag as proven right here in daring:

FROM promenade/prometheus:v2.54.1

ADD prometheus.yml /and many others/prometheus

ENTRYPOINT [ "prometheus" ]

CMD [ "--config.file=/etc/prometheus/prometheus.yml", "--enable-feature=exemplar-storage" ]

Utilizing this configuration we will now rebuild our Prometheus occasion with:

$ podman construct -t workshop-prometheus:v2.54.1 -f ./metrics/prometheus/Construct file-prom

Now let’s discover the best way to add metrics to our instance software.

Software Metrics

Open the metrics/app.py file and make sure the import for the Counter metric from Prometheus Shopper has been added as proven in daring:

import random
import re
import urllib3

import requests
from flask import Flask, render_template, request
from breeds import breeds

from opentelemetry.hint import set_tracer_provider
from opentelemetry.sdk.hint import TracerProvider
from opentelemetry.sdk.hint.export import SimpleSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.jinja2 import Jinja2Instrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from prometheus_flask_exporter import PrometheusMetrics
from prometheus_client import Counter
...

On this similar software file, additional down, be sure that a hits counter for the variety of homepage masses is added as proven in daring:

...
supplier = TracerProvider()
processor = SimpleSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
supplier.add_span_processor(processor)

set_tracer_provider(supplier)

app = Flask("hello-otel")
FlaskInstrumentor().instrument_app(app)
Jinja2Instrumentor().instrument()
RequestsInstrumentor().instrument()
metrics = PrometheusMetrics(app)

HITS_COUNTER = Counter('hits_counter', 'depend of homepage masses')
...

Lastly within the software file, guarantee code is added to the index() to increment (enhance) the hits_counter and configure exemplars to connect to the metric as proven in daring. Save the file when completed:

...
@app.route("https://dzone.com/")
def index():
  international HITS
  span = hint.get_current_span()
  trace_id = '{:032x}'.format(span.get_span_context().trace_id)
  HITS = HITS + 1
  span.set_attribute("hits", HITS)
  HITS_COUNTER.inc(1, exemplar={"trace_id": trace_id, "trace_url": f"http://localhost:16686/trace/{trace_id}"})
  msg = f'This webpage has been seen {HITS} instances'
  return msg
...

Now we have to replace our software container picture by including the command to put in prometheus_flask_exporter to the file metrics/Buildfile-metrics as proven in daring:

FROM python:3.12-bullseye

WORKDIR /app

COPY necessities.txt necessities.txt

RUN pip set up -r necessities.txt

RUN pip set up opentelemetry-api 
    opentelemetry-sdk 
    opentelemetry-exporter-otlp 
    opentelemetry-instrumentation-flask 
    opentelemetry-instrumentation-jinja2 
    opentelemetry-instrumentation-requests 
    prometheus-flask-exporter

COPY . .

CMD [ "flask", "run", "--host=0.0.0.0"]

Rebuild the applying picture as follows:

$ podman construct -t hello-otel:metrics -f metrics/Buildfile-metrics

...
Efficiently tagged localhost/hello-otel:metrics
81039de9e73baf0c2ee04d75f7c4ed0361cd97cf927f46020e295a30ec34af8f

Now we have to confirm that this all works.

Verifying Exemplars

Now to confirm our exemplars, we run a pod configuration with our instance software, a Prometheus occasion, and the Jaeger tooling to visualise our telemetry information as follows:

$ podman play kube metrics/app_pod.yaml 

As soon as this has began, we will open a browser and make a number of requests (over time, refresh the browser to generate extra within the ensuing graph on the following slide) to the homepage to generate metrics and traces at http://localhost:8001.

Open the Prometheus console in your browser http://localhost:9090 and question hits_counter_total in Graph view. Choose by clicking on the Present Exemplars button within the center to see them as little blue diamonds (observe it’s good to scale back the time window down to five minutes or so to see a graph plot):

Click on on one of many exemplars (any of the blue dots) on the chart and replica the trace_url into a brand new browser tab as proven:

If all is working as we count on it to, then we should always see the hint waterfall for one of many requests made one thing like this:

This verifies that we’re linking our software metrics to our software traces with exemplars. To be neat about issues, we should always shut down our pod and working containers as follows:

$ podman play kube metrics/app_pod.yaml --down

These examples use code from a Python software which you could discover within the offered hands-on workshop.

What’s Subsequent?

This text, half two, concludes the journey to linking metrics to our hint information with exemplars utilizing Prometheus and OpenTelemetry with our instance software.

Keep tuned for extra hands-on materials that will help you together with your cloud-native observability journey.