Kotlin Coroutines and OpenTelemetry Tracing – DZone – Uplaza

I lately in contrast three OpenTelemetry approaches on the JVM: Java Agent v1, v2, and Micrometer. I used Kotlin and coroutines with out overthinking. I acquired fascinating suggestions on the utilization of @WithSpan with coroutines:

Certainly, the @WithSpan annotation has labored flawlessly along with coroutines for a while already. Nonetheless, it made me take into consideration the underlying workings of OpenTelemetry. Listed here are my findings.

The @WithSpan Annotation Processor

@WithSpan is an easy annotation. To be of any use, one wants an annotation processor. In case you want a refresher on annotation processors, please verify this not-so-new however still-relevant publish.

A fast search on the OpenTelemetry repository reveals that the processor concerned is WithSpanInstrumentation.

This is an abridged abstract of the courses concerned:

WithSpanInstrumentation does the annotation processing half; it delegates to WithSpanSingleton. In flip, the latter bridges the decision to the Instrumenter class. Instrumenter comprises the core of making spans and interacting with the OpenTelemetry collector.

Instrumenter and Context

The Instrumenter encapsulates your complete logic for gathering telemetry, from accumulating the info, to beginning and ending spans, to recording values utilizing metrics devices.

An Instrumenter is known as at the beginning and the top of a request/response lifecycle.
When instrumenting a library, there’ll usually be 4 steps.

1. Create an Instrumenter utilizing InstrumenterBuilder. Use the builder to
   configure any library-specific customizations, and likewise expose helpful knobs to your person.
2. Name Instrumenter#shouldStart(Context, Object) and don’t proceed if it returns
   false.
3. Name Instrumenter#begin(Context, Object) initially of a request.
4. Name Instrumenter#finish(Context, Object, Object, Throwable) on the finish of a request.

For extra detailed details about utilizing the Instrumenter see the Utilizing the Instrumenter API web page.

– Instrumenter class

Instrumenter works along with Context. OpenTelemetry API customers must be acquainted with it, particularly the decision to Context.present(). Let’s describe it in additional element.

Context shops knowledge in a ContextStorage occasion, whose default is ThreadLocal. The ThreadLocal class has been the old-age strategy to cross knowledge round with out interfering with methodology signatures. It shops knowledge within the present thread.

Kotlin’s OpenTelemetry Extension

ThreadLocal works completely — till you spawn different threads. On this case, you have to explicitly cross knowledge round. So-called Reactive Programming frameworks, similar to Spring WebFlux, do spawn different threads; most, if not all, present utilities to deal with the passing mechanically.

Coroutines implement Reactive Programming. Not solely do they spawn threads, however in addition they decouple coroutine from threads. A coroutine might “jump” throughout a number of threads in its lifetime. Thus, storing the OpenTelemetry context in a ThreadLocal would not work.

But, coroutines present a devoted storage mechanism, the coroutine context. We’d like a strategy to transfer the OpenTelemetry context from the ThreadLocal to the coroutine context and again once more. The best way exists within the opentelemetry-extension-kotlin jar:

The one half that must be added is the place these capabilities are referred to as. Unsurprisingly, the magic occurs within the Java Agent and all different instrumentation courses. You would possibly keep in mind the TypeInstrumentation interface on the primary diagram, which the category WithSpanInstrumentation applied. The Java Agent caters to many alternative frameworks and libraries, e.g., Spring WebFlux, and Kotlin Coroutines. Its builders designed it so every TypeInstrumentation concrete class focuses on the instrumentation of a selected facet of the framework or library; coroutines are not any exception.

Be aware that the code supplies a extra particular instrumentation of WithSpanInstrumentation, which is devoted to coroutines.

It seems that the KotlinCoroutinesInstrumentationHelper comprises the magic to repeat the context from the ThreadLocal to the coroutine context:

package deal io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines;

import io.opentelemetry.context.Context;
import io.opentelemetry.extension.kotlin.ContextExtensionsKt;
import kotlin.coroutines.CoroutineContext;

public last class KotlinCoroutinesInstrumentationHelper {

  public static CoroutineContext addOpenTelemetryContext(CoroutineContext coroutineContext) {
    Context present = Context.present();                                                      //1
    Context inCoroutine = ContextExtensionsKt.getOpenTelemetryContext(coroutineContext);
    if (present == inCoroutine || inCoroutine != Context.root()) {
      return coroutineContext;
    }
    return coroutineContext.plus(ContextExtensionsKt.asContextElement(present));              //2
  }

  personal KotlinCoroutinesInstrumentationHelper() {}
}

1. Get the OpenTelemetry context – from the ThreadLocal. 
2. Add the context to the coroutine context.

And that is a wrap.

Abstract

On this publish, I’ve analyzed the workings of @WithSpan generally and within the context of Kotlin Coroutines. The Java Agent supplies many alternative instrumenting courses, every devoted to a novel side of a framework or library. The WithSpanInstrumentation within the io.opentelemetry.javaagent.instrumentation.extensionannotations manages “regular” code; the one in io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines manages coroutines.

The largest problem is that OpenTelemetry shops knowledge in a ThreadLocal by default. The coroutine library would not assure the identical thread can be used. Quite the opposite, a coroutine will seemingly bounce throughout completely different threads throughout its lifetime.

The Java Agent supplies the mechanism to deal with it. One half focuses on shifting OpenTelemetry knowledge from the ThreadLocal to the coroutine context; the opposite supplies a devoted instrumentation to name the above code when it enters the latter.

To Go Additional