I lately in contrast three OpenTelemetry approaches on the JVM: Java Agent v1, v2, and Micrometer. I used Kotlin and coroutines with out overthinking. I acquired fascinating suggestions on the utilization of @WithSpan
with coroutines:
Certainly, the @WithSpan
annotation has labored flawlessly along with coroutines for a while already. Nonetheless, it made me take into consideration the underlying workings of OpenTelemetry. Listed here are my findings.
The @WithSpan Annotation Processor
@WithSpan
is an easy annotation. To be of any use, one wants an annotation processor. In case you want a refresher on annotation processors, please verify this not-so-new however still-relevant publish.
A fast search on the OpenTelemetry repository reveals that the processor concerned is WithSpanInstrumentation
.
This is an abridged abstract of the courses concerned:
WithSpanInstrumentation
does the annotation processing half; it delegates to WithSpanSingleton
. In flip, the latter bridges the decision to the Instrumenter
class. Instrumenter
comprises the core of making spans and interacting with the OpenTelemetry collector.
Instrumenter and Context
The
Instrumenter
encapsulates your complete logic for gathering telemetry, from accumulating the info, to beginning and ending spans, to recording values utilizing metrics devices.An
Instrumenter
is known as at the beginning and the top of a request/response lifecycle.
When instrumenting a library, there’ll usually be 4 steps.
- Create an
Instrumenter
utilizingInstrumenterBuilder
. Use the builder to
configure any library-specific customizations, and likewise expose helpful knobs to your person.- Name
Instrumenter#shouldStart(Context, Object)
and don’t proceed if it returnsfalse
.- Name
Instrumenter#begin(Context, Object)
initially of a request.- Name
Instrumenter#finish(Context, Object, Object, Throwable)
on the finish of a request.For extra detailed details about utilizing the
Instrumenter
see the Utilizing the Instrumenter API web page.– Instrumenter class
Instrumenter
works along with Context
. OpenTelemetry API customers must be acquainted with it, particularly the decision to Context.present()
. Let’s describe it in additional element.
Context
shops knowledge in a ContextStorage
occasion, whose default is ThreadLocal
. The ThreadLocal
class has been the old-age strategy to cross knowledge round with out interfering with methodology signatures. It shops knowledge within the present thread.
Kotlin’s OpenTelemetry Extension
ThreadLocal
works completely — till you spawn different threads. On this case, you have to explicitly cross knowledge round. So-called Reactive Programming frameworks, similar to Spring WebFlux, do spawn different threads; most, if not all, present utilities to deal with the passing mechanically.
Coroutines implement Reactive Programming. Not solely do they spawn threads, however in addition they decouple coroutine from threads. A coroutine might “jump” throughout a number of threads in its lifetime. Thus, storing the OpenTelemetry context in a ThreadLocal
would not work.
But, coroutines present a devoted storage mechanism, the coroutine context. We’d like a strategy to transfer the OpenTelemetry context from the ThreadLocal
to the coroutine context and again once more. The best way exists within the opentelemetry-extension-kotlin
jar:
The one half that must be added is the place these capabilities are referred to as. Unsurprisingly, the magic occurs within the Java Agent and all different instrumentation courses. You would possibly keep in mind the TypeInstrumentation
interface on the primary diagram, which the category WithSpanInstrumentation
applied. The Java Agent caters to many alternative frameworks and libraries, e.g., Spring WebFlux, and Kotlin Coroutines. Its builders designed it so every TypeInstrumentation
concrete class focuses on the instrumentation of a selected facet of the framework or library; coroutines are not any exception.
Be aware that the code supplies a extra particular instrumentation of WithSpanInstrumentation
, which is devoted to coroutines.
It seems that the KotlinCoroutinesInstrumentationHelper
comprises the magic to repeat the context from the ThreadLocal
to the coroutine context:
package deal io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines;
import io.opentelemetry.context.Context;
import io.opentelemetry.extension.kotlin.ContextExtensionsKt;
import kotlin.coroutines.CoroutineContext;
public last class KotlinCoroutinesInstrumentationHelper {
public static CoroutineContext addOpenTelemetryContext(CoroutineContext coroutineContext) {
Context present = Context.present(); //1
Context inCoroutine = ContextExtensionsKt.getOpenTelemetryContext(coroutineContext);
if (present == inCoroutine || inCoroutine != Context.root()) {
return coroutineContext;
}
return coroutineContext.plus(ContextExtensionsKt.asContextElement(present)); //2
}
personal KotlinCoroutinesInstrumentationHelper() {}
}
- Get the OpenTelemetry context – from the
ThreadLocal
. - Add the context to the coroutine context.
And that is a wrap.
Abstract
On this publish, I’ve analyzed the workings of @WithSpan
generally and within the context of Kotlin Coroutines. The Java Agent supplies many alternative instrumenting courses, every devoted to a novel side of a framework or library. The WithSpanInstrumentation
within the io.opentelemetry.javaagent.instrumentation.extensionannotations
manages “regular” code; the one in io.opentelemetry.javaagent.instrumentation.kotlinxcoroutines
manages coroutines.
The largest problem is that OpenTelemetry shops knowledge in a ThreadLocal
by default. The coroutine library would not assure the identical thread can be used. Quite the opposite, a coroutine will seemingly bounce throughout completely different threads throughout its lifetime.
The Java Agent supplies the mechanism to deal with it. One half focuses on shifting OpenTelemetry knowledge from the ThreadLocal
to the coroutine context; the opposite supplies a devoted instrumentation to name the above code when it enters the latter.