Are you able to get began with cloud-native observability and telemetry pipelines?
This text is a part of a sequence exploring a workshop guiding you thru the open supply mission Fluent Bit, what it’s, a fundamental set up, and organising the primary telemetry pipeline mission. Discover ways to handle your cloud-native knowledge from supply to vacation spot utilizing the telemetry pipeline phases masking assortment, aggregation, transformation, and forwarding from any supply to any vacation spot.
Within the earlier article on this sequence, we appeared on the first a part of integrating Fluent Bit with OpenTelemetry to be used circumstances the place this connectivity in your telemetry knowledge infrastructure is crucial. On this subsequent article, we and finalize our integration of Fluent Bit with OpenTelemetry to be used circumstances the place this connectivity in your telemetry knowledge infrastructure is crucial.
You will discover extra particulars within the accompanying workshop lab.
There are lots of the reason why a company would possibly wish to cross their telemetry pipeline output onwards to OpenTelemetry (OTel), particularly via an OpenTelemetry Collector. To facilitate this integration, Fluent Bit from the discharge of model 3.1 added help for changing its telemetry knowledge into the OTel format.
On this article, we’ll finalize our resolution for integrating Fluent Bit with OpenTelemetry. Word that we decide up the place we left off beforehand, so when you have not but learn the earlier article, you must return and accomplish that.
Pushing to the Collector
When sending telemetry knowledge within the OTel Envelope, we’re utilizing the OpenTelemetry Protocol (OTLP), which is the usual to ingest right into a collector. The following step shall be for us to configure an OTel Collector to obtain telemetry knowledge and level our Fluent Bit situations at that collector as depicted on this diagram:
There are three issues to configure when processing telemetry knowledge via an OTel Collector: a receiver, processor, and exporter as proven beneath. Let’s get began with our personal OTel Collector configuration:
The configuration of our collector receivers part is the place we outline a supply vacation spot that Fluent Bit can push its telemetry (logs) to. Create a brand new configuration file workshop-otel.yaml and add the next to offer an OLTP endpoint at localhost:4317 as proven:
# This file is our workshop OpenTelemetry configuration.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4317
Now develop the configuration file workshop-otel.yaml with an exporters part to outline the place our telemetry (logs) shall be despatched after processing. On this case, we wish to present output to the console as follows:
# This file is our workshop OpenTelemetry configuration.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4317
exporters:
logging:
loglevel: information
Lastly, we added a service part to finish our collector configuration. Right here we’re defining the extent of log reporting (information) and making a collector pipeline for our logs. We assign particular receivers and exporters in order that we’re receiving via the OTLP protocol (utilizing our Fluent Bit OTel Envelope) and exporting to the console solely:
# This file is our workshop OpenTelemetry configuration.
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4317
exporters:
logging:
loglevel: information
service:
telemetry:
logs:
stage: debug
pipelines:
logs:
receivers: [otlp]
exporters: [logging]
To confirm the OTel Collector configuration utilizing a container set up, we first construct a brand new container that makes use of our new configuration. Create a brand new file Buildfile-otel and set it up as follows:
FROM otel/opentelemetry-collector-contrib:0.105.0 COPY workshop-otel.yaml /and many others/otel-collector-contrib/workshop-otel.yaml CMD [ "--config", "/etc/otel-collector-contrib/workshop-otel.yaml"]
Now we’ll construct a brand new container picture, naming it with a model tag as follows utilizing the Buildfile-otel and assuming you might be in the identical listing:
$ podman construct -t workshop-otel:v0.105.0 -f Buildfile-otel STEP 1/3: FROM otel/opentelemetry-collector-contrib:0.105.0 Resolving "otel/opentelemetry-collector-contrib" utilizing unqualified-search registries (/and many others/containers/registries.conf.d/999-podman-machine.conf) Attempting to drag docker.io/otel/opentelemetry-collector-contrib:0.105.0... Getting picture supply signatures Copying blob sha256:50658aaad663b91752461d59f121b17d0fd7f859d9575edfa60b077771286a63 Copying blob sha256:3a40a3a6b6e2d53bc3df71d348344a451c6adc16b61742cebcba70352539a859 Copying blob sha256:8339825c9e32785bf25d2686ad61ed0bb338f4ca2d8e7d04dc17abe5fe1768ec Copying config sha256:8920124fc5c615bbeeed5cdd2c55697b9430cd2100bea8e73aa7a36c8f645a90 Writing manifest to picture vacation spot STEP 2/3: COPY workshop-otel.yaml /and many others/otel-collector-contrib/workshop-otel.yaml --> 845307d97fa9 STEP 3/3: CMD [ "--config", "/etc/otel-collector-contrib/workshop-otel.yaml"] COMMIT workshop-otel:v0.105.0 --> da268a9ffa0d Efficiently tagged localhost/workshop-otel:v0.105.0 da268a9ffa0d27cf17d96af550d8d755973cbbb90db8866099fcc07b671d61e7
Run the collector utilizing this container command, noting port 4317 has been mapped to obtain telemetry knowledge and it has a reputation otelcol:
$ podman run --rm -p 4317:4317 --name otelcol workshop-otel:v0.105.0
The console output ought to look one thing like this (scroll to the precise to view longer strains). Word the highlighted line reveals that the port is open and receiving on localhost:4317. This runs till exiting with CTRL+C:
2024-08-02T07:42:45.349Z information service@v0.105.0/service.go:116 Establishing personal telemetry...
2024-08-02T07:42:45.349Z information service@v0.105.0/service.go:119 OpenCensus bridge is disabled for Collector telemetry and shall be eliminated in a future model, use --feature-gates=-service.disableOpenCensusBridge to re-enable
2024-08-02T07:42:45.349Z information service@v0.105.0/telemetry.go:96 Serving metrics {"address": ":8888", "metrics level": "Normal"}
2024-08-02T07:42:45.350Z information exporter@v0.105.0/exporter.go:280 Deprecated element. Shall be eliminated in future releases. {"kind": "exporter", "data_type": "logs", "name": "logging"}
2024-08-02T07:42:45.350Z warn widespread/manufacturing facility.go:68 'loglevel' possibility is deprecated in favor of 'verbosity'. Set 'verbosity' to equal worth to protect habits. {"kind": "exporter", "data_type": "logs", "name": "logging", "loglevel": "info", "equivalent verbosity level": "Normal"}
2024-08-02T07:42:45.350Z debug receiver@v0.105.0/receiver.go:313 Beta element. Could change sooner or later. {"kind": "receiver", "name": "otlp", "data_type": "logs"}
2024-08-02T07:42:45.350Z information service@v0.105.0/service.go:198 Beginning otelcol-contrib... {"Version": "0.105.0", "NumCPU": 4}
2024-08-02T07:42:45.350Z information extensions/extensions.go:34 Beginning extensions...
2024-08-02T07:42:45.350Z information otlpreceiver@v0.105.0/otlp.go:152 Beginning HTTP server {"kind": "receiver", "name": "otlp", "data_type": "logs", "endpoint": "0.0.0.0:4317"}
2024-08-02T07:42:45.350Z information service@v0.105.0/service.go:224 The whole lot is prepared. Start working and processing knowledge.
2024-08-02T07:42:45.351Z information localhostgate/featuregate.go:63 The default endpoints for all servers in parts have modified to make use of localhost as a substitute of 0.0.0.0. Disable the function gate to briefly revert to the earlier default. {"feature gate ID": "component.UseLocalHostAsDefaultHost"}
...
Now we head again to our Fluent Bit pipeline so as to add an output vacation spot to push our telemetry knowledge from our pipeline to the collector endpoint. Add a second entry known as opentelemetry to the exiting outputs part in our configuration file workshop-fb.yaml as proven (and you’ll want to save the file when executed):
...
# This entry directs all tags (it matches any we encounter)
# to print to plain output, which is our console.
outputs:
- title: stdout
match: '*'
format: json_lines
# this entry is for pushing logs to an OTel collector.
- title: opentelemetry
match: '*'
host: ${OTEL_HOST}
port: 4317
logs_body_key_attributes: true # permits for OTel attribute modification.
Word that the host attribute has been set to a variable which shall be crammed with a command line worth after we run the container. The command will present a lookup of the OTel collector container IP handle in order that Fluent Bit can discover it.
Now we’ll construct a brand new container picture, naming it with a model tag, as follows utilizing the Buildfile-fb and assuming you might be in the identical listing:
$ podman construct -t workshop-fb:v14 -f Buildfile-fb STEP 1/3: FROM cr.fluentbit.io/fluent/fluent-bit:3.1.4 STEP 2/3: COPY ./workshop-fb.yaml /fluent-bit/and many others/workshop-fb.yaml --> b4ed3356a842 STEP 3/3: CMD [ "fluent-bit", "-c", "/fluent-bit/etc/workshop-fb.yaml"] COMMIT workshop-fb:v4 --> bcd69f8a85a0 Efficiently tagged localhost/workshop-fb:v14 bcd69f8a85a024ac39604013bdf847131ddb06b1827aae91812b57479009e79a
Run this pipeline utilizing this container command, which features a seek for the interior container IP handle to cross as a variable to our configuration utilizing the variable OTEL_HOST:
$ podman run --rm -e OTEL_HOST=$(podman examine -f "{{.NetworkSettings.IPAddress}}"otelcol) workshop-fb:v14
The console output from the Fluent Bit pipeline appears to be like one thing like this, IF and ONLY IF you will have the OTel collector container working on the identical time! Word that each time a log occasion is ingested, it is processed into an OTel Envelope with three strains within the console, after which despatched to the collector on port 4317. Let’s return to the OTel collector console and test ingestion:
...
[2024/08/02 14:45:53] [ info] [input:dummy:dummy.0] initializing
[2024/08/02 14:45:53] [ info] [input:dummy:dummy.0] storage_strategy='reminiscence' (reminiscence solely)
[2024/08/02 14:45:53] [ info] [output:stdout:stdout.0] employee #0 began
[2024/08/02 14:45:53] [ info] [sp] stream processor began
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722609954.095659,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":4294967294.0}
[2024/08/02 14:45:55] [ info] [output:opentelemetry:opentelemetry.1] 10.88.0.3:4317, HTTP standing=200
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722609955.096505,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":4294967294.0}
[2024/08/02 14:45:56] [ info] [output:opentelemetry:opentelemetry.1] 10.88.0.3:4317, HTTP standing=200
{"date":4294967295.0,"resource":{},"scope":{}}
...
The console output from the OTel collector appears to be like one thing like this, with one line for every log occasion despatched from our Fluent Bit-based pipeline. This implies it is working high-quality! What we do not see is the log occasion particulars, so let’s add a brand new exporter to ship our log knowledge to a file.
...
2024-08-02T14:45:55.110Z information LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1}
2024-08-02T14:45:56.100Z information LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1}
2024-08-02T14:45:57.099Z information LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1}
...
Including File Output
Cease the OTel collector container, open the collector configuration file workshop-otel.yaml, and add the next to write down log output to a file in /tmp/output.json. We additionally want so as to add the brand new exporter within the pipeline’s part for logs as proven beneath:
...
exporters:
file:
path: /tmp/output.json
logging:
loglevel: information
service:
telemetry:
logs:
stage: debug
pipelines:
logs:
receivers: [otlp]
exporters: [file, logging]
Now we’ll construct a brand new OTel collector container picture, naming it with a model tag as follows utilizing the Buildfile-otel and assuming you might be in the identical listing:
$ podman construct -t workshop-otel:v0.105.0 -f Buildfile-otel STEP 1/3: FROM otel/opentelemetry-collector-contrib:0.105.0 Resolving "otel/opentelemetry-collector-contrib" utilizing unqualified-search registries (/and many others/containers/registries.conf.d/999-podman-machine.conf) Attempting to drag docker.io/otel/opentelemetry-collector-contrib:0.105.0... Getting picture supply signatures Copying blob sha256:50658aaad663b91752461d59f121b17d0fd7f859d9575edfa60b077771286a63 Copying blob sha256:3a40a3a6b6e2d53bc3df71d348344a451c6adc16b61742cebcba70352539a859 Copying blob sha256:8339825c9e32785bf25d2686ad61ed0bb338f4ca2d8e7d04dc17abe5fe1768ec Copying config sha256:8920124fc5c615bbeeed5cdd2c55697b9430cd2100bea8e73aa7a36c8f645a90 Writing manifest to picture vacation spot STEP 2/3: COPY workshop-otel.yaml /and many others/otel-collector-contrib/workshop-otel.yaml --> 845307d97fa9 STEP 3/3: CMD [ "--config", "/etc/otel-collector-contrib/workshop-otel.yaml"] COMMIT workshop-otel:v0.105.0 --> da268a9ffa0d Efficiently tagged localhost/workshop-otel:v0.105.0 da268a9ffa0d27cf17d96af550d8d755973cbbb90db8866099fcc07b671d61e7
Run the collector utilizing this container command, noting port 4317 has been mapped to obtain telemetry knowledge, it has the title otelcol, and we at the moment are mapping the interior container /tmp listing to our native listing for the output.json file:
$ podman run --rm -p 4317:4317 --name otelcol -v ./:/tmp workshop-otel:v0.105.0
Whereas the collector is ready on telemetry knowledge, we will (re)run the pipeline to begin ingesting, processing, and pushing log occasions to our collector:
$ podman run --rm -e OTEL_HOST=$(podman examine -f "{{.NetworkSettings.IPAddress}}" otelcol) workshop-fb:v14
Confirm that the pipeline with Fluent Bit is ingesting, processing, and forwarding logs (console):
# Pipeline with Fluent Bit ingest, processing and output.
...
{"date":4294967295.0,"resource":{},"scope":{}}
{"date":1722611305.09833,"service":"backend","log_entry":"Generating a 200 success code."}
{"date":4294967294.0}
[2024/08/02 15:08:26] [ info] [output:opentelemetry:opentelemetry.1] 10.88.0.10:4317, HTTP standing=200
...
Test that the collector is receiving these logs (console):
# OTel collector ingesting occasions.
...
2024-08-02T14:45:55.110Z information LogsExporter {"kind": "exporter", "data_type": "logs", "name": "logging", "resource logs": 1, "log records": 1}
...
Lastly, confirm that the collector is exporting to output.json file as proven beneath:
# Viewing file output.json
$ tail -f ./output.json
{"resourceLogs":[{"resource":{},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1722611307093849438",
"body":{"kvlistValue":{"values":[{"key":"service","value":{"stringValue":"backend"}},{"key":"log_entry",
"value":{"stringValue":"Generating a 200 success code."}}]}},"traceId":"","spanId":""}]}]}]}
...
A extra visually pleasing view of the JSON file may be achieved as follows with every log occasion offered in structured JSON:
# Viewing structured JSON file output.json utilizing jq.
$ tail -f ./output.json | jq
{
"resourceLogs": [
{
"resource": {},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"timeUnixNano": "1722611323093561678",
"body": {
"kvlistValue": {
"values": [
{
"key": "service",
"value": { "stringValue": "backend"}
},
{
"key": "log_entry",
"value": {
"stringValue": "Generating a 200 success code."
}
}
]
}
},
"traceId": "", "spanId": ""
...
This completes our use case for this text, you’ll want to discover this hands-on expertise with the accompanying workshop lab, linked earlier within the article.
What’s Subsequent?
This text walked us via the second a part of integrating Fluent Bit with OpenTelemetry to be used circumstances the place this connectivity in your telemetry knowledge infrastructure is crucial.
Keep tuned for extra hands-on materials that can assist you together with your cloud-native observability journey.