Skip to main content

Testing Kafka in a Go API with OpenTelemetry and Tracetest

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

Sample Go APIs with OpenTelemetry Collector, Jaeger and Tracetest

This is a simple quick start on how to configure two Go APIs to communicate via Kafka. They use OpenTelemetry instrumentation with traces, Jaeger as a trace data store, and Tracetest for enhancing your E2E and integration tests with trace-based testing.

Prerequisites

Tracetest Account:

Docker: Have Docker and Docker Compose installed on your machine.

Run This Quckstart Example

The example below is provided as part of the Tracetest project. You can download and run the example by following these steps:

Clone the Tracetest project and go to the Node.js Quickstart with Manual Instrumentation:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/quick-start-go-and-kafka

Follow these instructions to run the quick start:

  1. Copy the .env.template file to .env.
  2. Log into the Tracetest app.
  3. This example is configured to use the Jaeger Tracing Backend. Ensure the environment you will be utilizing to run this example is also configured to use the Jaeger Tracing Backend by clicking on Settings, Tracing Backend, Jaeger, endpoint: jaeger:16685, Save.
  4. Fill out the token and API key details by editing your .env file. You can find these values in the Settings area for your environment.
  5. Run docker compose up -d.
  6. Run tests from the Tracetest Web UI by accessing the producer URL http://producer-api:8080/publish, and by using a Kafka trigger on URL kafka:9092 of the consumer.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

Project Structure

The quick start Kafka + Go project is built with Docker Compose and contains the Tracetest Agent, Jaeger, OpenTelemetry Collector and a both Go apps.

The docker-compose.yaml file in the root directory of the quick start runs the Go apps, Jaeger, OpenTelemetry Collector, and the Tracetest Agent setup.

Go Producer API

The Go API is a simple HTTP server, contained in the main.go file.

The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file. Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the producer-api in the docker-compose.yaml is:

  • OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317

The server starts by running the main.go file as you can see in the Dockerfile.

The docker-compose.yaml contains one service for the Go Producer API.

  producer-api:
image: quick-start-producer-api
platform: linux/amd64
build: ./producer-api
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- 8080:8080
environment:
- OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
- OTEL_SERVICE_NAME=producer-api
- KAFKA_BROKER_URL=kafka:9092
- KAFKA_TOPIC=messaging
depends_on:
otel-collector:
condition: service_started
kafka:
condition: service_healthy

Go Consumer Worker

The Go API is a simple HTTP server, contained in the main.go file.

The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file. Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the consumer-worker in the docker-compose.yaml is:

  • OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317

The server starts by running the main.go file. As you can see in the Dockerfile.

The docker-compose.yaml contains one service for the Go Producer API.

  consumer-worker:
image: quick-start-consumer-worker
platform: linux/amd64
build: ./consumer-worker
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
- OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
- OTEL_SERVICE_NAME=consumer-worker
- KAFKA_BROKER_URL=kafka:9092
- KAFKA_TOPIC=messaging
depends_on:
otel-collector:
condition: service_started
kafka:
condition: service_healthy
producer-api:
condition: service_started

Running the Go Apps, Kafka, and Tracetest

Configure the .env like shown below.

TRACETEST_API_KEY="<YOUR_TRACETEST_API_KEY>"

To start the full setup, run the following command:

docker compose up -d

This example is configured to use the Jaeger Tracing Backend. Ensure the environment you're using is configured to use the Jaeger Tracing Backend by clicking on Settings, Tracing Backend, Jaeger, endpoint: jaeger:16685, Save.

Or, use the CLI and the tracetest-tracing-backend.yaml file sets the tracing backend to Jaeger, and tells Tracetest Agent to poll for traces from Jaeger.

./tracetest/tracetest-tracing-backend.yaml
---
type: DataStore
spec:
name: Jaeger
type: jaeger
default: true
jaeger:
endpoint: jaeger:16685
tls:
insecure: true
tracetest configure -t <YOUR_TRACETEST_API_KEY>
tracetest apply datastore -f ./tracetest/tracetest-tracing-backend.yaml

Run Tracetest Tests

Open the URL and start creating tests! Make sure to use the http://producer-api:8080/publish URL in your test creation, because your Go app and Tracetest Agent are in the same network. And, use the Kafka trigger for the Consumer Worker.

Here are two sample tests you can get started with quickly.

👉 Testing the Consumer

👉 Testing the Producer

To trigger tests in the CLI, first install the CLI, configure it, and run a test.

From the root of the quick start directory, run:

tracetest configure -t <YOUR_API_TOKEN>
tracetest run test -f ./test-producer.yaml
tracetest run test -f ./test-consumer.yaml

Learn More

Feel free to check out our examples in GitHub and join our Slack Community for more info!