Testing Kafka in a Go API with OpenTelemetry and Tracetest

note

Check out the source code on GitHub here.

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

Sample Go APIs with OpenTelemetry Collector, Jaeger and Tracetest

This is a simple quick start on how to configure two Go APIs to communicate via Kafka. They use OpenTelemetry instrumentation with traces, Jaeger as a trace data store, and Tracetest for enhancing your E2E and integration tests with trace-based testing.

Cloud-based Managed Tracetest

Prerequisites

Tracetest Account:

• Sign up to app.tracetest.io or follow the get started docs.
• Create an environment.
• Create an environment token.
• Have access to the environment's agent API key.

Docker: Have Docker and Docker Compose installed on your machine.

Run This Quckstart Example

The example below is provided as part of the Tracetest project. You can download and run the example by following these steps:

Clone the Tracetest project and go to the Node.js Quickstart with Manual Instrumentation:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/quick-start-go-and-kafka

Follow these instructions to run the quick start:

1. Copy the .env.template file to .env.
2. Log into the Tracetest app.
3. This example is configured to use the Jaeger Tracing Backend. Ensure the environment you will be utilizing to run this example is also configured to use the Jaeger Tracing Backend by clicking on Settings, Tracing Backend, Jaeger, endpoint: jaeger:16685, Save.
4. Fill out the token and API key details by editing your .env file. You can find these values in the Settings area for your environment.
5. Run docker compose up -d.
6. Run tests from the Tracetest Web UI by accessing the producer URL http://producer-api:8080/publish, and by using a Kafka trigger on URL kafka:9092 of the consumer.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

Project Structure

The quick start Kafka + Go project is built with Docker Compose and contains the Tracetest Agent, Jaeger, OpenTelemetry Collector and a both Go apps.

The docker-compose.yaml file in the root directory of the quick start runs the Go apps, Jaeger, OpenTelemetry Collector, and the Tracetest Agent setup.

Go Producer API

The Go API is a simple HTTP server, contained in the main.go file.

The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file. Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the producer-api in the docker-compose.yaml is:

• OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317

The server starts by running the main.go file as you can see in the Dockerfile.

The docker-compose.yaml contains one service for the Go Producer API.

  producer-api:
    image: quick-start-producer-api
    platform: linux/amd64
    build: ./producer-api
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 8080:8080
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
      - OTEL_SERVICE_NAME=producer-api
      - KAFKA_BROKER_URL=kafka:9092
      - KAFKA_TOPIC=messaging
    depends_on:
      otel-collector:
        condition: service_started
      kafka:
        condition: service_healthy

Go Consumer Worker

The Go API is a simple HTTP server, contained in the main.go file.

The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file. Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the consumer-worker in the docker-compose.yaml is:

• OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317

The server starts by running the main.go file. As you can see in the Dockerfile.

The docker-compose.yaml contains one service for the Go Producer API.

  consumer-worker:
    image: quick-start-consumer-worker
    platform: linux/amd64
    build: ./consumer-worker
    extra_hosts:
      - "host.docker.internal:host-gateway"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
      - OTEL_SERVICE_NAME=consumer-worker
      - KAFKA_BROKER_URL=kafka:9092
      - KAFKA_TOPIC=messaging
    depends_on:
      otel-collector:
        condition: service_started
      kafka:
        condition: service_healthy
      producer-api:
        condition: service_started

Running the Go Apps, Kafka, and Tracetest

Configure the .env like shown below.

TRACETEST_API_KEY="<YOUR_TRACETEST_API_KEY>"

To start the full setup, run the following command:

docker compose up -d

This example is configured to use the Jaeger Tracing Backend. Ensure the environment you're using is configured to use the Jaeger Tracing Backend by clicking on Settings, Tracing Backend, Jaeger, endpoint: jaeger:16685, Save.

Or, use the CLI and the tracetest-tracing-backend.yaml file sets the tracing backend to Jaeger, and tells Tracetest Agent to poll for traces from Jaeger.

./tracetest/tracetest-tracing-backend.yaml

---
type: DataStore
spec:
  name: Jaeger
  type: jaeger
  default: true
  jaeger:
    endpoint: jaeger:16685
    tls:
      insecure: true

tracetest configure -t <YOUR_TRACETEST_API_KEY>
tracetest apply datastore -f ./tracetest/tracetest-tracing-backend.yaml

Run Tracetest Tests

Open the URL and start creating tests! Make sure to use the http://producer-api:8080/publish URL in your test creation, because your Go app and Tracetest Agent are in the same network. And, use the Kafka trigger for the Consumer Worker.

Here are two sample tests you can get started with quickly.

👉 Testing the Consumer

👉 Testing the Producer

To trigger tests in the CLI, first install the CLI, configure it, and run a test.

From the root of the quick start directory, run:

tracetest configure -t <YOUR_API_TOKEN>
tracetest run test -f ./test-producer.yaml
tracetest run test -f ./test-consumer.yaml

Hobby Open-Source Tracetest Core

Prerequisites

You will need Docker and Docker Compose installed on your machine to run this quick start app!

Project Structure

The project is built with Docker Compose. It contains one docker-compose.yaml file that configures Tracetest, Jaeger, OpenTelemetry Collector, and both Go apps.

1. Go APIs

The docker-compose.yaml file references two Go apps, and one Kafka stream.

• Producer API: Builds the ./producer-api/Dockerfile
• Consumer Worker: Builds the ./consumer-worker/Dockerfile

The producer publishes a message to Kafka when receiving a POST request to the /publish endpoint.

The consumer accepts a message through Kafka.

2. Tracetest

The collector.config.yaml, tracetest-provision.yaml, and tracetest-config.yaml in the tracetest directory are for the setting up Tracetest and the OpenTelemetry Collector.

Docker Compose Network

All services in the docker-compose.yaml are on the same network and will be reachable by hostname from within other services. E.g. jaeger:4317 in the collector.config.yaml will map to the jaeger service, where the port 4317 is the port where Jaeger accepts traces.

Go Producer API

The Go API is a simple HTTP server, contained in the main.go file.

The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file. Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the producer-api in the docker-compose.yaml is:

• OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317

The server starts by running the main.go file.

As you can see in the Dockerfile.

FROM golang:alpine as builder
ENV GO111MODULE=on
RUN apk update && apk add --no-cache git

WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o ./bin/main .

FROM scratch
COPY --from=builder /app/bin/main .
CMD ["./main"]

And, the docker-compose.yaml contains one service for the Go Producer API.

  producer-api:
    image: quick-start-producer-api
    platform: linux/amd64
    build: ./producer-api
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 8080:8080
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
      - OTEL_SERVICE_NAME=producer-api
      - KAFKA_BROKER_URL=kafka:9092
      - KAFKA_TOPIC=messaging
    depends_on:
      otel-collector:
        condition: service_started
      kafka:
        condition: service_healthy

Go Consumer Worker

The Go API is a simple HTTP server, contained in the main.go file.

The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file. Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.

Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the producer-api in the docker-compose.yaml is:

• OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317

The server starts by running the main.go file.

As you can see in the Dockerfile.

FROM golang:alpine as builder
ENV GO111MODULE=on
RUN apk update && apk add --no-cache git

WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o ./bin/main .

FROM scratch
COPY --from=builder /app/bin/main .
CMD ["./main"]

And, the docker-compose.yaml contains one service for the Go Producer API.

  producer-api:
    image: quick-start-producer-api
    platform: linux/amd64
    build: ./producer-api
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - 8080:8080
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
      - OTEL_SERVICE_NAME=producer-api
      - KAFKA_BROKER_URL=kafka:9092
      - KAFKA_TOPIC=messaging
    depends_on:
      otel-collector:
        condition: service_started
      kafka:
        condition: service_healthy

Tracetest

The docker-compose.yaml configures four services more including Tracetest.

• Postgres - Postgres is a prerequisite for Tracetest to work. It stores trace data when running the trace-based tests.
• OpenTelemetry Collector - A vendor-agnostic implementation of how to receive, process and export telemetry data.
• Jaeger - Open source, distributed tracing platform. Monitor and troubleshoot workflows in complex distributed systems.
• Tracetest - Trace-based testing that generates end-to-end tests automatically from traces.

  tracetest:
    image: kubeshop/tracetest:${TAG:-latest}
    platform: linux/amd64
    volumes:
      - type: bind
        source: ./tracetest/tracetest-config.yaml
        target: /app/tracetest.yaml
      - type: bind
        source: ./tracetest/tracetest-provision.yaml
        target: /app/provisioning.yaml
    ports:
      - 11633:11633
    command: --provisioning-file /app/provisioning.yaml
    depends_on:
      postgres:
        condition: service_healthy
      otel-collector:
        condition: service_started
      jaeger:
        condition: service_started
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:11633"]
      interval: 1s
      timeout: 3s
      retries: 60
    environment:
      TRACETEST_DEV: ${TRACETEST_DEV}

  postgres:
    image: postgres:14
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_USER: postgres
    healthcheck:
      test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
      interval: 1s
      timeout: 5s
      retries: 60
    ports:
      - 5432:5432

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.59.0
    command:
      - "--config"
      - "/otel-local-config.yaml"
    volumes:
      - ./tracetest/collector.config.yaml:/otel-local-config.yaml
    ports:
      - 4317:4317
    depends_on:
      jaeger:
        condition: service_started

  jaeger:
    image: jaegertracing/all-in-one:latest
    restart: unless-stopped
    ports:
      - 16686:16686
      - 16685:16685
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    healthcheck:
      test: ["CMD", "wget", "--spider", "localhost:16686"]
      interval: 1s
      timeout: 3s
      retries: 60

Tracetest depends on all three services, Postgres, Jaeger, and the OpenTelemetry Collector. Both Tracetest and the OpenTelemetry Collector require config files to be loaded via a volume. The volumes are mapped from the root directory into the tracetest directory and the respective config files.

The tracetest-config.yaml file contains the basic setup of connecting Tracetest to the Postgres instance.

postgres:
  host: postgres
  user: postgres
  password: postgres
  port: 5432
  dbname: postgres
  params: sslmode=disable

The tracetest-provision.yaml file provisions the trace data store and polling to store in the Postgres database. The data store is set to OTLP, meaning the traces will be stored in Tracetest itself.

---
type: PollingProfile
spec:
  name: Default
  strategy: periodic
  default: true
  periodic:
    retryDelay: 5s
    timeout: 10m

---
type: DataStore
spec:
  name: Jaeger
  type: jaeger
  default: true
  jaeger:
    endpoint: jaeger:16685
    tls:
      insecure: true

But how are traces sent to Tracetest?

The collector.config.yaml explains that. It receives traces via the grpc or http. Then, exports them to Jaeger's OLTP endpoint jaeger:4317.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 100ms

exporters:
  logging:
    loglevel: debug
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces/1:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

Run Both the Go Apps and Tracetest

To start both Go apps and Tracetest we will run this command:

docker compose -f docker-compose.core.yaml up # add --build if the images are not built already

This will start your Tracetest instance on http://localhost:11633/.

Open the URL and start creating tests! Make sure to use the http://producer-api:8080/publish URL in your test creation, because your Go app and Tracetest are in the same network.

Here are two sample tests you can get started with quickly.

👉 Testing the Consumer

👉 Testing the Producer

Learn More

Feel free to check out our examples in GitHub and join our Slack Community for more info!