Testing Kafka in a Go API with OpenTelemetry and Tracetest
Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.
Sample Go APIs with OpenTelemetry Collector, Jaeger and Tracetest
This is a simple quick start on how to configure two Go APIs to communicate via Kafka. They use OpenTelemetry instrumentation with traces, Jaeger as a trace data store, and Tracetest for enhancing your E2E and integration tests with trace-based testing.
Prerequisites
You will need Docker and Docker Compose installed on your machine to run this quick start app!
Project Structure
The project is built with Docker Compose. It contains one docker-compose.yaml file that configures Tracetest, Jaeger, OpenTelemetry Collector, and both Go apps.
1. Go APIs
The docker-compose.yaml file references two Go apps, and one Kafka stream.
- Producer API: Builds the
./producer-api/Dockerfile - Consumer Worker: Builds the
./consumer-worker/Dockerfile
The producer publishes a message to Kafka when receiving a POST request to the /publish endpoint.
The consumer accepts a message through Kafka.
2. Tracetest
The collector.config.yaml, tracetest-provision.yaml, and tracetest-config.yaml in the tracetest directory are for the setting up Tracetest and the OpenTelemetry Collector.
Docker Compose Network
All services in the docker-compose.yaml are on the same network and will be reachable by hostname from within other services. E.g. jaeger:4317 in the collector.config.yaml will map to the jaeger service, where the port 4317 is the port where Jaeger accepts traces.
Go Producer API
The Go API is a simple HTTP server, contained in the main.go file.
The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file.
Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.
Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the producer-api in the docker-compose.yaml is:
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
The server starts by running the main.go file.
As you can see in the Dockerfile.
FROM golang:alpine as builder
ENV GO111MODULE=on
RUN apk update && apk add --no-cache git
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o ./bin/main .
FROM scratch
COPY --from=builder /app/bin/main .
CMD ["./main"]
And, the docker-compose.yaml contains one service for the Go Producer API.
producer-api:
image: quick-start-producer-api
platform: linux/amd64
build: ./producer-api
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- 8080:8080
environment:
- OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
- OTEL_SERVICE_NAME=producer-api
- KAFKA_BROKER_URL=kafka:9092
- KAFKA_TOPIC=messaging
depends_on:
otel-collector:
condition: service_started
kafka:
condition: service_healthy
Go Consumer Worker
The Go API is a simple HTTP server, contained in the main.go file.
The OpenTelemetry tracing is contained in the ./telemetry/telemetry.go file.
Traces will be sent to the OpenTelemetry Collector, and forwarded to Jaeger.
Traces will be sent to either the grpc endpoint. The hostname and port as seen in the env section of the producer-api in the docker-compose.yaml is:
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
The server starts by running the main.go file.
As you can see in the Dockerfile.
FROM golang:alpine as builder
ENV GO111MODULE=on
RUN apk update && apk add --no-cache git
WORKDIR /app
COPY go.mod ./
COPY go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o ./bin/main .
FROM scratch
COPY --from=builder /app/bin/main .
CMD ["./main"]
And, the docker-compose.yaml contains one service for the Go Producer API.
producer-api:
image: quick-start-producer-api
platform: linux/amd64
build: ./producer-api
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- 8080:8080
environment:
- OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
- OTEL_SERVICE_NAME=producer-api
- KAFKA_BROKER_URL=kafka:9092
- KAFKA_TOPIC=messaging
depends_on:
otel-collector:
condition: service_started
kafka:
condition: service_healthy
Tracetest
The docker-compose.yaml configures four services more including Tracetest.
- Postgres - Postgres is a prerequisite for Tracetest to work. It stores trace data when running the trace-based tests.
- OpenTelemetry Collector - A vendor-agnostic implementation of how to receive, process and export telemetry data.
- Jaeger - Open source, distributed tracing platform. Monitor and troubleshoot workflows in complex distributed systems.
- Tracetest - Trace-based testing that generates end-to-end tests automatically from traces.
tracetest:
image: kubeshop/tracetest:${TAG:-latest}
platform: linux/amd64
volumes:
- type: bind
source: ./tracetest/tracetest-config.yaml
target: /app/tracetest.yaml
- type: bind
source: ./tracetest/tracetest-provision.yaml
target: /app/provisioning.yaml
ports:
- 11633:11633
command: --provisioning-file /app/provisioning.yaml
depends_on:
postgres:
condition: service_healthy
otel-collector:
condition: service_started
jaeger:
condition: service_started
healthcheck:
test: ["CMD", "wget", "--spider", "localhost:11633"]
interval: 1s
timeout: 3s
retries: 60
environment:
TRACETEST_DEV: ${TRACETEST_DEV}
postgres:
image: postgres:14
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
healthcheck:
test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
interval: 1s
timeout: 5s
retries: 60
ports:
- 5432:5432
otel-collector:
image: otel/opentelemetry-collector-contrib:0.59.0
command:
- "--config"
- "/otel-local-config.yaml"
volumes:
- ./tracetest/collector.config.yaml:/otel-local-config.yaml
ports:
- 4317:4317
depends_on:
jaeger:
condition: service_started
jaeger:
image: jaegertracing/all-in-one:latest
restart: unless-stopped
ports:
- 16686:16686
- 16685:16685
environment:
- COLLECTOR_OTLP_ENABLED=true
healthcheck:
test: ["CMD", "wget", "--spider", "localhost:16686"]
interval: 1s
timeout: 3s
retries: 60
Tracetest depends on all three services, Postgres, Jaeger, and the OpenTelemetry Collector. Both Tracetest and the OpenTelemetry Collector require config files to be loaded via a volume. The volumes are mapped from the root directory into the tracetest directory and the respective config files.
The tracetest-config.yaml file contains the basic setup of connecting Tracetest to the Postgres instance.
postgres:
host: postgres
user: postgres
password: postgres
port: 5432
dbname: postgres
params: sslmode=disable
The tracetest-provision.yaml file provisions the trace data store and polling to store in the Postgres database. The data store is set to OTLP, meaning the traces will be stored in Tracetest itself.
---
type: PollingProfile
spec:
name: Default
strategy: periodic
default: true
periodic:
retryDelay: 5s
timeout: 10m
---
type: DataStore
spec:
name: Jaeger
type: jaeger
default: true
jaeger:
endpoint: jaeger:16685
tls:
insecure: true
But how are traces sent to Tracetest?
The collector.config.yaml explains that. It receives traces via the grpc or http. Then, exports them to Jaeger's OLTP endpoint jaeger:4317.
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
timeout: 100ms
exporters:
logging:
loglevel: debug
otlp/jaeger:
endpoint: jaeger:4317
tls:
insecure: true
service:
pipelines:
traces/1:
receivers: [otlp]
processors: [batch]
exporters: [otlp/jaeger]
Run Both the Go Apps and Tracetest
To start both Go apps and Tracetest we will run this command:
docker-compose up # add --build if the images are not built already
This will start your Tracetest instance on http://localhost:11633/.
Open the URL and start creating tests! Make sure to use the http://producer-api:8080/publish URL in your test creation, because your Go app and Tracetest are in the same network.
Here are two sample tests you can get started with quickly.
Learn More
Feel free to check out our examples in GitHub and join our Slack Community for more info!