Skip to main content

Pokeshop and Grafana Tempo

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

Grafana Tempo is an open-source, high-scale distributed tracing data store. It utilizes object storage to minimize costs, allowing you to store more information at a lower cost. It is deeply integrated with Grafana, Prometheus, and Loki. Grafana Tempo is also available as a paid hosted service at Grafana Cloud.

Pokeshop API is a testing ground, the team at Tracetest has implemented a sample instrumented API around the PokeAPI.

Pokeshop API with Grafana Tempo, OpenTelemetry and Tracetest

This is a simple quick start guide on how to configure a fully instrumented API to be used with Tracetest for enhancing your E2E and integration tests with trace-based testing. The infrastructure will use Grafana Tempo as the trace data store and the Pokeshop API to generate the telemetry data.

Prerequisites

You will need Docker and Docker Compose installed on your machine to run this quick start app!

Project Structure

The project is built with Docker Compose.

1. Tracetest

The collector.config.yaml file, tracetest.provision.yaml, and tracetest.config.yaml in the root directory are for the configuring Tracetest and it's OpenTelemetry Collector.

2. Grafana Tempo

The grafana.config.yaml and tempo.config.yaml in the root directory contains all files required to configure Grafana and Tempo.

Docker Compose Network

All services in the docker-compose.yaml are on the same network and will be reachable by hostname from within other services. E.g. tracetest:4317 in the tracetest/collector.config.yaml will map to the tracetest service, where the port 4317 is the port where Tracetest accepts telemetry data.

Pokeshop API

The Pokeshop API is a fully instrumented REST API that makes use of different services to mimic a real life scenario.

It is instrumented using the OpenTelemetry standards for Node.js, sending the data to the ADOT collector that will be pushing the telemetry information to both the AWS X-Ray service.

This is a fragment from the main tracing file from the Pokeshop API repo.

import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import * as opentelemetry from '@opentelemetry/api';
import { api, NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
import * as dotenv from 'dotenv';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { IORedisInstrumentation } from '@opentelemetry/instrumentation-ioredis';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';
import { AmqplibInstrumentation } from '@opentelemetry/instrumentation-amqplib';
import { SpanStatusCode } from '@opentelemetry/api';
import { B3Propagator } from '@opentelemetry/propagator-b3';

dotenv.config();
api.propagation.setGlobalPropagator(new B3Propagator());

const { COLLECTOR_ENDPOINT = '', SERVICE_NAME = 'pokeshop' } = process.env;

let globalTracer: opentelemetry.Tracer | null = null;

async function createTracer(): Promise<opentelemetry.Tracer> {
const collectorExporter = new OTLPTraceExporter({
url: COLLECTOR_ENDPOINT,
});

const spanProcessor = new BatchSpanProcessor(collectorExporter);
const sdk = new NodeSDK({
traceExporter: collectorExporter,
// @ts-ignore
instrumentations: [new IORedisInstrumentation(), new PgInstrumentation(), new AmqplibInstrumentation()],
});

sdk.configureTracerProvider({}, spanProcessor);
sdk.addResource(
new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: SERVICE_NAME,
})
);

await sdk.start();
process.on('SIGTERM', () => {
sdk
.shutdown()
.then(
() => console.log('SDK shut down successfully'),
err => console.log('Error shutting down SDK', err)
)
.finally(() => process.exit(0));
});

const tracer = opentelemetry.trace.getTracer(SERVICE_NAME);

globalTracer = tracer;

return globalTracer;
}

The docker-compose.yaml file includes the definitions for all of the required services by the Pokeshop API, which includes:

  • Postgres - To save Pokemon information.
  • Redis - For in memory strage.
  • RabbitMQ - For async processing use cases.
  • API - The Pokeshop API main container.
  • Worker - The Pokeshop worker instance.
version: "3"

#...

services:
#...

# Demo
postgres:
image: postgres:14
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
healthcheck:
test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
interval: 1s
timeout: 5s
retries: 60
ports:
- 5432:5432

demo-cache:
image: redis:6
restart: unless-stopped
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 60

demo-queue:
image: rabbitmq:3.8-management
restart: unless-stopped
healthcheck:
test: rabbitmq-diagnostics -q check_running
interval: 1s
timeout: 5s
retries: 60

demo-api:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: demo-cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: demo-queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://otel-collector:4317
NPM_RUN_COMMAND: api
ports:
- "8081:8081"
healthcheck:
test: ["CMD", "wget", "--spider", "localhost:8081"]
interval: 1s
timeout: 3s
retries: 60
depends_on:
postgres:
condition: service_healthy
demo-cache:
condition: service_healthy
demo-queue:
condition: service_healthy

demo-worker:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: demo-cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: demo-queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://otel-collector:4317
NPM_RUN_COMMAND: worker
depends_on:
postgres:
condition: service_healthy
demo-cache:
condition: service_healthy
demo-queue:
condition: service_healthy

demo-rpc:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: demo-cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: demo-queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://otel-collector:4317
NPM_RUN_COMMAND: rpc
ports:
- 8082:8082
healthcheck:
test: ["CMD", "lsof", "-i", "8082"]
interval: 1s
timeout: 3s
retries: 60
depends_on:
postgres:
condition: service_healthy
demo-cache:
condition: service_healthy
demo-queue:
condition: service_healthy
# Demo End

Tracetest

The docker-compose.yaml includes two services related to Tracetest. The Tracetest instance also connects to the postgres service.

  • Postgres - Postgres is a prerequisite for Tracetest to work. It stores trace data when running the trace-based tests.
  • OpenTelemetry Collector) - Vendor-agnostic way to receive, process and export telemetry data.
  • Tracetest - Trace-based testing that generates end-to-end tests automatically from traces.
version: "3"

# ...
services:

# ...

# Tracetest
tracetest:
image: kubeshop/tracetest:${TAG:-latest}
platform: linux/amd64
volumes:
- type: bind
source: ./tracetest/tracetest-config.yaml
target: /app/tracetest.yaml
- type: bind
source: ./tracetest/tracetest-provision.yaml
target: /app/provision.yaml
command: --provisioning-file /app/provision.yaml
ports:
- 11633:11633
extra_hosts:
- "host.docker.internal:host-gateway"
depends_on:
postgres:
condition: service_healthy
otel-collector:
condition: service_started
healthcheck:
test: [ "CMD", "wget", "--spider", "localhost:11633" ]
interval: 1s
timeout: 3s
retries: 60
environment:
TRACETEST_DEV: ${TRACETEST_DEV}

# Tracetest End

Tracetest depends on Postgres and the OpenTelemetry Collector. Tracetest requires config files to be loaded via a volume. The volumes are mapped from the tracetest directory into the root directory of the Tracetest container instance and the respective config files.

The collector.config.yaml file contains the OpenTelemetry Collector configuration that enables routing traces from the Pokeshop API to Tempo.

# collector.config.yaml

receivers:
otlp:
protocols:
grpc:
http:

processors:
batch:
timeout: 100ms

exporters:
logging:
loglevel: debug
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/tempo]

The tracetest.config.yaml file contains the basic setup of connecting Tracetest to the Postgres instance. It also enables forwarding Tracetest's internal telemetry to Grafana Tempo as well, with the telemetry and server config.

# tracetest-config.yaml

postgres:
host: postgres
user: postgres
password: postgres
port: 5432
dbname: postgres
params: sslmode=disable

telemetry:
exporters:
collector:
serviceName: tracetest
sampling: 100
exporter:
type: collector
collector:
endpoint: otel-collector:4317

server:
telemetry:
exporter: collector
applicationExporter: collector

The tracetest.provision.yaml file defines the trace data store, set to Grafana Tempo, meaning the traces will be forwarded via the OpenTelemetry Collector to Tempo where they are stored.

# tracetest-provision.yaml

---
type: DataStore
spec:
name: Tempo
type: tempo
tempo:
type: grpc
grpc:
endpoint: tempo:9095
tls:
insecure: true

---
type: Demo
spec:
type: pokeshop
enabled: true
name: pokeshop
opentelemetryStore: {}
pokeshop:
httpEndpoint: http://demo-api:8081
grpcEndpoint: demo-rpc:8082

How do traces reach Grafana Tempo?

The Pokeshop API code uses the native Node.js OpenTelemetry modules which send information to the OpenTelemetry Collector to be processed and then sent to Grafana Tempo.

Grafana Tempo

The docker-compose.yaml includes 3 services related to Grafana Tempo.

  • OpenTelemetry Collector
  • Grafana Tempo
  • Grafana (Dashboards)
version: "3"

# ...
services:

# ...
# Grafana
otel-collector:
image: otel/opentelemetry-collector-contrib:0.59.0
command:
- "--config"
- "/otel-local-config.yaml"
volumes:
- ./collector.config.yaml:/otel-local-config.yaml
depends_on:
- tempo
tempo:
image: grafana/tempo:latest
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./tempo.config.yaml:/etc/tempo.yaml
- ./tempo-data:/tmp/tempo
ports:
- "3200:3200" # tempo http
- "9095:9095" # tempo grpc
- "4317" # otlp grpc
- "4318" # otlp http
grafana:
image: grafana/grafana:9.4.3
volumes:
- ./grafana.config.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
- GF_FEATURE_TOGGLES_ENABLE=traceqlEditor
ports:
- "3000:3000"
# Grafana End

Run the Pokeshop API, Grafana and Tracetest

To start all the services, run this command:

docker-compose up -d

This will start your Tracetest instance on http://localhost:11633/. Open it and start creating tests!

Make sure to use the http://demo-api:8081/ URL in your test creation, because your Pokeshop API and Tracetest are in the same network.

Run Tracetest Tests with the Tracetest CLI

First, install the CLI. Then, configure the CLI:

tracetest configure --server-url http://localhost:11633

Once configured, you can run a test against the Tracetest instance via the terminal.

Check out the tests/test.yaml file.

# tests/test.yaml

type: Test
spec:
id: ZVJwkpC4g
name: Pokeshop - Import
description: Import a Pokemon
trigger:
type: http
httpRequest:
method: POST
url: http://demo-api:8081/pokemon/import
body: '{"id":6}'
headers:
- key: Content-Type
value: application/json
specs:
- selector: span[tracetest.span.type="http"]
name: "All HTTP Spans: Status code is 200"
assertions:
- attr:http.status_code = 200
- selector: span[tracetest.span.type="general" name="import pokemon"]
name: Validate that this span always exists after the message queue
assertions:
- attr:tracetest.selected_spans.count = 1
- attr:span.events not-contains "exception"
- selector: span[tracetest.span.type="database" name="get pokemon_6" db.system="redis" db.operation="get" db.redis.database_index="0"]
name: Validate that Redis is using Charizard.
assertions:
- attr:db.payload = '{"key":"pokemon_6"}'
- selector: span[tracetest.span.type="database" name="create postgres.pokemon" db.system="postgres" db.name="postgres" db.user="postgres" db.operation="create" db.sql.table="pokemon"]
name: Validate that the Postgres has Charizard.
assertions:
- attr:db.result contains "charizard"

This file defines a test the same way you would through the Web UI.

To run the test, run this command in the terminal:

tracetest run test -f ./tests/test.yaml
Output:
✔ Pokeshop - Import (http://localhost:11633/test/ZVJwkpC4g/run/1/test) - trace id: 4eff1e124f67cf7a802b3c4fc51c19d4
✔ All HTTP Spans: Status code is 200
✔ Validate that this span always exists after the message queue
✔ Validate that Redis is using Charizard.
✔ Validate that the Postgres has Charizard.

tracetest web ui overview

View Trace Spans Over Time in Grafana

To access a historical overview of all the trace spans the Pokeshop App generates, jump over to Grafana on http://localhost:3000.

With Grafana and Tracetest you can run trace-based tests and automate running E2E and integration tests against real trace data. And, use Grafana to get a historical overview of all traces your distributed application generates.

Learn More

Feel free to check out our examples in GitHub and join our Slack Community for more info!