Skip to main content

Pokeshop API and Azure Application Insights with OpenTelemetry Collector

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

Azure Application Insights is an extension of Azure Monitor and provides application performance monitoring (APM) features. APM tools are useful to monitor applications from development, through test, and into production in the following ways:

  • Proactively understand how an application is performing.
  • Reactively review application execution data to determine the cause of an incident.

OpenTelemetry Collector Contrib - The official OpenTelemetry Distribution for packages outside of the core collector.

Pokeshop API As a testing ground, the team at Tracetest has implemented a sample instrumented API around the PokeAPI.

Pokeshop API with Azure Application Insights, OpenTelemetry Collector and Tracetest

This is a simple quick start guide on how to configure a fully instrumented API to be used with Tracetest for enhancing your E2E and integration tests with trace-based testing. The infrastructure will use Azure App Insights as the trace data store, the OpenTelemetry Collector as a middleware and the Pokeshop API to generate the telemetry data.

Prerequisites

You will need Docker and Docker Compose installed on your machine to run this quick start app!

You will also need an App Insights API Access Key and the resource ARM ID for your App Insights instance.

Project Structure

The project is built with Docker Compose.

1. Pokeshop API

The Pokeshop API is a fully instrumented REST API that makes use of different services to mimic a real life scenario. Learn more about it, here.

2. Tracetest

The docker-compose.yaml file, tracetest.provision.yaml, and tracetest.config.yaml in the tracetest directory are for the setting up the Pokeshop API, Tracetest and OpenTelemetry Collector.

Docker Compose Network

All services in the docker-compose.yaml are on the same network and will be reachable by hostname from within other services. E.g. otel-collector:4317 in the src/index.js will map to the otel-collector service.

Pokeshop API

The Pokeshop API is instrumented using the OpenTelemetry standards for Node.js, sending the data to the OpenTelemetry collector that will be pushing the telemetry information to the Azure cloud.

This is a fragment from the main tracing file from the Pokeshop API repo.

import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import * as opentelemetry from '@opentelemetry/api';
import { api, NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
import * as dotenv from 'dotenv';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { IORedisInstrumentation } from '@opentelemetry/instrumentation-ioredis';
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg';
import { AmqplibInstrumentation } from '@opentelemetry/instrumentation-amqplib';
import { SpanStatusCode } from '@opentelemetry/api';
import { B3Propagator } from '@opentelemetry/propagator-b3';

dotenv.config();
api.propagation.setGlobalPropagator(new B3Propagator());

const { COLLECTOR_ENDPOINT = '', SERVICE_NAME = 'pokeshop' } = process.env;

let globalTracer: opentelemetry.Tracer | null = null;

async function createTracer(): Promise<opentelemetry.Tracer> {
const collectorExporter = new OTLPTraceExporter({
url: COLLECTOR_ENDPOINT,
});

const spanProcessor = new BatchSpanProcessor(collectorExporter);
const sdk = new NodeSDK({
traceExporter: collectorExporter,
// @ts-ignore
instrumentations: [new IORedisInstrumentation(), new PgInstrumentation(), new AmqplibInstrumentation()],
});

sdk.configureTracerProvider({}, spanProcessor);
sdk.addResource(
new Resource({
[SemanticResourceAttributes.SERVICE_NAME]: SERVICE_NAME,
})
);

await sdk.start();
process.on('SIGTERM', () => {
sdk
.shutdown()
.then(
() => console.log('SDK shut down successfully'),
err => console.log('Error shutting down SDK', err)
)
.finally(() => process.exit(0));
});

const tracer = opentelemetry.trace.getTracer(SERVICE_NAME);

globalTracer = tracer;

return globalTracer;
}

The docker-compose.yaml file includes the definitions for all of the required services by the Pokeshop API, which includes:

  • Postgres - To save Pokemon information.
  • Redis - For in memory storage.
  • RabbitMQ - For async processing use cases.
  • API - The Pokeshop API main container.
  • Worker - The Pokeshop worker instance.
version: "3"

services:
cache:
image: redis:6
restart: unless-stopped
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 60

queue:
image: rabbitmq:3.8-management
restart: unless-stopped
healthcheck:
test: rabbitmq-diagnostics -q check_running
interval: 1s
timeout: 5s
retries: 60

demo-api:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://adot-collector:4317
NPM_RUN_COMMAND: api
ports:
- "8081:8081"
healthcheck:
test: ["CMD", "wget", "--spider", "localhost:8081"]
interval: 1s
timeout: 3s
retries: 60
depends_on:
postgres:
condition: service_healthy
cache:
condition: service_healthy
queue:
condition: service_healthy

demo-worker:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://adot-collector:4317
NPM_RUN_COMMAND: worker
depends_on:
postgres:
condition: service_healthy
cache:
condition: service_healthy
queue:
condition: service_healthy

demo-rpc:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://adot-collector:4317
NPM_RUN_COMMAND: rpc
ports:
- 8082:8082
healthcheck:
test: ["CMD", "lsof", "-i", "8082"]
interval: 1s
timeout: 3s
retries: 60
depends_on:
postgres:
condition: service_healthy
cache:
condition: service_healthy
queue:
condition: service_healthy

Tracetest

The docker-compose.yaml includes three other services.

  • Postgres - Postgres is a prerequisite for Tracetest to work. It stores trace data when running the trace-based tests.
  • OpenTelemetry Collector Contrib - The official Open Telemetry Distribution for packages outside of the core collector.
  • Tracetest - Trace-based testing that generates end-to-end tests automatically from traces.
version: "3"
services:
tracetest:
image: kubeshop/tracetest:${TAG:-latest}
platform: linux/amd64
volumes:
- type: bind
source: ./tracetest.config.yaml
target: /app/tracetest.yaml
- type: bind
source: ./tracetest.provision.yaml
target: /app/provisioning.yaml
ports:
- 11633:11633
command: --provisioning-file /app/provisioning.yaml
extra_hosts:
- "host.docker.internal:host-gateway"
depends_on:
postgres:
condition: service_healthy
adot-collector:
condition: service_started
healthcheck:
test: ["CMD", "wget", "--spider", "localhost:11633"]
interval: 1s
timeout: 3s
retries: 60

postgres:
image: postgres:14
environment:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
healthcheck:
test: pg_isready -U "$$POSTGRES_USER" -d "$$POSTGRES_DB"
interval: 1s
timeout: 5s
retries: 60
ports:
- 5432:5432

otel-collector:
image: otel/opentelemetry-collector-contrib:latest
command:
- "--config"
- "/otel-local-config.yaml"
volumes:
- ./collector.config.yaml:/otel-local-config.yaml
environment:
INSTRUMENTATION_KEY: ${INSTRUMENTATION_KEY}
ports:
- 4317:4317

Tracetest depends on Postgres and the OpenTelemetry collector. Tracetest requires config files to be loaded via a volume. The volumes are mapped from the root directory into the root directory and the respective config files.

The tracetest.config.yaml file contains the basic setup of connecting Tracetest to the Postgres instance.

postgres:
host: postgres
user: postgres
password: postgres
port: 5432
dbname: postgres
params: sslmode=disable

The tracetest.provision.yaml file defines the trace data store, set to Azure App Insights using the collector connection, meaning the traces will be sent to the OpenTelemetry collector to be processed and routed to the Azure cloud. Then, Tracetest will fetch them from the App Insights instance to execute tests.

But how does Tracetest fetch traces?

Tracetest uses the Golang Azure SDK library to pull to fetch trace data.

type: DataStore
spec:
name: AzureAppInsights
type: azureappinsights
default: true
azureappinsights:
connectionType: direct
resourceArmId: <your-arm-id>
accessToken: <your-access-token>
useAzureActiveDirectoryAuth: false

How do traces reach Azure App Insights?

The Pokeshop API code uses the native Node.js OpenTelemetry modules which sends information to the OpenTelemetry Collector to be processed and then sent to the configured Azure App Insights instance.

Run Both the Pokeshop API and Tracetest

To start both the Pokeshop API and Tracetest, run this command:

docker compose -f ./docker-compose.yaml -f ./tracetest/docker-compose.yaml up -d

This will start your Tracetest instance on http://localhost:11633/. Open it and start creating tests! Make sure to use the http://demo-api:8081/ url in your test creation, because your Pokeshop API and Tracetest are in the same network.

Learn More

Please visit our examples in GitHub and join our Discord Community for more info!