Testing Distributed Services Proxied by Tyk Gateway (API Gateway) with OpenTelemetry and Tracetest
Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.
Tyk Gateway is an open-source API gateway and management platform designed to help developers control and manage their APIs. It serves as an intermediary layer between client applications and backend services, providing functionalities like authentication, rate limiting, access control, analytics, and traffic management.
Why is this important?​
Testing Distributed Services behind API Gateways has been a pain point for years. Not having visibility into the infrastructure and not knowing where a test fails causes the MTTR to be higher than for other tools. Including OpenTelemetry in your stack, allows you to expose telemetry from the tools you use like Tyk Gateway and your services that you can use for both production visibility and trace-based testing.
This sample shows how to run integration tests against a Node.js API behind Tyk Gateway, using OpenTelemetry and Tracetest.
The Node.js Services will fetch data from an external API, transform the data, and insert it into a Postgres table. This particular flow has two failure points that are difficult to test.
- Validating that an external API request from the worker function is successful.
- Validating that the Postgress insert operation is successful.
Prerequisites​
Tyk Gateway Example:
Clone the Tracetest GitHub Repo to your local machine, and open the quick start tyk quick start example app.
git clone https://github.com/kubeshop/tracetest.git
cd tracetest/examples/quick-start-tyk
Tracetest Account:
- Sign up to
app.tracetest.ioor follow the get started docs. - Have access to the environment's agent API key.
Docker: Have Docker and Docker Compose installed on your machine.
Project Structure​
This is a Docker Compose project you can find the setup in the docker-compose.yml file
1. The Tyk Gateway​
In the docker-compose.yml file you can find the Tyk Gateway setup.
# Tyk Gateway
tyk-gateway:
image: tykio/tyk-gateway:v5.2.1
ports:
- 8080:8080
environment:
- TYK_GW_OPENTELEMETRY_ENABLED=true
- TYK_GW_OPENTELEMETRY_EXPORTER=grpc
- TYK_GW_OPENTELEMETRY_ENDPOINT=otel-collector:4317
volumes:
- ./deployments/tyk-gateway/apps:/opt/tyk-gateway/apps
- ./deployments/tyk-gateway/tyk.conf:/opt/tyk-gateway/tyk.conf
depends_on:
- tyk-redis
tyk-redis:
image: redis:6.0.4
volumes:
- tyk-redis-data:/data
Adding the configuration for OpenTelemetry to the Tyk Gateway is as simple as setting the environment variables TYK_GW_OPENTELEMETRY_ENABLED, TYK_GW_OPENTELEMETRY_EXPORTER, and TYK_GW_OPENTELEMETRY_ENDPOINT.
You can find the Tyk Gateway apps and configuration in the deployments/tyk-gateway file.
{
"name": "pokeshop",
"api_id": "1",
"org_id": "default",
"active": true,
"use_keyless": false,
"detailed_tracing": true,
"definition": { "location": "header", "key": "version" },
"auth": { "auth_header_name": "authorization" },
"version_data": { "not_versioned": true, "versions": { "Default": { "name": "Default" } } },
"proxy":
{ "listen_path": "/", "target_url": "http://api:8081/", "strip_listen_path": true, "preserve_host_header": true },
}
2. Tracetest​
The tracetest setup is composed by the tracetest-e2e and the tracetest-agent services under the docker-compose.yml file.
# Cloud-based Managed Tracetest
tracetest-agent:
image: kubeshop/tracetest-agent:latest
environment:
# Get the required information here: https://app.tracetest.io/retrieve-token
- TRACETEST_API_KEY=${TRACETEST_TOKEN}
- TRACETEST_ENVIRONMENT_ID=${TRACETEST_ENVIRONMENT_ID}
tracetest-apply:
build:
dockerfile: Dockerfile.tracetest
volumes:
- ./resources:/resources
environment:
TRACETEST_TOKEN: ${TRACETEST_TOKEN}
TRACETEST_ENVIRONMENT_ID: ${TRACETEST_ENVIRONMENT_ID}
entrypoint:
- bash
- /resources/apply.sh
networks:
default: null
depends_on:
api:
condition: service_started
tracetest-agent:
condition: service_started
tracetest-run:
build:
dockerfile: Dockerfile.tracetest
volumes:
- ./resources:/resources
environment:
TRACETEST_TOKEN: ${TRACETEST_TOKEN}
TRACETEST_ENVIRONMENT_ID: ${TRACETEST_ENVIRONMENT_ID}
entrypoint:
- bash
- /resources/run.sh
networks:
default: null
depends_on:
tracetest-apply:
condition: service_completed_successfully
3. Services under Test​
As a testing ground, the example uses Tracetest's own Pokeshop Demo APP which includes the api and worker services.
# Demo services
# pokeshop demo services
postgres:
image: postgres:14
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
interval: 1s
timeout: 5s
retries: 60
cache:
image: redis:6
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 60
queue:
image: rabbitmq:3.12
restart: unless-stopped
healthcheck:
test: rabbitmq-diagnostics -q check_running
interval: 1s
timeout: 5s
retries: 60
api:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://otel-collector:4317
NPM_RUN_COMMAND: api
healthcheck:
test: ["CMD", "wget", "--spider", "localhost:8081"]
interval: 1s
timeout: 3s
retries: 60
depends_on:
postgres:
condition: service_healthy
cache:
condition: service_healthy
queue:
condition: service_healthy
worker:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://otel-collector:4317
NPM_RUN_COMMAND: worker
depends_on:
postgres:
condition: service_healthy
cache:
condition: service_healthy
queue:
condition: service_healthy
Set up Environment Variables​
Copy the .env.template and create a .env file in the same directory. Add token and Cloud Agent endpoint.
# Get the required information here: https://app.tracetest.io/retrieve-token
TRACETEST_TOKEN="<YOUR_TRACETEST_TOKEN>"
TRACETEST_ENVIRONMENT_ID="<YOUR_ENV_ID>"
POKESHOP_DEMO_URL=http://tyk-gateway:8080
TYK_AUTH_KEY=28d220fd77974a4facfb07dc1e49c2aa
The Tracetest End To End Script​
The resources/script.js file contains the script that will execute the trace-based tests against the Pokeshop App proxied by a Tyk endpoint.
Steps Executed by the Script​
- Create a new key in the Tyk Gateway.
- Import a Pokemon.
const { expect } = require('@playwright/test');
const URL = 'http://tyk-gateway:8080';
const API_KEY = '28d220fd77974a4facfb07dc1e49c2aa';
const getKey = async () => {
const params = {
headers: {
'Content-Type': 'application/json',
'x-tyk-authorization': API_KEY,
'Response-Type': 'application/json',
},
};
const data = {
alias: 'website',
expires: -1,
access_rights: {
1: {
api_id: '1',
api_name: 'pokeshop',
versions: ['Default'],
},
},
};
const res = await fetch(`${URL}/tyk/keys/create`, {
...params,
method: 'POST',
body: JSON.stringify(data),
});
const { key } = await res.json();
return key;
};
async function importPokemon(page) {
const key = await getKey();
await page.setExtraHTTPHeaders({
Authorization: `Bearer ${key}`,
});
await page.goto(URL);
expect(await page.getByText('Pokeshop')).toBeTruthy();
await page.click('text=Import');
await page.getByLabel('ID').fill('143');
await Promise.all([
page.waitForResponse((resp) => resp.url().includes('/pokemon/import') && resp.status() === 200),
page.getByRole('button', { name: 'OK', exact: true }).click(),
]);
}
module.exports = { importPokemon };
The output from the Playwright tests will show the test results with links to the Tracetest App.
[+] Running 2/2
✔ worker Pulled 0.9s
✔ api Pulled 0.8s
[+] Creating 8/0
✔ Container quick-start-tyk-tracetest-agent-1 Running 0.0s
✔ Container quick-start-tyk-postgres-1 Running 0.0s
✔ Container quick-start-tyk-jaeger-1 Running 0.0s
✔ Container quick-start-tyk-tyk-redis-1 Running 0.0s
✔ Container quick-start-tyk-cache-1 Running 0.0s
✔ Container quick-start-tyk-otel-collector-1 Running 0.0s
✔ Container quick-start-tyk-queue-1 Running 0.0s
✔ Container quick-start-tyk-tyk-gateway-1 Running 0.0s
[+] Running 7/7
✔ Container quick-start-tyk-jaeger-1 Healthy 0.5s
✔ Container quick-start-tyk-postgres-1 Healthy 3.2s
✔ Container quick-start-tyk-cache-1 Healthy 3.2s
✔ Container quick-start-tyk-queue-1 Healthy 3.2s
✔ Container quick-start-tyk-worker-1 Started 0.2s
✔ Container quick-start-tyk-api-1 Started 0.1s
✔ Container quick-start-tyk-tracetest-apply-1 Started 0.2s
[+] Running 2/2
✔ api Pulled 0.8s
✔ worker Pulled 0.8s
Configuring Tracetest
SUCCESS Successfully configured Tracetest CLI
Running Trace-Based Tests...
✔ RunGroup: #QMjWZ5jIR (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments/ttenv_b42fa137465c6e04/run/QMjWZ5jIR)
Summary: 1 passed, 0 failed, 0 pending
✔ Import Pokemon (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments/ttenv_b42fa137465c6e04/test/4_IKfPlIg/run/13/test) - trace id: 1554d5d5904daa17f2a6854a7d7608e5
✔ Import should return 200
✔ All Database Spans: Processing time is less than 1s
✔ Import Pokemon Span Should be Present
✔ Redis should return in 20ms
Tracetest App Results​
Running the Example​
Spin up the deployment and test execution.
docker-compose run tracetest-run
This will trigger the Docker Compose setup and immediately run the trace-based tests using the Tracetest Playwright integration as part of the tracetest-e2e service.
Learn More​
Feel free to check out our examples in GitHub and join our Slack Community for more info!