Testing Distributed Services Proxied by Tyk Gateway (API Gateway) with OpenTelemetry and Tracetest
Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.
Tyk Gateway is an open-source API gateway and management platform designed to help developers control and manage their APIs. It serves as an intermediary layer between client applications and backend services, providing functionalities like authentication, rate limiting, access control, analytics, and traffic management.
Why is this important?
Testing Distributed Services behind API Gateways has been a pain point for years. Not having visibility into the infrastructure and not knowing where a test fails causes the MTTR to be higher than for other tools. Including OpenTelemetry in your stack, allows you to expose telemetry from the tools you use like Tyk Gateway and your services that you can use for both production visibility and trace-based testing.
This sample shows how to run integration tests against a Node.js API behind Tyk Gateway, using OpenTelemetry and Tracetest.
The Node.js Services will fetch data from an external API, transform the data, and insert it into a Postgres table. This particular flow has two failure points that are difficult to test.
- Validating that an external API request from the worker function is successful.
- Validating that the Postgress insert operation is successful.
Prerequisites
Tyk Gateway Example:
Clone the Tracetest GitHub Repo to your local machine, and open the quick start tyk quick start example app.
git clone https://github.com/kubeshop/tracetest.git
cd tracetest/examples/quick-start-tyk
Tracetest Account:
- Copy the
.env.templatefile to.env. - Log into the Tracetest app.
- This example is configured to use the OpenTelemetry Collector. Ensure the environment you will be utilizing to run this example is also configured to use the OpenTelemetry Tracing Backend by clicking on Settings, Tracing Backend, OpenTelemetry, and Save.
- Configure your environment to use the agent, then click the Settings link, and from the Agent tab select the "Run Agent Locally" option.
- Fill out the token and agent api key details by editing your .env file. You can find these values in the Settings area for your environment.
Project Structure
This is a Docker Compose project you can find the setup in the docker-compose.yml file
1. The Tyk Gateway
In the docker-compose.yml file you can find the Tyk Gateway setup.
# Tyk Gateway
tyk-gateway:
image: tykio/tyk-gateway:v5.2.1
ports:
- 8080:8080
environment:
- TYK_GW_OPENTELEMETRY_ENABLED=true
- TYK_GW_OPENTELEMETRY_EXPORTER=grpc
- TYK_GW_OPENTELEMETRY_ENDPOINT=otel-collector:4317
volumes:
- ./deployments/tyk-gateway/apps:/opt/tyk-gateway/apps
- ./deployments/tyk-gateway/tyk.conf:/opt/tyk-gateway/tyk.conf
depends_on:
- tyk-redis
tyk-redis:
image: redis:6.0.4
volumes:
- tyk-redis-data:/data
Adding the configuration for OpenTelemetry to the Tyk Gateway is as simple as setting the environment variables TYK_GW_OPENTELEMETRY_ENABLED, TYK_GW_OPENTELEMETRY_EXPORTER, and TYK_GW_OPENTELEMETRY_ENDPOINT.
You can find the Tyk Gateway apps and configuration in the deployments/tyk-gateway file.
{
"name": "pokeshop",
"api_id": "1",
"org_id": "default",
"active": true,
"use_keyless": false,
"detailed_tracing": true,
"definition": { "location": "header", "key": "version" },
"auth": { "auth_header_name": "authorization" },
"version_data": { "not_versioned": true, "versions": { "Default": { "name": "Default" } } },
"proxy":
{ "listen_path": "/", "target_url": "http://api:8081/", "strip_listen_path": true, "preserve_host_header": true },
}
2. Tracetest
The tracetest setup is composed by the tracetest-e2e and the tracetest-agent services under the docker-compose.yml file.
# Tracetest x Playwright
tracetest-e2e:
build: .
command: npm test
environment:
- POKESHOP_DEMO_URL=${POKESHOP_DEMO_URL}
- TRACETEST_API_TOKEN=${TRACETEST_API_TOKEN}
- TYK_AUTH_KEY=${TYK_AUTH_KEY}
depends_on:
tracetest-agent:
condition: service_started
api:
condition: service_healthy
worker:
condition: service_started
tyk-gateway:
condition: service_started
tracetest-agent:
environment:
TRACETEST_DEV: ${TRACETEST_DEV}
TRACETEST_API_KEY: ${TRACETEST_AGENT_API_KEY}
TRACETEST_SERVER_URL: ${TRACETEST_SERVER_URL}
image: kubeshop/tracetest-agent:latest
3. Services under Test
As a testing ground, the example uses Tracetest's own Pokeshop Demo APP which includes the api and worker services.
# Demo services
# pokeshop demo services
postgres:
image: postgres:14
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
interval: 1s
timeout: 5s
retries: 60
cache:
image: redis:6
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 60
queue:
image: rabbitmq:3.12
restart: unless-stopped
healthcheck:
test: rabbitmq-diagnostics -q check_running
interval: 1s
timeout: 5s
retries: 60
api:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://otel-collector:4317
NPM_RUN_COMMAND: api
healthcheck:
test: ["CMD", "wget", "--spider", "localhost:8081"]
interval: 1s
timeout: 3s
retries: 60
depends_on:
postgres:
condition: service_healthy
cache:
condition: service_healthy
queue:
condition: service_healthy
worker:
image: kubeshop/demo-pokemon-api:latest
restart: unless-stopped
pull_policy: always
environment:
REDIS_URL: cache
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres?schema=public
RABBITMQ_HOST: queue
POKE_API_BASE_URL: https://pokeapi.co/api/v2
COLLECTOR_ENDPOINT: http://otel-collector:4317
NPM_RUN_COMMAND: worker
depends_on:
postgres:
condition: service_healthy
cache:
condition: service_healthy
queue:
condition: service_healthy
Set up Environment Variables
Copy the .env.template and create a .env file in the same directory. Add token and Cloud Agent endpoint.
TRACETEST_API_TOKEN=<my_token_with_engineer_access>
TRACETEST_AGENT_API_KEY=<my_agent_api_key>
POKESHOP_DEMO_URL=http://tyk-gateway:8080
TYK_AUTH_KEY=28d220fd77974a4facfb07dc1e49c2aa
The Tracetest End To End Script
The playwright/home.spec.ts file contains the script that will execute the trace-based tests against the Pokeshop App proxied by a Tyk endpoint.
Steps Executed by the Script
- Create a new key in the Tyk Gateway.
- Create a new test definition in Tracetest (using the async import service from the Pokeshop Demo).
- Run the Playwright Test.
- Report back the results.
import { test, expect } from "@playwright/test";
import { getKey } from "./auth";
import Tracetest, { Types } from "@tracetest/playwright";
const { TRACETEST_API_TOKEN = "" } = process.env;
let tracetest: Types.TracetestPlaywright | undefined = undefined;
test.describe.configure({ mode: "serial" });
const definition = `
type: Test
spec:
id: UGxheXdyaWdodDogaW1wb3J0cyBhIHBva2Vtb24=
name: "Playwright: imports a pokemon"
trigger:
type: playwright
specs:
- selector: span[tracetest.span.type="http"] span[tracetest.span.type="http"]
name: "All HTTP Spans: Status code is 200"
assertions:
- attr:http.status_code = 200
- selector: span[tracetest.span.type="database"]
name: "All Database Spans: Processing time is less than 100ms"
assertions:
- attr:tracetest.span.duration < 2s
outputs:
- name: MY_OUTPUT
selector: span[tracetest.span.type="general" name="Tracetest trigger"]
value: attr:name
`;
test.beforeAll(async () => {
tracetest = await Tracetest({ apiToken: TRACETEST_API_TOKEN });
await tracetest.setOptions({
"Playwright: imports a pokemon": {
definition,
},
});
});
test.beforeEach(async ({ page, context }, { title }) => {
// sets the auth headers for the page requests
const key = await getKey();
await context.setExtraHTTPHeaders({
Authorization: `Bearer ${key}`,
});
await page.goto("/");
await tracetest?.capture(title, page);
});
// optional step to break the playwright script in case a Tracetest test fails
test.afterAll(async ({}, testInfo) => {
testInfo.setTimeout(80000);
await tracetest?.summary();
});
test("Playwright: imports a pokemon", async ({ page }) => {
expect(await page.getByText("Pokeshop")).toBeTruthy();
await page.click("text=Import");
await page.getByLabel("ID").fill(Math.floor(Math.random() * 101).toString());
await page.getByRole("button", { name: "OK", exact: true }).click();
});
The output from the Playwright tests will show the test results with links to the Tracetest App.
WARN[0000] The "TRACETEST_SERVER_URL" variable is not set. Defaulting to a blank string.
[+] Running 2/2
✔ worker Pulled 1.1s
✔ api Pulled 1.1s
[+] Creating 9/0
✔ Container quick-start-tyk-tyk-redis-1 Running 0.0s
✔ Container quick-start-tyk-otel-collector-1 Running 0.0s
✔ Container quick-start-tyk-cache-1 Running 0.0s
✔ Container quick-start-tyk-queue-1 Running 0.0s
✔ Container quick-start-tyk-tracetest-agent-1 Running 0.0s
✔ Container quick-start-tyk-postgres-1 Running 0.0s
✔ Container quick-start-tyk-tyk-gateway-1 Running 0.0s
✔ Container quick-start-tyk-worker-1 Running 0.0s
✔ Container quick-start-tyk-api-1 Running 0.0s
[+] Running 3/3
✔ Container quick-start-tyk-postgres-1 Healthy 0.5s
✔ Container quick-start-tyk-cache-1 Healthy 0.5s
✔ Container quick-start-tyk-queue-1 Healthy 0.5s
[+] Running 2/2
✔ worker Pulled 1.2s
✔ api Pulled 1.1s
> quick-start-tyk@1.0.0 test
> playwright test
Running 1 test using 1 worker
[chromium] › home.spec.ts:59:5 › Playwright: imports a pokemon
Successful: 1
Failed: 0
[✔️ Playwright: imports a pokemon] #20 - https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments/ttenv_b42fa137465c6e04/test/UGxheXdyaWdodDogaW1wb3J0cyBhIHBva2Vtb24=/run/20
1 passed (14.1s)
To open last HTML report run:
npx playwright show-report
Tracetest App Results
Running the Example
Spin up the deployment and test execution.
docker-compose run tracetest-e2e
This will trigger the Docker Compose setup and immediately run the trace-based tests using the Tracetest Playwright integration as part of the tracetest-e2e service.
Learn More
Feel free to check out our examples in GitHub and join our Slack Community for more info!