Performance Testing with Distributed Tracing using Artillery, Playwright and Tracetest
Find out the requirements for your instrumented app to start using Tracetest x Playwright.
This integration is compatible with Artillery v2.0.10 and above.
Tracetest is a testing tool based on OpenTelemetry that permits you to test your distributed application. It allows you to use the trace data generated by your OpenTelemetry tools to check and assert if your application has the desired behavior defined by your test definitions.
Playwright is an open-source automation framework developed by Microsoft that enables cross-browser automation for web applications. It provides a set of APIs and libraries for automating interactions with web browsers such as Chrome, Firefox, and Microsoft Edge.
Artillery is a modern, powerful load-testing toolkit. Artillery is designed to help developers and testers simulate traffic to their applications, APIs, and microservices. It allows users to define scenarios to test how their systems behave under different loads.
Why is this important?​
The Tracetest integration for Playwright enables your current Playwright tests to easily capture a full distributed trace from your OpenTelemetry instrumented front-end and back-end system.
You can embed a Tracetest test in this Playwright test, and allow trace-based testing assertions to be applied across this entire flow, enabling true end-to-end tests across your entire system. Taking it a step further you can use the Playwright engine in Artillery to run performance tests on top if it.
Because you're using traces as test specs you can:
- Get faster MTTR for failing performance tests
- Assert against the Artillery test execution, Playwright test execution, and the system under test
- Validate functionality of other parts of your system that may be broken, even when performance tests are passing
The @tracetest/playwright npm Package​
The @tracetest/playwright npm package is a Playwright plugin that allows you to run trace-based testing using Tracetest and Playwright. It is a wrapper around the Tracetest API that allows you to configure, orchestrate and run tests from Playwright.
The Artillery Playwright Engine​
The artillery-engine-playwright npm package is an Artillery engine lets you combine Playwright with Artillery to launch real browsers for full browser load testing.
Requirements​
Tracetest Account:
- Sign up to
app.tracetest.ioor follow the get started docs. - Have access to the environment's agent API key.
Docker: Have Docker and Docker Compose installed on your machine.
Run This Quckstart Example​
The example below is provided as part of the Tracetest GitHub repo. You can download and run the example by following these steps:
Clone the Tracetest project and go to the Artillery + Playwright Quickstart:
git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/quick-start-artillery-playwright
Follow these instructions to run the included demo app and TypeScript example:
- Copy the
.env.templatefile to.env. - Fill out the TRACETEST_TOKEN and ENVIRONMENT_ID details by editing your
.envfile. - Run
docker compose run tracetest-run. - Follow the links in the output to view the test results.
Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.
Project Structure ​
The quick start Artillery + Playwright project is built with Docker Compose.
The Pokeshop Demo App is a complete example of a distributed application using different back-end and front-end services. We will be launching it and running tests against it as part of this example.
The docker-compose.yaml file in the root directory of the quick start runs the Pokeshop Demo app, the OpenTelemetry Collector, Jaeger, and the Tracetest Agent setup.
The Artillery Plugin quick start has two primary files:
- Playwright Test Script file
resources/test.tsthat includes the@tracetest/playwrightnpm package and the Tracetest test definition. To view the Tracetest test definition as a YAML file you can also see theimport-pokemon.yamlfile. - Artillery Playwright engine file
resources/artillery-playwright.yamlthat defines that the Artillery test will use the Playwright engine.
The resources/test.ts Test Script​
The resources/test.ts test script contains one test based on the Pokeshop Demo UI features:
- Import a Pokemon (using an async process)
Tracetest Library Setup​
If you go to the package.json file you will find the inclusion of a Tracetest package for Playwright @tracetest/playwright.
The test script imports the package, grabs the Tracetest API token from the environment variables and creates the Tracetest instance. It then defines the importPokemon test and uses the Tracetest test definition YAML.
The complete test script looks like this:
import { Page } from 'playwright';
import { expect, TestInfo } from '@playwright/test';
import Tracetest from '@tracetest/playwright';
import { config } from 'dotenv';
config();
const { TRACETEST_TOKEN = '', TRACETEST_ENVIRONMENT_ID = '' } = process.env;
const definition = `
type: Test
spec:
id: artillery-playwight-import-pokemon
name: "Artillery Playwright - Import Pokemon"
trigger:
type: playwright
specs:
- selector: span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"]
name: "All HTTP Spans: Status code is 200"
assertions:
- attr:http.status_code = 200
- selector: span[tracetest.span.type="http" name="GET" http.method="GET"]
assertions:
- attr:http.route = "/api/v2/pokemon/\${var:POKEMON_ID}"
- selector: span[tracetest.span.type="database"]
name: "All Database Spans: Processing time is less than 1s"
assertions:
- attr:tracetest.span.duration < 1s
outputs:
- name: DATABASE_POKEMON_ID
selector: span[tracetest.span.type="database" name="create postgres.pokemon" db.system="postgres" db.name="postgres" db.user="postgres" db.operation="create" db.sql.table="pokemon"]
value: attr:db.result | json_path '$.id'
`;
export async function importPokemon(page: Page) {
const tracetest = await Tracetest({ apiToken: TRACETEST_TOKEN, environmentId: TRACETEST_ENVIRONMENT_ID });
const title = 'Artillery Playwright - Import Pokemon';
const pokemonId = Math.floor(Math.random() * 101).toString();
await page.goto('/');
await tracetest?.setOptions({
[title]: {
definition,
runInfo: {
variables: [
{
key: 'POKEMON_ID',
value: pokemonId,
},
],
},
},
});
await tracetest?.capture(page, { title: 'Artillery Playwright - Import Pokemon', config: {} } as TestInfo);
expect(await page.getByText('Pokeshop')).toBeTruthy();
await page.click('text=Import');
await page.getByLabel('ID').fill(pokemonId);
await page.getByRole('button', { name: 'OK', exact: true }).click();
await tracetest?.summary();
}
Creating the Artillery Test Script​
The resources/artilery-playwright.yaml file contains the Artillery Test Script that will be used to trigger requests against the Pokeshop Demo and run trace-based tests. The steps executed by this script are the following:
- Defines a scenario using the
playwrightengine running theimportPokemonfromresources/test.tswhich is a Playwright test using the Tracetest integration using a random number as the Pokemon ID. - Defines the target as the Pokeshop Demo App running on
http://localhost:8081.
config:
target: http://localhost:8081
engines:
playwright: {}
processor: test.ts
scenarios:
- engine: playwright
testFunction: importPokemon
Setting the Environment Variables​
Copy the .env.template file to .env and add the Tracetest API token and agent tokens to the TRACETEST_TOKEN and TRACETEST_ENVIRONMENT_ID variables.
Running the Full Example​
To start the Pokeshop Demo App, run the following command from the root directory:
docker compose run tracetest-run
Finding the Results​
The output from the Tracetest Engine script should be visible in the console log after running the test command. This log will show links to Tracetest for each of the test runs invoked by the Artillery Testing Script. Click a link to launch Tracetest and view the test result.
[+] Running 2/2
✔ worker Pulled 0.9s
✔ api Pulled 0.9s
[+] Building 0.7s (10/10) FINISHED docker:desktop-linux
=> [tracetest-apply internal] load build definition from Dockerfile.tracetest 0.0s
=> => transferring dockerfile: 336B 0.0s
=> [tracetest-apply internal] load metadata for docker.io/library/alpine:latest 0.6s
=> [tracetest-apply auth] library/alpine:pull token for registry-1.docker.io 0.0s
=> [tracetest-apply internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [tracetest-apply 1/5] FROM docker.io/library/alpine:latest@sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5 0.0s
=> CACHED [tracetest-apply 2/5] WORKDIR /app 0.0s
=> CACHED [tracetest-apply 3/5] RUN apk --update add bash jq curl 0.0s
=> CACHED [tracetest-apply 4/5] RUN curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash -s -- v1.4.0 0.0s
=> CACHED [tracetest-apply 5/5] WORKDIR /resources 0.0s
=> [tracetest-apply] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:746b8de43a912dc68c81345fd5b2a54740f3d8b0b73724dc79a31abea045d748 0.0s
=> => naming to docker.io/library/pokeshop-tracetest-apply 0.0s
[+] Creating 10/8
✔ Network pokeshop_default Created 0.0s
✔ Container pokeshop-postgres-1 Created 0.2s
✔ Container pokeshop-jaeger-1 Created 0.2s
✔ Container pokeshop-queue-1 Created 0.2s
✔ Container pokeshop-cache-1 Created 0.2s
✔ Container pokeshop-otel-collector-1 Created 0.1s
✔ Container pokeshop-worker-1 Created 0.1s
✔ Container pokeshop-api-1 Created 0.0s
✔ Container pokeshop-tracetest-agent-1 Created 0.0s
✔ Container pokeshop-tracetest-apply-1 Created 0.1s
[+] Running 9/9
✔ Container pokeshop-queue-1 Healthy 12.2s
✔ Container pokeshop-jaeger-1 Healthy 1.9s
✔ Container pokeshop-cache-1 Healthy 12.2s
✔ Container pokeshop-postgres-1 Healthy 12.2s
✔ Container pokeshop-otel-collector-1 Started 0.4s
✔ Container pokeshop-worker-1 Started 0.1s
✔ Container pokeshop-api-1 Healthy 1.6s
✔ Container pokeshop-tracetest-agent-1 Started 0.1s
✔ Container pokeshop-tracetest-apply-1 Started 0.1s
[+] Running 2/2
✔ api Pulled 0.8s
✔ worker Pulled 0.8s
> quick-start-artillery-playwright@1.0.0 test
> artillery run resources/artillery-playwright.yaml
Test run id: twmey_ngwbckeqpwm9ja9p99hqhfjtbhefp_bw8c
Bundled Typescript file into JS. New processor path: /app/resources/dist/test.js
Phase started: unnamed (index: 0, duration: 1s) 16:55:42(+0000)
Phase completed: unnamed (index: 0, duration: 1s) 16:55:43(+0000)
--------------------------------------
Metrics for period to: 16:55:50(+0000) (width: 2.21s)
--------------------------------------
browser.http_requests: ......................................................... 10
browser.page.FCP.http://api:8081/:
min: ......................................................................... 175.3
max: ......................................................................... 175.3
mean: ........................................................................ 175.3
median: ...................................................................... 175.9
p95: ......................................................................... 175.9
p99: ......................................................................... 175.9
browser.page.FID.http://api:8081/:
min: ......................................................................... 1.6
max: ......................................................................... 1.6
mean: ........................................................................ 1.6
median: ...................................................................... 1.6
p95: ......................................................................... 1.6
p99: ......................................................................... 1.6
browser.page.LCP.http://api:8081/:
min: ......................................................................... 193.4
max: ......................................................................... 193.4
mean: ........................................................................ 193.4
median: ...................................................................... 194.4
p95: ......................................................................... 194.4
p99: ......................................................................... 194.4
browser.page.TTFB.http://api:8081/:
min: ......................................................................... 14.9
max: ......................................................................... 14.9
mean: ........................................................................ 14.9
median: ...................................................................... 15
p95: ......................................................................... 15
p99: ......................................................................... 15
browser.page.codes.200: ........................................................ 10
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
â ´ Run Group: #e2fae28c-ac06-4001-8dc1-2328eee32e5a (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments//run/e2fae28c-ac06-4001-8dc1-2328eee32e5a)
Failed: 0
Succeed: 1
Pending: 0
Runs:
✔ Artillery Playwright - Import Pokemon (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments//test/artillery-playwight-import-pokemon/run/7) - trace id: 1b3ba6a22f595dbfad1c5d32120aad20
--------------------------------------
Metrics for period to: 16:56:00(+0000) (width: 0s)
--------------------------------------
vusers.completed: .............................................................. 1
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 15189.2
max: ......................................................................... 15189.2
mean: ........................................................................ 15189.2
median: ...................................................................... 15218.6
p95: ......................................................................... 15218.6
p99: ......................................................................... 15218.6
All VUs finished. Total time: 16 seconds
--------------------------------
Summary report @ 16:56:00(+0000)
--------------------------------
browser.http_requests: ......................................................... 10
browser.page.FCP.http://api:8081/:
min: ......................................................................... 175.3
max: ......................................................................... 175.3
mean: ........................................................................ 175.3
median: ...................................................................... 175.9
p95: ......................................................................... 175.9
p99: ......................................................................... 175.9
browser.page.FID.http://api:8081/:
min: ......................................................................... 1.6
max: ......................................................................... 1.6
mean: ........................................................................ 1.6
median: ...................................................................... 1.6
p95: ......................................................................... 1.6
p99: ......................................................................... 1.6
browser.page.LCP.http://api:8081/:
min: ......................................................................... 193.4
max: ......................................................................... 193.4
mean: ........................................................................ 193.4
median: ...................................................................... 194.4
p95: ......................................................................... 194.4
p99: ......................................................................... 194.4
browser.page.TTFB.http://api:8081/:
min: ......................................................................... 14.9
max: ......................................................................... 14.9
mean: ........................................................................ 14.9
median: ...................................................................... 15
p95: ......................................................................... 15
p99: ......................................................................... 15
browser.page.codes.200: ........................................................ 10
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
min: ......................................................................... 15189.2
max: ......................................................................... 15189.2
mean: ........................................................................ 15189.2
median: ...................................................................... 15218.6
p95: ......................................................................... 15218.6
p99: ......................................................................... 15218.6
What's Next?​
After running the test, you can click the run link for any of them, update the assertions, and run the scripts once more. This flow enables complete a trace-based TDD flow.
Learn More​
Please visit our examples in GitHub and join our Slack Community for more info!