Performance Testing with Distributed Tracing using Artillery, Playwright and Tracetest

Version Compatibility

This integration is compatible with Artillery v2.0.10 and above.

note

Check out the source code on GitHub here.

Tracetest is a testing tool based on OpenTelemetry that permits you to test your distributed application. It allows you to use the trace data generated by your OpenTelemetry tools to check and assert if your application has the desired behavior defined by your test definitions.

Playwright is an open-source automation framework developed by Microsoft that enables cross-browser automation for web applications. It provides a set of APIs and libraries for automating interactions with web browsers such as Chrome, Firefox, and Microsoft Edge.

Artillery is a modern, powerful load-testing toolkit. Artillery is designed to help developers and testers simulate traffic to their applications, APIs, and microservices. It allows users to define scenarios to test how their systems behave under different loads.

Why is this important?

The Tracetest integration for Playwright enables your current Playwright tests to easily capture a full distributed trace from your OpenTelemetry instrumented front-end and back-end system.

You can embed a Tracetest test in this Playwright test, and allow trace-based testing assertions to be applied across this entire flow, enabling true end-to-end tests across your entire system. Taking it a step further you can use the Playwright engine in Artillery to run performance tests on top if it.

Because you're using traces as test specs you can:

Get faster MTTR for failing performance tests
Assert against the Artillery test execution, Playwright test execution, and the system under test
Validate functionality of other parts of your system that may be broken, even when performance tests are passing

The `@tracetest/playwright` npm Package

The @tracetest/playwright npm package is a Playwright plugin that allows you to run trace-based testing using Tracetest and Playwright. It is a wrapper around the Tracetest API that allows you to configure, orchestrate and run tests from Playwright.

The Artillery Playwright Engine

The artillery-engine-playwright npm package is an Artillery engine lets you combine Playwright with Artillery to launch real browsers for full browser load testing.

Requirements

Tracetest Account:

Sign up to app.tracetest.io or follow the get started docs.
Create an environment.
Create an environment token.
Have access to the environment's agent API key.

Docker: Have Docker and Docker Compose installed on your machine.

Run This Quckstart Example

The example below is provided as part of the Tracetest GitHub repo. You can download and run the example by following these steps:

Clone the Tracetest project and go to the Artillery + Playwright Quickstart:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/quick-start-artillery-playwright

Follow these instructions to run the included demo app and TypeScript example:

Log into the Tracetest app.
Copy the .env.template file to .env.
Fill out the token and agent API key details by editing your .env file. You can find these values in the Settings area for your environment.
Run docker compose up -d.
This example is configured to use Jaeger. Ensure the environment you will be utilizing to run this example is also configured to use the Jaeger Tracing Backend by clicking on Settings, Tracing Backend, Jaeger, updating the url to jaeger:16685, Test Connection and Save.
Run npm i to install the required dependencies.
Run npm run test to run the example.
The output will show the test results and the Tracetest URL for each test run.
Follow the links in the log to view the test runs programmatically created by the Atillery execution using the Playwright engine.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

Project Structure

The quick start Artillery + Playwright project is built with Docker Compose.

The Pokeshop Demo App is a complete example of a distributed application using different back-end and front-end services. We will be launching it and running tests against it as part of this example.

The docker-compose.yaml file in the root directory of the quick start runs the Pokeshop Demo app, the OpenTelemetry Collector, Jaeger, and the Tracetest Agent setup.

The Artillery Plugin quick start has two primary files:

Playwright Test Script file home.spec.ts that includes the @tracetest/playwright npm package and the Tracetest test definition. To view the Tracetest test definition as a YAML file you can also see the import-pokemon.yaml file.
Artillery Playwright engine file artillery-playwright.yaml that defines that the Artillery test will use the Playwright engine.

The Playwright Setup

Before moving forward, run npm i in the root folder to install the dependencies.

The `home.spec.ts` Test Script

The home.spec.ts test script contains one test based on the Pokeshop Demo UI features:

Import a Pokemon (using an async process)

Tracetest Library Setup

If you go to the package.json file you will find the inclusion of a Tracetest package for Playwright @tracetest/playwright.

The test script imports the package, grabs the Tracetest API token from the environment variables and creates the Tracetest instance. It then defines the importPokemon test and uses the Tracetest test definition YAML.

The complete test script looks like this:

import { Page } from 'playwright';
import { expect } from '@playwright/test';
import Tracetest from '@tracetest/playwright';
import { config } from 'dotenv';

config();

const { TRACETEST_API_TOKEN = '' } = process.env;

const definition = `
type: Test
spec:
  id: artillery-playwight-import-pokemon
  name: "Artillery Playwright - Import Pokemon"
  trigger:
    type: playwright
  specs:
    - selector: span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"]
      name: "All HTTP Spans: Status  code is 200"
      assertions:
        - attr:http.status_code = 200
    - selector: span[tracetest.span.type="http" name="GET" http.method="GET"]
      assertions:
        - attr:http.route = "/api/v2/pokemon/\${var:POKEMON_ID}"
    - selector: span[tracetest.span.type="database"]
      name: "All Database Spans: Processing time is less than 1s"
      assertions:
        - attr:tracetest.span.duration < 1s
  outputs:
    - name: DATABASE_POKEMON_ID
      selector: span[tracetest.span.type="database" name="create postgres.pokemon" db.system="postgres" db.name="postgres" db.user="postgres" db.operation="create" db.sql.table="pokemon"]
      value: attr:db.result | json_path '$.id'
`;

export async function importPokemon(page: Page) {
  const tracetest = await Tracetest({ apiToken: TRACETEST_API_TOKEN });
  const title = 'Artillery Playwright - Import Pokemon';
  const pokemonId = Math.floor(Math.random() * 101).toString();
  await page.goto('/');

  await tracetest?.setOptions({
    [title]: {
      definition,
      runInfo: {
        variables: [
          {
            key: 'POKEMON_ID',
            value: pokemonId,
          },
        ],
      },
    },
  });

  await tracetest?.capture(title, page);

  expect(await page.getByText('Pokeshop')).toBeTruthy();

  await page.click('text=Import');

  await page.getByLabel('ID').fill(pokemonId);
  await page.getByRole('button', { name: 'OK', exact: true }).click();

  await tracetest?.summary();
}

Creating the Artillery Test Script

The artilery-playwright.yaml file contains the Artillery Test Script that will be used to trigger requests against the Pokeshop Demo and run trace-based tests. The steps executed by this script are the following:

Defines a scenario using the playwright engine running the importPokemon from home.spec.ts which is a Playwright test using the Tracetest integration using a random number as the Pokemon ID.
Defines the target as the Pokeshop Demo App running on http://localhost:8081.

artilery-playwright.yaml
config:
  target: http://localhost:8081
  engines:
    playwright: {}
  processor: home.spec.ts
scenarios:
  - engine: playwright
    testFunction: importPokemon

Setting the Environment Variables

Copy the .env.template file to .env and add the Tracetest API token and agent tokens to the TRACETEST_API_TOKEN and TRACETEST_AGENT_API_KEY variables.

Running the Full Example

To start the Pokeshop Demo App, run the following command from the root directory:

docker compose up -d

This example is configured to use Jaeger. Ensure the environment you will be utilizing to run this example is also configured to use the Jaeger Tracing Backend.

Go to your Tracetest account, click on Settings, Tracing Backend, Jaeger, update the URL to jaeger:16685, Test Connection and Save.

Run the Artillery test.

npm run test

Finding the Results

The output from the Tracetest Engine script should be visible in the console log after running the test command. This log will show links to Tracetest for each of the test runs invoked by the Artillery Testing Script. Click a link to launch Tracetest and view the test result.

> quick-start-artillery-playwright@1.0.0 test
> artillery run artillery-playwright.yaml

Test run id: tx4gt_gppedez5zrnwdc9f5yt7zaqcpkxgp_xjc9
Bundled Typescript file into JS. New processor path: /Users/adnanrahic/Code/kubeshop/tracetest/examples/quick-start-artillery-playwright/dist/home.spec.js
Phase started: unnamed (index: 0, duration: 1s) 17:37:50(+0100)

Phase completed: unnamed (index: 0, duration: 1s) 17:37:51(+0100)

--------------------------------------
Metrics for period to: 17:38:00(+0100) (width: 4.128s)
--------------------------------------

browser.http_requests: ......................................................... 32
browser.page.FCP.http://localhost:8081/:
  min: ......................................................................... 307.5
  max: ......................................................................... 307.5
  mean: ........................................................................ 307.5
  median: ...................................................................... 308
  p95: ......................................................................... 308
  p99: ......................................................................... 308
browser.page.FID.http://localhost:8081/:
  min: ......................................................................... 4.9
  max: ......................................................................... 4.9
  mean: ........................................................................ 4.9
  median: ...................................................................... 4.9
  p95: ......................................................................... 4.9
  p99: ......................................................................... 4.9
browser.page.LCP.http://localhost:8081/:
  min: ......................................................................... 335.2
  max: ......................................................................... 335.2
  mean: ........................................................................ 335.2
  median: ...................................................................... 333.7
  p95: ......................................................................... 333.7
  p99: ......................................................................... 333.7
browser.page.TTFB.http://localhost:8081/:
  min: ......................................................................... 17.8
  max: ......................................................................... 17.8
  mean: ........................................................................ 17.8
  median: ...................................................................... 17.6
  p95: ......................................................................... 17.6
  p99: ......................................................................... 17.6
browser.page.codes.200: ........................................................ 32
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1


⠧
✔ Artillery Playwright - Import Pokemon (https://app.tracetest.io/organizations/ttorg_e66318ba6544b856/environments/ttenv_956484fe2078461e/test/artillery-playwight-import-pokemon/run/3) - trace id: bf18cbd2fdbc0be765e1a44ecee738ad

--------------------------------------
Metrics for period to: 17:38:10(+0100) (width: 0s)
--------------------------------------

vusers.completed: .............................................................. 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 16943.4
  max: ......................................................................... 16943.4
  mean: ........................................................................ 16943.4
  median: ...................................................................... 16819.2
  p95: ......................................................................... 16819.2
  p99: ......................................................................... 16819.2


All VUs finished. Total time: 18 seconds

--------------------------------
Summary report @ 17:38:09(+0100)
--------------------------------

browser.http_requests: ......................................................... 32
browser.page.FCP.http://localhost:8081/:
  min: ......................................................................... 307.5
  max: ......................................................................... 307.5
  mean: ........................................................................ 307.5
  median: ...................................................................... 308
  p95: ......................................................................... 308
  p99: ......................................................................... 308
browser.page.FID.http://localhost:8081/:
  min: ......................................................................... 4.9
  max: ......................................................................... 4.9
  mean: ........................................................................ 4.9
  median: ...................................................................... 4.9
  p95: ......................................................................... 4.9
  p99: ......................................................................... 4.9
browser.page.LCP.http://localhost:8081/:
  min: ......................................................................... 335.2
  max: ......................................................................... 335.2
  mean: ........................................................................ 335.2
  median: ...................................................................... 333.7
  p95: ......................................................................... 333.7
  p99: ......................................................................... 333.7
browser.page.TTFB.http://localhost:8081/:
  min: ......................................................................... 17.8
  max: ......................................................................... 17.8
  mean: ........................................................................ 17.8
  median: ...................................................................... 17.6
  p95: ......................................................................... 17.6
  p99: ......................................................................... 17.6
browser.page.codes.200: ........................................................ 32
vusers.completed: .............................................................. 1
vusers.created: ................................................................ 1
vusers.created_by_name.0: ...................................................... 1
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 16943.4
  max: ......................................................................... 16943.4
  mean: ........................................................................ 16943.4
  median: ...................................................................... 16819.2
  p95: ......................................................................... 16819.2
  p99: ......................................................................... 16819.2

What's Next?

After running the test, you can click the run link for any of them, update the assertions, and run the scripts once more. This flow enables complete a trace-based TDD flow.

assertions

Learn More

Please visit our examples in GitHub and join our Slack Community for more info!

Why is this important?​

The @tracetest/playwright npm Package​

The Artillery Playwright Engine​

Requirements​

Run This Quckstart Example​

Project Structure​

The Playwright Setup​

The home.spec.ts Test Script​

Tracetest Library Setup​

Creating the Artillery Test Script​

Setting the Environment Variables​

Running the Full Example​

Finding the Results​

What's Next?​

Learn More​