Synthetic Monitoring with Trace-based Playwright Tests

Version Compatibility

The features described here are compatible with the Tracetest CLI v1.4.1 and above.

note

Check out the source code on GitHub here.

Tracetest is a synthetic monitoring and testing tool based on OpenTelemetry that allows you to test distributed apps. You can use data from distributed traces generated by OpenTelemetry to validate and assert the functionality of your apps.

Playwright is an open-source automation framework developed by Microsoft that enables cross-browser automation for web applications. It provides a set of APIs and libraries for automating interactions with web browsers such as Chrome, Firefox, and Microsoft Edge.

Why is this important?

This recipe uses the Tracetest Playwright Engine trigger with Tracetest Monitors for synthetic monitoring.

With these two working together you can combine the power of end-to-end tests with trace-based testing to easily capture a full distributed trace from your OpenTelemetry instrumented front-end and back-end system, but also run them on a schedule with enabled alerting when tests fail.

Benefits of using traces as test specs alongside synthetic monitoring:

Get faster MTTR for failing performance tests
Assert against the Playwright test execution and the system under test
Validate functionality of other parts of your system that may be broken, even when end-to-end tests are passing
Create synthetic tests that run in defined intervals
Get alerted when synthetic tests fail via a Webhook

Requirements

Tracetest Account:

Sign up to app.tracetest.io or follow the get started docs.
Have access to the environment's agent API key.

Docker: Have Docker and Docker Compose installed on your machine.

Run This Example

View Playwright recipe

This recipe uses the official Playwright Engine recipe.

The example below is provided as part of the Tracetest GitHub repo. You can download and run the example by following these steps:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/tracetest-synthetic-monitoring

Follow these instructions to run the quick start:

Copy the .env.template file to .env.
Fill out the TRACETEST_TOKEN and ENVIRONMENT_ID details by editing your .env file.
Run docker compose run tracetest-run.
Follow the links in the output to view the test results.

Project Structure

The project structure for running Tracetest Playwright Engine tests is as follows:

.env.template
.gitignore
.Dockerfile
collector.config.yaml
docker-compose.yaml
/resources
  apply.sh
  datastore.yaml
  test.yaml
  script.js
  monitor.yaml

The Pokeshop Demo App is a complete example of a distributed application using different back-end and front-end services. We will be launching it and running tests against it as part of this example. The docker-compose.yaml file in the root directory of the quick start runs the Pokeshop Demo app, the OpenTelemetry Collector, Jaeger, and the Tracetest Agent setup.

The Tracetest resource definitions and scripts are defined under the /resources directory. The resources include tests and the tracing backend definition, while the scripts include the apply.sh script to apply the resources.

Provisioned Resources

The example provisions the following resources:

Import Pokemon Test

resources/test.yaml
type: Test
spec:
  id: import-pokemon
  name: Import Pokemon
  trigger:
    type: playwrightengine
    playwrightEngine:
      target: http://api:8081
      script: ./script.js
      method: importPokemon
  specs:
    - selector: span[tracetest.span.type="general" name="documentLoad"]
      name: Document Load Should be fast
      assertions:
        - attr:tracetest.span.duration < 500ms
    - selector: span[tracetest.span.type="http" http.scheme="http"]
      name: All HTTP request should return 200
      assertions:
        - attr:http.status_code = 200
    - selector: span[tracetest.span.type="messaging" name="queue.synchronizePokemon
        process" messaging.system="rabbitmq"
        messaging.destination="queue.synchronizePokemon"
        messaging.operation="process"]
      name: The worker should be processed
      assertions:
        - attr:tracetest.selected_spans.count = 1
    - selector: span[tracetest.span.type="database"]
      name: "All Database Spans: Processing time is less than 100ms"
      assertions:
        - attr:tracetest.span.duration  <  250ms

Playwright Synthetic Monitor

resources/monitor.yaml
type: Monitor
spec:
  id: playwright-monitor
  name: Playwright Synthetic Monitor
  enabled: true
  variableSetId: ""
  tests:
  - import-pokemon
  schedule:
    cron: "*/5 * * * *"
    timeZone: Etc/UTC
  alerts:
  - id: slack-webhook-message
    type: webhook
    webhook:
      body: "{\n \"text\": \"Monitor ${.Monitor.Name} has failed, follow the link to find the <${.URL}|results>\"\n}"
      method: POST
      url: <your-webhook-url>
      headers:
      - key: Content-Type
        value: application/json
    events:
    - FAILED

Playwright Script

resources/script.js
const { expect } = require("@playwright/test");

async function importPokemon(page) {
  expect(await page.getByText("Pokeshop")).toBeTruthy();

  await page.click("text=Import");
  await page.getByLabel("ID").fill("143");

  await Promise.all([
    page.waitForResponse((resp) => resp.url().includes("/pokemon/import") && resp.status() === 200),
    page.getByRole("button", { name: "OK", exact: true }).click(),
  ]);
}

module.exports = { importPokemon };

Jaeger Tracing Backend

resources/datastore.yaml
type: DataStore
spec:
  id: current
  name: jaeger
  type: jaeger
  default: true
  jaeger:
    endpoint: jaeger:16685
    headers:
      "": ""
    tls:
      insecure: true

The Apply Script

The apply script configures and provisions the resources in the Tracetest environment:

resources/apply.sh
#!/bin/sh

set -e

TOKEN=$TRACETEST_API_KEY
ENVIRONMENT_ID=$TRACETEST_ENVIRONMENT_ID

apply() {
  echo "Configuring Tracetest"
  tracetest configure --token $TOKEN --environment $ENVIRONMENT_ID

  echo "Applying Resources"
  tracetest apply datastore -f /resources/datastore.yaml
  tracetest apply test -f /resources/import-pokemon.yaml
  tracetest apply monitor -f /resources/monitor.yaml
  tracetest list monitor
}

apply

resources/run.sh
#!/bin/sh

set -e

TOKEN=$TRACETEST_TOKEN
ENVIRONMENT_ID=$TRACETEST_ENVIRONMENT_ID

run() {
  echo "Configuring Tracetest"
  tracetest configure --token $TOKEN --environment $ENVIRONMENT_ID

  echo "Running Trace-Based Tests..."
  tracetest run test -f /resources/test.yaml
  tracetest list monitor
}

run

Setting the Environment Variables

Copy the .env.template file to .env and add the Tracetest API token and environment id to the TRACETEST_TOKEN and TRACETEST_ENVIRONMENT_ID variables.

Running the Full Example

Creating the resources and running the tests is automated for you. You only need to run the following command:

docker compose run tracetest-run

Viewing the Created Resources

The output from the Tracetest resource run script should be visible in the console log after running the run command.

Output
WARN[0000] /Users/oscar/Documents/kubeshop/t/examples/tracetest-synthetic-monitoring/docker-compose.yaml: `version` is obsolete
[+] Running 2/2
 ✔ api Pulled                                                                                                                                                                        0.8s
 ✔ worker Pulled                                                                                                                                                                     1.1s
[+] Creating 9/9
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-otel-collector-1   Running                                                                                             0.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-queue-1            Running                                                                                             0.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-postgres-1         Running                                                                                             0.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-jaeger-1           Running                                                                                             0.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-cache-1            Running                                                                                             0.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-worker-1           Running                                                                                             0.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-tracetest-agent-1  Recreated                                                                                           0.1s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-api-1              Running                                                                                             0.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-tracetest-apply-1  Recreated                                                                                           0.1s
[+] Running 6/6
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-tracetest-agent-1  Started                                                                                             0.3s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-queue-1            Healthy                                                                                             1.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-postgres-1         Healthy                                                                                             1.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-cache-1            Healthy                                                                                             1.0s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-api-1              Healthy                                                                                             0.5s
 ✔ Container tracetest-synthetic-monitoring-playwright-engine-tracetest-apply-1  Started                                                                                             0.1s
[+] Running 2/2
 ✔ worker Pulled                                                                                                                                                                     1.0s
 ✔ api Pulled                                                                                                                                                                        0.9s
Configuring Tracetest
 SUCCESS  Successfully configured Tracetest CLI
Running Trace-Based Tests...
✔ RunGroup: #2c3uThCSg (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments/ttenv_b42fa137465c6e04/run/2c3uThCSg)
 Summary: 1 passed, 0 failed, 0 pending
  ✔ Import Pokemon (https://app.tracetest.io/organizations/ttorg_ced62e34638d965e/environments/ttenv_b42fa137465c6e04/test/import-pokemon/run/66/test) - trace id: f09a1370ec7a6178aa9ba398f05c2f0f
	✔ Document Load Should be fast
	✔ All HTTP request should return 200
	✔ The worker should be processed
	✔ All Database Spans: Processing time is less than 100ms

 ID                   NAME                            VERSION   RUNS   LAST RUN TIME         LAST RUN STATE   URL
-------------------- ------------------------------- --------- ------ --------------------- ---------------- -----------------------------------------------------
 playwright-monitor   Playwright Synthetic Monitor    1         0                                             https://api.tracetest.io/monitor/playwright-monitor
 JtFYYc9SR            Quick Start Synthetic Monitor   4         6      2024-08-15 19:01:53                    https://api.tracetest.io/monitor/JtFYYc9SR

Running Synthetic Monitoring

Select Monitors in the Tracetest sidebar.
You'll see the Monitor that was created in the run script.
It will start running based on the schedule you selected. With this setup, your Monitor will trigger the Playwright test every 5 minutes.

Learn More

Please visit our examples in GitHub and join our Slack Community for more info!

Why is this important?​

Requirements​

Run This Example​

Project Structure​

Provisioned Resources​

Import Pokemon Test​

Playwright Synthetic Monitor​

Playwright Script​

Jaeger Tracing Backend​

The Apply Script​

Setting the Environment Variables​

Running the Full Example​

Viewing the Created Resources​

Running Synthetic Monitoring​

Learn More​