Performance and Trace-Based Tests with Tracetest and Artillery Plugin

Commercial Feature

Feature available only in Cloud-based Managed Tracetest & Enterprise Self-hosted Tracetest.

Version Compatibility

This integration is compatible with Artillery v2.0.10 and above.

note

Check out the source code on GitHub here.

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

Artillery is a modern, powerful load-testing toolkit. Artillery is designed to help developers and testers simulate traffic to their applications, APIs, and microservices. It allows users to define scenarios to test how their systems behave under different loads.

Why is this important?

Artillery is it's a great tool in its own right that allows you to replicate most of the production challenges you might encounter. But, as with all of the tools that only test the initial transaction between the client side and the server, you can only run validations against the immediate response from the service.

The `artillery-plugin-tracetest` NPM Package

The artillery-plugin-tracetest package allows you to extend the Artillery's HTTP engine to enhance your performance tests with trace-based testing. It uses Artillery's plugin system to intercept requests and attaches the trace ID to the request headers. This allows you to run trace-based tests using the Tracetest API.

note

Do you want to use Tracetest Triggers as an entry point for your performance tests? Take a look at the artillery-engine-tracetest for a complete guide on how to do it.

Today You'll Learn How to integrate Trace-Based Tests with your Aritllery Test Scripts

This is a simple quick-start guide on how to use the Tracetest artillery-plugin-tracetest NPM package to enhance your Artillery Test Scripts with trace-based testing. The infrastructure will use the Pokeshop Demo as a testing ground, triggering requests against it and generating telemetry data.

Requirements

Tracetest Account:

Sign up to app.tracetest.io or follow the get started docs.
Create an environment.
Create an environment token.
Have access to the environment's agent API key.

Docker: Have Docker and Docker Compose installed on your machine.

Run This Quckstart Example

The example below is provided as part of the Tracetest project. You can download and run the example by following these steps:

Clone the Tracetest project and go to the TypeScript Quickstart:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/quick-start-artillery

Follow these instructions to run the included demo app and TypeScript example:

Copy the .env.template file to .env.
Log into the Tracetest app.
This example is configured to use Jaeger. Ensure the environment you will be utilizing to run this example is also configured to use the Jaeger Tracing Backend by clicking on Settings, Tracing Backend, Jaeger, updating the URL to jaeger:16685, Test Connection and Save.
Fill out the token and agent API key details by editing your .env file. You can find these values in the Settings area for your environment.
Run docker compose up -d.
Run npm i to install the required dependencies.
Run npm run test:plugin to run the example.
The output will show the test results and the Tracetest URL for each test run.
Follow the links in the log to view the test runs programmatically created by the Atillery execution.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

Project Structure

The quick-start Artillery project is built with Docker Compose.

The Pokeshop Demo App is a complete example of a distributed application using different back-end and front-end services. We will be launching it and running tests against it as part of this example.

The docker-compose.yaml file in the root directory of the quick start runs the Pokeshop Demo app, the OpenTelemetry Collector, Jaeger, and the Tracetest Agent setup.

The Artillery Plugin quick start has two primary files: a Test Script file plugin-test.yaml that defines the Artillery execution, and a Tracetest Definition file import-pokemon-trace-id.yaml that contains the specs and execution of the trace-based tests.

Installing the `artillery-plugin-tracetest` NPM Package

The first step when using the Artillery Plugin NPM package is to install the artillery-plugin-tracetest NPM Package. It is as easy as running the following command:

npm i artillery-plugin-tracetest

Once you have installed the artillery-plugin-tracetest package, you can use it as part of your Artillery Test Scripts to trigger trace-based tests and run checks against the resulting telemetry data.

Tracetest Test Definitions

The import-pokemon-trace-id.yaml file contains the YAML version of the test definitions that will be used to run the tests. It uses the Artillery trigger to execute requests against the Pokeshop Demo.

type: Test
spec:
  id: artillery-plugin-import-pokemon
  name: "Artillery Plugin: Import a Pokemon"
  trigger:
    type: traceid
    traceid:
      id: ${var:TRACE_ID}

  # test specs and assertions
  specs:
    - selector: span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"]
      name: "All HTTP Spans: Status  code is 200"
      assertions:
        - attr:http.status_code = 200
    - selector: span[tracetest.span.type="http" name="GET" http.method="GET"]
      assertions:
        - attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"
    - selector: span[tracetest.span.type="database"]
      name: "All Database Spans: Processing time is less than 1s"
      assertions:
        - attr:tracetest.span.duration < 1s
    - selector: span[tracetest.span.type="http" name="post" http.method="POST"]
      name: Response time should be fast
      assertions:
        - attr:response.time.ms < 100

  # generated output
  outputs:
    - name: DATABASE_POKEMON_ID
      selector: span[tracetest.span.type="database" name="create postgres.pokemon" db.system="postgres" db.name="postgres" db.user="postgres" db.operation="create" db.sql.table="pokemon"]
      value: attr:db.result | json_path '$.id'

Creating the Artillery Test Script

The plugin-test.yaml file contains the Artillery Test Script that will be used to trigger requests against the Pokeshop Demo and run trace-based tests. The steps executed by this script are the following:

Congures a phase that will execute 10 requests against the Pokeshop Demo url http://localhost:8081.
Defines two plugins, the public-metrics to send Artillery's telmetry data to the OpenTelemetry Collector and the tracetest plugin.
Includes the API token to access the Tracetest APIs and the Tracetest definition.
Defines the scenarios to execute triggering a request to the POST pokemon/import endpoint sending 6 (Charizard) as the Pokemon Id.

config:
  target: "http://localhost:8081"
  phases:
    - duration: 2
      arrivalRate: 5
  plugins:
    publish-metrics:
      - type: "open-telemetry"
        serviceName: "artillery"
        metrics:
          reporter: otlp-http
          endpoint: "http://localhost:4318/v1/metrics"
          attributes:
            environment: "test"
            tool: "Artillery"
            type: "Load test"

        traces:
          reporter: otlp-http
          endpoint: "http://localhost:4318/v1/traces"
          attributes:
            environment: "test"
            tool: "Artillery"
    tracetest:
      token: tttoken_dfa1e954d47d1857
      definition: import-pokemon-trace-id.yaml
scenarios:
  - name: "Import Pokemon"
    flow:
      - post:
          url: "/pokemon/import"
          json:
            id: 6

Running the Full Example

To start the full setup, run the following command:

docker compose up -d
npm run test:plugin

Finding the Results

The output from the Tracetest Plugin script should be visible in the console log after running the test command. This log will show links to Tracetest for each of the test runs invoked by the Artillery Testing Script. Click a link to launch Tracetest and view the test result.

> quick-start-artillery@1.0.0 test:plugin
> artillery run plugin-test.yaml

Test run id: t6pjz_zbaqcap7rhab59xn9kygn66bg8cfq_fqjc
Phase started: unnamed (index: 0, duration: 2s) 15:59:46(-0600)

Phase completed: unnamed (index: 0, duration: 2s) 15:59:48(-0600)

--------------------------------------
Metrics for period to: 15:59:50(-0600) (width: 2.509s)
--------------------------------------

http.codes.200: ................................................................ 10
http.downloaded_bytes: ......................................................... 80
http.request_rate: ............................................................. 10/sec
http.requests: ................................................................. 10
http.response_time:
  min: ......................................................................... 1
  max: ......................................................................... 2
  mean: ........................................................................ 1.6
  median: ...................................................................... 2
  p95: ......................................................................... 2
  p99: ......................................................................... 2
http.responses: ................................................................ 10
tracetest.tests_started: ....................................................... 10
vusers.created: ................................................................ 10
vusers.created_by_name.Import Pokemon: ......................................... 10


--------------------------------------
Metrics for period to: 16:00:10(-0600) (width: 5.71s)
--------------------------------------

tracetest.tests_failed: ........................................................ 2
tracetest.tests_succeeded: ..................................................... 1
vusers.completed: .............................................................. 3
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 16656.5
  max: ......................................................................... 22462.5
  mean: ........................................................................ 18737.5
  median: ...................................................................... 17158.9
  p95: ......................................................................... 17158.9
  p99: ......................................................................... 17158.9


All VUs finished. Total time: 30 seconds

--------------------------------
Summary report @ 16:00:17(-0600)
--------------------------------

http.codes.200: ................................................................ 10
http.downloaded_bytes: ......................................................... 80
http.request_rate: ............................................................. 5/sec
http.requests: ................................................................. 10
http.response_time:
  min: ......................................................................... 1
  max: ......................................................................... 2
  mean: ........................................................................ 1.6
  median: ...................................................................... 2
  p95: ......................................................................... 2
  p99: ......................................................................... 2
http.responses: ................................................................ 10
tracetest.tests_failed: ........................................................ 9
tracetest.tests_started: ....................................................... 10
tracetest.tests_succeeded: ..................................................... 1
vusers.completed: .............................................................. 10
vusers.created: ................................................................ 10
vusers.created_by_name.Import Pokemon: ......................................... 10
vusers.failed: ................................................................. 0
vusers.session_length:
  min: ......................................................................... 16656.5
  max: ......................................................................... 27889.9
  mean: ........................................................................ 23730.5
  median: ...................................................................... 25091.6
  p95: ......................................................................... 27181.5
  p99: ......................................................................... 27181.5
⠸ ✔ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/104) - trace id: 787ff20c554c1e21cc90a7c3eec8d2ee

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/105) - trace id: 44a208bbe579c513c5f79ac8838aeb46
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/106) - trace id: a677b4c7b68f4d692087c860686699fc
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/107) - trace id: e6e3963c0975367d89308acd4e3c379c
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/108) - trace id: 22684fc7a5d43ed9e44e24c7135a6aa5
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/109) - trace id: c73da8f9a009dc365f19f8701924c9cc
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/110) - trace id: 98c74bd6e6501f6721f058be4f5bf311
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/111) - trace id: 5c722a53b527f7334800dd42d98f5a49
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/112) - trace id: 53ea977eb682fc5dc416f0425dc0ac5e
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

✘ Artillery Plugin: Import a Pokemon (https://app.tracetest.io/organizations/ttorg_2179a9cd8ba8dfa5/environments/ttenv_231b49e808c29e6a/test/artillery-plugin-import-pokemon/run/113) - trace id: d1de80e71c85119b62c8495f3ec42e81
  > All HTTP Spans: Status  code is 200 (span[tracetest.span.type="general" name = "validate request"] span[tracetest.span.type="http"])

    • Expected: attr:http.status_code = 200

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.status_code" from. To fix this error, create a selector matching at least one span.
  > span[tracetest.span.type="http" name="GET" http.method="GET"]

    • Expected: attr:http.route = "/api/v2/pokemon/${var:POKEMON_ID}"

      No Spans
      Received: resolution error: there are no matching spans to retrieve the attribute "http.route" from. To fix this error, create a selector matching at least one span.
  > All Database Spans: Processing time is less than 1s (span[tracetest.span.type="database"])
  > Response time should be fast (span[tracetest.span.type="http" name="post" http.method="POST"])

note

Most of the tests will fail as the import Pokemon flow reads from memory if the info already exists. This is expected behavior.

View these tests in our Demo environment

👉 Join our shared Pokeshop API Demo environment.

What's Next?

After running the initial set of tests, you can click the run link for any of them, update the assertions, and run the scripts once more. This flow enables complete a trace-based TDD flow.

assertions

Learn More

Please visit our examples in GitHub and join our Slack Community for more info!

Why is this important?​

The artillery-plugin-tracetest NPM Package​

Today You'll Learn How to integrate Trace-Based Tests with your Aritllery Test Scripts​

Requirements​

Run This Quckstart Example​

Project Structure​

Installing the artillery-plugin-tracetest NPM Package​

Tracetest Test Definitions​

Creating the Artillery Test Script​

Running the Full Example​

Finding the Results​

What's Next?​

Learn More​

Why is this important?

The `artillery-plugin-tracetest` NPM Package

Today You'll Learn How to integrate Trace-Based Tests with your Aritllery Test Scripts

Requirements

Run This Quckstart Example

Project Structure

Installing the `artillery-plugin-tracetest` NPM Package

Tracetest Test Definitions

Creating the Artillery Test Script

Running the Full Example

Finding the Results

What's Next?

Learn More