Skip to main content

Defining Tests as Text Files

One important aspect of testing your code is the ability to quickly implement changes while not breaking your application. If you change your application, it is important to be able to update your tests and run them against your new implementation as soon as possible for a timely development feedback loop.

As Tracetest is mainly a visual tool, this might make it difficult to update tests in an auditable way and execute those changes only when we are sure the application has been deployed with the new changes. With that in mind, we built a new way for you to define your tests: using a YAML test definition!

Motivation​

Imagine that you were assigned a ticket to improve your application database usage. You notice that every time a specific endpoint is called, your application executes N+1 select statements on the database instead of only one statement. You probably already have a test in place to ensure the correct functionality of that endpoint: it inserts the necessary information into the database, calls that specific endpoint using our tool and ensures you get the expected results using the trace generated by your application. It works fine, but there is a problem. That test is managed by Tracetest on its server and the test cannot be changed until the new patch is deployed. Otherwise, if the test is run using a non-patched version of the application, the test would fail.

To solve that, the best approach would be to enable developers to define their tests as text files and allow them to run those tests using a CLI, so you can integrate the execution of those tests to your existing CI pipeline. There are many benefits of this functionality for your tests:

  • Peers can review your tests before merging them to the main branch.
  • Ensure your test works before merging it to the main branch.
  • Have different versions of the same test running in parallel in different branches, so you and your peers can work on the same code modules and update the same test without interfering with each other.

Definition​

The definition can be broken into three parts: test information including triggering transaction, assertions, and outputs. Here is a real test we have on Tracetest to test our Pokeshop Demo API.

type: Test
spec:
  name: DEMO Pokemon - Import - Import a Pokemon
  description: "Import a pokemon"
  trigger:
    type: http
    httpRequest:
      url: http://demo-pokemon-api.demo/pokemon/import
      method: POST
      headers:
      - key: Content-Type
        value: application/json
      body: '{ "id": 52 }'
  specs:
    - selector: span[name = "POST /pokemon/import"]
      assertions:
        - attr:tracetest.span.duration <= 500ms
        - attr:http.status_code = 200
    - selector: span[name = "send message to queue"]
      assertions:
        - attr:messaging.message.payload contains 52
    - selector: span[name = "consume message from queue"]:last
      assertions:
        - attr:messaging.message.payload contains 52
    - selector: span[name = "consume message from queue"]:last span[name = "import pokemon
        from pokeapi"]
      assertions:
        - attr:http.status_code = 200
    - selector: span[name = "consume message from queue"]:last span[name = "save pokemon
        on database"]
      assertions:
        - attr:db.repository.operation = "create"
        - attr:tracetest.span.duration <= 500ms
  outputs:
    - name: POKEMON_ID
      selector: span[name = "POST /pokemon/import"]
      value: attr:http.response.body | json_path '.id'

Test Information​

Currently, you can define:

  • test name
  • test description
  • test id (if it is known)

Trigger​

This section defines how Tracetest will interact with your application. You can send an HTTP request, a GRPC call, send a message to a message broker, use an existing TraceID, etc.

The attribute type defines which trigger method you are going to use to interact with your application. The rest of the attributes in this section rely on the value you define there.

trigger:
  type: http|grpc|kafka|traceid|cypress|playwright|k6|artillery

Choose the kind of trigger to initiate the trace:

Or, choose to use an external integration to trigger Tracetest:

Specs​

This section defines test specifications. You can use the selector language to pick which spans to test against. Then, use the assertions to validate attribute values.

specs:
  - selector: span[tracetest.span.type="http"]
    name: "All HTTP Spans: Status  code is 200"
    assertions:
    - attr:http.status_code   =   200
  - selector: span[tracetest.span.type="database"]
    name: "All Database Spans: Processing time is less than 100ms"
    assertions:
    - attr:tracetest.span.duration < 2s

Outputs​

This section defines test outputs. You can define an output value from a test to use as input for subsequent tests.

outputs:
  - name: MY_OUTPUT
    selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    value: attr:name

Defining a polling profile​

Some tests have different requirements than others, it means that some will take longer to run than other tests, so it's crucial to be able to define the trace polling strategy at the test level.

type: Test
spec:
  name: my test
  pollingProfile: ./my-polling-profile.yaml
  trigger: http
  httpRequest:
    ...

This will make all runs from this test to use this specific test profile, and in case no value is defined, the test will use your default polling profile.