Defining Tests as Text Files

One important aspect of testing your code is the ability to quickly implement changes while not breaking your application. If you change your application, it is important to be able to update your tests and run them against your new implementation as soon as possible for a timely development feedback loop.

As Tracetest is mainly a visual tool, this might make it difficult to update tests in an auditable way and execute those changes only when we are sure the application has been deployed with the new changes. With that in mind, we built a new way for you to define your tests: using a YAML test definition!

Motivation

Imagine that you were assigned a ticket to improve your application database usage. You notice that every time a specific endpoint is called, your application executes N+1 select statements on the database instead of only one statement. You probably already have a test in place to ensure the correct functionality of that endpoint: it inserts the necessary information into the database, calls that specific endpoint using our tool and ensures you get the expected results using the trace generated by your application. It works fine, but there is a problem. That test is managed by Tracetest on its server and the test cannot be changed until the new patch is deployed. Otherwise, if the test is run using a non-patched version of the application, the test would fail.

To solve that, the best approach would be to enable developers to define their tests as text files and allow them to run those tests using a CLI, so you can integrate the execution of those tests to your existing CI pipeline. There are many benefits of this functionality for your tests:

• Peers can review your tests before merging them to the main branch.
• Ensure your test works before merging it to the main branch.
• Have different versions of the same test running in parallel in different branches, so you and your peers can work on the same code modules and update the same test without interfering with each other.

Definition

The definition can be broken into three parts: test information including triggering transaction, assertions, and outputs. Here is a real test we have on Tracetest to test our Pokemon demo API:

type: Test
spec:
  name: DEMO Pokemon - Import - Import a Pokemon
  description: "Import a pokemon"
  trigger:
    type: http
    httpRequest:
      url: http://demo-pokemon-api.demo/pokemon/import
      method: POST
      headers:
      - key: Content-Type
        value: application/json
      body: '{ "id": 52 }'
  specs:
    - selector: span[name = "POST /pokemon/import"]
      assertions:
        - attr:tracetest.span.duration <= 500ms
        - attr:http.status_code = 200
    - selector: span[name = "send message to queue"]
      assertions:
        - attr:messaging.message.payload contains 52
    - selector: span[name = "consume message from queue"]:last
      assertions:
        - attr:messaging.message.payload contains 52
    - selector: span[name = "consume message from queue"]:last span[name = "import pokemon
        from pokeapi"]
      assertions:
        - attr:http.status_code = 200
    - selector: span[name = "consume message from queue"]:last span[name = "save pokemon
        on database"]
      assertions:
        - attr:db.repository.operation = "create"
        - attr:tracetest.span.duration <= 500ms
  outputs:
    - name: POKEMON_ID
      selector: span[name = "POST /pokemon/import"]
      value: attr:http.response.body | json_path '.id'

Test Information

Currently, you can only define the test name.

Trigger

This section defines how Tracetest will interact with your application: send an HTTP request, a GRPC call, send a message to a message broker, etc. Currently, only HTTP calls are supported, please let us know any other triggering mechanism that you require to test your application.

The attribute type defines which trigger method you are going to use to interact with your application. The rest of the attributes in this section rely on the value you define there.

HTTP Trigger

When defining a HTTP trigger, you are required to define a httpRequest object containing the request Tracetest will send to your system, this is where you define: url, method, headers, authentication, and body.

Note: Some APIs require the Content-Type header to respond. If you are not able to trigger your application, check if you are sending this header and if its value is correct.

trigger:
  type: http
  httpRequest:
    url: http://demo-pokemon-api.demo/pokemon/import
    method: POST
    headers:
    - key: Content-Type
      value: application/json
    body: '{ "id": 52 }'

Authentication

Currently, we support three authentication methods for HTTP requests: basic authentication, api key, and bearer token. Here is one example of each authentication method:

Basic Authentication

trigger:
    type: http
    httpRequest:
        url: http://my-api.com
        method: GET
        authentication:
            type: basic
            basic:
                user: my-username
                password: mypassword

API Key Authentication

trigger:
    type: http
    httpRequest:
        url: http://my-api.com
        method: GET
        authentication:
            type: apiKey
            apiKey:
                key: X-Key
                value: my-key
                in: header # Either "header" or "query"

Bearer Token Authentication

trigger:
    type: http
    httpRequest:
        url: http://my-api.com
        method: GET
        authentication:
            type: bearer
            bearer:
                token: my-token

Body

Currently, Testkube supports raw body types that enable you to send text formats over HTTP: JSON, for example.

trigger:
    type: http
    httpRequest:
        url: http://my-api.com
        method: POST
        body: '{"name": "my Json Object"}'

Generator Functions

Sometimes we want to randomize our test data. Maybe we want to try new values or maybe we know our API will fail if the same id is provided more than once. For this use case, you can define generator functions in the test trigger.

Generator functions can be invoked as part of expressions. Therefore, you only need to invoke it as uuid(). However, you might want to generate values and concatenate them with static texts as well. For this, you can use the string interpolation feature: "your user id is ${uuid()}.

Available functions:

Function	Description
uuid()	Generates a random v4 uuid.
firstName()	Generates a random English first name.
lastName()	Generates a random English last name.
fullName()	Generates a random English first and last name.
email()	Generates a random email address.
phone()	Generates a random phone number.
creditCard()	Generates a random credit card number (from 12 to 19 digits).
creditCardCvv()	Generates a random credit card cvv (3 digits).
creditCardExpDate()	Generates a random credit card expiration date (mm/yy).
randomInt(min, max)	Generates a random integer contained in the closed interval defined by [min, max].
date(format?)	Get the current date and formats it. Default is YYYY-MM-DD but you can specify other formats.
dateTime(format?)	Get the current datetime and formats it. Default is RFC3339 but you can specify other formats.

tip

Continue reading about date and datetime formats here.

tip

Continue reading about Test Specs here.

tip

Continue reading about Test Outputs, here.