Skip to main content

Pokeshop and Grafana Cloud Tempo

note

Check out the source code on GitHub here.

Tracetest is a testing tool based on OpenTelemetry that allows you to test your distributed application. It allows you to use data from distributed traces generated by OpenTelemetry to validate and assert if your application has the desired behavior defined by your test definitions.

Grafana Tempo is an open-source, high-scale distributed tracing data store. It utilizes object storage to minimize costs, allowing you to store more information at a lower cost. It is deeply integrated with Grafana, Prometheus, and Loki. Grafana Tempo is also available as a paid hosted service at Grafana Cloud.

Grafana Cloud is a managed service offering from Grafana Labs that provides scalable and secure monitoring, visualization, and alerting for metrics, logs, and traces. It integrates seamlessly with various data sources, enabling users to create dynamic, interactive dashboards and gain insights into their system performance and reliability. With features like team collaboration, alerting, and plugins, Grafana Cloud is designed to simplify observability and help teams efficiently monitor their applications and infrastructure.

Pokeshop API is a testing ground, the team at Tracetest has implemented a sample instrumented API around the PokeAPI.

Pokeshop API with Grafana Cloud Tempo, OpenTelemetry and Tracetest​

This is a simple quick start guide on how to configure a fully instrumented API to be used with Tracetest for enhancing your E2E and integration tests with trace-based testing. The infrastructure will use Grafana Cloud Tempo as the trace data store and the Pokeshop API to generate the telemetry data.

Prerequisites​

Tracetest Account:

Grafana Cloud Account

Docker: Have Docker and Docker Compose installed on your machine.

Run This Quckstart Example​

The example below is provided as part of the Tracetest project. You can download and run the example by following these steps:

Clone the Tracetest project and go to the Grafana Cloud Tempo Pokeshop Quickstart:

git clone https://github.com/kubeshop/tracetest
cd tracetest/examples/tracetest-grafana-cloud-tempo-pokeshop

Follow these instructions to run the quick start:

  1. Copy the .env.template file to .env.
  2. Log into the Tracetest app.
  3. Fill out the token and API key details by editing your .env file. You can find these values in the Settings area for your environment.
  4. Run docker compose up -d.
  5. This example is configured to use the Grafana Tempo Tracing Backend. Ensure the environment you're using to run this example is configured to use the Grafana Tempo Tracing Backend by clicking on Settings, Tracing Backend, Grafana Tempo, Save. Or, use the CLI as explained below.
  6. Run tests from the Tracetest Web UI by accessing the app with the URL http://demo-api:8081/pokemon/import.

Follow along with the sections below for an in detail breakdown of what the example you just ran did and how it works.

Project Structure​

The project contains the Tracetest Agent, and the Pokeshop Demo app.

The docker-compose.yaml file in the root directory of the quick start runs the Pokeshop Demo app, OpenTelemetry Collector, and the Tracetest Agent setup.

Configuring the Pokeshop Demo App​

The Pokeshop API is a fully instrumented REST API that makes use of different services to mimic a real life scenario.

It is instrumented using the OpenTelemetry standards for Node.js, sending the data to the ADOT collector that will be pushing the telemetry information to both the AWS X-Ray service.

This is a fragment from the main tracing file from the Pokeshop API repo.

Configure the .env like shown below.

TRACETEST_API_KEY="<YOUR_TRACETEST_API_KEY>"
TRACETEST_API_TOKEN="<YOUR_TRACETEST_TOKEN>"

Configuring OpenTelemetry Collector​

Configure OpenTelemetry Collector to send traces to Grafana Cloud Tempo.

collector.config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    verbosity: detailed
  otlp:
    endpoint: tempo-us-central1.grafana.net:443
    headers:
      authorization: Basic <base64 encoded username:password>

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp]

Fill the authorization header in the collector.config.yaml file from your Grafana Tempo Setup. It should be encoded base64 with the format of username:token. Follow this guide to learn how.

Configuring Grafana Tempo​

Configure Tempo as a Tracing Backend:

tracetest-tracing-backend.yaml
---
type: DataStore
spec:
  id: tempo-cloud
  name: Tempo
  type: tempo
  tempo:
    type: http
    http:
      url: https://tempo-us-central1.grafana.net/tempo
      headers:
        authorization: Basic <base64 encoded username:password>
      tls: {}

Fill in the details of your Grafana Cloud Tempo instance by using the HTTP integration authorization header. It should be encoded base64 with the format of username:token. Follow this guide to learn how.

tracetest config -t <YOUR_API_TOKEN>
tracetest apply datastore -f ./tracetest-tracing-backend.yaml

Run the Pokeshop Demo App, OpenTelemetry Collector, and Tracetest Agent with Docker Compose​

The docker-compose.yaml file in the root directory contains the Pokeshop Demo app services.

The docker-compose.yaml file also contains the Tracetest Agent and OpenTelemetry Collector.

To start it, run this command:

docker compose up -d

This will start the Pokeshop Demo app, OpenTelemetry Collector, and Tracetest Agent.

Run Tracetest Tests​

  1. Open Tracetest
  2. Configure Grafana Tempo as a tracing backend if you have not already as explained above.
  3. Start creating tests! Make sure to use the http://demo-api:8081/pokemon/import URL in your test creation.
  4. To trigger tests in the CLI, first install the CLI, configure it, and run a test. From the root of the quick start directory, run:
tracetest configure -t <YOUR_API_TOKEN>
tracetest run test -f ./tests/test.yaml

Learn More​

Feel free to check out our examples in GitHub and join our Slack Community for more info!