Skip to main content

Sampling Tracetest Spans

Suppose you are considering Tracetest to run tests against a high-volume environment, such as production. In that case, you probably will hit some drawbacks, such as having your test spans not sampled by your probabilistic sampler. There are a couple of things that you can do to avoid those problems:

Add a Separate Pipeline for Tracetest in your OpenTelemetry Collector​

Your existing OpenTelemetry Collector already receives traces from your applications and sends them to your datastore and you have a set of processors configured to ensure the quality of the traces your datastore is receiving. It probably looks like this one:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 5.0

  batch:

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces/jaeger:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/jaeger]

We don't want to change this pipeline for your testing to work, so let's create a new pipeline for collecting only traces generated by Tracetest tests:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 5.0

  # If this configuration fails on your collector, make sure to update it to a newer version.
  # This is the recommended way of filtering spans based on the `trace_state`. It's faster and less
  # resource intensive than using a `tail_sampling` approach.
  filter/tracetest:
    error_mode: ignore
    traces:
      span:
        - 'trace_state["tracetest"] != "true"'

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces/jaeger:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/jaeger]

  pipelines:
    traces/tracetest:
      receivers: [otlp]
      processors: [filter/tracetest, batch]
      exporters: [otlp/jaeger]

Tail Sampling Approach​

Before December 2023, we were suggesting people to use tail sampling to filter the traces generated by Tracetest. However, the new filter capabilities are better for performance than tail sampling as it requires less memory to decide if a trace should be sampled or not. If you are not concerned with memory usage and still want to use the tail sampling approach, this is the processor you can use:

processors:
  tail_sampling:
    decision_wait: 5s
    policies:
      - name: tracetest-spans
        type: trace_state
        trace_state: { key: tracetest, values: ["true"] }

With this configuration, you will still get 5% of all your traces, but you will also ensure that all your test traces are collected and sent to Jaeger.

When exposing something to the internet, it's common to ignore all unknown headers from an HTTP request. This usually includes the traceparent and tracestate headers, which are the two headers defined in the W3C specification.

However, supporting those two fields can be beneficial. For starters, you can start testing your application in production using Tracetest. If your product APIs are used directly by your customers, you can also share the same Trace-ID as them and use this to correlate a faulty operation on their side with your trace.