Observability

What it is: TramAI provides OpenTelemetry instrumentation through OpenTelemetryOperationObserver. It records one span per provider attempt, emits latency and token metrics, and works with your existing OpenTelemetry pipeline.

When to use it: Production deployments where you need visibility into AI call latency, token consumption, error rates, retry behaviour, and provider attribution.

Optionality rule: tramai-observability is optional at the dependency level, and it stays that way. If you do not add the module, TramAI runs with zero OpenTelemetry dependencies — no transitive OTel SDK, no exporter, no runtime cost. Do not add it "just in case"; add it when you have a collector to send to.

Dependency

Declare the platform once, then the module without a version:

dependencies {
    implementation(platform("dev.tramai:tramai-bom:0.6.0"))
    implementation("dev.tramai:tramai-observability")
}

Basic Setup

import io.opentelemetry.api.OpenTelemetry
import dev.tramai.observability.OpenTelemetryOperationObserver

val observer = OpenTelemetryOperationObserver(openTelemetry)

val tramai = Tramai {
    provider(OpenAiProvider(System.getenv("OPENAI_API_KEY")), name = "openai", default = true)
    model("gpt-4o", "openai")
    observer(observer)
}

OpenTelemetryOperationObserver is an OperationObserver, so it composes with any observer of your own. It can be constructed either from an OpenTelemetry instance or directly from a Tracer plus an optional Meter; the instrumentation name defaults to dev.tramai.observability.

Every provider attempt now produces a span, and metrics are emitted when your SDK has a meter provider configured.

What Is Recorded

Spans

One span per provider attempt, named ai.<methodName>.

Set when the attempt starts:

Set when the provider responds:

Set when the attempt completes:

Span events:

A cancelled attempt ends with tramai.outcome = cancelled and is recorded as a non-error terminal — cancellation is not a failure.

Metrics

The tramai.engine.events counter carries a tramai.event.name attribute identifying the specific event — for example tramai.retry.scheduled, tramai.route.selected, tramai.circuit.opened, or tramai.token_budget.soft_limit_exceeded. Event names and attribute keys are declared in RuntimeEventCatalogue, so they are a stable contract rather than an implementation detail.

Token Usage Tracking

Token counts appear when the provider reports them:

• span attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
• counters tramai.operation.input_tokens and tramai.operation.output_tokens
• histograms tramai.operation.input_tokens.per_attempt and tramai.operation.output_tokens.per_attempt

OpenAI, Anthropic, and Gemini report usage per response; Ollama varies by model. Where a provider does not report usage, the attributes and per-attempt observations are simply absent — nothing is estimated.

Error Monitoring

Workflow-Level Visibility

When you use tramai-orchestration, OpenTelemetryWorkflowObserver adds a workflow trace layer above the provider-attempt spans. OpenTelemetryTramaiWorkerObserver covers worker lifecycle.

It records:

• one span per workflow run, with start and completion
• checkpoint load and save events
• lease claim, renew, and release events
• step start, completion, and failure events
• workflow counters and durations through tramai.workflow.runs, tramai.workflow.duration, and tramai.workflow.events

The stable workflow correlation model is the pair tramai.workflow.name and tramai.workflow.id. Reusing the same workflowId across different workflow definitions is supported; running two concurrent executions with the same workflow name and the same workflowId is a caller error — use unique workflow IDs or lease-based ownership.

A retry renders as sibling attempt spans under one operation span:

Method: SupportAgent.handle()
 └── [Span] ai.handle (gen_ai.system=ollama, gen_ai.request.model=gemma4:e2b)
      ├── [Span] Provider Call Attempt 1
      │    └── Event: tramai.parse.failure
      ├── [Span] Provider Call Attempt 2
      │    └── Event: tramai.parse.failure
      └── [Span] Provider Call Attempt 3 → Success

Why The Observer Sits Outside Providers

Providers create requests, do HTTP transport, and map responses. The engine wraps provider calls with the observation hooks, so providers carry no tracing logic and every provider — including one you write yourself — gets observability for free. Remove the observer and no tracing code runs.

Spring Boot

Expose the observer as a bean and auto-configuration picks it up:

@Configuration
class ObservabilityConfig {
    @Bean
    fun tramaiObserver(openTelemetry: OpenTelemetry): OpenTelemetryOperationObserver =
        OpenTelemetryOperationObserver(openTelemetry)
}

Spring Boot's OpenTelemetry auto-configuration provides the OpenTelemetry instance. TramAI's auto-configuration resolves any OperationObserver bean.

Verifying Traces In Tests

Assert emitted spans and metrics with InMemorySpanExporter and InMemoryMetricReader from the OpenTelemetry SDK testing artifacts. Match the version to your OTel SDK; TramAI does not pin it for you.

class MyObserverTest {
    private val spanExporter = InMemorySpanExporter.create()
    private val metricReader = InMemoryMetricReader.create()

    private val tracerProvider = SdkTracerProvider.builder()
        .addSpanProcessor(SimpleSpanProcessor.create(spanExporter))
        .build()

    private val meterProvider = SdkMeterProvider.builder()
        .registerMetricReader(metricReader)
        .build()

    private val openTelemetry = OpenTelemetrySdk.builder()
        .setTracerProvider(tracerProvider)
        .setMeterProvider(meterProvider)
        .build()

    @AfterEach
    fun tearDown() = spanExporter.reset()
}

Asserting Spans

@Test
fun `records span attributes for a successful provider call`() = runBlocking {
    val provider = MockAiProvider {
        onMethod("classify") respondWith """{"priority":"high"}"""
    }

    val tramai = Tramai {
        provider(provider, default = true)
        model("gpt-4o", "mock")
        observer(OpenTelemetryOperationObserver(openTelemetry))
    }

    tramai.create<Classifier>().classify("Urgent issue")

    val span = spanExporter.finishedSpanItems.single()
    assertThat(span.name).isEqualTo("ai.classify")
    assertThat(span.attributes.asMap())
        .containsEntry(AttributeKey.stringKey("gen_ai.system"), "mock")
        .containsEntry(AttributeKey.stringKey("gen_ai.request.model"), "gpt-4o")
        .containsEntry(AttributeKey.stringKey("tramai.operation.method"), "classify")
}

Asserting Metrics

val metrics = metricReader.collectAllMetrics()

val attempts = metrics.single { it.name == "tramai.operation.attempts" }
assertThat(attempts.longSumData.points.single().value).isEqualTo(1L)

val duration = metrics.single { it.name == "tramai.operation.duration" }
assertThat(duration.histogramData.points.single().count).isEqualTo(1L)

Asserting Parse Failures And Retries

val provider = MockAiProvider {
    onMethod("classify") respondWith "not json"
    onMethod("classify") respondWith """{"priority":"high"}"""
}

// ... run the call ...

// One span per attempt
assertThat(spanExporter.finishedSpanItems).hasSize(2)

val firstAttempt = spanExporter.finishedSpanItems.first()
assertThat(firstAttempt.events).anySatisfy { event ->
    assertThat(event.name).isEqualTo("tramai.parse.failure")
}

val parseFailures = metricReader.collectAllMetrics()
    .single { it.name == "tramai.operation.parse_failures" }
assertThat(parseFailures.longSumData.points.single().value).isEqualTo(1L)

Limitations

• No automatic exporter setup. You configure the OTel SDK, meter provider, and exporter pipeline yourself.
• No dashboards. The module provides instrumentation, not pre-built panels.
• No extra trace-correlation helpers beyond standard OpenTelemetry usage.
• No per-chunk streaming metrics. Streaming operations produce completion metrics, not per-chunk metrics.
• Provider-specific attributes vary. Anthropic thinking tokens and similar provider details are captured only when the provider reports them.

Next Steps

• Testing — the mock providers and assertion DSL used above
• CI Integration — running these tests in a pipeline
• Structured Output — the parse-and-repair contract behind tramai.parse.failure
• Production Hardening — token budgets and circuit breakers