Testing & Observability
Reliable AI integration requires two things: deterministic validation before release, and clear runtime visibility after release. This section groups the guides that deliver both.
In This Section
| Guide | Focus |
|---|---|
| Testing TramAI Code | Mock providers, failure simulation, the assertion DSL, and the shared contract TCKs |
| Observability | OpenTelemetry spans, metrics, token tracking, and workflow-level visibility |
| CI Integration | Running deterministic AI tests in a pipeline, and the gate set TramAI itself enforces |
When To Start Here
Start here when you already understand the core APIs and want confidence, traceability, and operational visibility around them.
What 0.6.0 Changed Here
Two things are worth knowing before you read on:
- Contract TCKs now ship.
tramai-testingcontainsProviderTckplus the shared store TCKs (approval, continuation, audit, suspended invocation, checkpoint, lease, chat memory, ops outbox), and enrolment is architecture-enforced. If you implement a provider or a store, the TCK is the contract. tramai-observabilitystays optional at the dependency level. If you do not add the module, TramAI runs with zero OpenTelemetry dependencies. Event, attribute, and metric names are declared inRuntimeEventCatalogue, so they are a stable contract rather than an implementation detail.
If You Are Extending TramAI
The testing surfaces back the extension points directly:
| You are adding | Start with |
|---|---|
| A provider adapter | Adding a Provider, then Custom Providers |
| A persistence store | Adding a Store |
| A runtime event | Adding an Event |
| An interceptor or observer | Custom Interceptors, Custom Observers |
Next Steps
- Contributor Change Guides — every change type, with its mandatory contract tests
- Structured Output — the parse, validate, and repair contract
- Providers & Routing — routing, fallback, and resilience
- Production Hardening — budgets, circuit breakers, secure defaults