Custom Observers

TramAI exposes observation through dev.tramai.core.observation. The primary surface is OperationObserver, which the engine calls once per provider attempt and which returns an OperationObservation for that attempt. Engine-owned resilience and routing events arrive through the same observation object, and EngineEventObserver provides a coarser engine-level hook.

What This Covers

• The OperationObserver / OperationObservation pair
• Observable engine events and their catalogue identities
• Custom metrics and alerting
• Diagnostic channels for provider, tool, and structured-output failures
• Standalone and Spring Boot registration

When To Use It

Use OperationObserver when you need:

• custom alerting on retries, circuit opens, DLP rejections, or token budget breaches
• cost tracking and token budget monitoring
• custom metrics in a non-OTel format (Prometheus, Micrometer, a custom dashboard)
• operational logging for security-relevant events
• per-attempt assertions in tests

For full OpenTelemetry tracing, use OpenTelemetryOperationObserver from tramai-observability — see Observability. For application tests, RecordingOperationObserver from tramai-testing is usually what you want.

The Observation Pair

OperationObserver is a fun interface (tramai-core/src/main/kotlin/dev/tramai/core/observation/OperationObservation.kt:24):

fun interface OperationObserver {
    fun onCallStarted(context: OperationCallContext): OperationObservation
}

It starts observation for a single provider attempt. OperationCallContext carries serviceInterface, methodName, providerId, requestedModel, and the zero-based attempt index.

The returned OperationObservation is a plain interface with these callbacks (OperationObservation.kt:34):

onEngineEvent, onCallCancelled, and the typed overload have defaults, so a minimal implementation only fills in what it needs.

onStructuredParseFailure is an ordinary observability surface: it no longer receives raw diagnostic material. The structured-output diagnostic observer is the sole privileged channel.

Minimum Setup

import dev.tramai.core.observation.OperationObservation
import dev.tramai.core.observation.OperationObserver
import dev.tramai.core.observation.OperationCallContext

class MyObserver : OperationObserver {
    override fun onCallStarted(context: OperationCallContext): OperationObservation =
        object : OperationObservation {
            override fun onProviderResponse(response: ModelResponse) = Unit

            override fun onProviderFailure(error: Throwable) = Unit

            override fun onStructuredParseFailure(rawResponse: String, errorSummary: String) = Unit

            override fun onCallCompleted(parseSuccess: Boolean?) = Unit
        }
}

Standalone Registration

val tramai = Tramai {
    provider(OpenAiProvider(System.getenv("OPENAI_API_KEY")), name = "openai", default = true)
    model("gpt-4o", "openai")
    observer(MyObserver())
}

Spring Boot Registration

@Bean
fun tramaiObserver(): OperationObserver = MyObserver()

TramAI's auto-configuration picks up OperationObserver, OperationInterceptor, DlpInterceptor, DlpRedactionAuditEmitter, and EngineEventObserver beans. DlpInterceptor and EngineEventObserver are single-bean surfaces; OperationInterceptor beans are composed in order.

Observable Engine Events

Event identities are declared in RuntimeEventCatalogue in tramai-core and referenced at compile time through the RuntimeEvents registry, so names and attribute keys are a stable contract in 0.6.0. Verified names include:

Attribute keys are the canonical RuntimeAttributeKey names in RuntimeAttributes.kt — for example tramai.event.name, provider_id, retry_index, delay_millis, failure_type, and reasonCode.

Example: Security Alerting

class SecurityAlertObserver : OperationObserver {
    override fun onCallStarted(context: OperationCallContext): OperationObservation =
        object : OperationObservation {
            override fun onProviderResponse(response: ModelResponse) = Unit
            override fun onProviderFailure(error: Throwable) = Unit
            override fun onStructuredParseFailure(rawResponse: String, errorSummary: String) = Unit
            override fun onCallCompleted(parseSuccess: Boolean?) = Unit

            override fun onEngineEvent(name: String, attributes: Map<String, Any?>) {
                when (name) {
                    "tramai.dlp.tool_result_rejected", "tramai.dlp.inspection_failed" ->
                        alertSecurityTeam("$name: $attributes")

                    "tramai.circuit.opened" ->
                        alertPlatformTeam("Provider circuit opened: ${attributes["providerId"]}")

                    else -> Unit
                }
            }
        }
}

Example: Custom Metrics

class MetricsObserver(
    private val meterRegistry: MeterRegistry,
) : OperationObserver {

    private val attempts = meterRegistry.counter("tramai.operation.attempts.local")
    private val parseFailures = meterRegistry.counter("tramai.parse.failures.local")
    private val circuitOpens = meterRegistry.counter("tramai.circuit.opens.local")

    override fun onCallStarted(context: OperationCallContext): OperationObservation =
        object : OperationObservation {
            override fun onProviderResponse(response: ModelResponse) = Unit
            override fun onProviderFailure(error: Throwable) = Unit

            override fun onStructuredParseFailure(rawResponse: String, errorSummary: String) {
                parseFailures.increment()
            }

            override fun onCallCompleted(parseSuccess: Boolean?) {
                attempts.increment()
            }

            override fun onEngineEvent(name: String, attributes: Map<String, Any?>) {
                if (name == "tramai.circuit.opened") circuitOpens.increment()
            }
        }
}

Diagnostic Channels

Three narrow diagnostic surfaces carry sanitized failure detail that the ordinary observation path deliberately does not receive. Each has a NoOp default:

Register the provider-side one through the builder with toolFailureDiagnosticObserver(...); defaults are NoOpProviderFailureDiagnosticObserver, NoOpToolFailureDiagnosticObserver, and NoOpStructuredOutputFailureDiagnosticObserver.

Relationship to OpenTelemetry

OpenTelemetryOperationObserver is itself an OperationObserver, so you can compose: register it alongside your own observer, and the engine dispatches to both. See Observability.

0.6.0 Constraints

• No sensitive-payload logging. Do not log prompts, responses, tool arguments, tokens, or credentials. verifyStaticSafetyGuards bans sensitive-payload logging in production code.
• No System.out / System.err. Use a logger.
• Cancellation must propagate. Call rethrowIfCancellation() from dev.tramai.core.coroutines.rethrowIfCancellation in any catch (e: Exception) block before wrapping or logging. onCallCancelled() is the observation hook for a cancelled attempt — it is not an error path.
• Observers are on the hot path. Heavy processing should be dispatched off the calling thread.

Limitations

• The (name, attributes) onEngineEvent callback is synchronous — do not block the invocation flow in it.
• There is no built-in event buffering or batching; implement it in your observer if you need it.
• At most one EngineEventObserver bean is supported in Spring Boot.

Next Steps

• Custom Interceptors — DLP, redaction, and request/response rewriting
• Observability — OpenTelemetry wiring
• Adding an Event — declare a new catalogue event
• Production Hardening — token budgets and circuit breakers