Testing Reference

TramAI testing support lives in tramai-testing and has two layers:

• Application-test layer — deterministic provider fakes, a recording observer, an assertion DSL, and a mock tool. Use these in your own tests.
• Contract layer — shared compatibility TCKs for providers and persistence stores. These are the same suites the TramAI adapters run; use them when you implement your own ModelProvider, store, or memory backend.

tramai-testing is classified internal maturity but is a published artifact in 0.6.0.

Application-Test Layer

MockAiProvider

val provider = MockAiProvider {
    onMethod("analyze") respondWith """{"status":"ok"}"""
}

• providerId() is mock.
• Requests are captured in invocation order on provider.requests.
• Responses are queued per method name; when a queue is exhausted the last response repeats. The request must carry operationMethod, which the engine sets for service operations.

SimulatedFailureProvider

val provider = SimulatedFailureProvider {
    onMethod("analyze").retryableFailure("rate limited", statusCode = 429)
    onMethod("analyze") respondWith """{"status":"ok"}"""
}

Available per-method outcomes: respondWith(String), respondWith(ModelResponse), retryableFailure(message, statusCode), nonRetryableFailure(message, statusCode), failWith(Throwable).

RecordingOperationObserver

observer.callRecords holds one CallRecord per provider attempt, with the call context, the response, the provider failure, the structured-parse failure summary, the final parse outcome, and any engine events.

MockTool

val tool = MockTool.fixed<LookupInput, LookupResult>(
    name = "vendor_lookup",
    description = "Looks up a vendor.",
    response = LookupResult("ok"),
)

tool.calls records every input it received.

TramaiAssertions

TramaiAssertions.assertThat(provider, observer)
    .whenCalled("analyze")
    .wasCalledTimes(2)
    .andRetried(1)
    .andParsedSuccessfully()
    .emittedProvider("mock")

MethodAssertion also provides andObservedParseFailure(), andObservedFailure(KClass), andFailedWith(KClass), andCalledTool(name), and andCalledToolTimes(name, expected).

Complete Application Test

@AiService
interface Analyzer {
    @Operation(prompt = "Analyze the invoice.", model = "gpt-4o")
    suspend fun analyze(invoiceId: String): Status
}

data class Status(val status: String)

@Test
fun `analyze returns the parsed status`() = runBlocking {
    val provider = MockAiProvider {
        onMethod("analyze") respondWith """{"status":"ok"}"""
    }
    val observer = RecordingOperationObserver()

    val tramai = Tramai {
        provider(provider, default = true)
        model("gpt-4o", "mock")
        observer(observer)
    }

    tramai.use {
        val service = it.create<Analyzer>()
        assertEquals(Status("ok"), service.analyze("invoice-1"))
    }

    TramaiAssertions.assertThat(provider, observer)
        .whenCalled("analyze")
        .wasCalledTimes(1)
        .andParsedSuccessfully()
        .emittedProvider("mock")
}

In Spring tests, the equivalent move is to replace the real provider bean with a deterministic ModelProvider bean and keep the rest of the context as it is.

Contract Layer: Shared TCKs

TCK classes live on the tramai-testing test-fixtures surface. You add them to your test source set and extend the abstract TCK class.

dependencies {
    testImplementation(platform("dev.tramai:tramai-bom:0.6.0"))
    testImplementation("dev.tramai:tramai-testing")
    testImplementation(testFixtures("dev.tramai:tramai-testing:0.6.0"))
}

Provider TCK

class MyProviderTckTest : ProviderTck() {
    private val observer = RecordingProviderFailureDiagnosticObserver()

    override val harness = ProviderTckHarness(
        expectedProviderId = "my-provider",
        expectedCapabilities = setOf(ProviderCapability.TOOL_CALLING, ProviderCapability.STREAMING),
        createProvider = { stubHttpClient -> MyProvider(httpClient = stubHttpClient) },
        diagnosticObserver = { observer },
        happyPathBody = "<protocol-shaped text completion>",
        happyPathExpectedContent = "Hello!",
        emptyBody = "<protocol-shaped empty completion>",
        malformedBody = "<protocol-shaped truncated payload>",
        tools = ToolSpec(
            toolCallBody = "<response with a tool call>",
            toolOnlyBody = "<response with only a tool call>",
            expectedToolCall = TCK_TOOL_CALL,
            requestToolNameMarker = "get_weather",
            requestToolSchemaMarker = "\"parameters\"",
        ),
        streaming = StreamingSpec(
            body = "<full stream body>",
            malformedBody = "<stream truncated mid-event>",
            expectedTokens = listOf("Hel", "lo!"),
        ),
    )
}

Rules the harness enforces:

• expectedProviderId and expectedCapabilities are pinned by you, not read from the provider — a provider cannot make a contract test disappear by changing its own supportsCapability(...).
• Declaring a capability requires the matching spec (TOOL_CALLING requires tools, VISION requires vision, STRUCTURED_OUTPUT requires structuredOutput, STREAMING requires streaming).
• HTTP providers must wire a diagnosticObserver, because the bounded-preview contract is asserted.

What ProviderTck covers, offline and with no credentials: provider identity, the exact capability set in both directions, cancellation escaping as cancellation (never wrapped in ProviderException), safe error codes and sanitized public messages, no credential or body leakage into public exceptions, empty and malformed responses, usage normalization, response-body closure on every path, request timeout propagation, retryable versus non-retryable status mapping, numeric Retry-After handling, the bounded diagnostic preview, tool-calling serialization and parsing, vision payload encoding, structured-output declaration, and streaming token order plus exactly one terminal Complete.

This is the important correction to older documentation: the HTTP wire contract of a provider is covered by a shared TCK — status mapping, Retry-After, timeouts, and bounded error bodies are all asserted against a stub HTTP client. What the TCK does not do is talk to a live endpoint, so a smoke test against the real service is still yours to run.

Store TCKs

Store TCKs are backend-agnostic on purpose. The ChatMemoryStoreTckHarness exposes two distinct store objects (primary and peer) over the same physical backend, so concurrency cases prove backend-level coordination rather than a per-instance mutex. Each call to a harness factory must return a fresh, isolated store — the runner owns temp directories, encryption keys, data sources, schema creation, and cleanup.

Example enrollment:

class MyApprovalStoreTckTest : ApprovalStoreTck() {
    override val harness = object : ApprovalStoreTckHarness {
        override fun createStore(clock: MutableClock): ApprovalStore =
            MyApprovalStore(clock = clock)
    }
}

Time is deterministic: MutableClock (approval and continuation TCKs) and MutableMillisClock (lease and chat-memory TCKs) are advanced explicitly by the TCK. There are no real sleeps in the contract suites.

Enrollment Gates

Enrollment is enforced, not suggested. Architecture tests fail the build when a provider or store loses its TCK runner — ProviderTckEnrollmentArchitectureTest, ApprovalStoreTckEnrollmentArchitectureTest, ApprovalContinuationStoreTckEnrollmentArchitectureTest, AuditStoreTckEnrollmentArchitectureTest, SuspendedInvocationStoreTckEnrollmentArchitectureTest, WorkflowCheckpointStoreTckEnrollmentArchitectureTest, WorkflowLeaseStoreTckEnrollmentArchitectureTest, WorkflowLeaseCheckpointFenceTckEnrollmentArchitectureTest, ChatMemoryStoreTckEnrollmentArchitectureTest, and SovereignOpsAuditOutboxStoreTckEnrollmentArchitectureTest.

Property And State-Machine Tests

Alongside the hand-written TCK cases, the persistence contracts are exercised as model-based property tests. Each family ships a lifecycle model and an action generator — ApprovalLifecycleModel / ApprovalLifecycleActionGenerator, ApprovalContinuationLifecycleModel, WorkflowCheckpointLifecycleModel, WorkflowLeaseLifecycleModel, SovereignOpsAuditOutboxLifecycleModel — that generate random operation sequences and compare the real store against a pure in-memory oracle.

State-machine suites cover step execution, supervisor recovery, lease coordination, and drain budgets. These are how TramAI tests its own stores; if you implement a store, they are the model of what your implementation should satisfy.

Consumer Compilation Gates

The stable surface is also tested by compilation, not just by behavior: examples/kotlin-consumer-smoke and examples/java-consumer-smoke compile real consumer code against the published artifacts. A change that breaks an ordinary application consumer fails CI. See API Stability.

What To Test Where

Related Pages

• Testing TramAI Code
• Writing A Custom Provider
• API Stability
• Quality Gates
• CI Integration