Testing TramAI Code

What it is: The tramai-testing module provides mock providers, failure simulation, request recording, and assertion utilities so you can test AI-dependent application code without network calls.

When to use it: Every test that involves a TramAI service. Never hit a live model in CI. Use mock providers for fast, deterministic, free tests.

What the Testing Module Provides

Component	Purpose
`MockAiProvider`	Deterministic in-memory provider with configurable responses per method
`SimulatedFailureProvider`	Simulates provider-side failures (rate limits, timeouts)
`RecordingOperationObserver`	Captures call metadata for assertions
`TramaiAssertions`	Fluent assertion DSL for verifying call counts, retries, parse success
`MockTool`	Create lightweight mock tools for tool-calling tests

Minimum Setup

Add the dependency:

dependencies {
   testImplementation("dev.tramai:tramai-testing:0.3.1")
}

Basic Test Pattern

// 1. Create a mock provider with canned responses
val provider = MockAiProvider {
   onMethod("analyze") respondWith """{"status":"ok"}"""
}

// 2. Create a recording observer
val observer = RecordingOperationObserver()

// 3. Build TramAI with the mock
val tramai = Tramai {
   provider(provider, default = true)
   model("gpt-4o", "mock")
   observer(observer)
}

// 4. Create and call the service
val service = tramai.create<Analyzer>()
val result = runBlocking { service.analyze("invoice-1") }

// 5. Assert result
assertEquals(Status("ok"), result)

// 6. Assert call behavior
TramaiAssertions.assertThat(provider, observer)
   .whenCalled("analyze")
   .wasCalledTimes(1)
   .andRetried(0)
   .andParsedSuccessfully()
   .emittedProvider("mock")

Testing Structured Output

Mock responses feed directly into the structured output pipeline (JSON extraction, parsing, validation, and retries).

data class Status(val status: String)

@AiService
interface Analyzer {
   @Operation(
       prompt = "Analyze the invoice",
       model = "gpt-4o",
   )
   suspend fun analyze(invoiceId: String): Status
}

@Test
fun `parses structured output correctly`() = runBlocking {
   val provider = MockAiProvider {
       onMethod("analyze") respondWith """{"status":"approved"}"""
   }

   val tramai = Tramai {
       provider(provider, default = true)
       model("gpt-4o", "mock")
   }

   val service = tramai.create<Analyzer>()
   val result = service.analyze("inv-1")

   assertEquals(Status("approved"), result)
}

Simulating Retries

Queue multiple responses for the same method. Each call consumes the next response in sequence.

val provider = MockAiProvider {
   onMethod("analyze") respondWith "not json"
   onMethod("analyze") respondWith """{"status":"ok"}"""
}

// First attempt: "not json" → parse failure → retry
// Second attempt: {"status":"ok"} → success
val service = tramai.create<Analyzer>()
val result = runBlocking { service.analyze("inv-1") }

assertEquals(Status("ok"), result)

This is useful for verifying:

Structured output retry behavior
Final parse success after correction
Call counts and retry indices

Simulating Provider Failures

Use SimulatedFailureProvider for provider-level failures (rate limits, server errors):

val provider = SimulatedFailureProvider {
   onMethod("analyze").retryableFailure("rate limited", statusCode = 429)
   onMethod("analyze") respondWith """{"status":"ok"}"""
}

// First attempt: 429 rate limit → provider retry
// Second attempt: success

This verifies:

Retryable provider failure handling
Non-retryable failure propagation
Recovery after transient provider errors

Recording and Replaying Responses

Use RecordingOperationObserver to capture what happened during a call:

val observer = RecordingOperationObserver()

// ... after running the test ...

val calls = observer.calls
assertEquals(1, calls.size)

val call = calls[0]
assertEquals("analyze", call.methodName)
assertEquals("gpt-4o", call.requestedModel)
assertEquals("mock", call.providerId)
assertEquals(true, call.parseSuccess)

You can replay recorded responses by extracting them from MockAiProvider.requests:

// Inspect what was sent to the provider
val sentRequest = provider.requests.single()
println(sentRequest.prompt.text) // "Analyze the invoice"
println(sentRequest.operationMethod) // "analyze"

Kotlin + Java Examples

Kotlin

@Test
fun `test support ticket routing`() = runBlocking {
   val provider = MockAiProvider {
       onMethod("classify") respondWith """{"priority":"high","department":"billing"}"""
   }

   val tramai = Tramai {
       provider(provider, default = true)
       model("gpt-4o", "mock")
   }

   val service = tramai.create<TicketClassifier>()
   val result = service.classify("Payment issue")

   assertEquals("high", result.priority)
   assertEquals("billing", result.department)
}

Java

@Test
void testSupportTicketRouting() {
   MockAiProvider provider = MockAiProvider.invoke(builder -> {
       builder.onMethod("classify").respondWith("{\"priority\":\"high\",\"department\":\"billing\"}");
   });

   Tramai tramai = Tramai.builder()
       .provider(provider, "mock", true)
       .model("gpt-4o", "mock")
       .build();

   TicketClassifier service = tramai.create(TicketClassifier.class);
   TicketClassification result = service.classify("Payment issue");

   assertEquals("high", result.getPriority());
   assertEquals("billing", result.getDepartment());
}

CI Integration Patterns

Pattern	How
No live calls in CI	Always use `MockAiProvider` in unit tests
Integration smoke tests	Use a real provider with a cheap model (gpt-4o-mini) in a separate test suite
Separate profiles	Spring: `@ActiveProfiles("test")` with mock provider bean
Parallel tests	Each test creates its own `MockAiProvider` — no shared state
Record-and-replay	Capture real responses once, replay in CI with `MockAiProvider`
Cost control	No API calls = no cost. Run AI tests on every commit.

What to Test

Recommended TramAI-facing application tests:

Prompt and argument wiring through service methods
Structured return type parsing
Retry behavior for malformed output
Provider routing decisions (model mapping, fallback)
Business logic wrapped around TramAI service calls

What NOT to Over-Test

You usually do not need to unit test:

Every internal prompt word choice (test the contract, not the prose)
Provider HTTP specifics in application tests (those belong in provider integration tests)
Jackson deserialization itself (TramAI tests it)

Limitations

No multi-provider state machines: Mock providers are per-test. Simulating complex failover chains requires orchestrating multiple mocks manually.
No streaming mocks: MockAiProvider does not support streaming. For stream testing, use end-to-end tests with Ollama or a lightweight model.
No built-in property-based testing: The module provides canned responses, not response generators based on property schemas.

Next Steps

Structured Output — Understanding parse-and-retry behavior
Observability — Monitoring real provider behavior in staging/production
Providers & Routing — Test different routing configurations