Design Deep Dive

Version: 0.6.0 Implementation language: Kotlin Framework policy: Framework-agnostic core, optional Spring adapter Serialization: Jackson (with kotlin-module)

Backend engineers on the JVM are increasingly asked to integrate AI capabilities into existing applications. The tooling available is either Python-native (LangChain, LlamaIndex) or a direct port of Python idioms into Java (LangChain4j) — neither of which feels idiomatic to a Kotlin/Java engineer who thinks in types, interfaces, and coroutines.

Complication

LangChain4j exists but inherits LangChain's mental model: chains, agents, and prompt templates as first-class abstractions. This model has known production problems — hard to test, hard to observe, leaky abstractions that expose model internals to application code. Structured outputs are an afterthought. Observability requires manual instrumentation.

Question

Can we build a JVM-native AI library where typed interfaces are the only abstraction, structured outputs are the default contract, and every AI interaction is automatically observable — with zero boilerplate?

Answer

Yes. TramAI is that library.

What 0.6.0 added to the answer

A capability is only trustworthy if its boundaries are enforced. 0.6.0 turned this design into an executable contract: 60 Gradle projects classified into 10 layers with machine-checked dependency rules, a frozen public API for the stable surface, fail-closed static safety and cancellation gates, and coverage and mutation ratchets over the critical runtime. See Module Architecture and Quality Gates.

Core Abstraction

The single unit of TramAI is the AI Operation — an annotated Kotlin interface method.

@AiService
interface InvoiceAnalyzer {

    @Operation(
        prompt = "Analyze this invoice and extract line items with cost categories",
        model = "claude-sonnet-4-20250514"
    )
    suspend fun analyze(invoice: Invoice): InvoiceAnalysis
}

Rules

Interface must be annotated with @AiService
Each method must be annotated with @Operation
Method parameters are serialized as structured context
Return type determines behavior:
- String -> raw model output, no parsing
- Unit -> fire-and-forget
- Any data class or POJO -> structured output with schema validation and retry
- suspend fun -> coroutine-native, non-blocking
- Blocking methods -> supported for Java consumers

Ownership for an invocation is explicit: TramaiInvocationHandler owns proxy dispatch, the engine's invocation coordinator owns governance enforcement points, and the provider-execution boundary owns routing, retry, and fallback. Provider adapters own transport only.

Structured Output Pipeline

This is the core intellectual work of the library. The authoritative artifact is the compiled structured descriptor — StructuredTypeDescriptor with a StructuredContractFingerprint — produced by StructuredTypeCompiler in tramai-structured and consumed by the engine.

1. SCHEMA GENERATION
   Return type -> JSON Schema
   Compiled into a typed descriptor, cached per method
   Descriptor is authoritative for parsing and validation

2. PROMPT CONSTRUCTION
   User prompt + input serialization + schema injection
   Schema injected as system instruction: "Respond ONLY with valid JSON matching this schema"

3. MODEL CALL
   Request dispatched through provider resolution
   Raw text response captured

4. RESPONSE PARSING
   Raw text -> JSON extraction (handles markdown fences, preamble)
   JSON -> deserialization into the return type
   `JacksonStructuredOutputHandler.analyze()` returns either success or a typed failure

5. VALIDATION FEEDBACK LOOP (if parsing fails)
   The failure carries a feedback message suitable for retry
   The broken response and the feedback are appended, and the full conversation is resubmitted
   Up to @Operation(maxRetries = N) — default 2

6. FAILURE
   After the retry limit: StructuredOutputException
   Fields: originalPrompt, lastRawResponse, validationError, attemptCount

The retry logic distinguishes three failure classes: the response is not JSON, the response is JSON with the wrong structure, and the response parses but fails validation. Each class produces different feedback, which is why "retry with feedback" is a real repair loop rather than a blind repeat.

Schema Annotations

data class SpendAnalysis(
    @AiDescription("Total spend in USD, always positive")
    val totalSpend: Double,

    @AiDescription("List of cost reduction recommendations, ordered by impact")
    @AiMinItems(1)
    val recommendations: List<Recommendation>,

    @AiDescription("Confidence score between 0.0 and 1.0")
    @AiRange(min = 0.0, max = 1.0)
    val confidence: Double
)

Provider Model

Provider Interface

interface ModelProvider {
    suspend fun complete(request: ModelRequest): ModelResponse
    fun providerId(): String = this::class.simpleName ?: "unknown"
    fun supportsCapability(capability: ProviderCapability): Boolean = false
}

ProviderCapability is an enum with VISION, TOOL_CALLING, STRUCTURED_OUTPUT, and STREAMING. Capability admission is checked before dispatch, so a provider that cannot accept an image fails fast instead of being sent one.

Provider Resolution

providers are registered explicitly
models are registered explicitly against providers
explicit override via @Operation(provider = "...") remains available
unknown or unregistered models produce a deterministic resolution error

Resolution order is @Operation(provider = ...), then the model-to-provider mapping, then the default provider. Each provider attempt is wrapped by the engine's retry/fallback policy; when the primary route fails, fallback routes are tried in order. Provider adapters must not implement retries, fallbacks, or circuit breaking themselves — that is the engine's boundary, enforced by the module rules.

Native Structured Output

Some providers can constrain the model to a schema natively (for example OpenAI's response_format). When the provider advertises the capability, the engine prefers it; otherwise the schema-in-prompt pipeline above applies. In both cases the same compiled descriptor validates the response.

Retry and Error Handling

Retry Hierarchy

Scenario	Default	Behavior
Structured output parse failure	`@Operation(maxRetries = 2)`	Feeds validation feedback back to the model
Provider transient error (429, 503)	`@Operation(providerRetries = 3)`	Exponential backoff
Non-retryable (auth, invalid model)	0 retries	Throws immediately

Per-attempt and per-operation token budgets are enforced by the engine, not by providers.

Exception Hierarchy

TramaiException (sealed, unchecked)
├─ StructuredOutputException        — exhausted retries on parsing/validation
│    originalPrompt, lastRawResponse, validationError, attemptCount
├─ ProviderException                — provider transport or API failure
│    statusCode, retryable, retryAfterMillis
├─ ConfigurationException           — misconfiguration detected at startup
├─ TimeoutException                 — a single provider attempt exceeded its timeout
├─ ProviderCapabilityException      — provider cannot serve the requested capability
├─ CircuitBreakerOpenException      — route temporarily unavailable (reopenAtEpochMillis)
├─ TokenBudgetExceededException     — engine-owned token budget exceeded
├─ ApprovalException (open)         — approval-domain failures
│    IllegalApprovalTransitionException, ApprovalNotFoundException, …
├─ ToolInvalidInputException        — tool input validation failure
│    safeModelMessage only when explicitly marked model-visible
├─ PolicyViolationException         — policy denial, carries decision.reason/reasonCode
├─ ApprovalSuspendedException       — invocation suspended pending approval
└─ ModelRegistryException (+ subtypes)

Engine-produced failures carry fixed public messages; the original cause is delivered separately to a diagnostic observer rather than through the exception a caller sees. Treat that as a feature: governance failures must not leak prompts, payloads, or raw driver errors into application logs.

Prompt Rendering

Parameter interpolation uses single-brace placeholders resolved by parameter name:

@Operation(prompt = "Analyze spend data for tenant {tenantId}")
suspend fun analyze(tenantId: String, data: SpendRecord): SpendAnalysis

For a class-level system message, use the interface-level annotation:

@AiService
@SystemPrompt("You are a cost optimization expert. Be concise and actionable.")
interface SpendAnalyzer

aiStep and the core operation path have no framework-owned prompt-defense layer. If you build an operation that injects untrusted text, sanitization is application code — TramAI provides the seam (PromptSanitizer) and the governance hooks, not a promise that the prompt was safe.

Framework Integration

TramAI has no framework dependency in its core. Framework adapters are thin modules that wire the standalone API into a framework's DI and configuration systems.

Spring Boot: tramai-spring-core provides AiServiceProxyAutoConfiguration, which registers @AiService proxies as beans, and @EnableTramai, which imports the configurations contributed by modules on the classpath. In 0.6.0 the unified tramai-spring-boot-starter composes standard and sovereign runtimes and selects between them with tramai.profile; @EnableTramai is not required to get beans.
Quarkus/Micronaut: Not shipped. The standalone API works in both — build TramAI explicitly and register it as a CDI bean or singleton.
Plain JVM/CLI: No framework, no classpath scanning, no magic. TramAI is a library, not a container.

Testing Support

The tramai-testing module (published, internal API stability) provides:

MockAiProvider — canned responses, no network calls
SimulatedFailureProvider — retryable and non-retryable failures
RecordingOperationObserver — verify call counts and behavior
TramaiAssertions — fluent assertion API

It also hosts the shared contract oracles — the provider TCK and the store TCKs — that independent implementations are enrolled in. Test behavior against those contracts rather than against a specific implementation.

Architecture Overview — runtime execution flow
Module Architecture — layers and boundary enforcement
API and Binary Compatibility — what is frozen and what is preview
Why TramAI? — comparison to alternatives