CI Integration

TramAI tests should never call real AI providers in CI. This guide covers running your TramAI application tests deterministically in a pipeline, and then the gate set TramAI itself enforces — useful when you contribute, or when you want to mirror the same discipline in your own repository.

What This Covers

• Why real provider calls in CI are a losing trade
• Mock provider setup and dependency pins
• Gradle patterns that separate fast tests from provider-dependent tests
• A GitHub Actions example
• The 0.6.0 gate set, with the actual task names
• The ratchets: coverage, mutation, API compatibility, link integrity

The Problem

Real provider calls in CI cost money, flake on network and rate limits, add seconds to minutes per test, and make results non-reproducible. The fix is to make the deterministic path the default and put real-provider tests behind an explicit switch.

Minimum Setup: Mock Providers

tramai-testing is a published artifact managed by the BOM:

dependencies {
    testImplementation(platform("dev.tramai:tramai-bom:0.6.0"))
    testImplementation("dev.tramai:tramai-testing")
}

@Test
fun `analyze returns structured result`() = runBlocking {
    val provider = MockAiProvider {
        onMethod("analyze") respondWith """{"status":"ok"}"""
    }

    val tramai = Tramai {
        provider(provider, default = true)
        model("gpt-4o", "mock")
    }

    val service = tramai.create<Analyzer>()
    assertEquals(Status("ok"), service.analyze("invoice-1"))
}

No network, no credentials, no cost. See Testing for the full double and assertion surface, including SimulatedFailureProvider for retryable and non-retryable failure paths.

Gradle Patterns

Separate Source Set For Provider-Dependent Tests

Keep provider-dependent tests out of test so the default build cannot reach the network:

sourceSets {
    create("integrationTest") {
        compileClasspath += sourceSets.main.get().output
        runtimeClasspath += sourceSets.main.get().output
    }
}

val integrationTest by tasks.registering(Test::class) {
    testClassesDirs = sourceSets["integrationTest"].output.classesDirs
    classpath = sourceSets["integrationTest"].runtimeClasspath
    // Never wired into `check` — runs only when invoked explicitly
}

Gate Real Providers Behind An Explicit Switch

TramAI does not read a magic system property for this, so wire it yourself:

val integrationTest by tasks.registering(Test::class) {
    testClassesDirs = sourceSets["integrationTest"].output.classesDirs
    classpath = sourceSets["integrationTest"].runtimeClasspath
    onlyIf { providers.gradleProperty("useRealProviders").orNull == "true" }
}

Then ./gradlew integrationTest runs with mocks only and does nothing; ./gradlew integrationTest -PuseRealProviders=true opts in, with credentials supplied through the environment.

Deterministic Replay

Queue responses to simulate repair and retry behaviour:

val provider = MockAiProvider {
    onMethod("analyze") respondWith "not json"
    onMethod("analyze") respondWith """{"status":"ok"}"""
}

Provider-side failures use SimulatedFailureProvider:

val provider = SimulatedFailureProvider {
    onMethod("analyze").retryableFailure("rate limited", statusCode = 429)
    onMethod("analyze") respondWith """{"status":"ok"}"""
}

Assert the observed behaviour, not just the final value:

TramaiAssertions.assertThat(provider, observer)
    .whenCalled("analyze")
    .wasCalledTimes(2)
    .andRetried(1)
    .andParsedSuccessfully()

Caching Recorded Responses

For acceptance tests that need realistic payloads, wrap a JSON fixture in a provider:

class CachedProvider(private val cache: Map<String, ModelResponse>) : ModelProvider {
    override fun providerId(): String = "cached"

    override suspend fun complete(request: ModelRequest): ModelResponse =
        cache[request.operationMethod] ?: error("No cached response for ${request.operationMethod}")
}

Store fixtures in src/test/resources/fixtures/. Recorded fixtures go stale when prompts or types change — regenerate them deliberately.

GitHub Actions Example

name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'

      - name: Cache Gradle
        uses: actions/cache@v4
        with:
          path: ~/.gradle/caches
          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}

      - name: Fast tests (no real providers)
        run: ./gradlew test

      - name: Provider-dependent tests (scheduled only)
        if: github.event_name == 'schedule'
        run: ./gradlew integrationTest -PuseRealProviders=true
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Parallel Test Execution

Mock providers are per-test and stateless, so test parallelises safely:

tasks.withType<Test> {
    maxParallelForks = Runtime.getRuntime().availableProcessors()
}

The Gate Set TramAI Actually Runs

If you contribute to TramAI — or want to copy the discipline — these are the real 0.6.0 task names. verifyPr is the primary day-to-day gate; verify060MaintainabilityRelease is the fail-closed release aggregator.

./gradlew verifyPr
./gradlew verify060Architecture
./gradlew verifyStaticSafetyGuards
./gradlew apiCheck
./gradlew verifyChangePolicy -PchangeClass=runtime-behaviour
./gradlew verify060MaintainabilityRelease --warning-mode=fail --no-daemon

Ratchets

Three gates are ratchets rather than binary checks. They fail when quality moves backwards:

• Critical coverage — 100% branch and instruction coverage over critical packages. New code in those packages must be covered, not merely "not worse".
• Mutation — a mutation ratchet over canonical failure modes. Weakening a test that kills a mutant fails the gate.
• API compatibility — apiCheck compares against the committed api/<module>.api dump. Run ./gradlew apiDump and commit the result when the public surface changes intentionally.

Link Integrity

verifyReleaseDocumentationIntegrity fails on broken relative links. Documentation is part of the build, not an afterthought: a page that links to a moved target breaks CI.

What To Test In CI

Always:

• prompt and argument wiring through service methods
• structured return behaviour, including retry and exhaustion
• retry and fallback behaviour
• provider routing decisions
• business logic wrapped around TramAI service calls
• cancellation propagation for anything that owns a coroutine scope

On demand only:

• real provider integration against a cheap model
• end-to-end workflows against a test provider

What Not To Test In CI

• provider HTTP specifics in application tests
• internal prompt wording
• Jackson serialisation itself
• model intelligence or answer quality

Limitations

• MockAiProvider simulates responses; it does not validate that your prompt is correct.
• Recorded fixtures go stale when prompts or data structures change.
• Cached responses do not catch provider API contract changes — only a real-provider run does.

Next Steps

• Testing — mock providers, TCKs, and assertions
• Observability — asserting spans and metrics in tests
• Contributor Change Guides — the contract tests each kind of change must satisfy