CI Integration

TramAI tests should never call real AI providers in CI. This guide covers how to run your TramAI application tests efficiently in CI environments.

What This Covers

Cached provider responses for deterministic test execution
Deterministic replay with mock providers
Avoiding API calls in CI
Gradle build patterns for TramAI testing
Parallel test execution considerations

The Problem

Real AI provider calls in CI cause:

Cost: every CI run incurs API charges
Flakiness: network issues, rate limits, and model latency cause non-deterministic failures
Speed: provider calls add seconds to minutes per test
Rate limits: concurrent CI jobs can trigger 429 responses

The solution: never call real providers in CI unless you are running explicit provider integration tests.

Minimum Setup: Mock Provider in Tests

TramAI ships MockAiProvider and SimulatedFailureProvider in the tramai-testing module.

Dependency

testImplementation("dev.tramai:tramai-testing:0.3.1")

Basic Test Pattern

@Test
fun `analyze returns structured result`() = runTest {
    val provider = MockAiProvider {
        onMethod("analyze") respondWith """{"status":"ok"}"""
    }

    val tramai = Tramai {
        provider(provider, default = true)
        model("gpt-4o", "mock")
    }

    val service = tramai.create<Analyzer>()
    val result = runBlocking { service.analyze("invoice-1") }

    assertEquals(Status("ok"), result)
}

Gradle Build Patterns

Separate Source Sets for Integration Tests

Separate provider-dependent tests from fast unit tests:

// build.gradle.kts
sourceSets {
    create("integrationTest") {
        compileClasspath += sourceSets.main.get().output
        runtimeClasspath += sourceSets.main.get().output
    }
}

val integrationTest by tasks.registering(Test::class) {
    testClassesDirs = sourceSets["integrationTest"].output.classesDirs
    classpath = sourceSets["integrationTest"].runtimeClasspath
    // Only run integration tests on demand
}

Skip Real Provider Calls by Default

Use a Gradle property to gate provider-dependent tests:

tasks.named<Test>("test") {
    // Fast unit tests only — no real providers
    systemProperty("tramai.test.skip-real-providers", "true")
}

tasks.register<Test>("providerTest") {
    // Explicit task for provider integration tests
    systemProperty("tramai.test.use-real-providers", 
        project.findProperty("useRealProviders")?.toString() ?: "false")
}

Deterministic Replay

Queuing Multiple Responses

Simulate retry scenarios in CI:

val provider = MockAiProvider {
    onMethod("analyze") respondWith "not json"    // first attempt fails
    onMethod("analyze") respondWith """{"status":"ok"}"""  // retry succeeds
}

Simulating Provider Failures

Use SimulatedFailureProvider to test resilience:

val provider = SimulatedFailureProvider {
    onMethod("analyze").retryableFailure("rate limited", statusCode = 429)
    onMethod("analyze") respondWith """{"status":"ok"}"""
}

Recording and Validating Behavior

val observer = RecordingOperationObserver()

TramaiAssertions.assertThat(provider, observer)
    .whenCalled("analyze")
    .wasCalledTimes(1)
    .andRetried(0)
    .andParsedSuccessfully()
    .emittedProvider("mock")

Caching Provider Responses

For acceptance tests that need realistic responses, cache provider output:

Simple JSON Fixture Pattern

class CachedProvider : ModelProvider {
    private val cache = loadFixture("provider-responses.json")

    override suspend fun complete(request: ModelRequest): ModelResponse {
        val key = cacheKey(request)
        return cache[key] ?: error("No cached response for $key")
    }

    override fun providerId(): String = "cached"
}

Store fixtures in src/test/resources/fixtures/:

{
  "analyze:InvoiceAnalyzer": {
    "content": "{\"status\":\"ok\"}",
    "modelUsed": "gpt-4o",
    "inputTokens": 150,
    "outputTokens": 25
  }
}

GitHub Actions Example

name: CI

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'

      - name: Cache Gradle
        uses: actions/cache@v4
        with:
          path: ~/.gradle/caches
          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}

      - name: Run fast tests (no real providers)
        run: ./gradlew test

      - name: Run provider integration tests (on demand)
        if: github.event_name == 'schedule'
        run: ./gradlew providerTest -DuseRealProviders=true
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Parallel Test Execution

Mock providers are stateless — TramAI tests can run in parallel safely.

// build.gradle.kts
tasks.withType<Test> {
    maxParallelForks = Runtime.getRuntime().availableProcessors()
    // Each test creates its own MockAiProvider — no shared state
}

What to Test in CI

Always test:

prompt and argument wiring through service methods
structured return behavior
retry behavior for malformed output
provider routing decisions
business logic wrapped around TramAI service calls

Test on demand (not every push):

real provider integration (scheduled or manual)
end-to-end workflows against a test provider

What Not to Test in CI

Do not test in every CI run:

provider HTTP specifics in application tests
internal prompt word choices
Jackson serialization (tested by TramAI module tests)
model intelligence or quality

Limitations

MockAiProvider simulates provider responses — it does not validate that the prompt is correct
recorded fixtures can become stale if your prompt or data structures change
cached responses do not catch provider API contract changes

Next Steps

Testing Reference — full testing API
Custom Providers — implement test doubles for custom providers