TramAI

Production Hardening & Security

TramAI is built for production environments where cost control, data privacy, and reliability matter. This guide covers the hardening mechanisms documented for the current 0.4.x line.

What This Covers

  • PII masking and data redaction with OperationInterceptor
  • DLP interceptors for model output and tool result scanning
  • Secret management with SecretValueResolver
  • Token budgets for cost control
  • Circuit breakers, retry pacing, and fallback routing
  • Sovereign Mode for hardened, auditable deployments

PII Masking with OperationInterceptor

When to use: You need to redact or transform sensitive data in request messages before they leave the JVM.

Minimum setup: Implement OperationInterceptor and register it on your TramAI builder or as a Spring bean.

The OperationInterceptor interface has two hooks:

  • interceptRequest(...) — inspect and modify messages before provider transport
  • interceptResponse(...) — inspect and modify provider responses before the engine processes them

Standalone

val tramai = Tramai {
    provider(OpenAiProvider(System.getenv("OPENAI_API_KEY")), name = "openai", default = true)
    model("gpt-4o", "openai")

    interceptor(object : OperationInterceptor {
        override fun interceptRequest(
            context: OperationCallContext,
            messages: List<Message>,
        ): List<Message> = messages.map { message ->
            message.copy(content = redactPii(message.content))
        }
    })
}

Spring Boot

Spring auto-configuration composes ordered OperationInterceptor beans into the generated TramAI instance.

@Configuration
class TramaiSecurityConfiguration {
    @Bean
    fun piiMaskingInterceptor(): OperationInterceptor = object : OperationInterceptor {
        override fun interceptRequest(
            context: OperationCallContext,
            messages: List<Message>,
        ): List<Message> = messages.map { message ->
            message.copy(content = maskSensitiveData(message.content))
        }
    }
}

Current limitations:

  • interceptors are engine-level request/response hooks
  • they are opt-in — no interceptor is applied by default
  • Spring auto-configuration composes registered interceptor beans in order via CompositeOperationInterceptor

DLP (Data Loss Prevention)

When to use: You need to scan model outputs and tool results for sensitive data before they reach the caller, structured parser, or cache.

Minimum setup: Register a DlpInterceptor on the TramAI builder or as a Spring bean.

The DLP system inspects model outputs and tool results after the provider returns but before the result reaches the caller or structured parser. The SPI is a single-function interface:

fun interface DlpInterceptor {
    fun inspect(context: DlpContext, text: String): DlpResult
}

Misconfiguration example — two DlpInterceptor beans will cause Spring auto-configuration to throw at startup. Define exactly one.

For complete DLP documentation including RuleBasedDlpInterceptor, DlpRedactionAuditEmitter, rule configuration, and cache bypass — see the dedicated DLP guide.

Secret Management

When to use: You need to resolve provider credentials without hard-coding them in application code.

Minimum setup: Use the built-in env: and file: resolvers, or implement SecretValueResolver for cloud secret stores.

Built-in Resolvers

  • env:NAME — resolves from environment variables
  • file:/path/to/secret.txt — resolves from a file on disk

Standalone

val secretResolver = CompositeSecretValueResolver(
    listOf(
        SecretValueResolver { secretRef ->
            if (!secretRef.startsWith("vault:")) null
            else vaultClient.read(secretRef.removePrefix("vault:"))
        },
        EnvironmentSecretValueResolver,
        FileSecretValueResolver,
    ),
)

val tramai = Tramai {
    provider(
        OpenAiProvider(
            apiKey = secretResolver.resolve("vault:providers/openai/api-key")
                ?: error("Missing OpenAI API key"),
        ),
        name = "openai",
        default = true,
    )
    model("gpt-4o", "openai")
}

Spring Boot

Spring Boot auto-configuration composes SecretValueResolver beans. Reference secrets with *-secret-ref properties:

tramai:
  providers:
    openai:
      api-key-secret-ref: vault:providers/openai/api-key

@Bean
fun vaultSecretValueResolver(): SecretValueResolver = SecretValueResolver { secretRef ->
    if (!secretRef.startsWith("vault:")) null
    else vaultClient.read(secretRef.removePrefix("vault:"))
}

Current limitations:

  • built-in support covers env: and file: only
  • cloud secret stores require custom SecretValueResolver implementations
  • bundled AWS Secrets Manager or Vault adapters are not shipped yet

Token Budgets

When to use: You need to cap token consumption per attempt or per logical operation to control cost.

Minimum setup: Configure TokenBudgetSettings on the engine.

Available Controls

  • hardMaxTokensPerAttempt — hard cap per single provider call
  • hardMaxTokensPerOperation — hard cap across the entire logical operation (including retries and tool loops)
  • softMaxTokensPerOperation — emits a warning event when crossed, does not fail

Standalone

val tramai = Tramai {
    provider(OpenAiProvider(System.getenv("OPENAI_API_KEY")), name = "openai", default = true)
    model("gpt-4o", "openai")

    tokenBudget(
        TokenBudgetSettings(
            hardMaxTokensPerAttempt = 4_000,
            hardMaxTokensPerOperation = 20_000,
            softMaxTokensPerOperation = 10_000,
        ),
    )
}

Spring Boot

tramai:
  cost:
    token-budget:
      hard-max-tokens-per-attempt: 4000
      hard-max-tokens-per-operation: 20000
      soft-max-tokens-per-operation: 10000

Practical notes:

  • budgets apply across retries
  • structured-output retries count toward the operation total
  • tool-call loops count toward the operation total
  • soft limit crossing emits an engine event instead of failing the call

Resilience Controls

When to use: You need fallback models, circuit breakers, and retry pacing for reliable provider access.

Fallback Routing

Fallback routes are explicit ordered routes for a requested model.

Circuit Breaking

Prevents repeated calls into an unhealthy provider after a configurable failure threshold.

Retry Pacing

Exponential backoff with jitter, honoring provider Retry-After hints.

Standalone

val tramai = Tramai {
    provider(OpenAiProvider(System.getenv("OPENAI_API_KEY")), name = "openai", default = true)
    provider(AnthropicProvider(System.getenv("ANTHROPIC_API_KEY")), name = "anthropic")

    model("gpt-4o", "openai")
    model("claude-sonnet-4-20250514", "anthropic")

    fallbackModel("gpt-4o", "claude-sonnet-4-20250514", "anthropic")

    circuitBreaker(
        CircuitBreakerSettings(
            enabled = true,
            failureThreshold = 3,
            openDurationMillis = 30_000,
        ),
    )
    retryPolicy(
        RetryPolicySettings(
            maxRetryAfterMillis = 20_000,
            jitterRatio = 0.1,
        ),
    )
}

Spring Boot

tramai:
  models:
    gpt-4o: openai
    claude-sonnet-4-20250514: anthropic
  fallbacks:
    gpt-4o:
      - provider: anthropic
        model: claude-sonnet-4-20250514
  resilience:
    circuit-breaker:
      enabled: true
      failure-threshold: 3
      open-duration-millis: 30000
    retry:
      max-retry-after-millis: 20000
      jitter-ratio: 0.1

Current limitations:

  • fallback routing is explicit, not heuristic
  • streaming failover is only allowed before the first emitted token
  • once a stream has emitted user-visible output, TramAI returns a terminal error rather than stitching providers together

Sovereign Mode

When to use: You need a hardened, auditable deployment profile with deny-by-default policy, approved-model routing, classification-aware provider trust zones, and hash-chained audit.

Sovereign Mode composes tramai-standalone with tramai-security into a secure-by-default embedded runtime. It enforces empty-allowlists-reject-everything policy, fail-fast build-time validation, and append-only hash-linked audit emission.

For full documentation including SovereignTramai builder, trust zones, and evidence packs, see the dedicated Sovereign Mode guide.

Next Steps

Previous
Next