AI Agent Tool Governance

Tool calling turns an AI system from a text generator into an actor. A model that can invoke a tool can write data, send email, delete records, create payments, invoke MCP tools, and call external APIs. That makes tool calling a security boundary — and the boundary needs governance.

Tool governance is the practice of deciding, at runtime, whether a tool may be exposed to the model and whether a specific tool call may execute.

Why Tool Calling Is a Security Boundary

A model's text output cannot directly mutate your systems. A tool call can. The moment you register a tool, the model gains the ability to trigger real side effects through your application's own code paths.

Prompt instructions are not a control here. "Tell the model not to delete records" is advice; the runtime decision "this delete call is DENIED" is enforcement.

The Three Decisions: ALLOW, DENY, REQUIRE_APPROVAL

TramAI's policy engine models tool governance with three decision types:

The decision type is evaluated at runtime against the actual call context, not configured once at registration.

Two Boundaries: Exposure and Execution

TramAI enforces tool policy at two distinct points, plus a third for the return path:

This distinction matters:

Exposure permission is not execution permission.

A tool can be visible to the model — so the model can plan around its existence — while the actual execution is denied or gated behind human approval. Both decisions are recorded as separate evidence events.

Example: Deny by Default, Then Deny One Tool

The production posture starts from PolicyConfiguration.secure() — every allowlist empty, so nothing is permitted until it is named explicitly:

val baselinePolicy = DefaultPolicyEngine(
    PolicyConfiguration.secure().copy(
        allowedModels = setOf("gpt-4o"),
        allowedProviders = setOf("openai"),
        allowedTools = setOf("customer_lookup", "account_delete"),
        allowedPermissions = setOf("customer.read", "account.delete"),
    )
)

PolicyConfiguration.preview() still exists, but it is a 0.4.x migration and testing preset — its wildcard allowlists bypass most registry checks. Do not use it as a production posture.

Policy wrappers compose on top of that baseline:

val denyingPolicy = PolicyEngine { context ->
    if (context.enforcementPoint == EnforcementPoint.BEFORE_TOOL_EXECUTION
        && context.toolName == "account_delete") {
        PolicyDecision.Deny(
            reason = "Account deletion is disabled",
            reasonCode = "account-delete-disabled"
        )
    } else {
        baselinePolicy.evaluate(context)
    }
}

val engine = TramaiEngine(
    provider = provider,
    toolRegistry = ToolRegistry(mapOf("account_delete" to tool)),
    policyEngine = denyingPolicy,
    policyDecisionAuditEmitter = emitter,
)

When the model calls account_delete:

1. BEFORE_TOOL_EXPOSURE — the tool is exposed, so the model may plan around it.
2. BEFORE_TOOL_EXECUTION — the policy engine returns Deny.
3. The tool never executes — the provider is not called again and the arguments are never applied.
4. PolicyViolationException is raised, carrying the decision: e.decision.reason and e.decision.reasonCode. The denial is audited before the exception propagates, and exported as tool.permission evidence — tool enforcement events are partitioned out of generic policy.decision evidence by design.

Approval-Gated Tools

For high-risk actions the runtime decision is RequireApproval: execution suspends at BEFORE_TOOL_EXECUTION, a cryptographically bound challenge is created (binding the workflow run ID, tool name, argument digest, policy version, and workflow digest), the continuation is persisted, and the engine raises ApprovalSuspendedException — which carries toolName and workflowRunId alongside the challenge (approval ID, token, expiry). The tool never runs until an external authorizer presents the matching token and the runtime resumes the workflow. See human approval for the lifecycle in depth.

DefaultPolicyEngine decides this from tool security metadata: risk levels in requireApprovalForRiskLevel (default HIGH and CRITICAL) and ApprovalMode.HUMAN_REQUIRED both produce RequireApproval, while unknown tools are denied outright.

Tool Permission Evidence

Every tool enforcement decision is exported as a dedicated tool.permission evidence record by ToolPermissionRuntimeEvidenceExporter, separate from generic policy.decision records — PolicyDecisionRuntimeEvidenceExporter excludes the three tool enforcement points. A reviewer can reconstruct, for each tool call:

• which enforcement point made the decision
• whether it was ALLOW, DENY, or REQUIRE_APPROVAL
• which tool the decision belongs to, plus its risk level and data classification
• the audit event it came from

Raw tool arguments, prompts, and secrets never appear in evidence records: exports are restricted to a safe metadata allowlist (toolName, enforcementPoint, riskLevel, classification, classificationSource).

This is the difference between "we have tool calling" and "we can prove what each tool call was permitted to do."

Security Metadata Per Tool

Tools carry security metadata that drives policy evaluation — permission (e.g. customer.read, account.delete, payment.execute), risk level, and whether human approval is required. The runtime combines this metadata with policy at each enforcement point. The deny-by-default posture means: no tool, permission, or model is allowed unless explicitly configured.

MCP and Tool Governance

MCP servers expose tools over a standardized protocol, which means the same governance questions apply to every MCP capability. TramAI's tool permission model covers MCP-bound tools through the same policy engine and enforcement points. Note the honest boundary:

• Available in 0.6.0 — tool governance applied to registered tools in the runtime: policy engine, the three enforcement points, ALLOW / DENY / REQUIRE_APPROVAL, and the tool.permission evidence family.
• Repository module, not published (0.6.0) — the MCP workflow server (tramai-mcp) exposes registered workflows as MCP tools (list_workflows, run_workflow, resume_workflow, get_workflow_status). The module is internal in 0.6.0 and is not a published artifact.
• Not implemented (future direction) — a governed remote MCP tool connector that routes remote MCP tool calls through the same runtime boundary.

See MCP governance for the full picture.

When Not to Use This

Tool governance is overhead when there is no tool surface to protect:

• Read-only workloads with no tool calls — nothing to gate.
• Tools with zero side effects and public data — Allow-everything policy is fine.
• Prototypes — use the plain builder; add the sovereign profile when tools can mutate state.

The rule of thumb: the moment a tool can write, send, delete, or pay, it needs a decision boundary, not a convention.

Related Documentation

• Runtime governance — the full enforcement chain
• Human approval — REQUIRE_APPROVAL lifecycle
• AI agent governance — the category pillar
• JVM AI governance — in-process governance for Java/Kotlin
• Governed tool use guide — tool calling in TramAI
• MCP governance — MCP tool surfaces
• Security documentation — policy, audit, evidence
• TramAI on GitHub