An AI agent becomes security-relevant when it calls tools rather than merely generating text. Reading, writing, sending, executing and deleting are different powers. A model must not derive those boundaries from a prompt or change them because a document contains instructions.
The model is not an authorisation system
A language model can propose an action. The application decides whether it is allowed. User role, tenant, project, data class and workflow state are checked outside the model. Even a perfectly formed tool request receives no additional privileges.
A common mistake is a generic execute_command or call_api tool. It moves the security decision into generated text. Narrow contracts such as search_documents, create_ticket_draft and run_authorised_scan expose known parameters and side effects.
Give every tool an explicit boundary
A useful contract defines:
- permitted parameters and values,
- required user and system permissions,
- accessible sources or targets,
- maximum runtime and result volume,
- possible side effects,
- recorded inputs and outputs,
- failure and retry behaviour.
Inputs are schema validated. Target IDs and tenants are resolved by the server. Free URLs, SQL and shell fragments are allowed only where that flexibility is essential and additionally isolated.
Treat prompt injection as a trust-boundary problem
An agent reads untrusted content. Websites, tickets, email and documents can contain text that resembles system instructions. A model cannot reliably distinguish data from commands by phrasing alone.
Read content must therefore never extend tool rights. Protection combines:
- fixed permissions outside model context,
- workflow separation of instructions and data,
- filtering and labelling of external content,
- confirmation for risky actions,
- output validation before tool use,
- minimal data returned by tools.
A prompt alone is not a dependable security boundary.
Scale approval with impact
Not every action needs a person. Internal search can run automatically when results are authorised correctly. Customer email, production data changes and active security tests often need approval.
| Impact | Example | Control |
|---|---|---|
| read only | authorised knowledge search | automatic policy check |
| reversible write | ticket draft | preview and user confirmation |
| external communication | send email | approval of final content |
| high impact | deployment or active scan | additional role and dual control |
Approval displays concrete parameters and expected effect. A generic “continue agent” button is too vague.
Budgets limit technical and financial damage
Agents can loop, load excessive context or repeat a tool unnecessarily. Each run needs limits on time, tokens, cost, tool calls and similar repetitions. When a limit is reached, it ends with a traceable partial result.
Writing tools need idempotency keys. A retry after timeout must not create two invoices, tickets or scans. Circuit breakers protect external services when their failure rate rises.
Audit without creating another data leak
An investigation needs to show which model proposed which action under which policy. Complete prompts can also contain passwords, personal data and confidential documents. Logging therefore needs minimisation, redaction, access control and retention limits.
Useful audit fields include run ID, user and tenant reference, model version, policy version, tool, normalised parameters, decision, approval and result status. Secrets and complete documents generally do not belong in logs.
Evaluate abuse cases
Alongside normal tasks, test malicious documents, unauthorised targets, permission changes, repeated tool responses, timeouts and conflicting instructions. Success is not the most creative answer. It is the correct controlled stop.
A tool gate as executable policy
Before each call, the application evaluates a structured request. The model supplies arguments but never identity, tenant or effective permissions:
{
"tool": "create_ticket_draft",
"actor_id": "usr_842",
"tenant_id": "tenant_17",
"arguments": {"project_id": "prj_9", "title": "..."},
"limits": {"calls_remaining": 4, "deadline_ms": 1500},
"required_approval": "before_external_send",
"policy_version": "agent-tools-12"
}
The gate validates schema, the actor-tenant-project relationship, data class, budget and side effect. It returns allow, deny or approval_required with a machine-readable reason. Tests exercise this decision independently of the prompt. A malicious document must produce the same denial as a direct unauthorised request.
When is an AI agent ready for production? covers the broader operating architecture. For sensitive data paths, the local versus frontier model decision completes the routing policy.
Sources and further reading
- OWASP Top 10 for LLM Applications
- NIST AI Risk Management Framework
- MITRE ATLAS
- OWASP Authorization Cheat Sheet
Conclusion
A safe agent has no general power over a system. It works through narrow tools, least privilege, measurable budgets and approvals at clear impact boundaries. The model proposes; the application enforces.