An AI agent can connect a model’s interpretation to actions in business systems. That connection changes the risk: an incorrect proposal may become a sent message, an altered record or a request to another service.
Assess the complete path from incoming information to external effect. The question is what the application permits and verifies, rather than whether the model usually gives a sensible answer.
Identify the authority behind each action
List the tools the workflow can request and the identity used for each one. Record the resources and operations that identity can access.
Broad administrative credentials can turn a narrow task into a wide exposure. Give the integration the permissions required for the agreed work and keep secrets out of the model context.
Check whether actions are performed on behalf of the user or under a service identity. That distinction affects authorisation, logging and how access revocation should work.
Treat retrieved material as data
A customer email or document can contain malicious instructions. An agent may encounter them while performing an otherwise legitimate task.
Do not let the text’s presence in context grant it authority. Application code should validate and authorise a requested tool action against the actual user, task and policy. A source document cannot approve a new recipient or expand the tool’s permissions.
OWASP’s prompt-injection guidance explains why this risk remains relevant even when retrieval and model adaptation improve ordinary answer quality.
Check outputs before execution
Validate the request’s shape, allowed values and business constraints. Where exact calculations are needed, use a tested calculation path rather than accepting a model-generated number without provenance.
For consequential actions, present the proposed change and evidence to the appropriate reviewer before execution. Bind that approval to the actual proposal so later edits do not reuse an earlier decision improperly.
A second model can be a useful evaluation component, but it is not an independent security boundary merely because it reviews the first model. It can share similar failure modes and cannot replace deterministic permission checks.
Handle uncertain outcomes honestly
A tool timeout does not establish that an external action failed. Record the request and reconcile the connected system before repeating a write.
Use idempotency protections when supported. If the outcome cannot be determined, expose an unresolved state and assign review rather than repeatedly trying until the model reports success.
Some actions can be reversed, some need compensation and some cannot be undone. The recovery design must reflect that difference. A stop control prevents future work; it does not automatically reverse completed changes.
Test the paths you hope will not happen
Include malicious documents, swapped identifiers, revoked access, duplicate events and missing approval in test scenarios. Verify that the system rejects or stops as intended.
Evaluate actual external outcomes as well as the final response. A convincing “completed” message can hide a failed request or an action on the wrong record.
Keep bounded limits on attempts, elapsed time and resource use. An agent that cannot complete the task needs a defined exit path, not unlimited opportunity to improvise.
Keep a usable investigation record
Capture observable proposals, checks, tool requests, responses and approvals with task and configuration identifiers. Restrict sensitive content and define retention rather than collecting everything indefinitely.
Generated reasoning can be incomplete or unfaithful. Logs should be described as evidence of application activity, not as a full explanation of the model’s internal causes.
Our AI Security and Governance and AI Harness pages describe how these controls fit the surrounding runtime. The objective is a bounded, inspectable system with an explicit operating owner.
Written by Anshul Kaushik, Managing Director at Ryzolv.
Back to Insights