In 30 seconds
Do not only test that the AI does the right things. Test that it cannot do the wrong ones. Deliberately try to make it exceed its limits, and confirm the guardrails stop it.
Why this comes up
Most testing checks that the workflow produces good results. But safety depends equally on the opposite: that prohibited actions are actually prevented. A workflow that does its job well and can also, under an odd input, do something forbidden is not safe. You only find out by testing the boundaries directly.
Test the limits, not just the tasks
For each limit and prohibited action you defined, construct cases that try to cross it:
- An amount above the allowed cap.
- An action on a record type that is off limits.
- A recipient not on the approved list.
- A request that tries to make the AI do something outside its job.
- An action outside allowed hours or conditions.
For each, confirm the workflow refuses or escalates, and does not perform the action.
Enforce with guardrails, then verify
Remember that a prohibition written only in the prompt is a request the model may not honour. Real limits should be enforced by guardrails in the surrounding system. Your test is checking that the guardrail actually blocks the action, not that the model politely declined. See how to define allowed and prohibited actions.
Try to break it on purpose
Include inputs designed to push the AI off its job: unusual phrasing, instructions embedded in the content it processes, requests that sound urgent or authoritative. The point is to confirm the boundaries hold even when something actively tries to cross them, not only during normal use.
A small example
An agent was limited to refunds under 100. Testing the boundary, the team submitted a case that should have triggered a 250 refund. The model, misreading it, tried to issue it, and the guardrail blocked it, exactly as designed. Without that test, they would not have known whether the limit was real or just written down.
What this does not solve
Boundary testing confirms the limits you thought to test; it cannot prove no boundary can ever be crossed. New inputs and new attack ideas appear. Combine it with logging and monitoring so any real breach is caught quickly, and keep adding cases. See the checklist before giving an agent write access.
Good enough to move on when
- Every limit and prohibited action has a test that tries to cross it.
- Guardrails, not just prompts, enforce the limits.
- Inputs designed to push the AI off its job have been tried.
- Blocked attempts are logged so real breaches would be visible.
