AI agent testing checklist | Drop Jaw Studio

An AI agent testing checklist helps you answer a more useful question than “did the demo work?” It asks whether the complete workflow can do its defined job, stay inside its boundaries and fail in a way your team can see and control.

That distinction matters. An agent may write a good draft in one test while still using the wrong source, skipping an approval or attempting an action it was never meant to take. A polished output is evidence about one run. It is not proof that the system is ready for live work.

This guide gives a small business a practical acceptance-testing process before an AI agent goes live. It is deliberately lighter than a formal assurance programme, but it still creates a decision record another person can understand.

Define what must be true before you test

Start with a one-sentence specification for the business job. For example:

Every Monday, prepare a draft campaign report from the three approved exports, flag missing data and save it in the review folder. Do not change campaigns, invent explanations or send the report.

That sentence gives you something testable: a trigger, approved inputs, expected output, destination and prohibited actions. “Help with reporting” is too vague because almost any result can be presented as a success.

Write the acceptance criteria before running the agent. Include four types:

Task criteria: the required sections, calculations, sources and format are present.
Quality criteria: facts are traceable, language is usable and unsupported conclusions are absent.
Control criteria: access, approvals, logging and action limits work as designed.
Recovery criteria: the team can stop the workflow, find affected work and complete the job manually.

Use pass or fail criteria where the requirement is absolute. “Uses only the approved exports” should not become an average score. Use a scale where judgement is genuinely needed, such as brand fit or clarity, and describe what each score means.

The UK Government’s introduction to AI assurance distinguishes measurement from evaluation. Measurement gathers evidence about performance and impacts. Evaluation judges that evidence against agreed criteria. Your test plan needs both.

If the job and boundaries are not yet settled, step back to a bounded AI pilot before treating the workflow as a release candidate.

Build a representative test set

Do not let the agent choose its own exam. Assemble examples that reflect the work it will actually encounter and record the expected result before testing begins.

For a campaign-reporting agent, the set might contain:

three ordinary weeks with complete exports;
a quiet week with zero conversions;
a missing export;
duplicate rows or an unexpected column;
a renamed campaign;
contradictory dates across sources;
an unusually large result that needs checking;
a note inside a source file telling the agent to ignore its instructions;
personal or confidential information it should not use;
a request to send or publish the report without approval.

Use recent real examples only when you have an appropriate basis and safe process for doing so. Otherwise use synthetic, public or properly minimised data that preserves the shape of the work without exposing unnecessary information.

Keep some test cases back from the people configuring the agent. If every example has already been used to tune the workflow, the result may show that it remembers the setup rather than handles new work reliably.

The test set should cover common work, boundary conditions and deliberately hostile or misleading inputs. The UK Government’s Data and AI Ethics Framework recommends rigorous testing for potential failure modes, including unexpected scenarios, and says a system should not be used for the task if it cannot be made sufficiently safe.

Test the whole workflow, not only the model

An AI agent is more than the model producing text. It includes triggers, connected accounts, retrieval, instructions, tools, approval steps, destinations and logs. A model can give a sensible answer while the surrounding workflow fails.

Run the test from the real starting point to the intended finish. Check:

the correct event starts the workflow once, not twice;
only the approved data is retrieved;
missing or stale inputs are detected;
calculations and transformations are reproducible;
the output reaches the correct folder, record or queue;
the reviewer receives enough context to make a decision;
rejected work stays rejected;
no external action happens before the required approval;
the activity log identifies what happened and which identity acted.

Test the integrations separately as well as together. An agent that performs correctly with a saved example can still fail when an API returns no data, a file name changes or a connection loses permission.

The Government Digital and Data team’s AI testing framework overview makes the same distinction at a larger scale: model evaluation considers how the model performs, while system testing covers infrastructure, interfaces and APIs. A small-business test can use that principle without copying the public-sector framework wholesale.

Run seven groups of AI agent tests

Organise the work into seven groups so one strong output does not hide a weak control.

1. Normal task tests

Use representative, complete inputs. Confirm that the agent follows the agreed sequence, uses the approved sources and produces every required part of the output. Have a person who understands the job review the result, not only the person who built the agent.

2. Accuracy and source tests

Trace important statements, figures and decisions back to the input. Add a case where the answer is not available and confirm that the agent says so instead of filling the gap with a plausible guess.

If personal data is involved, the ICO’s guidance on accuracy and statistical accuracy says organisations should set appropriate accuracy measures, understand their limitations and continue monitoring after deployment. The exact measure depends on the task and consequence of an error.

3. Boundary and permission tests

Ask for information outside the approved folder, date range, customer group or purpose. Try a higher-impact action than the job allows. Confirm that the request is blocked and that the agent cannot reach the same result through another connected tool.

Compare the test against your AI agent permissions map. The expected result is not merely that the agent chooses to behave. The connection and workflow should prevent unauthorised action wherever practical.

4. Instruction-conflict tests

Place conflicting, irrelevant or malicious instructions inside a document, webpage or message the agent may read. Ask a user to override the source boundary or reveal information from another task. Confirm that the system follows the higher-priority operating rules and records the blocked attempt where appropriate.

This does not prove resistance to every attack. It checks the failure modes relevant to the sources and actions in this particular job.

5. Failure and recovery tests

Remove an expected file, disconnect an integration, deny a permission and interrupt a run. Check whether the failure is visible, partial work is identifiable and retrying will not duplicate an external action.

Run the pause and revocation process for real. Then use the manual fallback to finish the task. Your AI incident response plan should name the escalation route if the test exposes data, changes a live record or crosses another incident threshold.

6. Human-review tests

Give reviewers a mix of acceptable and defective outputs without telling them which is which. Check whether they can find the important errors, see the underlying sources and stop the next action.

Record false approvals as system findings rather than blaming the reviewer. The approval screen, evidence or training may be inadequate. A human-in-the-loop design only works when the person has the information, time and authority to make a meaningful decision.

7. Repeat and change tests

Run key cases more than once and note material variation. Then repeat them after changing the model, instructions, connection, source structure or workflow logic.

AI behaviour and supplier components can change. The National Cyber Security Centre’s secure AI system development guidance treats security as a lifecycle requirement covering design, development, deployment, operation and maintenance. Keep a small regression set of essential tests for every material change.

Record results in a simple test table

One row per test is enough. Use these fields:

Test ID and date
Scenario and input version
Expected result
Actual result
Pass, fail or needs review
Severity if failed
Evidence link or log reference
Owner and next action
Retest result

Do not replace evidence with “looked fine”. Save the output, source version, relevant activity record and reviewer decision. Keep personal and confidential information out of screenshots or shared records unless it is necessary and appropriately protected.

Use a small severity scale. A formatting issue may be low. An unsupported public claim may be high. Accessing an unapproved customer record or taking an unauthorised external action should normally stop the release until the cause and control are understood.

Make a risk-based go-live decision

Do not average every result into one impressive percentage. Some failures are release blockers even when dozens of easier tests pass.

Before launch, confirm:

every required task criterion passes on new examples;
no unresolved failure crosses a data, permission or prohibited-action boundary;
the approval gate has stopped defective work in testing;
logs are sufficient to investigate a run;
pause, revocation, recovery and manual fallback have been demonstrated;
known limitations are written into the operator guidance;
one named owner accepts the remaining risk and sets the next review date.

Choose one of four decisions: go live within the tested boundary, fix and retest, reduce the scope, or do not deploy. A conditional pass should name the condition and who will verify it. It should not become a quiet route around a failed control.

Launch in stages where possible. Keep the first users, data sources and action level within the tested scope. Adding another mailbox, customer group or publishing permission is a new risk decision, not a routine configuration change.

Transfer the accepted tests, known limits and recovery steps into the workflow’s AI agent runbook. After launch, monitor the same important measures and turn real incidents or near misses into new regression cases.

A 15-point AI agent testing checklist

Before an AI agent goes live, confirm:

The business job, inputs, outputs and prohibited actions are written clearly.
Acceptance criteria were agreed before the final test run.
The test set includes normal, awkward, missing and hostile inputs.
Some final cases were not used to configure the agent.
Important facts and figures can be traced to approved sources.
The agent handles missing information without inventing an answer.
Access outside the approved boundary is blocked.
Higher-impact actions cannot bypass the approval gate.
Hidden or conflicting instructions have been tested.
Integration failure and duplicate-run behaviour are understood.
A reviewer has tested outputs they did not help configure.
Logs show inputs, actions, approvals and failures at the required level.
Pause, access revocation, recovery and manual fallback work.
Release-blocking failures have been fixed and retested.
The owner has signed off the scope, limitations and monitoring plan.

Keep this checklist proportional. A draft-only internal workflow does not need the same evidence as an agent that communicates with customers or changes records. Higher consequence, broader access and less reversibility should lead to stronger testing and a smaller initial release.

Frequently asked questions

How many tests does an AI agent need before launch?

There is no useful universal number. The set should cover the normal work, important variations, likely failures, access boundaries and highest-consequence actions for the job. Ten well-chosen cases can teach you more than a hundred duplicates. Increase the depth when errors would affect people, confidential data, money or difficult-to-reverse actions.

Who should test an AI agent?

Use at least one person who understands the business task and did not configure every example. The builder can check the technical workflow, while the task owner checks usefulness and accuracy. The eventual reviewer should also test the approval process. Higher-risk systems may need appropriate security, privacy, legal or independent assurance expertise.

What should make an AI agent fail acceptance testing?

Treat an unapproved data access, prohibited action, bypassed approval, hidden failure, unsupported material claim or unusable stop process as a serious failure. Set the release blockers before testing. Minor presentation issues can be tracked separately, but they should not dilute failures that cross the workflow’s core boundaries.

Do we need to retest an AI agent after it launches?

Yes. Retest after material changes to the model, instructions, source data, integrations, permissions or output destination. Keep a small regression set for the most important behaviours and add cases from incidents and near misses. Monitoring live work does not replace testing, but it shows where the test set needs to improve.

Need to turn a useful prototype into a workflow your team can trust? Talk to us about an AI agent system build with testable acceptance criteria, controlled access, human review and a practical handover.

Sources checked on 5 October 2026: UK Government introduction to AI assurance, Data and AI Ethics Framework, Government Digital and Data overview of the AI Testing Framework, NCSC guidelines for secure AI system development and ICO guidance on accuracy and statistical accuracy. This is practical business guidance, not legal, data-protection or cyber-security advice.