What an AI evaluation set actually tests

An evaluation set checks an agent against real work, including when it should stop and ask a person. It provides a repeatable check before launch and after changes.

An AI agent can give a convincing demonstration and still be unsuitable for your workflow. A tidy example shows that it can handle that example. It does not show what happens when a customer leaves out an order number, two policies conflict, or a connected system is unavailable.

An evaluation set makes those situations testable. It is a list of real cases from your own work, with the correct handling written down for each one. You run the agent against those cases and compare what it does with what your team agreed it should do.

The purpose is not to prove that the agent sounds helpful. It is to establish whether it handles a defined job well enough, within clear limits.

Start with cases your team recognizes

A useful evaluation set starts with the workflow, not a collection of clever questions designed to confuse AI.

For a customer support agent, the source material might be past tickets and the policies used to resolve them. For an operations agent, it might be incoming documents, processing rules and records of exceptions.

Select material your team is authorized to use, and remove personal information that is not needed for the test. Preserve the details that affect the decision.

Each case needs enough context to judge the result:

• The request or document the agent receives.
• The relevant policy, records or system state.
• The expected answer or action.
• Anything the agent must not do.
• The point at which a person must take over.

The expected result is not always a single sentence to copy. Several phrasings may be acceptable. What matters is whether the agent uses the right facts, follows the process and respects its permissions.

Writing this down can expose gaps in the workflow itself. If experienced staff disagree about the correct handling, that case needs a business decision before it can become a fair test.

Include ordinary work, exceptions and handoffs

Typical cases establish whether the agent can do the job it was built for. Include common requests in the forms your team actually receives, including short messages, imperfect wording and incomplete context.

Then add awkward cases. These should reflect plausible difficulties in your workflow, rather than puzzles unrelated to the job.

Consider a hypothetical agent that helps with order enquiries. Its evaluation set could include:

• A status request with a valid order reference and an available shipping record.
• A request without enough information to identify the order.
• A cancellation request for an order already dispatched.
• A request to see another customer's order details.
• An order lookup when the connected system is unavailable.
• A complaint that needs a supervisor's decision.

Each tests a different behaviour. Finding a shipping date is not the same as recognizing missing information. Recognizing a cancellation request is not the same as having permission to cancel.

A correct handoff is a successful result, not a failure to answer. If the agent lacks evidence, authority or access, the expected handling may be to stop, explain the limitation and pass the relevant context to a person.

The test should specify what that handoff contains. A vague instruction to contact support may not be enough if the workflow requires creating a review item for staff.

Check behaviour, not just the final wording

A polished answer can conceal a wrong action. Evaluation needs to cover both what the agent says and what it does.

For an answer, check factual accuracy, use of the relevant source, completeness and whether it makes unsupported claims. For an action, check the target record, the requested change, permissions and any required approval.

In the hypothetical order workflow, an agent might correctly explain the cancellation policy but change the wrong order. Or it might report that a cancellation succeeded when the system rejected it. Neither should pass because the message reads well.

Useful checks include whether the agent:

• Retrieves the information needed for the decision.
• Avoids guessing when evidence is missing.
• Requests approval before a restricted action.
• Reports system failures honestly.
• Leaves enough context for staff to continue the work.

Some checks can be automated, such as whether a record changed or a required field is present. Others need human judgement, particularly whether an explanation is clear and a handoff is useful.

TEFY's agent service includes testing on the client's own cases before launch. That makes the evaluation relevant to the actual workflow, rather than a generic demonstration of AI capability.

Agree the pass conditions before launch

A threshold is the agreed boundary between acceptable performance and work that still needs fixing.

Start by separating failures according to their consequences. An awkward phrase, a missed request and an unauthorised record change should not carry the same weight.

The business owner or workflow lead should help decide which errors are tolerable, which require review and which must block launch. The build team then turns those decisions into measurable checks.

For example, a hypothetical pilot could require that every restricted action in the test set waits for approval. It could separately track whether routine enquiries are answered correctly and whether handoffs include the required context.

Do not rely on one overall score. Strong performance on easy requests can hide a serious weakness in a smaller group of sensitive cases.

Passing the set is evidence within the tested scope, not a guarantee that every future request will be handled correctly. That is why evaluation belongs alongside limited permissions, approvals and monitoring. TEFY's delivery approach includes agreed test cases and thresholds before launch.

Keep the set useful after launch

The evaluation set should remain part of maintaining the agent. Run it after every change to check whether an improvement has also broken something that previously worked.

A prompt update might make answers shorter but remove a necessary qualification. A model update might handle document extraction differently. A revised policy might make an old expected answer wrong.

Keep the cases current, record which version was tested and add newly discovered failures. Preserve older cases when they still represent valid work. They help prevent the same mistake from returning.

For a next step, choose one workflow and gather representative requests, exceptions and handoffs. Write down the correct handling with the person accountable for that work.

If you cannot yet define those expectations, clarify the process first. If you can, you have the beginnings of a useful evaluation set and a concrete starting point for a fit call.