Build a task evaluation set before choosing a model

Choosing a model means deciding whether a component can perform your task. It does not mean picking the winner of a leaderboard. Tasks differ in acceptable latency, output requirements, and the cost of failure.

Define completion before comparing candidates. A longer answer or a more confident tone can otherwise be mistaken for better performance.

Define success using the task

For extracting a product name from a public description, success might require complete fields, no invented names, and an explicit state when the information is missing. For tool use, also check whether the selected action is allowed.

Include ordinary inputs, missing information, ambiguity, long content, and untrusted instructions. Keep an expected result or scoring rule for every case and record its source category.

Do not put accounts, customer material, or private documents into a public evaluation set. Start with public or constructed examples, then use a controlled private process to evaluate a real workload.

Hold variables constant and repeat

Fix task instructions, retrieved evidence, and the output contract when comparing models. Record versions, relevant parameters, and run times. Repeat cases to observe stability instead of selecting one successful answer.

Make scores interpretable. Valid-format rate, evidence support, task completion, and the ability to stop when action is inappropriate are more useful than an undefined overall score.

Generate a report from teaching data

Every number below is invented. The example demonstrates aggregation and failure reporting; it says nothing about the performance of any actual model.

from statistics import median

runs = [
    {"case": "clear-input", "passed": True, "latency_ms": 220},
    {"case": "missing-evidence", "passed": False, "latency_ms": 180},
    {"case": "untrusted-text", "passed": True, "latency_ms": 310},
]
report = {
    "pass_rate": sum(r["passed"] for r in runs) / len(runs),
    "median_latency_ms": median(r["latency_ms"] for r in runs),
    "failed_cases": [r["case"] for r in runs if not r["passed"]],
}
assert report["failed_cases"] == ["missing-evidence"]
assert report["median_latency_ms"] == 220
print(report)

A real report should group results by task category and preserve failure reasons such as timeout, refusal, or invalid fields. Different failures deserve different remedies.

Compare quality, time, and cost together

Eliminate candidates that fail a minimum quality requirement before optimizing latency and cost. A model with a good average score may still be unsuitable for actions if it frequently crosses permission boundaries.

Total cost includes retrieval, tools, retries, and human effort spent on failures. Prices and service conditions can change. Do not treat a one-time estimate as a permanent fact.

Evaluation rules can run locally and do not require a particular hosted platform. Check the current lifecycle of an external evaluation service before adopting it, and keep scoring logic portable.

Try it next

Choose ten public or constructed cases representative of your task and write their scoring rules. Include a case with no answer and one that attempts to induce an unauthorized action.

Inspect failures manually before deciding to change the model, prompt, or task decomposition. Rerun the same cases after each change and add newly discovered failures to the set.