# Turn model capabilities into testable task contracts

Canonical page: https://yishto.com/en/learn/ai-foundations/prompts-context-and-capabilities/

Author: [Yishto](https://yishto.com/en/about/)

Define capabilities, context, outputs, and abstention for classification and extraction, then verify facts, provenance, and execution permissions beyond valid structured output.

Published: 2026-11-10
Updated: 2026-11-10

A document assistant receives text and proposes a category and summary. “Understand the document accurately” is insufficient as an acceptance rule. Define provenance, allowed context, fields, evidence, and abstention so that model suggestions and program behavior can be checked separately.

A fictional price sentence demonstrates an extraction subtask, then extends the same approach to classification. No model call runs here. The code validates a supplied answer and shows why prompts also need deterministic constraints.

## Connect capabilities to tasks

Models can classify, extract, summarize, and propose content; some support vision, speech, or tools. A capability label does not establish task quality. Classification needs confusing-category checks, extraction needs field evidence, summaries need omission checks, and tool use needs selection and argument validation.

Break the workflow into observable actions: classify into permitted categories, extract a title and source, or return needs_review. Images and damaged text need an appropriate modality path or an explicit rejection, not forced text processing.

Compare models using the same samples, latency, cost, and available interfaces. Product descriptions help narrow choices; task evaluation determines suitability. Different steps can use different processing methods and budgets.

## Separate instructions from source material

Prompts can define tasks, allowed evidence, outputs, and examples. Message roles and fields differ by API, so system, developer, and tool semantics are not universal. Verify the selected interface's official documentation.

Instructions embedded in a document remain data without authority. Clear separation helps communicate the boundary, while execution permission still requires program checks. Delimiters do not eliminate prompt injection.

Examples should include supported edge cases and insufficient evidence, using synthetic material. Specify an unknown result instead of inventing an answer to maintain the format. No magic phrase guarantees success; compare changes through fixed examples and error categories.

## Select context within a budget

Context can include questions, rules, retrieved passages, tool output, and state. Limits use tokens and interface-specific input/output budgeting. Character counts cannot guarantee fit. Old decisions in lengthy histories can interfere with current tasks.

Select relevant state, retrieve needed material, and retain document IDs, versions, and passage provenance. Summaries reduce length but should identify what they replace and what detail may be lost. Preserve originals for exact prices, identifiers, and quotations.

Truncation needs an observable outcome such as incomplete_material or segmented processing. Missing essential constraints means the output cannot represent the full task. Tool errors and stale results must not become authoritative context.

## Validate shape and facts separately

Structured Outputs can constrain supported schemas on supported interfaces. Official documentation also states that correctly structured output can contain mistakes. Handle support limits, refusals, incomplete responses, and truncation instead of assuming every call returns a business object.

An object containing item, price, and quote is only parseable structure. Its item must belong to the input, the price must follow extraction rules, and its quotation must exist in the source. Validate before persistence or execution.

This fixed example expects item A, price 12, and a matching quotation. Missing quotes and boolean prices are rejected.

```python
source = "Item A costs 12 units."
def verify(answer):
    if not isinstance(answer, dict):
        return False
    quote = answer.get("quote")
    return (answer.get("item") == "A" and type(answer.get("price")) is int
            and answer["price"] == 12 and isinstance(quote, str)
            and quote == "costs 12 units" and quote in source)
assert verify({"item": "A", "price": 12, "quote": "costs 12 units"})
assert not verify({"item": "B", "price": 12, "quote": "costs 12 units"})
assert not verify({"item": "A", "price": 12})
assert not verify({"item": "A", "price": True, "quote": "costs 12 units"})
print(verify({"item": "A", "price": 12, "quote": "costs 12 units"}))
```

The actual output is `True`. The valid answer passes; the wrong item, missing quotation, and invalid price type fail. The function hardcodes this sample's correct facts. It is a contract test, not a general extractor or a verifier of another document's truth.

## Treat abstention as ordinary behavior

Missing evidence, contradictory evidence, damaged inputs, and unsupported tasks require different states. Treating all as service errors wastes retries; filling all with confident answers hides content problems. Define UI and backend behavior for each.

A model-reported confidence number is not automatically a calibrated probability. Its relationship to errors needs independent evaluation. Field completeness, source coverage, category conflict, and deterministic rule results can provide more directly inspectable evidence.

Human review should expose the candidate category, supporting passages, missing information, and editing controls. Review outcomes may support evaluation after checking permissions and label quality. Acceptance by a user does not itself prove factual correctness.

## Use evaluation to choose the repair

Build a fixed set covering normal, short, ambiguous, unsupported, and injected inputs. Version context selection, prompts, models, schemas, and validation rules. Compare changes on the same samples and inspect both target improvements and new failures.

Missing evidence needs retrieval repair. Missing field checks need program changes. Ambiguous categories need clearer definitions and instructions. Unstable results may require narrower tasks or human review. One success does not replace coverage across failure types.

## Exercise: deliver a classification contract

Specify inputs, categories, fields, evidence rules, abstention, and prohibited side effects. Include an unclassifiable document and malicious instructions inside a source. Extend the price example to multiple objects and validate object-price relationships.

Accept it when a wrong object cannot pass with a valid quote, unsupported facts remain absent, refusals and incomplete responses cannot appear successful, and external content cannot grant tool permissions. Keep failing samples and before-and-after results. Offline checks validate contract rules; model quality, token use, and latency need separate measurement.

## Sources & further reading

- [Text generation guide](https://developers.openai.com/api/docs/guides/text)
- [Structured outputs guide](https://developers.openai.com/api/docs/guides/structured-outputs)
- [Evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices)
- [LLM01: Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/)

Sources checked on 2026-11-10. Examples are teaching data, not production measurements.

## Related guides

- [LLM API basics: tokens and context budgets](https://yishto.com/en/learn/models/context-and-tokens/)
- [Streaming and structured output solve different problems](https://yishto.com/en/learn/models/streaming-and-structured-output/)