From demo to deployment: design the failure paths

Deploying an AI application is more than moving one successful demo to a server. Real use brings timeouts, rate limits, missing evidence, permission changes, and uncertain outcomes.

Acceptance should cover both success and what the user sees, what the system saves, and how execution recovers after failure.

Define completion as a business result

Returned text is not necessarily usable output. Check fields, evidence, business constraints, and authorization. For writes, confirm the external result rather than only the model's claim that it completed the action.

Map the deployment boundaries: application pages, model service, task executor, and storage. Static websites suit product information and learning content; browser code must not contain a model service credential.

Separating frontend and backend responsibilities makes it clear which failures permit a user retry and which require server-side recovery.

Build a failure matrix

Failure	User feedback	System behavior
Timeout	Explain that the task is unfinished and offer safe recovery	Save explicit state and bound retries
Rate limit	Explain temporary unavailability	Respect waiting instructions and remaining budget
Missing evidence	Explain insufficient material	Avoid invention and allow clarification
Denied permission	Explain unavailable access	Stop rather than bypassing with another tool
Uncertain write result	Explain that the result is being checked	Query status or await confirmation instead of repeating blindly

You can test these branches with constructed results before making real model calls. A generic “try again later” message hides different recovery requirements.

Bound retries in code

This example makes no network request. Even a potentially transient error requires an idempotent operation, remaining attempts, and remaining time before another try.

def can_retry(status, attempts, idempotent, remaining_seconds):
    transient = {408, 429, 500, 502, 503, 504}
    return (
        status in transient
        and idempotent
        and attempts < 3
        and remaining_seconds > 0
    )

assert can_retry(503, 1, True, 10)
assert not can_retry(503, 1, False, 10)
assert not can_retry(401, 1, True, 10)
assert not can_retry(429, 3, True, 10)
assert not can_retry(503, 1, True, 0)
print("Retry boundaries checked")

Real execution also needs per-attempt timeouts, backoff, and a total deadline. Follow rate-limit waiting guidance and verify actual idempotency semantics for side effects instead of simply setting a flag to true.

Accept deployment with evidence

Check actual public routes, HTTPS, missing-page status codes, and static asset caching. Searchable articles should return their body directly rather than a shell requiring JavaScript to fill it.

Check authorization, cancellation, duplicate requests, and process interruption in the application. Record versions and necessary metrics while avoiding user-content logging by default.

Use task evaluations for model quality and constructed faults for recovery behavior. Both are needed to show that the application can answer and can handle being unable to finish.

Try it next

Fill out a failure matrix and trigger every row. Combine user feedback, saved task state, and recovery behavior into one acceptance condition.

Keep a recoverable version before release. After deployment, use observed failures to improve evaluation cases, capability boundaries, and recovery instead of treating a running server as the end of delivery.