Observe AI apps: first-token latency, total time, and cost

A fast experience may mean earlier first output or earlier task completion. First-token latency and completion time are different metrics. Streaming does not automatically shorten total model execution.

Understand the stages of a request before deciding whether to optimize retrieval, the model, tools, or presentation.

Observe along the request path

A task may include queuing, retrieval, a model request, tool execution, and final validation. Correlate these stages with one request identifier and retain their timing and status.

A trace describes the request path; a span represents a unit of work. Parallel span durations cannot simply be added to obtain total latency because they may overlap.

Define where first-token latency is measured: from server receipt to the first displayable output, or from the browser request to visible content. These measurements include different work.

Record enough to locate a failure

Keep model versions, error categories, retry counts, token usage, and tool types. Avoid logging full prompts, user content, credentials, or document bodies by default.

Use a separate controlled process for failures requiring content inspection. Telemetry should help diagnose behavior rather than collect every piece of data.

Practice percentiles with a small sample

This example uses the nearest-rank percentile definition and invented numbers. Five observations demonstrate the calculation, not stable production P95 performance.

from math import ceil

def percentile(values, p):
    if not values or not 0 < p <= 1:
        raise ValueError("samples and valid percentile required")
    ordered = sorted(values)
    return ordered[ceil(p * len(ordered)) - 1]

latencies = [100, 120, 200, 250, 600]
assert percentile(latencies, 0.5) == 200
assert percentile(latencies, 0.95) == 600

# Invented rate units, not any service's actual price.
input_tokens, output_tokens = 1200, 200
input_rate, output_rate = 0.2, 0.8
estimated_units = (input_tokens * input_rate + output_tokens * output_rate) / 1000
assert abs(estimated_units - 0.4) < 1e-9
print({"p95_ms": percentile(latencies, 0.95), "cost_units": estimated_units})

Actual cost depends on current billing rules and may include cache, reasoning, tools, and service fees. The example rates illustrate arithmetic only and must not be used for a real budget.

Diagnose before optimizing

For slow retrieval, inspect the index and query scope. For slow first output, separate queuing, input length, and tools that must run first. For slow completion, inspect output length and repeated calls.

Retries can improve success while increasing tail latency and cost. Caches can reduce repeated work, but must account for document versions and permissions. Do not share restricted results across users.

Check quality after a performance change. A faster answer that misses required evidence is not an equivalent completion of the same task.

Try it next

Record first output, completion time, and stage spans for one task. Keep measurement definitions fixed and inspect median latency, tail latency, and failure rate by task category.

Change one bottleneck at a time and rerun the quality cases. The next guide combines telemetry with explicit failure behavior in deployment acceptance.