Handle load with jobs, caches, and backpressure

Document-preview jobs take time. Holding every concurrent job inside an HTTP connection can exhaust resources. A queue separates acceptance from completion and a cache reuses results, but neither supplies unlimited capacity or automatically prevents duplicate effects.

The synthetic queue holds two entries and produces one effect for repeated work. It uses an in-process deque, without a broker, Redis, or workers. Define overload, retry, and results before evaluating the guarantees missing from this model.

Separate acceptance from completion

An asynchronous API can return a job ID and expose progress separately. Acceptance must validate input, permission, capacity, and reliable recording. Appending to memory does not guarantee completion after restart.

Distinguish queued, running, succeeded, and failed states. Cancellation needs explicit semantics; stopping running work depends on its implementation. Result retrieval needs ownership, lifetime, and storage checks rather than access granted by a guessed ID.

Queue capacity and worker concurrency bound different in-flight work. Unlimited queues move overload into memory, disk, or waiting time. Define whether excess input is rejected, delayed, or admitted within a budget, and show its actual status.

Distinguish admission and execution controls

Rate limiting bounds requests over an interval. Backpressure slows producers when downstream capacity is unavailable. A concurrency limit bounds simultaneous work; queue capacity bounds waiting entries. These controls can cooperate without being interchangeable.

Per-tenant budgets can prevent one source consuming everything, alongside global limits. Costs may depend on file size, processing stages, and dependencies rather than job count alone. Release capacity after cancellation, timeout, and completion.

Clients need total waiting and retry budgets too. Immediate unlimited retries amplify overload. Observe queue wait, processing time, rejection, and failure categories before choosing more workers or tighter input bounds.

Confirmations and deduplication differ

RabbitMQ publisher confirms address acceptance between publisher and broker. Consumer acknowledgements address processing of deliveries. Automatic acknowledgement can mark a delivery handled before work completes and lose a redelivery opportunity if the consumer crashes.

Manual acknowledgement after durable work reduces some loss, but a crash after commit and before acknowledgement can still redeliver. Business effects need durable deduplication, uniqueness, or coordinated transactions. Acknowledgement alone does not establish exactly-once effects.

Check that repeated IDs carry the same input. Deduplication records must coordinate with effects: marking completed before work risks loss, while independently marking afterward leaves a duplicate window.

Give retries and poison messages an endpoint

Retry transient failures within attempt and time limits, with backoff and jitter where appropriate. Invalid format, denied access, and unsupported input generally need terminal errors rather than unchanged retry.

Isolate repeatedly failing messages and retain safe diagnostic reasons and a review path. Immediate requeue can create an endless loop. Isolation also needs capacity and monitoring; moving failures elsewhere is not itself recovery.

A timeout may stop waiting without stopping external work. A retry can then create a second result. Use result correlation, state lookup, and compensation for uncertain completion rather than blindly repeat side effects.

Observe capacity and duplicate effects offline

Two copies of J-1 occupy both waiting slots. J-2 is rejected. During consumption, the processed set prevents a second effect. Capacity control and effect deduplication are separate; duplicates are not combined on admission.

from collections import deque
queue = deque()
processed = set()
def accept(job):
    if len(queue) >= 2:
        return False
    queue.append(job)
    return True
assert accept("J-1") and accept("J-1")
assert not accept("J-2")
effects = []
while queue:
    job = queue.popleft()
    if job not in processed:
        effects.append(job)
        processed.add(job)
assert effects == ["J-1"]
print(effects)

The output is ['J-1']. The queue and set disappear on restart and do not provide atomic checks across consumers. This verifies sequential model behavior, not a durable exactly-once job implementation.

Checking processed during admission still leaves in-progress duplicates and concurrent races. A real service may create a unique request record and coordinate work with transactions, depending on whether the effect shares the same database.

State the cache's staleness policy

Include resource, version, and required tenant scope in cache keys. TTL bounds lifetime but does not promise current data on each read. Choose invalidation, versioned keys, or explicit staleness according to the use case.

Redis memory limits and eviction policies mean entries may disappear before expiration. The application must recover from misses. Simultaneous misses can overwhelm the origin; request coalescing or bounded refresh may help. Cached authorization needs revocation-aware treatment.

A cache should not be the only job record. Rebuild cached results without losing ownership or reexecuting an irreversible effect. On a miss, retrieve the original result where required.

Waiting also needs fairness. A large job can occupy workers while short jobs time out. Separate cost or category budgets may help, but must not permanently starve one category.

Exercise: verify overload and recovery

Define capacity, concurrency, tenant budgets, states, retries, and deduplicated results. Model duplicates, overload, transient and permanent failures, and a crash after processing but before acknowledgement.

Overload must remain visible and bounded, repeated success must not repeat effects, permanent failures must terminate, exhausted retries must enter isolation, and cache misses must recover results safely. Broker redelivery and restart behavior need actual integration tests. Deque assertions do not test durability or concurrent consumers.