Design failure and recovery before splitting services

An order service works on one machine, and its inventory module is about to become a separate service. The change introduces business states such as an order being created while inventory reservation fails, or a successful reservation whose reply disappears. Define those states before choosing service boundaries so that the extra deployment units solve a concrete problem.

The fictional case is a textbook order, O-17, for two sets. Inventory owns reservations; orders own confirmation. All identifiers and quantities are teaching data. The goal is to prevent duplicate reservation and make unfinished orders discoverable and recoverable.

Connect service boundaries to constraints

A monolith can contain separate order, inventory, and billing modules while maintaining constraints in one database transaction. A modular monolith is a reasonable starting point because calls are easier to trace, deployment units are fewer, and transaction boundaries are explicit. Independent scaling, ownership, failure isolation, or release schedules can provide specific reasons to split it.

Services add protocols, timeouts, authentication, version compatibility, and observability costs. An order service that requires a synchronous inventory reply cannot confirm orders when inventory is unavailable, even if more order replicas are running. Draw dependencies along the request path before counting instances.

Write the business constraints first: inventory cannot become negative; an order reserves only once; users can inspect processing status; an unknown result cannot be treated as a definitive failure followed immediately by a new order. Set latency and availability objectives separately for reads, creation, and recovery. They may require different consistency behavior.

A timeout means an unknown result

A remote operation has at least three stages: sending the request, committing on the receiver, and returning the reply. If the reply disappears, the caller sees a timeout after inventory has changed. Retries should reuse a stable business key, such as the order ID combined with the reservation action. Inventory must persist the processing record and return its previous result for a duplicate request.

A client-side sent flag disappears after a restart. Bind an idempotency key to parameters as well: a request for two sets followed by a request for three sets under the same key should be rejected as a conflict. Keep the processing record and inventory change consistent within inventory's own transaction boundary.

An order can move through pending, reserved, confirmed, and cancelled states. Transitions need preconditions. An old timeout task must not cancel an order that has already been confirmed. A recovery task queries the reservation result and advances the order, or records why it was cancelled. Releasing inventory is also an idempotent action tied to the original reservation.

Preserve evidence for cross-service recovery

Two local database transactions do not automatically form one cross-service atomic transaction. An eventually consistent process accepts intermediate states and therefore needs a waiting limit, retry budget, and manual investigation path. Compensation can reverse only effects that the business allows to be reversed. Changing a database row cannot retrieve textbooks that have already shipped.

Messaging has a gap between committing data and sending an event. An outbox approach writes business data and a pending event in one transaction, then publishes it in the background. Consumers deduplicate by event ID. Duplicate delivery, backlog, and replay still need handling. Event versions and order correlation distinguish an initial request from later attempts.

Recovery jobs must be safe to run repeatedly. Select pending records that have exceeded the chosen waiting interval, then reread their current state before acting. Retain the next retry time and last error category so one unresolved record cannot consume an unlimited loop of work.

Majority counting has a limited role

Raft separates consensus for a replicated log into leader election, log replication, and safety rules. A fixed group of five members requires three distinct members for a majority. Majorities intersect, but safety also depends on terms, log matching, election restrictions, and durable state.

A network partition can divide those members into groups of three and two. If a write requires majority confirmation, the smaller group must stop confirming that kind of write. CAP describes limits under particular definitions of consistency and availability during a partition. It is not a universal instruction to select two properties for every operation at all times.

This example only counts votes in a fixed membership set. Duplicate votes do not add members, and unknown members are rejected.

members = {"A", "B", "C", "D", "E"}
def majority(votes):
    distinct = set(votes)
    if not distinct <= members:
        raise ValueError("unknown member")
    return len(distinct) >= len(members)//2 + 1
assert majority(["A", "B", "C"])
assert not majority(["A", "A", "B"])
assert not majority(["D", "E"])
print(majority(["A", "B", "C"]))

The actual output is True. Three distinct members form a majority; A, A, B represents two members, as does D, E. These assertions check counting. They do not simulate Raft, log replication, clocks, disk failures, membership changes, or arbitrary malicious nodes. Installing the function in an application does not give its database consensus guarantees.

Verify replicas, backups, and recovery objectives

Replicas can help survive a machine failure, while an accidental deletion may propagate to every replica. Backups preserve recoverable history and should be separated from the current failure domain. Replicas sharing power, networking, or administrator credentials can fail together. Geographic separation also introduces latency, synchronization, and operational costs.

RTO is the target time to restore service. RPO is the acceptable window of data loss. A teaching requirement of two hours for restoration and fifteen minutes of data loss requires checks on backup frequency, restore permissions, and log replay. These values are proposed requirements, not measured results from this exercise.

In an isolated environment, select a backup, restore it, check order and inventory relationships, and verify read and create operations. Record the time until service is usable and the range of missing data after the recovery point. A successful backup job alone does not establish readability, schema compatibility, or usable permissions.

Exercise: resolve unknown outcomes

Create a transition table for O-17. For each transition, name its trigger, preconditions, persistent evidence, and next action after failure. Simulate reservation with a lost reply, duplicate requests, a restart before confirmation, duplicate compensation, and prolonged loss of contact with inventory.

Accept the design when repeated requests cannot consume additional inventory, conflicting parameters are rejected, and a restart can continue from durable records. Unresolved states must remain queryable without appearing completed. Add a restore record that reports service recovery time separately from confirmed data loss. If the design cannot meet its objectives, revise the architecture or the objectives and retain the reason.