# Make every release verifiable and recoverable Canonical page: https://yishto.com/en/learn/delivery/cicd-monitoring-and-incidents/ Author: [Yishto](https://yishto.com/en/about/) Connect artifact tracking, required checks, user health, telemetry, and incident response through a search release, while separating code rollback from data recovery. Published: 2026-11-10 Updated: 2026-11-10 A document service releases search. The homepage opens, but queries containing spaces fail. Building an artifact, deploying it, and allowing users to complete a task are separate facts that need evidence. A delivery process connects them and retains a verified version for recovery. In this case, build-1 is the last stable artifact, build-2 is a failed candidate, and build-3 passes the teaching checks. The identifiers do not refer to a real pipeline. The goal is to define release gates, online verification, and incident records. ## Trace a change to its artifact CI checks a change through dependency installation, compilation, and tests appropriate to its risk. CD delivers a verified artifact to a specified environment. Continuous delivery may retain approval steps, while continuous deployment usually emphasizes automatic environment promotion. State the workflow's actual rules. GitHub Actions uses YAML workflows with triggers and jobs in `.github/workflows`. Separate ordinary branch checks from jobs with release permissions. A build identity should not automatically have production credentials, and untrusted contributed code must not gain access to protected environments. Record the source commit, dependency locks, runtime, build parameters, and artifact identity. Where feasible, build once and promote that artifact to reduce differences caused by rebuilding. Manage environment configuration separately because identical artifacts can behave differently under different settings. ## Make gates reflect the risk Search touches input parsing, database access, and empty-result presentation. Check normal queries, blank input, no matches, and permission isolation. Coverage percentages do not establish that those paths work, and a homepage HTTP 200 does not prove search succeeds. A document site may also check content links, language routes, and metadata. Fix the names of required checks and demand a result for each. Missing, timed-out, unexecuted, and failed checks are different states, useful for separating product defects from verification infrastructure problems. Database migrations need compatibility planning. Adding compatible fields before switching code and later removing old structures is one approach to assess against the database and business constraints. Reverting application code cannot safely reverse written data or restore a removed column. Backup and restoration require separate verification. ## Encode the release rule This offline model requires two checks and a health result. Missing or failed checks preserve the previous version, as does failed health verification. It does not deploy anything. ```python stable = "build-1" required = {"tests", "content"} def release(candidate, checks, healthy): global stable if not required <= checks.keys(): return stable if not all(checks[name] is True for name in required) or not healthy: return stable stable = candidate return stable assert release("build-2", {}, True) == "build-1" assert release("build-2", {"tests": True, "content": False}, True) == "build-1" assert release("build-2", {"tests": True, "content": True}, False) == "build-1" assert release("build-3", {"tests": True, "content": True}, True) == "build-3" print(stable) ``` The actual output is `build-3`. An empty check dictionary, a content failure, and an unhealthy candidate all preserve build-1. Only the final complete success updates stable. Checking required names is deliberate: Python's `all([])` returns True, so an empty collection must not imply completed verification. Real health checks require a time window and request samples rather than one instantaneous boolean. Concurrent releases, locks, promotion permissions, and rollback actions need handling. An in-memory variable does not survive restarts; stable artifact identity and execution records belong in trusted durable storage. ## Locate failures with complementary signals Metrics summarize behavior such as request volume, failure ratios, and latency distributions. Logs describe individual events and error causes. Traces represent a request across components through related spans. Request IDs and deployment versions can connect them without copying user bodies, tokens, or credentials into telemetry. For a search incident, inspect the affected proportion and start time, then error categories for the same release, then the parsing, data-access, and response stages in traces. One slow request does not establish general slowness. Average latency can hide a small group of severe timeouts, so choose distributions and sampling appropriate to the user task. OpenTelemetry provides components and conventions for collecting, processing, and exporting signals. It is not automatically a complete monitoring backend. Storage, retention, redaction, and sampling omissions remain application and observability-system decisions. Correlation also does not guarantee that every causal relationship was captured. ## Give alerts an actionable response An SLO quantifies user-visible behavior, such as successful searches over a window. Alerts should identify sustained user failures or objective consumption, with a service, environment, scope, and investigation entry point. A brief CPU increase is evidence for analysis; whether it warrants waking someone depends on its user impact. Define responses to specific failures. Stop expanding a release when new-version failures rise. Inspect dependency status and retry pressure when dependencies fail. Limit incoming work and restore processing capacity during backlog growth. Alerts without ownership or a response become repeated noise. During an incident, mitigate impact before completing the causal investigation. Assign a coordinator to prevent conflicting configuration edits. Keep observed facts separate from hypotheses in a timestamped record of actions, owners, and outcomes. After rollback, verify the user task, deployed version, and error trend instead of relying only on the deployment tool's success message. Use one version identity across delivery and telemetry. A page identifying build-3, logs identifying only a branch, and cached build-2 content make attribution ambiguous. Include the artifact version in health responses and log context without exposing configuration secrets. Preserve before-and-after windows to investigate timing; temporal overlap is a clue and still requires a reproducing input and error path. ## Exercise: retain recovery evidence Create a release record containing the candidate commit, artifact, required checks, configuration summary, rollout method, health window, stable version, and recovery criteria. Cover queries with spaces, empty results, and denied access. Simulate a missing check, an online failure, and a migration that has already run. Accept the process when omitted checks cannot promote a release, verification refers to the promoted artifact, and recovery distinguishes code, configuration, and data. Write an incident record with impact, timeline, mitigation, and testable follow-up changes. A new test reproducing the space-query failure is a completion criterion; “improve testing” is not. ## Sources & further reading - [Workflow syntax](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax) - [OpenTelemetry concepts](https://opentelemetry.io/docs/concepts/) - [Site Reliability Engineering](https://sre.google/sre-book/table-of-contents/) Sources checked on 2026-11-10. Examples are teaching data, not production measurements. ## Related guides - [Gather evidence before changing code](https://yishto.com/en/learn/developer-workflow/debugging-with-evidence/) - [AI application release checklist: timeouts, retries, permissions, and rollback](https://yishto.com/en/learn/production/shipping-checklist/)