Software Incident Response and Recovery Operations Guide

Define incidents from service impact and required authority

An incident is a condition that requires coordinated action beyond ordinary work because it threatens a service, data, security, safety, financial integrity, or another important outcome. Define examples and severity criteria using user impact, scope, duration, sensitivity, exploitability, legal or contractual concern, and recovery complexity. Avoid declaring severity from one infrastructure metric. A database can be online while transactions are wrong, and an application error can affect only one critical customer workflow.

Specify who may declare, lead, escalate, contain, restore, communicate, engage vendors, preserve evidence, and close each incident class. Name backups and out-of-hours coverage. Separate the incident commander’s coordination role from technical investigation when possible. The latest NIST SP 800-61 revision integrates incident response across governance, identification, protection, detection, response, and recovery instead of treating it as an isolated sequence that begins only after an alert.

Prepare service knowledge before a stressful event

Maintain a service inventory connecting owners, repositories, environments, dependencies, data stores, queues, scheduled jobs, third parties, domains, certificates, credentials, dashboards, backups, recovery objectives, support contacts, and customer-facing outcomes. Keep architecture and data-flow views current enough to support decisions. An incident is a poor time to discover that a former contractor owns the domain, an undocumented job changes records nightly, or a critical webhook has no replay procedure.

Create concise runbooks for likely high-impact conditions while preserving room for judgment. Include detection evidence, initial safety checks, access required, diagnostic queries, containment choices, validation, rollback or forward-repair limits, communication triggers, and escalation. Record when the runbook was last exercised and which service version it describes. Exercise expired certificates, compromised credentials, cross-tenant access, provider outage, failed database change, corrupted data, queue backlog, messaging failure, and loss of primary administration. Tests should expose missing authority and access, not merely confirm that participants can read a script.

Build detection from user journeys and protected evidence

Instrument the outcomes users depend on: authentication, record changes, payment, upload, notification, synchronization, report generation, or another critical transaction. Combine application and infrastructure metrics, logs, traces, security events, synthetic checks, business reconciliation, and support reports. OpenTelemetry describes observability as the ability to ask new questions from emitted signals; collecting a large log volume without correlation, ownership, and useful retention does not create that ability.

Define what to log, where it is generated, which identity and resource context is necessary, who can access it, how integrity is protected, and how long it remains useful. CISA recommends enabling and centralizing relevant logs, monitoring high-risk events, protecting logs from unauthorized access or deletion, and assigning response roles. Avoid secrets, full documents, and unnecessary personal data in telemetry. Test that evidence survives the failure and that investigators can correlate it across services and time sources.

Coordinate investigation without losing facts or decisions

Use one incident record for status, severity, affected services, timeline, owners, verified facts, hypotheses, actions, approvals, communications, and next checkpoints. Mark observations separately from interpretation. Preserve the command, query, deployment, configuration, and customer evidence supporting consequential decisions. A chat channel is useful for rapid collaboration but weak as the only permanent record because messages are edited, fragmented, inaccessible, or mixed with speculation.

Set an update cadence appropriate to impact and uncertainty. Ask repeatedly: what changed, what evidence supports it, what is the safest next action, who owns that action, and when will the team reassess? Use a handoff format for long events. Limit broad access to sensitive security details while keeping decision-makers informed. Record why an apparently relevant alert was rejected so the same dead end does not consume the next shift.

Choose containment and recovery as explicit tradeoffs

Containment can disable an account, revoke a token, isolate a tenant, block traffic, pause processing, disable a feature, stop a rollout, quarantine a device, or take a service offline. Each option changes user impact, evidence, attacker access, data consistency, and recovery work. Define who may authorize it and how the action is verified. Avoid a fleet-wide or tenant-wide control whose scope cannot be previewed and whose retry can apply twice.

Recovery may involve rollback, forward repair, failover, restoration, replay, reconciliation, credential rotation, rebuilding from known sources, or controlled manual service. A successful deployment or provider status does not prove the business workflow recovered. Verify representative transactions, data integrity, queued work, notifications, integrations, access boundaries, and monitoring. DORA’s continuous-delivery guidance emphasizes keeping software deployable and using fast feedback; incident recovery benefits from the same immutable artifacts, tested changes, and visible release evidence.

Communicate facts, uncertainty, and next actions

Define internal, customer, partner, leadership, legal, regulatory, insurance, and public-safety audiences according to context. Use approved templates as completeness prompts, then write plain messages with affected service, observed impact, known scope, current action, available workaround, next update, and support route. Do not speculate about root cause or promise restoration times without evidence. Correct earlier statements visibly when facts change rather than silently rewriting the timeline.

Keep customer communication separate from confidential investigation and formal regulatory reporting. Qualified owners must determine notification and reporting obligations. Provide alternate publishing access if the primary identity or network is unavailable, with strong protection and later review. After restoration, remove stale banners and scripts across every channel. Measure approval and delivery delays, but do not optimize message speed at the expense of accuracy or expose protected evidence to demonstrate transparency.

Review the system without converting learning into blame

Reconstruct the event from detection through recovery using timestamps, evidence, decisions, organizational conditions, and customer impact. Ask why safeguards, tests, monitoring, review, or recovery behaved as they did. Distinguish triggering change from deeper contributors such as unclear ownership, unsafe defaults, architectural coupling, inaccessible runbooks, incentive pressure, missing environments, or vendor limits. A blameless review still assigns named owners and deadlines to bounded improvements.

Track recurring causes, repeated manual work, overdue actions, detection gaps, restoration failures, and whether prior improvements changed later outcomes. Share appropriate lessons with product and support teams. Protect sensitive incident material and personal information. Review the review process itself: if reports are lengthy documents nobody uses, replace them with concise evidence and linked work. Verify corrective actions through tests, production signals, or exercises instead of closing tasks when code is merged.

Commission incident readiness as a production deliverable

Require a service catalog, ownership and escalation map, critical-journey indicators, protected logging, alert routes, incident roles, runbooks, recovery procedures, communication paths, vendor contacts, backup evidence, and at least one realistic exercise before launch. Acceptance should include unavailable responders, an identity outage, incomplete telemetry, conflicting hypotheses, a failed containment step, and data reconciliation. Small systems may use simple tools, but they still need decisions and access that survive pressure.

Preserve client control of repositories, cloud and monitoring accounts, domains, backup keys, deployment pipelines, incident records, status systems, and vendor relationships. Use the DevOps and platform engineering improvement guide for delivery foundations and the application security requirements checklist for preventative controls. Then share the service, critical workflows, architecture, current incidents, operating constraints, response coverage, and desired recovery outcome through the project questionnaire.

Authoritative references

• NIST SP 800-61 Revision 3 incident response guidance
• CISA guidance for logging business systems
• OpenTelemetry observability primer
• DORA continuous delivery capability

Related software planning guides

• Continuous Delivery Pipeline Cost and Budget Planning Guide
• Continuous Delivery Pipeline Requirements Checklist
• DevOps and Platform Engineering: A Practical Improvement Guide