In 30 seconds

After a serious failure, write down what happened, what it affected, why it happened, and what you changed so it will not happen again. The point is learning and prevention, not blame.

Why this comes up

In the rush to fix a serious problem, the understanding of what went wrong lives only in the heads of the people who fixed it. Weeks later, nobody remembers the details, and a similar failure catches everyone by surprise. A short, honest record turns a bad day into a lasting improvement.

What to record

Keep it brief and factual:

  • What happened: a plain description of the failure.
  • When: the timeline, from when it started to when it was resolved. Your audit trail helps here.
  • The impact: what and who was affected, and how much.
  • The cause: why it happened, traced to the real root, not just the surface symptom.
  • The response: what you did to stop it and to correct the affected cases.
  • The prevention: what you changed so it will not recur, including any new test cases or monitoring.

Focus on the system, not the person

A good incident record asks how the system allowed the failure, not who to blame. Blame makes people hide problems; a blameless record makes them surface them. If a person made a mistake, the useful question is why the workflow made that mistake easy or its consequences large.

Turn the incident into prevention

The most valuable part is the last one. For each incident, ask:

  • What test case would have caught this? Add it to your golden test set.
  • What monitoring would have caught it sooner?
  • What guardrail would have limited the damage?

An incident you learn nothing from is likely to happen again.

A small example

An agent sent duplicate confirmations after a retry went wrong. The incident record traced it to a missing duplicate check, noted the customers affected, and led to two changes: a guardrail preventing duplicate sends, and a test case for the retry scenario. The same failure could not recur unseen.

What this does not solve

Documenting an incident helps prevent its recurrence; it does not undo the harm already done, and it only helps if the prevention is actually implemented. A record that leads to no change is just paperwork. Close the loop by making the fixes real.

Good enough to move on when

  • The incident record covers what, when, impact, cause, response and prevention.
  • The cause is traced to its root, blamelessly.
  • At least one concrete prevention was implemented.
  • A test case or alert now covers the failure.