How to Perform a Successful Incident Postmortem

When growing a business from a startup to a large enterprise, it’s software systems also expand in complexity, meaning that encountering incidents is inevitable. These incidents have indirect costs that can include loss of trust in the product or brand. However, incidents occurring are not necessarily a bad thing — it provides the business with new learning opportunities and the chance to improve its operational practices. But how do we learn from such a failure?

What is a postmortem?

An incident postmortem is a meeting that brings together all of the people that were involved, whether directly or indirectly, to discuss, document and derive value. Postmortems should remain positive to avoid demotivating the technical teams further. Postmortems should also be blameless — this empowers engineers to provide details of their contribution, and prevents the establishment of a fear culture. Learning should be the focus rather than dwelling on what went wrong, and so any past tense discussions should stick to facts instead of opinions — avoiding phrases that include would have, could have or should have is essential.

Who should get invites?

A variety of different stakeholders should get invites to the postmortem, including:

  • Individuals/Teams that Responded to the Incident
  • Individuals/Teams that Diagnosed the Issue(s)
  • Individuals/Teams that Rectified the Issue(s)

What should get documented?

Thorough documentation of the event is vital. Memories are short, and in time important details fade into obscurity. Some key facts to document are:

  • Date and Time that the Incident was First Logged
  • Which Teams/Individuals Responded to the Incident
  • Number of Users/Accounts Affected
  • Number of Support Requests Raised
  • Date and Time that the Incident was Fixed
  • Any Solutions or Mitigations

What should you do after the postmortem?

The postmortem produces discussion points to bring into further sessions and documentation that will prove valuable in the future.

Welsh Vegan Senior Software Engineer. https://kylejones.io

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store