When Incidents Refuse to End

The presentation titled When Incidents Refuse to End by Vanessa Huerta Granda explores the complexities and impacts of long-running technology incidents.

Key Points Discussed:

• Long-running incidents stretch teams and systems, revealing the distinction between "work as imagined" and "work as done" in the chaos of resolving them.
• These incidents expose organizational and system fragility, showing where there is a need for communication improvement and priority adjustments.
• The speaker emphasizes the importance of resilience, not only in technical systems but also in the people managing these incidents. This includes building systems for endurance, and better paging and escalation strategies.
• Three examples illustrate different incident experiences: a data center fire, internal systems complexity, and recurring holiday instability.
• The presentation highlights the need for a socio-technical approach to incident management, which includes understanding stress responses and leadership roles in high-pressure situations.
• Insights into handling incidents involve not only solving immediate technical problems but also managing human factors and learning from systemic issues exposed during prolonged outages.

The talk advocates for an integrated approach to incident management that considers both the technological and human elements to build stronger and more resilient systems and teams.

This is the end of the AI-generated content.

As engineers, we’re used to managing failure, but long-running outages hit differently. They stretch teams, systems, and assumptions about how incidents “should” play out. In this talk, we’ll dive into real examples of incidents that dragged on far longer than anyone expected, and unpack what they revealed about our systems, processes, and mental models.

We’ll explore what these situations taught us about coordination under pressure, shifting system behavior, and the limitations of our current practices for detection and response. We will also look at how a mindset of curiosity helped us make sense of the mess — not just to resolve the immediate situation, but to improve how we adapt, learn, and build stronger systems and teams.