Evidence note

How to Fight Production Incidents? An Empirical Study on a Large-Scale Cloud Service

Peer-Reviewed Research2022Microsoft Research, ACM SoCC 2022

What was examined?

An empirical analysis of high-severity Microsoft Teams production incidents covering root cause, detection, mitigation and operational learning.

152 high-severity production incidents over one year.

Key findings

1Approximately 60% of the incidents arose from infrastructure, deployment or service dependencies rather than direct code/configuration faults.
2Approximately 17% lacked adequate monitoring or telemetry coverage.
3Many code/configuration incidents were initially mitigated through operational mechanisms rather than an immediate permanent code fix.
4For configuration incidents, rollback was a common mitigation route.

Why it matters. Lornets interpretation.

Production reliability extends beyond application source code. Deployment, infrastructure, dependencies, observability, rollback and recovery are part of the technical position of an operating service.

This is the Lornets reading of the source, not a finding of the source itself.

What it does not establish

1The study concerns one large cloud service at one company.
2Incident percentages should not be generalised to all SaaS applications.
3The data reflects the operational environment and incident process of Microsoft Teams.