Evidence note

How to Fight Production Incidents? An Empirical Study on a Large-Scale Cloud Service

Peer-Reviewed Research2022Microsoft Research, ACM SoCC 2022

What was examined?

An empirical analysis of high-severity Microsoft Teams production incidents covering root cause, detection, mitigation and operational learning.

152 high-severity production incidents over one year.

Key findings

01Approximately 60% of the incidents arose from infrastructure, deployment or service dependencies rather than direct code/configuration faults.
02Approximately 17% lacked adequate monitoring or telemetry coverage.
03Many code/configuration incidents were initially mitigated through operational mechanisms rather than an immediate permanent code fix.
04For configuration incidents, rollback was a common mitigation route.

Why it matters. Lornets interpretation.

Production reliability extends beyond application source code. Deployment, infrastructure, dependencies, observability, rollback and recovery are part of the technical position of an operating service.

This is the Lornets reading of the source, not a finding of the source itself.

What it does not establish

01The study concerns one large cloud service at one company.
02Incident percentages should not be generalised to all SaaS applications.
03The data reflects the operational environment and incident process of Microsoft Teams.