23、Analysis of a Google Satellite Machine Outage Post - Mortem

postgres8guard

于 2025-09-22 10:27:29 发布

阅读量20

点赞数

CC 4.0 BY-SA版权

分类专栏： SRE实战：从理论到落地文章标签： Google satellite outage post-mortem

本文链接：https://blog.youkuaiyun.com/postgres8guard/article/details/153315426

SRE实战：从理论到落地专栏收录该内容

46 篇文章 ¥499.90

订阅专栏¥69.90

会员秒杀 ¥9.9 重磅福利

超级会员免费看

Analysis of a Google Satellite Machine Outage Post - Mortem

1. Incident Overview

On August 11, 2014, from 17:10 to 17:50 PST8PDT, Google experienced a significant incident where all satellite machines were sent to diskerase. This incident had far - reaching impacts on frontend services.
- Impact :
- Frontend queries dropped.
- Some ads were not served.
- There was a latency increase for all services normally served from satellite for nearly two days.
- Root Cause : A long - standing input validation bug in the Traffic Admin server was triggered by the manual re - execution of a workflow to decommission the a12bcd34 satellite. This bug removed the machine constraint on the decom action, sending all sa