post-mortems

A List of Post-mortems! Table of Contents Config Errors Hardware/Power Failures Conflicts Time Database Uncategorized Other lists of postmortems Analysis Contributors Config Errors Allegro. E-commerce site went down after a sudden traffic spike caused by a marketing campaign. The outage was caused by a configuration error in cluster resource management which prevented more service instances from starting even though hardware resources were available. Amazon. A configuration change in the Seoul region incorrectly removed the setting that specifies the minimum healthy hosts for the EC2 DNS resolver fleet, so the system fell back to a very low default. The fleet's healthy host count dropped and in-VPC DNS queries from EC2 instances failed for ~84 minutes until capacity was manually restored. AWS added semantic config validation and per-hour throttling on host removal as remediations. CircleCI. A deploy changed the type of a field in the PostgreSQL database used by the job-distribution service. Newly-written rows used the new type and old rows used the old type, so the distributor's strict schema validation failed on each scan and it stopped distributing work. Rolling back made the rows written between the two deploys unreadable, so distribution kept failing; recovery required hand-deploying a build that ignored the offending field, plus manual scaling. CircleCI. An IAM-role gap permitted out-of-band changes to AWS WAF outside of CircleCI's Terraform pipeline; an operator performing what they believed were read-only investigation actions modified WAF in a way that began blocking legitimate traffic to the api.circleci.com and circleci.com CloudFront distributions. Because the change wasn't recorded in Terraform, responders deprioritized WAF as a suspect and chased CORS errors and recent deploys until automated drift detection surfaced the discrepancy. Cloudflare. A bad config (router rule) caused all of their edge routers to crash, taking down all of Cloudflare. Cloudflare. During a maintenance of their private backbone network, an engineer made a typo in the Atlanta datacenter network configuration, causing all traffic coming from America and Europe flowing to this only datacenter, crushing it. Cloudflare. An incorrect ordering of the disabled BGP advertised prefixes caused malfunction on 19 datacenters. Cloudflare. A change to our Tiered Cache system caused some requests to fail for users with status code 530. The impact lasted for almost six hours in total. We estimate that about 5% of all requests failed at peak. Because of the complexity of our system and a blind spot in our tests, we did not spot this when the change was released to our test environment. Cloudflare. Several Cloudflare services became unavailable for 121 minutes on January 24, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products including aspects of our Workers platform, our Zero Trust solution, and control plane functions in our content delivery network (CDN). Cloudflare. On 4 October 2023, Cloudflare experienced DNS resolution problems starting at 07:00 UTC and ending at 11:00 UTC. Some users of 1.1.1.1 or products like WARP, Zero Trust, or third party DNS resolvers which use 1.1.1.1 may have received SERVFAIL DNS responses to valid queries. We’re very sorry for this outage. This outage was an internal software error and not the result of an attack. In this blog, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again. Cloudflare. On 18 Nov. 2025, a change in permissions in a database in Cloudflare's bot-detection systems caused a file to be output that exceeded the limits of the software that runs that system. That file was propagated througout Cloudflare's network, causing a systemwide outage. Cloudflare. A configuration change to disable an internal WAF testing tool propagated globally without a gradual rollout, triggering a Lua exception in the proxy's rulesets module that caused HTTP 500 errors for customers using the Cloudflare Managed Ruleset. Cloudflare. A bug in an automated cleanup task queried the Addressing API incorrectly, causing the system to interpret all BYOIP prefixes as pending deletion and systematically remove them. Cloudflare. Human error during abuse remediation inadvertently disabled the R2 Gateway service instead of a specific endpoint, due to insufficient validation safeguards in admin tooling. Cloudflare. During credential rotation, the R2 engineering team accidentally deployed new storage credentials to the default environment instead of production by omitting the --env production flag, causing the R2 Gateway to fail authentication once old credentials were removed. Cloudflare. A newly deployed DDoS mitigation rule exposed a latent bug in the rate-limiting system where flawed cookie validation logic caused HTTP request handler processes to enter infinite loops consuming 100% CPU, propagating across data centers. Cloudflare. A deployment tool bug caused the production environment to reference a staging build version, routing traffic to an unauthorized endpoint returning HTTP 401 errors, cascading failures across Workers KV, Pages, Access, and Turnstile. Cloudflare. A React component bug in the dashboard caused excessive API calls to the Tenant Service due to an improperly configured dependency array, overwhelming the system and cascading into broader API and dashboard failures. Cloudflare. An automated routing policy configuration error removed prefix list filters from BGP export policies in Cloudflare's Miami router, causing it to advertise internal IPv6 prefixes to external peers and resulting in a route leak lasting 25 minutes. Datadog. A bad service discovery config in one of the clients brought down service discovery globally when a dependent client went down. Enom. On January 15, 2022, at 9:00 AM ET, Tucows’ engineering team began planned mainte