We wanted to provide a detailed postmortem of the service outage that occurred on Saturday, June 15th, from 16:11 UTC to 20:11 UTC.
Summary of the Issue:
The outage was caused by issues with our Load Balancer. While not every store was impacted, those with older DNS configurations experienced interruptions.
Background:
Last year, we upgraded our Load Balancer Services to utilize Google Cloud Services, offering enhanced reliability and security. This upgrade ensures faster and more reliable service for our customers’ catalogs. However, stores still using our older DNS configuration were affected during this outage.
Steps Taken:
- Immediate Response: Our team quickly identified the Load Balancer as the root cause and worked to restore service.
- Resolution: Service was fully restored by 20:11 UTC.
- Preventive Measures: We added another layer of WAF at the edge using Fastly, in addition to the Google Cloud Armor security layer.
Next Steps for Affected Stores:
Starting this week, to prevent future issues, stores using the older DNS configuration will see a new banner on their dashboard. The banner provides steps to confirm domain ownership and transfer DNS configurations to our new servers. This process takes several hours but will not result in downtime for your members.
For those who do not manage their DNS settings, please coordinate with the team member responsible for this task.
Attached is an example of the banner you will see.
We apologize for the inconvenience caused and appreciate your understanding and cooperation in making these necessary updates.
