Some web storefronts may be unavailable

Incident Report for Uscreen

Postmortem

Incident Summary
On October 12, 2024, users with domains connected to our legacy DigitalOcean load balancer experienced outages due to a large-scale attack targeting DigitalOcean servers.

Impact
Customers using the older infrastructure suffered service disruptions as the load balancer was temporarily disabled.

Root Cause
The legacy DigitalOcean infrastructure, which lacks modern DDoS protection, became vulnerable.

Resolution
We strongly recommend updating domain records to point to our new Google Cloud (GCP) load balancers. GCP’s built-in Cloud Armor provides robust DDoS protection, ensuring higher reliability and security.

Next Steps
To prevent future issues, we urge all customers to move to our GCP infrastructure, which offers far superior performance and protection compared to older systems on DigitalOcean. This transition ensures the platform’s resilience against future DDoS attacks and other risks. For more information regarding this migration please review our help article.

Posted Oct 15, 2024 - 14:40 UTC

Resolved

Our 3rd Party Server Partner, DigitalOcean, has restored service. Web Pages should be back up and running now.

If your site was impacted by this outage, we highly recommend following our Migration guide and moving your site to our more reliable servers. Please visit https://help.uscreen.tv/en/articles/9492966 for more information.

Posted Oct 12, 2024 - 22:49 UTC

Identified

If you are affected by the issue with DigitalOcean, please follow these steps:

1. Go to your Domain Settings.
2. Click "Use a different domain" and select a Uscreen subdomain.
3. Repeat the process, but this time select "Custom domain" to generate a new set of DNS records.

Posted Oct 12, 2024 - 20:03 UTC

Update

Our team continues to investigate this issue with DigitalOcean, our 3rd party server partner.

In the meantime, you can switch your DNS to our other servers. Please note that DNS changes may take some time to propagate, depending on your DNS host, and in some cases, this can take several hours.

Here are the steps to do that:
1. Go to your Domain Settings.
2. Click "Use a different domain" and select a Uscreen subdomain.
3. Repeat the process, but this time select "Custom domain" to generate a new set of DNS records.

This will begin the migration to our newer servers.

Posted Oct 12, 2024 - 18:46 UTC

Investigating

Our team is currently investigating an issue with our 3rd Party Server Partners regarding some web pages being unavailable.

We will continue to update this incident as our investigation proceeds.

Posted Oct 12, 2024 - 17:32 UTC

This incident affected: Storefront.