Postmortem

Date of Incident: 1/6/25

Summary:

We recently experienced degraded performance and limited service interruptions caused by unprecedented traffic growth and an issue within our database infrastructure. A CPU resource leak in our database provider’s system contributed to high resource utilization, compounding the challenge of meeting demand. This incident tested our system’s capacity, and while it caused temporary disruptions, it also highlighted opportunities for immediate and long-term improvements.

Root Cause

• Increased Demand:

  • Platform traffic and user activity tripled compared to typical levels, leading to an unexpected surge in database load.
  • This spike caused memory saturation and an overabundance of connections.

• CPU Resource Leak:

  • A resource leak in the database provider’s infrastructure led to persistent CPU spikes, limiting system efficiency.
  • This issue prevented the system from scaling effectively to handle workloads.

• Inefficient Resource Allocation:

  • Idle database connections and unoptimized queries further stressed the infrastructure, reducing its ability to respond to peak demand.

Resolution

Immediate Actions:

• Increased Database Capacity:

  • Doubled CPU and memory resources in our database instance to handle the increased load.

• Instance Restarts:

  • Refreshed instances to eliminate stale connections and stabilize performance.

• Cleared Stale Connections:

  • Optimized active connection counts, reducing strain on the system.

Short-Term Adjustments:

• Adjusted query handling to improve load distribution.
• Enhanced monitoring tools to identify potential resource leaks sooner.

‌

This incident underscores the challenges of balancing rapid growth with infrastructure resilience. By immediately increasing database capacity and addressing inefficiencies, we stabilized the platform for now. Moving forward, we are committed to strengthening our systems through proactive scaling, deeper collaboration with our providers, and better resource management to support your continued success on our platform.