Increased error rates

Summary

On Friday, April 22nd, 2022 at 12:20 UTC, Slido users experienced errors and/or increased latency for around 45 minutes. We’d like to apologize to our customers who were impacted by this disruption. There was no impact to your data and Slido’s security wasn’t breached by this incident. 

Rest assured that we are taking several actions to improve the resiliency and performance of our service to prevent this type of problem happening again. 

Root cause

This incident was triggered by continuous, machine-generated requests sent to one of our API endpoints in our EU cluster. The nature and volume of this adverse traffic led to the max_prepared_stmt_count limit [1] on our MySQL database being reached. As a result, subsequent requests started failing. 

Remediation

We detected the incident through our monitoring at 12:27 UTC. At 12:48 UTC, we manually blocked the source of the adverse traffic on our firewall. At this stage the value for our max_prepared_stmt_count limit [1] was increased. 

At 12:59 UTC, we blocked a second batch of IP addresses and the max_prepared_stmt_count value was increased again. The incident was fully resolved at 13:05 UTC.

Prevention

We are taking the following actions to ensure this does not happen again:

• Control the number of database prepared statements created by our backend service
• Improve monitoring and add alerting of the number of prepared statements in our database clusters
• Improve rate limiting on our API
• Improve tools and procedures for posting external communications during outages

– – – –

[1] https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_prepared_stmt_count