- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
During a scale of the cluster (+20% capacity), one core service got flooded thus generated an increase of 50x http errors for our s3 High Perf and Standard users in GRA region.
Our technical teams have observed ~2% error rate from January 4th, 13:11 UTC to January 5th, 14:52 UTC, mitigated by team operations, until a snowball effect that leaded to several internal services crashes, from 14:52 UTC to 16:41 UTC, reaching up to ~80% error rate. From 16:41 UTC to 18:18 UTC, the average rate decreased to ~7%, until a configuration change.
The incident was due to a low cache timeout that was widely used on the cluster.
Once the issue found and the parameter has been fixed, the service quickly came back to nominal state and performances are now better than before the scale, as expected.
We would like to point out that the incident resolution was not a workaround, but a stable, permanent and definitive solution to the problem.
This never seen issue is now specifically monitored to prevent further disruption.
End time : 05/01/2024 19:26 UTC
Service impact : Customers might have experienced 500 and 503 errors
Our technical teams resolved the issue. All impacted services are now operational.
Our teams will continue to monitor services.
Update will be posted as significant progress is made.
Our technical teams are working on the issue.
Update will be posted as significant progress is made.
Our technical teams are working on the issue.
Update will be posted as significant progress is made.
Service impact : Customers may experience 500 and 503 errors.
Root cause : Performance degradation due to overload.
Ongoing actions : Our technical teams are working on the issue. Update will be posted as significant progress is made.
Service impact : Customers may experience 500 and 503 errors
Ongoing actions : Investigating
Our technical teams are working on the issue. Update will be posted as significant progress is made.