Rainbow Services have experienced some troubles Thursday, November 04, 2021 from 16:55PM CET to 18:17PM CET.
What Happened:
Incident description:
While deploying a new Data Center and operating some software updates to extend our geographic presence and improve the quality, we have experienced an exceptional inflow traffic impacting one of the backend load-balancing components.
This resulted in slowness of the connection services affecting some users when they tried to login with latency in the services access.
This has been amplified by the load of a massive inflow of reconnections following the software update delivery.
Incident Time frame:
- From 16:55PM CET to 17:00PM CET: A slowness was detected by the operation team.
- From 17:00PM CET to 17:26PM CET: While the traffic congestion was increasing, the operation team started the procedure to mitigate the impact on the users’ access
- From 17:26PM CET to 18:17PM CET: The different services came back to a normal situation, completing with the hybrid telephony. All functionalities are available and a monitoring of the infrastructure remains active.
Incident impact:
Discover the detailed impacts by Region.
Remember that the region of the Rainbow Company prevails, not the Rainbow user's region.
Remember that the region of the Rainbow Company prevails, not the Rainbow user's region.
Region: [Worldwide]
- From 16:55PM CET to 17:00PM CET:
- Slowness in the application provides a degraded level of service for several users.
- From 17:00PM CET to 18:17PM CET
- All features can be affected by the slowness.
Corrective Measures:
- Adjust configuration of the faulty servers: Already done.
- Database optimizations. Planned in November.
- Adapt delivery time slots according to the observed traffic . All deliveries.
COMMUNICATION HISTORY:
The communication was managed through the site status.openrainbow.com:
https://status.openrainbow.com/incident/ckvl5ivb210129368hn0wpe688nk
Comments
0 comments
Please sign in to leave a comment.