Rainbow Services in APAC Region have experienced some troubles from Wednesday, Feburary 16, 2022 ,16:40 CET to Thurdsay, Feburary 17, 09:30 CET.
Rainbow's infrastructure is based on geographic redundancy of servers, allowing upgrades to be performed with minimal impact to users.
Following a planned upgrade of a server component in the APAC region, a small portion of the redundant servers did not reconnect all connections. As a result, some services experienced difficulties for a few hours.
Incident Time frame:
Wednesday, February 16, 2022
From 16:40 CET to 00:00 CET: Following an update, some API errors have been reported. The number of failures remains below the alert threshold.
Thursday, February 17, 2022
- From 00:01 CET to 02:25 CET: Following an update, some API errors have been reported. The number of failures remains below the alert threshold.
- From 02:26 CET to 04:50 CET: Alert thresholds are exceeded. Some server components are restarted to restore Rainbow services to all users. These restarts restore most services.
- From 04:51 CET to 09:30 CET: Other components are restarted, restoring full Rainbow services.
Remember that the region of the Rainbow Company prevails, not the Rainbow user's region.
- Wednesday, Feburary 16, 2022 ,16:40 CET to Thurdsay, Feburary 17, 09:30 CET :
Some users may have experienced troubles when using:
- Telephony services
- File sharing
- Web conferencing
These regions were not impacted by the outage.
- Further increase the monitoring of the infrastructure after an upgrade.
- Modify the error alert thresholds on the API to detect this type of failure more quickly.
The communication was managed through the site status.openrainbow.com: