Outage on Wednesday, July 1, 2020 [EN]

Rainbow Services have experienced some troubles in NA & CALA on Wednesday, 1st July from 11:49AM UTC to 05:55 UTC. The disruption took place in two phases.

What Happened:

Incident Time frame:

• [Phase 1] From 11:49AM UTC to 12:30AM UTC: Backbone network issues in the NA datacenter of our provider impacting Rainbow users in NA & CALA. Our provider published two incidents impacting all hosted applications in the datacenter.
• From 12:40AM UTC to 12:55AM UTC: New network errors detected having the same impacts.
• From 12:55AM UTC to 03:58PM UTC: The entire Rainbow Operations team is working to restore Rainbow service while controlling possible overloads and traffic spikes when users reconnect.
• [Phase 2] From 05:08PM UTC to 05:50PM UTC: Our operations team again detected packet loss in our provider's network, resulting in degraded and slowed Rainbow service for LATAM and NAR Rainbow users.
• At 05:50PM UTC: Network is stable, no more errors detected. we verify that all Rainbow services are available and stable.
• At 05:56PM UTC: Incident is closed. All Rainbow services are WW available and stable.

 Incident impact:

• [NA, CALA] From 11:49AM UTC to 12:55AM UTC:  Full Rainbow Services (Connection, Messaging, Bubbles, Conferences, Telephony Services ...) down for all Rainbow users.
• [EMEA, NA, CALA] From 12:55AM UTC to 03:58PM UTC: Restoration of services (Web Call, Web Conference, Telephony services) in a phased manner. Users in EMEA may have been impacted when restoring services and some may have noticed empty bubbles or may not be able to create, join, delete a bubble or make a conference call
• [EMEA, NA, CALA] From 05:08PM UTC to 05:50PM UTC:  Full Rainbow Services (Connection, Messaging, Bubbles, Conferences, Telephony Services ...) were degraded and slowed for NA & CALA users. Users in EMEA may have been impacted when restoring services and some may have noticed empty bubbles or may not be able to create, join, delete a bubble or make a conference call

Incident description:

Our datacenters provider faced a network issue in a NA datacenter resulting also network issues in CALA datacenter. This network issue resulted in server intermittent loss of internal network access. Despite high-availability is in place, secondary datacenter did not offload the traffic back

Once access to the data center was restored, our teams were obliged to control possible overloads and traffic spikes when users reconnect. Bubble traffic was handled despite the fact that some users could see empty bubbles for a short period of time. Rainbow's teams did everything possible to restore services and shorten the disruption time after the network outage.