The situation is now stable enough for us to consider this incident resolved. All clients that were previously served by the London data centre are now back to being served there. We have equipment that is handling the load well. The replacement part for the 2nd core router will be replaced next week via internal maintenance as it is not a client-impacting effort and should be 100% transparent to our clients.
Posted May 31, 2023 - 19:53 UTC
Update
Our team is still waiting on the part to arrive at our facility. New shipping estimates that it should arrive Thursday. The situation remains stable with the traffic currently served by the London site. More information to come.
Posted May 31, 2023 - 13:05 UTC
Update
Our team continues to closely monitor the situation while we wait for the replacement part to arrive at our London Data Centre. Shipping estimates appear to indicate that the part will arrive on Monday May 29th. We will then be able to replace the part and restore full functionality to this site. In the mean time, our clients are instructed to contact our Support Team in case of any issues. More information to come.
Posted May 26, 2023 - 17:48 UTC
Update
Power has been restored to all equipment in our London data centre. However, our team is still awaiting delivery on the replacement part for our core router which is preventing us from fully restoring this site. During the past 14 hours, we experienced issues with the streaming and ad management traffic which had been previously restored to this data centre. During this time listeners experienced longer than usual connection times and some listeners might have experienced streaming drops. Ad delivery for certain listeners was also affected during this time. These issue are now mitigated and under control. More information to come.
Posted May 25, 2023 - 15:23 UTC
Update
Our team has been able to restore partial service in the London data centre. A portion of the ad traffic has been restored as well as Player Services. Our team continues its efforts to restore full functionality. More information to come.
Posted May 24, 2023 - 23:41 UTC
Update
As our team has been able to restore power to a large portion of our equipment in our London data centre. However, a core equipment requires parts replacement in order to function. We are working with our local partner to replace the components. Please note that since we were forced to redirect traffic to other data centres in our infrastructure, some DSPs might have difficulty returning YO ads in time, which can affect the number of ads sourced by YO for some clients. More information to come.
Posted May 24, 2023 - 18:55 UTC
Identified
The infrastructure team is still hard at work to restore functionality at our London Data Centre. Most of our equipment is now powered back up. However, a core router is still affected and being worked on. Please note that this issue has affected some data routing (no data loss). Some UI might not be displaying correct listener and impression data until this incident is resolved. More information to come.
Posted May 24, 2023 - 13:58 UTC
Update
Our team has determined the nature of the issue to be an overloaded power supply component. We are working with our London facility to replace the component ASAP. In the mean time, our team is redirecting traffic to other data centres. Please note that this issue can also cause certain web players to become unavailable as one of our player services servers is housed in London. Player traffic is also being redirected. More information to come.
Posted May 24, 2023 - 13:14 UTC
Investigating
We are currently investigating a network related issue in our London datacenter. This investigation is in the early stages. More information will follow.
Posted May 24, 2023 - 11:57 UTC
This incident affected: EMEA (Live Streaming Delivery, Podcast Delivery, Live Streaming Ad Delivery, Yield-Op Ad Delivery).