Service disruption May 8 2015

TIMELINE

14:50 GMT: We are currently investigating reports of problems with Voice and Help Center for some customers. More info to follow once available.

15:30 GMT: Our Operations team have resolved the issue affecting Voice and Help Center

POST-MORTEM

The root-cause of this issue was found to be due to a release that inadvertently had a bug in one of the processes.  This bug caused overtime some of our services to become un-responsive due to the number of jobs that were queued up to be processed. Once the cause was verified, we rolled back the release that introduced this issue which then promptly resolved the problem. 

The next steps to prevent this issue from happening in the future is to do more extensive testing of these service updates prior to being released.  Although our testing covered the process in question, the issue would not have been noticed without extensive run-time testing.