Voice Service Incident September 28th

SUMMARY

As of 14:12 GMT+1 / 06:12 PDT, we are working to resolve the following service incident:

We're currently investigating reports of issues with Voice recordings with our Voice service provider

14:52 GMT+1 / 06:52 PDT

We are continuing to work with our voice provider on issues affecting the ability to playback some voice recordings.

 19:00 GMT+1 / 11:00 PDT

Our voice provider has implemented a change at 11am PDT to resolve the errors retrieving call recordings.  They noted:

Starting at 6:16 AM PDT we experiences elevated error rate for Conferences Recordings in IE affecting 33% of Recorded Conferences. The issue should be now resolved.

Our independent tests confirm resolution. However, call recordings that were made before 11am PDT may not be retrievable.

POST-MORTEM

Due to a production incident experienced by our Voice provider, some recordings were not created for voice tickets and 404 errors were served.

In addition, confusion was caused for customers as we had rolled out a new player in Zendesk which indicated that missing recordings (i.e. 404s) were in fact deleted. 

We're following up with our Voice service provider to ensure escalation procedures are adhered to for faster resolution turnaround. We have also reverted the player changes as they were causing a lot of confusion and are working on an improved version of player for our recordings.