As of 11:05 GMT / 03:05 PST we are working to resolve the following service incident:
We are investigating performance issues with Zopim. We will provide more information shortly.
11:24 GMT / 03:24 PST:
We are happy to report the performance issues affecting Zopim are now resolved. A Postmortem will be available here shortly.
This incident began due to a database pile-up, which caused by an emergency backup job and resulted in performance and login issues across the Zopim application.
An emergency database backup routine was executed to generate formal point-in-time backup to repair MySQL read services. High parallel thread concurrency from a backup tool caused memory and encrypted disk starvation which resulted in a pile-up of SQL queries.
To resolve the problems, we killed the offending backup routine as soon as we identified the root cause, and restarted MySQL to ensure full resolution.