Service Incident January 20th 2016

 

 17:09 UTC | 09:09 PT
Trial creation has returned to normal at this time, we will continue to keep an eye on this moving forward.

15:59 UTC | 07:59 PT
Efforts continue to investigate of the trial creation issues encountered by some of our customers.

13:05 UTC | 05:05 PT
We're currently investigating issues with creating new Zendesk trials.

14:46 UTC | 06:46 PT
We continue to investigate issues with trial creation. Apologies for the inconvenience.

POST-MORTEM

For the purpose of Zendesk trial creation, we hold an ample supply of pre-created accounts to easily set up trials as new signups come in. On January 20th, an increase in network traffic to our European data centre caused the supply of pre-created accounts to be depleted and, therefore, new trials were routed through the old creation code path rather than fast account creation. This resulted in timeouts on the marketing site, since the old creation code path can take 30-60 seconds - exceeding our default timeout on responding to a request.

This has been resolved by further extending the timeout on the marketing server, as well as increasing the size of the pool of pre-created accounts.

To prevent recurrences, we're working to further investigate the timeout error handling, so as to avoid failures when our resources are exhausted. We'll also be setting up monitoring and alerting systems that will be triggered in the event of an increase in account creation failures, to enable us to take action faster. 

FOR MORE INFORMATION