Planning customer service during a major outage

Dave and Ben (our directors at Campaign Monitor) often used to go on holidays together. This time, the worst time, they were off surfing from a boat in the ocean somewhere near Indonesia.

That's why we couldn't contact them when we found out one day in 2009 that hackers were attacking our servers. It would be over a day before we could get a message to them, and another before they could even get back to land. In the meantime, we were on our own.

Handling customer service is a tricky job at the best of times; when things are going really wrong it can be terrifying. In my 2012 UserConf presentation I talked about how to be prepared when things go really wrong.

You can watch the video right here; I've included some summary notes below.

What's a crisis plan?

A crisis response plan is a document that you and your team can use when things have gone badly wrong (like a major service outage or a security issue). It will help you make better decisions under pressure, and give better customer service.

What to do before a crisis happens

The more time you put in before you need it, the easier you will have it when things are falling apart.

Understand your company

There’s no point writing a plan up about your fanatical customer service if your company doesn’t believe it to be important. Work out realistic options based on the resources and support you have available.

For example, are you a Ryanair “low costs are more important than any customer” or a Southwest “Customer Service company that happens to fly”

Decide which support channels you will offer

Will you respond on email, on twitter, by phone, on Facebook? Can you realistically do all that in a crisis? Or should you funnel people from those places into one full support channel?

At Campaign Monitor we will watch and post updates on Facebook and twitter but we may point people to our email support channel for individual help.

Nominate team members

Decide which person (or which role) should be responding on which channel, so in a pinch you can avoid double handling tickets; or worse, not responding at all.

Include a list of escalation points; how and when to contact developers,managers and operational teams.

Setup detection systems

Catching problems early makes everything easier to handle. Investigate technical options (like Nagios for one), human options (how easy is it for support staff and customers to report issues) and third party options (can your up-stream providers give you notifications of problems)?

Create a notification system

Create an easily updatable page where your customers can self-serve to see if there are any known issues, but only if you promise to actually keep it up to date.

Build up a library of shared language, tone and structure

Rather than having to come up with the right words on the fly, spend time in advance deciding on what terms to use (the ones your customers will understand!) and share lots of specific examples with your team.

Example of a general response structure:

  • Acknowledge the issue exists, and let the customer know it isn’t their fault
  • Explain the situation to the best of your current knowledge
  • Explain what is being done to resolve it, and give a timeline where you can
  • Let the customer know how and when they will get further updates

During a crisis

This is when you put your plan into action. So make sure the plan is readily available and up to date.

How bad is it?

Your early warning systems will help, but the faster you can see the scope of the problem the better your customer service can be.

Who can help?

Determine who you need to get in touch with (developers? managers? software providers?) and bring them in.

Work on a rapid response

You may have many people contacting you in a short period. Work out a shared basic response that accepts the problem and explains what is happening and get your team using that to quickly respond to every person.

Important: Take the time to modify the response to the specifics of each customer’s case. It makes a big difference to them.

Work out the right update frequency

A fast first response is great but if the problem continues with no further details being shared the goodwill evaporates. Work out how often you need to update people depending on the type and severity of the problem. Read the blog post I wrote when Campaign Monitor was attacked for an example.

Brave the wrath of developers (if relevant)

Someone in customer service needs to be a little pushy in finding out what is going on, on behalf of the customer.

Keep up internal communication

Don’t leave your support team on their own to guess at what is happening and when it will be fixed. Help them to help your customers by keeping them informed.

Maintain a list of people to follow up with

Some customers will need to know immediately that the problem is resolved. If you don’t keep a list it is hard to do that in a timely way.

Aftermath

Once the immediate issue is resolved, take some time to apologise to the affected people and improve your system for next time.

Make an apology

Avoid the fake “It wasn’t so bad” or “it wasn’t our fault” apologies. A real apology is:

  • Genuinely meant
  • Detailed as to the cause and impact
  • Open to a response from the customer
  • Useful in fixing the problem (or compensating for it)

Modify processes and procedures

Review the whole event and see what can be learned for next time. Some good questions to ask:

  • Was the plan used?
  • Did your team have the tools they needed to help customers?
  • Was information shared quickly, and accurately?
  • Were there answers given that should be saved and used as a model for the future?
  • Maintaining quality when the team grows - A small team (and one in a shared space) will absorb knowledge by osmosis. When you have 10 or 30 or 50 support staff, that process doesn’t work so well.
  • Create a home for internal documentation that is updated
  • Focus on soft skills for new people, maybe above technical training
  • Keep working on the company culture, especially for remote staff

If you found this post useful, please do share it. I'd love to hear from you too, tweet me @mrpatto.