Call Center Catastrophe: Fend Off the Crippling Effects of Hacking, Mother Nature and More

When the Fukushima I nuclear power plant on the East Coast of Japan overcooked after being battered by first an earthquake then a tsunami, Sony, the Japanese electrical manufacturer, temporarily closed its online gaming network, PSN, and suspended its PlayStation information call center.

Who could have foreseen that within three months, the company would be facing what gamers and business recovery specialists are now describing as the Fukushima of data theft? Perhaps it was impossible to predict. But perhaps the two are not so disparate.

It seems facetious to compare the hacking of Sony’s PSN with the disaster at Fukushima, but the response by the organizations involved provides a salutary lesson in the value of communication. Losing the personal data of millions of customers would have been a public relations disaster for Sony come what may, but waiting for a week to inform its customers after the hack was detected multiplied the PR gaff many times over.

We can assume two things from this. First, the business continuity plan (BCP) – or disaster recovery plan -- was not properly thought through, which probably also means responsibility for BCP did not reside high enough up the organizational chain. And second, communication was not considered a high enough priority. For a company the size of Sony, both of these errors was inexcusable. Of course, contact center managers know better than anyone the value of communication. But with electricity still temperamental after the accident in Japan, and Katrina still an uncomfortable memory, it is inexcusable for contact centers of any size to be neglecting their own BCPs.

According the Scott Hammer , principle at North Highland, a management and technology consultancy, the first thing to remember is that a BCP incorporates preparation for all aspects of an organization - people, process and technology - and for all internal and external stakeholders, including customers, vendors and partners. A strong BCP therefore can:

Ensure consistent performance of critical business processes throughout the crisis
Provide stability and confidence to vendors, partners and customers
Mitigate economic loss throughout the crisis scenario

He adds that even if your contact center does not yet have a BCP, it is not too late to look closely at your processes. He says, “Look at your value chain, decide what critical services your organization must continue to provide, and develop the detailed plans to provide these services. Look beyond IT, where most organizations focus disaster recovery plans, to your operations and business. Your business is where your people are, and that is where [some events will] have the greatest impact."

For Ted Brown, of Ketch Consulting, neglecting this human aspect is the classic mistake in continuity planning. “Historically, disaster recovery plans focused on technology (hot site) and data (back up and off-site storage),” he says. “Very little attention was given to the people component. The assumption was made that the required people would be available and would simply go to the internal or external hot site, even if it were hundreds or thousands of miles away. It was also assumed that they would stay away for weeks or even months.”

This is no exaggeration.  In Japan, backup power plants have remained in operation to ensure the power supply, but an essential shortfall in Japan’s ability to satisfy the power demand this summer is still enforcing 15% energy cuts for businesses. Moreover, a moratorium on nuclear power in the country, suggests rolling power outages are a likely scenario even later on this year.

According to AMI Perspective’s 2011 Disaster’s Impact on Japan SMBs study, having already seen or experienced vital data loss or system damages, these ongoing problems mean Japanese businesses have realized the risk of in-house installed systems with little or no off site backup. Small and medium sized businesses (SMBs) who previously could not afford BCPs are now convinced of their necessity. According to AMI, cloud solutions that are easier and quicker to implement and do not require large initial financial outlays are optimal solutions for Japan SMBs experiencing serious cash-flow issues as a result of decreasing revenue in the aftermath of the disaster.

But even if contact centers choose to locate in the cloud, it is not enough to simply sit back and think that is enough. According to Jeff Weil of Empirix, the IP communications vendor, the critical thing is to have a clear, and properly tested BCP; and to have that BCP championed at the highest level of the organization because different organizations may be willing to accept different trade-offs.

In many cases, in shifting sites after a disaster, for example, a primary site might benefit from state-of-the art technology, including speech recognition and CTI, while a secondary site relies on older equipment relocated from the primary site after an upgrade. In this case if there is a failover, customers may navigate the IVR/ACD system using DTMF technology and calls may go to a pool of agents rather than be directed to agents with a certain level of expertise. Or, agents may experience a flood of calls.

During testing and ongoing monitoring of the secondary site, the organization may learn that customers will experience service levels that are slower than normal, and this may be a trade-off that senior managers are willing to accept under the disaster recovery plan. But, only through testing and monitoring will an organization understand that the possibility of such a trade-off exists at all.

"Even though you can never completely mitigate risk," adds Dodge McCord, a Senior Specialist with North Highland, "the right preparation is a roadmap around the panic that invariably accompanies catastrophe. Don't make the mistake of thinking you can just deal with it when it occurs - it'll be too late then." Sony is finding this out the hard way.
Mark Lewis is the former editor of Call Centre Focus, a UK sister publication to Call Center Insider; he's now a freelance editor covering the call center industry from Norway.
Comments (4)

Leave a comment

Please sign in to leave a comment. If you don't have an account you can register for free here.

Forgot username or password?


Reggie Bautista — 1:40PM on Jul 28, 2011

It's absolutely true that organizations should invest in the development and testing of BCPs that include contingencies for personnel. BCPs should also account for security breaches and service disruptions in addition to natural disasters, as Sony illustrates. The question remains how organizations can be persuaded to invest in such comprehensive planning and testing, when the perception within those organizations is that risk is minimal.

Christina D’Airo — 11:20AM on Jul 29, 2011

This is a great topic! It seems that having an effective BCP is a crucial consideration when setting up a contact center, or any organization that handles a high volume of customer data. It would be interesting to find out if there are many companies who rely on "the cloud" as form of BCP and what their experiences have been.

jeremy sanders — 7:50AM on Aug 1, 2011

I was one of those affected by the Playstation Network shutdown. Funny that I didn't hear about it directly from Sony. I had to find out from a friend a week later. When the network went back up I received a welcome back email with a code to download 2 free games. So it would seem that they were lacking any disaster plan but the marketing team had their stuff together.

Rose Polchin — 5:39PM on Sep 8, 2011

it is absolutely critical to have a plan...perhaps as critial, if not more so is actually TESTING the plan!