Technology Blog

Armanino’s Technology Blog informs technology and software CFOs and executives of the latest industry trends, rule changes and best practices. Our professionals bring you their insights and advice from an accounting and organization perspective to help your company reach its goals.

Thursday, February 7, 2013

A “Super” Reason to Revisit Your Business Continuity Plan

Posted by Jeremy Sucharski

Sunday afternoon and the weather was great. Do I stay at home and watch the Super Bowl live or head out for a quick bike ride? Bike ride it is…why else were DVR’s invented. So I get ready to head out and when I go to set up the DVR I am asked if I want to add extra time to my live recording. Sure…never know if the game will go to overtime.

…Or have a 34 minute power outage that would have made you miss the end of the game.

This past Sunday, less than 30 seconds into the 3rd quarter of Super Bowl XLVII, the Mercedes Benz Super Dome went dark…literally. The stadium had a power outage lasting 34 minutes that put half of the enclosed dome in darkness. While this was the break for some to hit he kitchen for more snacks, I doubt the Super Dome and NFL officials saw it as a positive. In reality, this event as a huge black eye for the Super Dome, their partners and the NFL.There is no bigger stage for a televised sporting event than the Super Bowl. Over the past 4 years, this broadcast has attracted more than 100 million viewers each year and consistently broken records for the largest television audience in history.

In the days after the game when the confetti was done falling and everyone has started looking forward for the day when pitchers and catchers report, we still don’t have a complete understanding of what caused the outage.

The issue was blamed on an unknown power issue or aging equipment. Great. But really went wrong? Why did I almost run out of DVR space?

The answer: lack of adequate business continuity planning.

The outage could have and should have been prevented or greatly shortened. Because it wasn’t, it highlighted several areas where the Super Dome’s plans could have been deficient.

First, there was a failure to adequately plan. The Super Dome and/or NFL Facilities team failed to adequately think of all possible outage scenarios. Despite making significant upgrades to the stadiums infrastructure, getting the power back on became dependent on automated equipment that took over 30 minutes to complete the power restoration. Given the magnitude of this game and the global spotlight placed on it, reliance on a slow power resumption system wasn’t the best option given their tolerance for downtime. Even Beyonce brought her own generators so she wouldn’t have issues with her half time show. This failure to adequately plan should have been identified through one of two BCP planning exercises:

  • Business Impact Analysis: Analysis performed to develop a common understanding of the processes that are critical to the survival of an organization.
  • Threat Analysis: Assessment of possible threats that could prevent the ongoing operation of critical business processes.

Secondly, the failure was exacerbated by the perceived lack of communication. There did not appear to be one single individual in charge. There were different reports provided to the media from different sources. A key tenant of BCP is to have a single control point in the form of a command team to manage the issue and work the plan. This did not appear to be solidly in place for the Super Bowl.

The end result was a 34 minute delay that some say shifted the momentum to another team. Others feel that it shifted the momentum to another city as it is now speculated that the NFL will not grant the 2018 Super Bowl to New Orleans.

So how does your organization avoid a PR and reputational black eye such as this?

Disaster Recovery Plan

First, if you don’t have a current (i.e. reviewed and revised in the last 6 months) Disaster Recovery and Business Continuity Plan, you need to create or revise one. The first starting point is to update your Business Impact Assessment. This will help you to understand your recovery priorities. And don’t forget to assess the peaks or seasonality of your business. Imagine a credit card company like Visa or MasterCard being down the day after Thanksgiving. I can assure you that their plans take into account key dates and times of year.

Benchmark Priorities

Next, you then need to benchmark those priorities against your ability to recover in those timeframes. If you can’t recovery in that timeframe, you need to update one of three things: People, Process or Technology.

Test Your Plan

Lastly, you need to test your plan. BCP plans aren’t created to collect dust. Unfortunately, I have seen more serve as door weights than useful business tools. This is because testing is overlooked. Testing serves several functions. First, it tells you if you can recovery in your anticipated timeframe. The ancillary benefit is that you can assess the efficiency of your processes as well. This can help to highlight areas where your processes are inefficient and therefore require updates.

At the end of the day, it comes down to a little common sense and a focused planning effort on the part of management. With an ongoing focus on the recoverability of your business processes, your company can avoid unplanned outages. Especially those that are seen by over 108 million viewers around the country.

Jeremy has over twelve years of experience delivering internal audit, and consultative services to clients in a variety of industries. Jeremy manages teams and works closely with clients to analyze risks associated with information systems and business processes, implement controls and test those controls for operational effectiveness. He has extensive experience in developing and executing a detailed audit plan to test effectiveness of systematic and manual information system and business process controls. Furthermore, Jeremy is the leader of the firm’s GRC practice. Jeremy is a member of the Information Systems Audit and Control Association (ISACA) and has a B.A. in Business Administration, MIS Concentration, from California State Polytechnic University, San Luis Obispo.


comments powered by Disqus
« | »