© 2006 Brian D. Voss

EDUCAUSE Review, vol. 41, no. 2 (March/April 2006): 76–77.

What Would Ozymandias
Think about Disaster Planning?

A friend once told me that no one can tell you how spectacular the Grand Canyon is—you need to see it with your own eyes. This is also true for New Orleans today: you cannot truly understand the extent of the devastation of Hurricane Katrina without personally seeing the breadth and depth of the disaster.

With much reservation and at the urging of my wife, I took a quick car trip down to New Orleans the second weekend in November 2005. My wife felt it was important for us to have a better understanding of Katrinas impact on our lives as Louisiana residents. My reservations had to do with a feeling of being a tourist through other peoples hardship and tragedy. In the end, I acquiesced and we went.

My visit gave me a sense of the scope of the disaster. This is not something one can grasp by looking at photographs or news footage. The perspective of the lens is too narrow—too focused on pieces of the puzzle. Driving back home, I realized that this is the same challenge we in higher education IT face as we try to craft disaster-recovery plans in a post-Katrina world. We can get individual pieces of the puzzle from those who have experienced it, but grasping the total scope will be difficult for any CIO who has not personally seen or lived through such a disaster. Nevertheless, we all must try to do so. We all need to think more broadly, using the puzzle-piece pictures from experiences post-Katrina to help us do that thinking.

How Prepared Are We for Someone Elses Disaster?

On Sunday, August 28, I felt I was fairly prepared for what would happen if a major hurricane hit Baton Rouge, which is far enough inland to avoid the most deadly and destructive part of such storms. We had the Louisiana State University (LSU) campus and machine room locked down, and our diesel tanks were full of fuel. The storm came late that afternoon. After lashing us with heavy winds for about twenty-four hours, it left on the afternoon of the twenty-ninth. By five p.m. that day, we were picking up debris and running on utility power. Everything was OK. And then the levees failed in New Orleans.

The next fourteen days were terra incognita. Our campus extended its closing and became the key point for a variety of aid and disaster-response activities around the region.¹ For me personally, my role as CIO of Louisianas flagship university expanded to include supporting a campus that was engaging in a broader mission. We installed hundreds of temporary phone lines and data drops. We distributed scores of laptop and desktop computers to be used by emergency-response agencies and personnel. We became the IT support for the use of technology by these personnel and immediately took on a variety of information systems development and support responsibilities. Their thirst for IT infrastructure was quenched, but their hunger for using it to support the flow of information was only fueled.

Another aspect of being on the edge of the disaster was that some of those who lost their data centers and campuses came to us, seeking refuge. All institutions in New Orleans were put out of business by Katrina (at least for the fall 2005 semester). The University of New Orleans, part of the LSU system, showed up on our doorstep seeking sanctuary (and a raised floor with power and HVAC). These dedicated IT professionals needed a place to begin restoring their service. We had enough floor space and power. We had equipment—on hand for other projects not yet deployed—to get them rolling until they were able to acquire their own. We had a training facility in the computing center building to offer as office space and home for them as they reassembled their IT staff.

We survived this challenge, but not because of excellent pre-event planning. Our plan envisioned LSU as playing a small but key role in a transient/transitional event—evacuation, storm passage, and return immediately thereafter. And our own internal disaster-planning efforts focused most specifically on the traditional loss of our data center event. In both cases, our plans were not broad enough to encompass the scope and duration of Katrina or the fact that someone elses disaster could require so much from us (since we were untouched directly by the disaster). In short, although we had planned for our own disaster, we had not planned for someone elses.

We survived because we were lucky and because we had some basic preparations in place. We had sound and service-oriented strategies for stocking IT equipment. We had spent year-end funds (the usual budget dust funding strategy) to acquire an inventory of key elements for the coming year—and the storm happened near the start of the fiscal year, when our storerooms were filled. We survived because the students were off-campus at the time, and we could borrow their lab computers and their check-out program laptops. We had vendors who rushed to our assistance and were generous. And we had an excellent and dedicated staff committed to doing whatever they were called on to do. We did a lot of things right, but we were fortunate in the timing of the events and in how we managed our steps in progress—not because we had a step-by-step plan of action.

In a Disaster, Do the Rules—and the Plan—Go Out the Window?

In the aftermath of these events, Im taking a broader look at my disaster-recovery planning. Of course, now I have a model for what Ill need to do if, again, Im on the edge of someone elses disaster. Im also adjusting my parochial planning, based on what I witnessed with Katrina, in case a disaster hits my data center, my campus, and my city. But I have to wonder, is it even possible to plan for these things? In the end, isnt the very nature of disaster its unpredictability? Can we really foresee every possible event or circumstance and have a fault-tolerant, step-by-step disaster plan capable of dealing with every contingency?

I dont think we can. During the Katrina crisis, I never once referred to a written disaster-recovery plan. My guess is that our written plans covered less than 10 percent of the events we handled—and those only in the very early stages of the crisis. I do hope that our new written plan will cover perhaps double or triple that. But even so, that means the vast majority of what will transpire will be handled via improvisation. Trying to develop disaster-recovery plans so detailed as to cover every possible scenario is a monumental and likely hopeless task that will consume far too many resources—and as the press of going on with normal business increases, such a process is likely to lose all momentum and be abandoned. The old saying If you want to make God laugh, tell him your plans certainly applies here. My experience with the Katrina aftermath was that it was very much a living thing, greatly chaotic. Day to day, new circumstances arose based on what had happened the previous day (or night); there was no script to the disaster. Hence, no script for responding to it was possible, let alone feasible.

So, should we do nothing? No, I am not advocating that. We can certainly take stock of what happened and make plans that encompass a broader array of possibilities than we might have pre-Katrina. For example, having back-ups located off-site but within a prospective disaster zone is an idea that needs rethinking. We can assemble lifeboat strategies—grab-and-go supplies to help us reestablish the college or university IT in another location. And we can work as a community of higher education IT leaders to find ways to leverage the national cyberinfrastructure and our own broader community into a better position to deal with future disasters, both regionally and nationally.

We should follow the advice I received from Capt. Joseph R. Castillo, Chief of Operations for the U.S. Coast Guard Eighth District in New Orleans: Focus on the process of planning, and not on building a plan. In so doing, we should establish a reasonable set of disaster-response principles to cover the fundamentals, and we should open our thinking to the broader aspects of a disasters impact. We should spend our time examining how we will position ourselves to be flexible in responding to the disaster, focusing on knowing exactly how we will do our jobs in a disaster setting but not on trying to script each and every scenario in terms of what we do.

The key to any success that LSU achieved in dealing with the aftermath of Katrina was not that we had a break glass in case of disaster, all-encompassing plan. The key was that we had resources to fall back on and we had a talented and committed group of people who managed the process of coping with the disaster by knowing how to do their jobs—even under these chaotic circumstances. I think our process would have been better had some of the overall elements been better organized (and thought through in a bit more detail ahead of time), and thats what were doing now. But in retrospect, I see no way that we could have created—or could even now create—a cookbook for handling a disaster of this magnitude. Plus, a future disaster wont be the same. It wont unfold in the same way. Katrina changed everything, including how the next disaster will affect us.

Plans need to have flexibility as their defining feature. Because whether the disaster is on your doorstep, next door, or down the road seventy miles, after youve done all you can do to be ready, events will unfold in unforeseen ways. The best hope is to be prepared to handle a core set of events as best you can and to be aware that you will have to deal with surprises. Then, when something you hadnt thought about does happen, youll be able to adapt, improvise, and—indeed—overcome.

Finally, we must act now, while we have the motivation and the fresh examples in front of us. We must revise our thinking and revisit our planning quickly, before our attention is pulled elsewhere. Soon—if it hasnt already begun to happen!—the lessons of Katrina regarding IT disaster recovery will fade. The press to get on with normal business or advance other strategic initiatives will take priority over investments in business continuity planning.

A key to doing so is to keep our planning fairly simple and straightforward and to avoid preparing overly complicated, detailed, and expensive disaster-recovery plans. We should establish a reasonable set of disaster-response principles to cover the fundamentals, and we should open our thinking to broader aspects of disasters as we do our planning. But in the end, we must rely on our ability to be flexible in the midst of a disaster.

We need to be as prepared as we can reasonably be. But we will still need to be lucky too.

The name Ozymandias in the column title comes from a poem by Percy Bysshe Shelley.

1. LSU has published a book describing what happened at the university during those two weeks, what we were called on to do, and how we responded as a flagship university: LSU in the Eye of the Storm: A University Model for Disaster Response (Baton Rouge: LSU Press, 2005). See http://www.lsu.edu/pa/book/ for more information.