Description
This edition of DISASTER RECOVERY TESTING: EXERCISING YOUR CONTINGENCY PLAN is a reprinting of the version originally published in 1994. In the fourteen years since I first began development of this book out of a sense of frustration with the lack of published resources on the subject, Business Continuity as a discipline, as an industry, and as a professional practice has evolved dramatically. In addition, the role of exercises in the continuity management program has grown significantly.
And yet – more aspects of business continuity have stayed pretty much the same over the years. Of course, there have been some changes – in 1994, the Internet was incidental to most organizations; today, it is indispensable. Email use was limited. Cell phones were prohibitively expensive and impractical. Amazon.com was a struggling startup. And don’t get me started on iPods! On the other hand, there are still plenty of organizations who have yet to implement meaningful continuity structures, let alone to conduct effective exercise programs.
Nevertheless, two factors prompted me to agree to reprint this book:
- Upon rereading this book, I was pleased to see that most of the concepts, processes, issues and recommendations remain current and effective with, of course, some allowance for current technologies; and,
- After fourteen years, nobody else had stepped up to the plate and written a book to replace it.
So, what has changed since I first set fingers to keyboard? Technology, certainly – both as a recovery issue and in offering recovery and exercising opportunities that did not exist in the mid-1990s; for example, there are now numerous web-based recovery planning and exercise management solutions. The Internet, of course, has become a major consideration to most any business, regardless of size; VoIP telephony, satellite phones and cellular phones have changed the way we think about and use telephones, and have provided new recovery support options as well as contingency issues. Remote workplaces as well as outsourcing have affected recovery options and exercise strategies. Ubiquitous dependence upon email and wireless networking have not only complicated recovery strategies but also the management of vital business records. As Walt Kelly said in the comic strip Pogo years ago, “we are confronted with insurmountable opportunities!”
I have said for as long as I can remember, “An unexercised contingency plan can be worse than no plan at all.” I believe this more than ever. It is far too easy to document a plan which looks good on paper (or on a screen), tuck in lots of impressive content, and assert an enterprise is protected from disruption. In reality, without exercise, there is no evidence that your plan and its supporting procedures will be functional when called upon; even more critical, there is no assurance that the underlying assumptions and strategies would be appropriate and effective.
The exercise process assures your team members fully understand your plans and procedures, along with their roles and relationships; that your resources and infrastructure will be in place and sufficient; that your plan and its underlying assumptions are up to date and adequate. There is no substitute for exercising your contingency plan!
As a profession, we have learned a great deal about exercising every type of contingency plan – from community emergency management to IT disaster recovery to departmental and business unit continuity to trading floor recovery to telecommunications network recovery, crisis management/communication, and more. I hope that this book will provide you with the benefit of the collective experience and wisdom of the thirty contributors. As I have been fond of saying over the past 25 years as a management consultant, “A consultant’s job is to have made all of the mistakes already – at somebody else’ expense.”
FROM THE INTRODUCTION TO THE ORIGINAL EDITION:
There are those of us who would argue that testing the business continuity or disaster recovery plan is at least as critical as actually developing the plan in the first place. Without testing, the continuity plan is little more than an exercise in speculation – or even futility. After all, how else could any organization assess the effectiveness of a continuity plan, short of the “ultimate” test: experiencing an actual disaster?
Regrettably, many organizations find it difficult enough to develop a plan, let alone to find the resources and time to conduct meaningful testing. Recovery testing has, for these organizations, held many similarities to the classic dilemma of application software documentation: always budgeted at the end of the project, yet seldom enough time or money left to do a decent job. Why is it that some organizations never get around to recovery testing, and others test aggressively?
Where recovery testing is planned, budgeted and conducted, the predicament is, as often as not, a dearth of practical experience or knowledge. The typical disaster recovery book has a chapter near the back entitled “Testing and Maintenance,” or something like that. In a handful of pages, the author communicates the desirability of testing, the basic types of tests, and little more. This is not meant to be a criticism of these valuable books, so much as an observation that they are usually focused on plan development and implementation. This book does not attempt to convey the process of recovery plan development – it assumes that the reader at least understands the mechanics of disaster recovery / business continuity / contingency planning.
Contents
Preface – Alan Freedman
Introduction – Philip Jan Rothstein
MANAGING RECOVERY TESTING:
1 Justifying Recovery Testing – Dan Muecke
2 Managing The Testing Process – Marvin Wainschel
3 Budgeting for Recovery Testing – Robbie Atabaigi
4 The First Test – John Nevola
5 The Test Plan – David Sobolow
6 Hot Site Recovery Testing Checklist – Judith Hinds
7 Choosing and Developing the Test Scenario – Melvyn Musson
8 The Politics of Recovery Testing – Philip Jan Rothstein
9 Closing The Loop – Patrick LaValla, Robert Stoffel & Charles Erwin
TEST PARTICIPANTS AND RESOURCES
10 The Test Team – Michael G. Courton
11 Management’s Role – Dan Muecke
12 The Psychology of Recovery Testing – David Doepel
13 The Consultant s Role In Disaster Recovery Testing – Marvin Wainschel
14 The Recovery Vendor s Perspective – John Sensenich
15 Client Participation In Testing – Michael G. Courton
16 Recovery Planning Software s Role In Testing – Kenneth J. Bauman
17 The Recovery Plan Document In Testing – William J. Krouslis
18 The Restoration Vendor’s Perspective – Ronald N. Chamberlain
19 The Risk Management Perspective – Melvyn Musson
TESTING METHODS AND PROCESSES
20 Testing Methods – Geoffrey H. Wold & Robert F. Shriver
21 Testing Levels and Resources – William J. Krouslis
22 The Test Report – David Sobolow
23 Surprise and Unannounced Testing – James Certoma
24 Disaster Recovery Testing Cycles – Judith Hinds
WHAT IS BEING TESTED
25 Data Center Recovery Testing – Joan Blum
26 Data Communications Recovery Testing – Paul F. Kirvan
27 Voice Communications Recovery Testing – Paul F. Kirvan
28 Local Area Network Recovery Testing – Paul F. Kirvan & Philip Jan Rothstein
29 Trading Floor Recovery Testing – James N. Loizides
30 Testing Public and Internal Communications – James E. Lukaszewski
31 The Emergency Management Exercise Process – Patrick LaValla, Robert Stoffel & Charles Erwin
32 Disaster Plan Simulates Plane Crash Into High-Rise Building – William H. Johnson & Warren R. Matthews
33 Rehearsing Your Crisis Management Plan – Alvin Arnell
34 Testing Emergency Plans and Capabilities: Two Case Studies – Roger W. Mickelson
35 Vital Records Testing – Federal Emergency Management Agency
36 The Bottom Line – Philip Jan Rothstein
About the Author
Philip Jan Rothstein, FBCI, is President of Rothstein Associates Inc., a management consultancy focused on business continuity, disaster avoidance and recovery, and crisis management, founded in 1985.
He was elected Fellow by The Business Continuity Institute (BCI) in 1994 in recognition of his substantial contributions to the business continuity industry.
Mr. Rothstein has edited, authored or published over 100 books. He has published over 200 magazine articles.
Mr. Rothstein is a frequent speaker on business continuity, disaster avoidance, disaster recovery and crisis management for top corporate management and at industry conferences. He is sought after by the media: he has been interviewed on dozens of radio and television programs, and by well over a hundred newspapers and magazines worldwide.
He has served from 1994-1999 on the U.S. Advisory Board of Survive! The Business Continuity Group; from 1987 to 1994 on the Business Advisory Board of the Contingency Planning Exchange; and, from 1996 to 1998 on the Advisory Board for the Westchester County American Red Cross Disaster Management Program. He has served as a member of the Board of Advisors of the City College of New York School of Engineering from 2002 – 2006.
He is Publisher of Rothstein Publishing, the premier content provider for the growing, global discipline of Business Continuity Management and related fields. Since 1989, we’ve published an extensive, informational suite of books, templates & software about Business Continuity, Disaster Recovery, and Emergency, Risk, and Crisis Management.
He holds a B.S. degree in Computer Sciences from the City College of New York, School of Engineering with Honors in Human Factors Engineering and Psycholinguistics, with extensive graduate studies in Business Administration and Industrial Engineering.
Excerpt from the Preface
In December 1992, a Northeaster ripped its way through the east coast of the United States causing severe flooding and business disruption. Some companies were put out of business or significantly hampered to the extent their business could not function.
On February 26, 1993, a bomb exploded in the World Trade Center in New York City. The result of this disaster included six people dead; over one thousand injured; 50,000 people displaced from their work locale; and 950 businesses put on the street. In addition to spending eight weeks around the-clock and significant dollars to make the World Trade Center habitable, an additional $70 million was spent to erect an interim air conditioning plant to provide temporary cooling to the World Trade Center during the summer. Business loss was estimated at $600 million and many believe that figure to be conservative.
“Perhaps the World Trade Center disaster had an even greater impact than described above. It took away whatever innocence we had that a terrorist attack would ever befall us at home. It also reinforced in a most tragic way the need for preparedness in the event of catastrophic circumstances.
Most disasters are not the work of terrorists but of mother nature or human error. The impacts of disasters take many forms with human and business devastation all too often the result. In business, we refer to these phenomena and the ability to recover from them as disaster recovery or contingency planning or business continuity. Regardless of label, this translates to the ability to conduct business at an alternate location when the primary business facility is no longer functional.
Business continuity is a necessity. Testing is crucial to business continuity. This book will help you work through a variety of testing scenarios and methods to better understand the requirement for a workable contingency plan. In examining these alternative test scenarios, you will in turn be able to better plan for your own business.
Viable business continuity plans need not be a three-inch book. They should only contain what is absolutely necessary for business resumption. First and foremost they must always address life-safety of the individual. Interviews with those who lived through Hurricane Andrew and other catastrophes have articulated two common themes: the well-being of the employee and immediate family is paramount; and, the recovery plan book is not used when it represents a symbol of an audit or regulatory requirement. It is used when its contents define what is required to recover.
No two businesses are identical. One business’ needs may be very different from those of another. This book will walk you through various test scenarios and approaches. It is designed to be a guide, to help you gain optimal results in business preparedness, to identify and quantify base needs, to facilitate response to the difficult questions of who, where, what, when, and how.
A business continuity plan that is not tested is of little to no value. Testing and learning from the experience is the only way to perfect the plan; to ensure the business needs are addressed. Ideally, a perfect test addresses all business requirements and thoroughly involves the real users. Anything less than optimum presents levels of business risk.
This book has been written to allow the reader to focus on a particular or multiple concerns (at the discretion of the reader); to examine and study one or a series of test scenarios. As this preface is being written the Midwest United States is trying to recover from the worst flood in its experience. The human devastation is incalculable while business losses have been estimated as high as eight billion dollars. The ability to recover in an appropriate time after a catastrophe is imperative. One difficulty has been creating a document and test mechanism that will do just that. You will discern what it takes to be prepared, to experience a catastrophe and survive without sacrifice of life-safety or going out of business.”
Alan Freedman
Vice President, Technology Strategic Planning
Bankers Trust Company
Fall, 1993
Reviews
” … The most widely read source of information on the subject to date… an excellent starting point for both novice and experienced business continuity planners… this book is a feast of ideas that challenge and stimulate.” – – – SURVIVE Magazine
==============================
Disaster Recovery Testing: Exercising Your Contingency Plan, written jointly by thirty disaster recovery professionals, drives home the importance of testing the company’s contingency plan, discussing necessary management skills, effective use of resources, and the elements of successful testing in various settings.
The authors assume that a disaster recovery plan has already been developed and approved. The book’s initial chapters are similar to what one might find in a disaster recovery planning book, including how to justify, budget, and manage the process – but all are geared towards testing.
One important chapter addresses the benefits of client participation. Disaster recovery managers know how important it is to develop a partnership with the client business leaders to address adequately all recovery aspects of business functionality.
Various testing methods and processes are also addressed. The authors provide numerous checklists and suggest topics to cover in any disaster recovery planning document. These lists alone make the book worthwhile.
Overall, … the material focuses much needed attention on disaster recovery testing. It is written in four well-planned, easy-to-follow sections, with numerous realistic scenarios and ideas to consider. This book will provide useful guidance for management of the entire disaster recovery testing process. – – – SECURITY MANAGEMENT Magazine.
==============================
Rather than providing readers with only one perspective on testing contingency plans, Mr. Rothstein has taken the powerful approach of carefully assembling a panel of thirty professionals to contribute their unique expertise in 36 detailed chapters, 350 pages, covering such subject areas as:
- test planning and management, including management’s role, budgeting, justification and politics
- test participants and resources, including professional development, human factors, the test team, self-assessment, the roles of vendors, consultants, auditors, clients, software
- testing methods, including walkthroughs, simulations, joint testing, surprise testing, real disasters as the ultimate test
- what is being tested, including business units, locations, data centers, voice / data communications, trading floors, local area networks
- any other practical considerations, such as test monitoring, first-time testing, feedback, reporting and follow-up;
even a sample test plan is included.
According to Mr. Rothstein, President of Rothstein Associates Inc., “Testing is critical to any business continuity or disaster recovery program. Yet, most books in the field devote at most a few pages to testing.”
