IT Emergency Preparedness

• By Michael Fickes
• 06/01/08

When reviewing your school district’s emergency preparedness plans, how do you evaluate the component that ensures the survival of your information technology (IT) networks? What do you look for? What threats must you prepare today with? How do you secure an IT network against those threats?

First, a definition of IT emergency preparedness: “It means that when the blizzard hits, you and your department will execute a blizzard-emergency plan that may have been prepared some time ago but is updated regularly,” says Doug Landoll, CISSP, CISA, and chief strategist with the security consulting firm of Lantego, LLC in Austin, TX. “The IT manager has assigned responsibilities to critical personnel. Everyone has drilled and practiced regularly. Transportation arrangements have been made so that everyone can get to work.”

Planning begins with the development of a document that lays out how an emergency might compromise the sensitive and critical school district data and systems that live on your IT network. The document also discusses threats, prevention, and recovery.

“All school districts should be doing general security risk assessments and emergency preparedness plans, and IT emergency preparedness should be integrated into those plans,” says Landoll.  

Landoll suggests a three-phase approach to emergency IT preparedness:

•    identify IT assets you want to protect;
•    list the threats confronting those IT assets; and
•    identify or find tools, policies, and procedures that you can use to protect those assets.

Identify IT Assets
IT assets consist of systems with data, Landoll says. Systems include software applications that record grades, manage capital budgets, write payroll checks, schedule maintenance, purchase supplies, automate heating and lighting systems, operate security systems, maintain health records as appropriate, provide information to the public… you name it.

While identifying systems, it is also important to note where they are and whether key systems are physically and logically separate from other systems.

Usually, data servers reside in a main IT building somewhere in the district. Individual schools will also house some data. Districts often outsource certain functions to third parties, and that data will reside outside district facilities.

Sites that store data from outsourced systems include local IT shops retained to backup data. Backing up data, of course, is part of an IT routine and important to IT preparedness.

A backup plan, according to Landoll, might call for an incremental backup every day and a full backup on the weekend, with regular restoration tests. The preparedness part of the backup plan would plan restoration needs. If the financial system must come back up within 12 hours, for example, the IT manager must make sure that the outsourcing backup firm can handle that requirement. Sometimes special restoration arrangements will be necessary.

Other outsourced data could include Web services — managing the Web server with district and school home pages. “This is actually a good idea,” Landoll says.

For instance, explains Landoll, a school district in Columbus, OH, could outsource Web services to a national provider and specify placing Web pages on a local server and also a physically distant server in, say, California. Suppose a tornado hit the local site and took it off line. Thanks to the redundant Website, administrators could post a message that the students were all okay and waiting out the storm in the auditorium. Parents logging on to get information will find the Website up and operating and be able to read the message.

Concerning the issue of physical and logical separation, Landoll advises maintaining financial systems separately from other systems. “If I put a school’s public Web server with the sports team schedules on the same server as financial systems, a hacker that finds a vulnerability in the Web server can get into the financial system,” he says.

Sensitive Systems, Critical Systems
After identifying the data and systems, Landoll analyzes each identified piece to determine how critical and how sensitive it is.

A critical system would damage the school district’s ability to operate if it were lost. A sensitive system would compromise district facilities and people in some way if it fell under the control of someone with malicious or criminal intentions.

“It wouldn’t be bad to lose the Webpage that displays the football schedule,” Landoll says. “That isn’t critical information. Payroll is critical as well as a sensitive.”

The analysis would rank how critical and how sensitive each data source and system is to school operations and school security. “The most sensitive and most critical systems will require the most controls and protections,” Landoll continues.

Catalog Threats
Threats include natural disasters, and malicious or criminal cyber activities.

What kinds of natural disasters affect your area? State disaster planning offices often have this kind of information. A book entitled Security Risk Assessment Handbook (2006, Auerbach Publishing) by Doug Landoll provides maps and charts that lay out the natural disasters that various regions should consider.

In central Maryland, for instance, severe thunderstorms and occasional hurricanes lead to flooding often enough that preparedness plans in the region address these disasters.

Landoll’s book recommends preparing to protect IT equipment and data from floods by assessing the location: are schools and data facilities close to rivers? If so, that will influence the construction of systems in the first place.

One step recommended for every IT department faced with the threat of floods: put the equipment on the second floor of the building — not the first floor and certainly not the basement.

Don’t forget about backup power. Most natural disasters cause power outages that can last for some time. Protecting the equipment is one thing; being able to use systems during recovery is important, too.

Lay in a supply of sandbags. Keep a hand truck in the equipment room to help move equipment if necessary. Access to a forklift would be even better.

Lightning arrestors and surge suppressors are important tools for any IT manager. In areas subject to lightning, they are essential.

Other natural disasters call for other preparations. Extra sturdy equipment racks braced securely to walls might be a sensible precaution in earthquake-prone areas.

A preparedness plan addresses each natural disaster that affects a region.

Cyber Vandals, Cyber Criminals
“Public school systems often have many vulnerabilities, especially in Web interfaces,” Landoll says. “Making a secure Web application costs too much for most district budgets. At the same time, kids are going to try to break in.

“Bright kids can hit these sites on their own. But they needn’t be bright. Many kids simply download automated hacking programs from the Internet. My experience with corporations and schools has shown me that unprotected IT networks are usually being hacked and will eventually lose control of sensitive data.”

Solutions include firewalls, encryption, logical and physical security, and back up and restore systems applied to systems called out by the criticality and sensitivity analysis.

Then there are employees engaged in criminal undertakings. Most school districts employ thousands of people. It would be unusual if no one had criminal tendencies. According to Landoll, IT departments must deal with IT administrators who sell grades, embezzle money, turn off security controls for pay, and come up with ideas that you will never expect.

Cyber crime can turn into a liability disaster for a school district. What if an employee sold personal information on district servers to a ring of identity thieves?

Controls related to employee cyber crime include oversight and audit logging. “If your school system has a 10 person IT department, it would be a good idea for the boss to get a report on what they did today,” Landoll says. “And oversight program might tell me that an assistant administrator modified grades today — but that’s okay because interim reports came out today and that’s when grades get changed. But what if this grade was changed from an “A” to a “C?” I might question that because it is unusual to change two grades.”

If an assistant administrator altered a firewall rule, the boss should get a report and check whether the administrator followed the procedure established for altering firewall rules, continues Landoll. In many IT departments, one person can suggest a change, but a supervisor must sign off on it.

Auditing is a different form of oversight. Instead of reporting on one person’s daily activities, auditing records all actions that carried out by various systems and looks for patterns. “You might want to audit systems ranked as high priorities on your sensitivity and criticality analysis,” Landoll says. “The idea is to monitor for exceptions. You would want to know, for example, if someone changed a grade on the one hand and then altered how much money that student has in his or her account.”

The IT Disaster Recovery Role
While it makes sense to develop an emergency IT preparedness plan to protect sensitive and critical information, protecting the IT system from harm is also important because it plays such an important role in disaster recovery.

As a member of the Illinois Terrorism Task Force subcommittee on school security, Paul Timm helped develop a template for crisis preparedness in Illinois. “We put this together by drawing on best practices used by a couple of different states,” says Timm, who is the president of RETA Security, a security consulting firm based in Lemont, IL.

The template contains a section on the role of IT in disaster recovery. In the wake of an emergency, the IT department will coordinate the use of technology, states the report. It will assist first responders in setting up an emergency communications network and establish communications with appropriate agencies. IT administrators will monitor the communications connections and provide student and staff information that first responders might request from the IT files. If necessary, administrators will set up freestanding computers with student and staff databases for use by emergency responders.

Under the plan, the IT department will also prepare and maintain an emergency kit with floor plans showing telephone line locations, computer locations, and the locations of other communications equipment.

To summarize, an emergency IT plan is what IT managers do when an emergency happens. It is written down, and IT employees regularly practice their assignments in connection to the emergencies covered by the plan: natural disasters, cyber vandalism, and cyber crime. So when disaster comes, there is a chance to mitigate if not prevent ill effects. Finally, the plan directs the role of IT during recovery.