Scheduled Data Center Outage, December 1

On November 27, the University experienced electrical equipment problems in its main data center that houses key servers for University computer systems. The electrical problem rendered all of the systems inoperable.

Since that time, the systems have remained on emergency power in order to ensure their continued functioning as University personnel continued their efforts to repair the UPS (uninterruptable power supply) and troubleshoot the other electrical system components. After extensive testing, a plan has been put in place to successfully transition off of the generators and back to the electrical grid.

The transition back to main power will take place during the early morning hours of December 1, 2007, between 12:01 a.m. and 7:30 a.m. To reduce the risk of another data center failure, all non-essential services will be shut down first, beginning at 12:01 a.m.

Core enterprise systems that are utilized by students will remain operational during the transition period. These systems include @u.northwestern.edu e-mail, Blackboard, Depot, NUINFO, and Library services. By 7:30 a.m. on December 1, work is expected to be successfully completed, and the data center will be operating normally.

The November 27 event occurred when emergency maintenance was being performed on the UPS equipment. We are continuing to investigate why the remaining UPS failed and caused the unexpected power outage. Without power, the servers shut down without warning.

University personnel immediately enacted disaster recovery plans to isolate the problem and begin the system recovery process, including putting the data center on generator power. As a result, 90 percent of the primary computing systems were re-enabled within two hours of the initial outage. The remaining systems were enabled in a phased manner throughout the remainder of the day. Some systems, such as the central e-mail servers, did not respond well to the sudden loss of power and took extra effort to revive. As of 6 p.m. that same day, all of the affected systems were restored and functioning properly on the backup emergency power system.

Every effort is being made to return the data center to the power grid in a non disruptive manner. New status messages will be posted on the Northwestern University Web site and NUIT home page with current information by mid-morning on December 1.

NUIT Support Center

Central helpdesk for faculty, staff, and students.
847-491-HELP (4357)
consultant@northwestern.edu