The Northeast blackout of 2003 was a widespread power outage throughout parts of the Northeastern and Midwestern United States, and most parts of the Canadian province of Ontario on Thursday, August 14, 2003, beginning just after 4:10 p.m. EDT.
Most places restored power by midnight (within 7 hours), some as early as 6 p.m. on August 14 (within 2 hours), while the New York City Subway resumed limited services around 8 p.m. Full power was restored to New York City and parts of Toronto on August 16. At the time, it was the world's second most widespread blackout in history, after the 1999 Southern Brazil blackout. The outage, which was much more widespread than the Northeast blackout of 1965, affected an estimated 55 million people, including 10 million people in southern and central Ontario and 45 million people in eight U.S. states.
The blackout was due to a software bug in the alarm system at the control room of FirstEnergy, which rendered operators unaware of the need to redistribute load after overloaded transmission lines dropped in voltage. What should have been a manageable local blackout cascaded into the collapse of much of the Northeast regional electricity distribution system.
The Northeast blackout of 2003 was a widespread power outage throughout parts of the Northeastern and Midwestern United States, and most parts of the Canadian province of Ontario on Thursday, August 14, 2003, beginning just after 4:10 p.m. EDT.
Most places restored power by midnight (within 7 hours), some as early as 6 p.m. on August 14 (within 2 hours), while the New York City Subway resumed limited services around 8 p.m. Full power was restored to New York City and parts of Toronto on August 16. At the time, it was the world's second most widespread blackout in history, after the 1999 Southern Brazil blackout. The outage, which was much more widespread than the Northeast blackout of 1965, affected an estimated 55 million people, including 10 million people in southern and central Ontario and 45 million people in eight U.S. states.
The blackout was due to a software bug in the alarm system at the control room of FirstEnergy, which rendered operators unaware of the need to redistribute load after overloaded transmission lines dropped in voltage. What should have been a manageable local blackout cascaded into the collapse of much of the Northeast regional electricity distribution system.
According to the New York Independent System Operator (NYISO)—the ISO, responsible for managing the New York state power grid—a 3,500 megawatt power surge (towards Ontario) affected the transmission grid at 4:10:39 p.m. EDT.
For the next 30 minutes, until 4:40 p.m. EDT, outages were reported in parts of Michigan (Detroit), Ohio (Cleveland, Akron, Toledo), Ontario (Toronto, Hamilton, London, Windsor), New Jersey (Newark), and New York (New York City, Suffolk, Nassau, Westchester, Orange and Rockland counties, Rochester, Syracuse, Binghamton, Albany).
This was followed by outages in other areas initially unaffected, including all of New York City, portions of southern New York state, New Jersey, Vermont, Connecticut, as well as most of the province of Ontario. Eventually, a large, somewhat triangular area bounded by Lansing, Michigan, Sault Ste. Marie, Ontario, the shore of James Bay, Ottawa, New York, and Toledo was left without power.
According to the official analysis of the blackout prepared by the US and Canadian governments, more than 508 generating units at 265 power plants shut down during the outage. In the minutes before the event, the NYISO-managed power system was carrying 28,700 MW of load. At the height of the outage, the load had dropped to 5,716 MW, a loss of 80%.
Essential services remained in operation in some of these areas. In others, backup generation systems failed. Telephone networks generally remained operational, but the increased demand triggered by the blackout left some circuits overloaded. Water systems in several cities lost pressure, forcing boil-water advisories to be put into effect. Cellular service was interrupted as mobile networks were overloaded with the increase in volume of calls. Multiple cell sites were out of commission due to power outages. Television and radio stations remained on the air, with the help of backup generators, although some stations were knocked off the air for periods ranging from several hours to the length of the entire blackout.
High heat played a role in the initial event that triggered the wider power outage. Much of the affected region reached temperatures higher than 31 °C (88 °F) that day, increasing energy demand as people across the region turned on fans and air conditioning. This caused the power lines to sag as higher currents heated the lines.
Most of the Amtrak Northeast Corridor service was interrupted, as it relied on electricity for its signaling and crossing systems; electrified commuter railways also shut down. Via Rail in Canada was able to continue most of its service. All airports in the affected area closed immediately, there were no departures, and incoming flights had to be diverted to airports with power.
The reliability of the electrical grid was called into question and required substantial investment to repair its shortcomings.
In areas where power remained off after nightfall, the Milky Way and orbiting artificial satellites became visible to the naked eye in metropolitan areas where they cannot ordinarily be seen due to light pollution.
Duration
Most places restored power by midnight, as early as 6 p.m. on August 14, and the New York City Subway had resumed limited services around 8 p.m. Some areas lost power for only four to eight hours; these are: Albany and parts of Long Island in New York; three‐quarters of New Jersey; parts of Pennsylvania, Ohio and Michigan; New London County, Connecticut; parts of downtown Toronto, Mississauga, and London in Ontario; portions of western Ottawa in Ontario, including Kanata and south to Kingston; a number of areas of the Regional Municipality of Niagara in Ontario; and parts of Southwestern Ontario, particularly areas near the Bruce Nuclear Generating Station.
By the next morning (August 15), some areas of Manhattan regained power around 5:00 a.m.; Staten Island regained power around 3:00 a.m. Half of the affected portions of Ontario regained power by the morning. By early evening of August 15, two airports, Cleveland Hopkins International Airport and Toronto Pearson International Airport, were back in service. By August 16, power was fully restored in New York and parts of Toronto. 500,000 Detroit Edison customers were still without power at 10:00pm August 15; all were restored by 6:30am August 16.
Unaffected regions
Within the area affected, about 200,000 people continued to have power—in the Niagara Peninsula of Ontario; the easternmost corner of Ontario (centered on Cornwall); northwestern Ontario (west of Wawa); and the Buffalo, New York area, excluding southern Erie county, along the shore of Lake Huron via a feeder line to Owen Sound from Bruce Nuclear Generating Station. Three of the four Bruce B units were able to throttle back their output without a complete shutdown, then reconnect to the grid within five hours: the portion of New York state including parts of Albany and north and west of Albany, a small pocket of mid-east Michigan, the Upper Peninsula of Michigan, and small pockets in New Jersey. The unaffected area was protected by transmission circuit devices at the Sir Adam Beck Hydroelectric Generating Stations in Niagara Falls (coincidentally, the starting point of the Northeast blackout of 1965) at a switching station of the hydroelectric power station in Cornwall, as well as central New York state. Philadelphia and the surrounding mid-Atlantic areas were also completely unaffected because PJM disconnected them. The Saint Clair power plant in East China Township, Michigan, remained online for about 36 hours, and residents were informed that the plant would have to shut down in order to facilitate the reboot of the whole system. Orrville, Ohio, was able to restore power within an hour disconnecting the local utility from the larger grid and restarting the coal-fired generator.
Causes
Power grid management concepts
The load on any power network must be immediately matched by its supply and its ability to transmit that power. Any overload of a power line or generator can cause costly damage, so the affected device is disconnected from the network if an overload is detected.
The electrical resistance of a power line causes it to produce more heat as the current it carries increases. If this heat is not sufficiently dissipated, the metal conductor in the line will expand and lengthen, so that it sags between supporting structures. If the line sags too low, a flash over to nearby objects (such as trees) may occur, causing a transient increase in current. Automatic protective relays detect the excessively high current and quickly disconnect the line, so the load previously carried by the line is transferred to other lines. If the other lines do not have enough spare capacity to accommodate the extra current, their overload protection will react as well, causing a cascading failure.
System operators are responsible for ensuring that power supply and loads remain balanced, and for keeping the system within safe operational limits such that no single fault can cause the system to fail. After a failure affecting their system, operators must obtain more power from generators or other regions or "shed load" (meaning to intentionally cut power to a given area, similar to a rolling blackout) until they can be sure that the worst remaining possible failure anywhere in the system will not cause a system collapse. In an emergency, they are expected to immediately shed load as required to bring the system into balance.
To assist the operators there are computer systems, with backups, which issue alarms when there are faults in the transmission or generation system. Power flow modeling tools let them analyze the state of their network, predict whether any parts of it may be overloaded, and predict the worst possible failure remaining, so that they can change the distribution of generation or reconfigure the transmission system to prevent a failure should this situation occur. If the computer systems and their backups fail, the operators are required to monitor the grid manually, instead of relying on computer alerts. If they cannot interpret the state of the power grid in such an event, they follow a contingency plan, contacting other plant and grid operators by telephone if necessary. If there is a failure, they are also required to notify adjacent areas which may be affected, so those can predict the possible effects on their own systems.
Investigation efforts
A joint federal task force was formed by the governments of Canada and the U.S. to oversee the investigation and report directly to Ottawa and Washington. The task force was led by then-Canadian Natural Resources Minister Herb Dhaliwal and U.S. Energy Secretary Spencer Abraham.
In addition to determining the initial cause of the cascading failure, the investigation of the incident also included an examination of the failure of safeguards designed to prevent a repetition of the Northeast blackout of 1965. The North American Electric Reliability Corporation, a joint Canada-U.S. council, is responsible for dealing with these issues.
On November 19, 2003, Abraham said his department would not seek to punish FirstEnergy Corp for its role in the blackout because current U.S. law does not require electric reliability standards. Abraham stated, "The absence of enforceable reliability standards creates a situation in which there are limits in terms of federal level punishment."
Findings
In April 2004, the U.S.-Canada Power System Outage Task Force released their final report, placing the causes of the blackout into four groups:
FirstEnergy (FE) and its reliability council "failed to assess and understand the inadequacies of FE's system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria."
FirstEnergy "did not recognize or understand the deteriorating condition of its system."
FirstEnergy "failed to manage adequately tree growth in its transmission rights-of-way."
Finally, the "failure of the interconnected grid's reliability organizations to provide effective real-time diagnostic support."
The report states that a generating plant in Eastlake, Ohio, a suburb northeast of Cleveland, went offline amid high electrical demand, putting a strain on high-voltage power lines (located in Walton Hills, Ohio, a southeast suburb of Cleveland) which later went out of service when they came in contact with "overgrown trees". This trip caused load to transfer to other transmission lines, which were not able to bear the load, tripping their breakers. Once these multiple trips occurred, a number of generators suddenly lost parts of their loads, so they accelerated out of phase with the grid at different rates, and tripped out to prevent damage. The cascading effect that resulted ultimately forced the shutdown of at least 265 power plants.
Computer failure
A software bug known as a race condition existed in General Electric Energy's Unix-based XA/21 energy management system. Once triggered, the bug stalled FirstEnergy's control room alarm system for over an hour. System operators were unaware of the malfunction. The failure deprived them of both audio and visual alerts for important changes in system state.
Unprocessed events queued up after the alarm system failure and the primary server failed within 30 minutes. Then all applications (including the stalled alarm system) were automatically transferred to the backup server, which itself failed at 14:54. The server failures slowed the screen refresh rate of the operators' computer consoles from 1–3 seconds to 59 seconds per screen. The lack of alarms led operators to dismiss a call from American Electric Power about the tripping and reclosure of a 345 kV shared line in northeast Ohio. But by 15:42, after the control room itself lost power, control room operators informed technical support (who were already troubleshooting the problem) of the alarm system problem.
Sequence of events
Outages in the Cleveland-Akron area
The cascading failure began in the FE-controlled Cleveland–Akron area; the area was significant because it was a "transmission-constrained load pocket with relatively limited generation." On August 14, a number of power generators in and around the area were offline. This reduced the available resources for voltage control and reactive power management in contingencies but did not affect normal operation. Four or five of the area's capacitor banks—also used to manage reactive power—were also offline for routine maintenance; this did not follow the best practice of performing such maintenance during low-demand periods, and their status was not reported for regional planning purposes because FE had deemed them to be non-critical infrastructure.
Incidents contributing to the blackout began after noon on August 14. FE's reliability coordinator was the Midwest Independent System Operator (MISO). MISO maintained a regularly updated model of its area of responsibility with a state estimator (SE). At 12:15 p.m., the SE was mistakenly shutdown after producing anomalous results due to old data. Unscheduled grid outages followed. In the Cleveland-Akron area, Eastlake Power Plant unit 5 went offline at 1:31 p.m. In Southern Ohio, Dayton Power and Light's (DPL) Stuart-Atlanta transmission line tripped from tree contact at 2:02 p.m. The outages did not affect normal operation. However, the loss of the Eastlake unit put the FE system at risk of overloads in certain contingency scenarios; FE did not notice the danger. At the time of the Eastlake unit outage, FE was asking for and receiving significant additional voltage support, but the situation was not unprecedented.
FE's situational awareness was reduced by the development of faults in its General Electric Harris XA21 energy management system (EMS) from 2:14 p.m. to 2:54 p.m. The alarm and logging software in the control room failed first, followed by some remote terminals and data links, then the primary and backup EMS servers. Control room operators were unaware of the alarm failure for over an hour. Operators began to suspect a failure after receiving a notification from American Electric Power (AEP) at 2:32 p.m. about a short AEP line outage that the FE EMS had failed to raise an alert for. FE IT was automatically notified of the remote terminal and server failures. IT performed a "warm reboot" of the primary server at 3:08 p.m. but did not realize that the alarm function remained broken. The control room notified IT of the alarm problem at 3:42 p.m. A "cold reboot"—later discovered to be the vendor-recommended action—was rejected because the power system situation was precarious and operators were concerned about the greater loss of functionality during the reboot procedure. The servers were offline again at 3:46 p.m. to 3:59 p.m. as they were rebooted in another attempt to re-enable the alarms. FE failed to notify MISO of its degraded situational awareness.