A device designed to bolster the Superdome electrical system instead caused it to shut down dramatically during Super Bowl XLVII, officials said Friday, but the equipment maker blames the settings keyed in by system operators in New Orleans.
Amid the volley of competing conclusions today, there was only slightly more clarity on the cause of the partial blackout of the Mercedes-Benz Superdome that halted action for 34 minutes Sunday night in the Baltimore Ravens 34-31 victory over the San Francisco 49ers. (See related story: "What Caused the Super Bowl Blackout at the Superdome?")
Entergy New Orleans, which provides power to the venue, said testing traced the source of the problem to an "electrical relay device" it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium.
At an emergency meeting of the New Orleans City Council utility regulatory committee, the power company had no answer to why the relay failed. Though there was some discussion that an independent probe might be needed, council members laced their questions with civic pride.
"The reason that you've identified is not a reason that was the fault of Entergy, or the Superdome or anyone here in New Orleans, is that correct?" asked city council member James Austin Gray II.
"As best as we can tell, the partial outage was due to a device that was manufactured in Chicago," said Charles Rice, president and chief executive of Entergy New Orleans.
But the manufacturer, S&C Electric, a 102-year-old designer of switching and protection products for electric power transmission and distribution, says the problem Sunday was caused by human error by system operators, not a fault with the device. (See related quiz: "What You Don't Know About Electricity.")
"In working with those involved, we found that the electric outage at the Super Bowl was a result of the electric load current exceeding the trip setting for the switchgear relay as set by the system operators," said Michael Edmonds, S&C's vice president for strategic solutions, in an email exchange with National Geographic News. "Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power.
"S&C continues to work with all those involved to get the system back online, and our customers can continue to rely on the quality and performance of our products," he added.
Root Cause?
Relays are devices that sense electrical current, voltages, and power and tell the circuits to operate. "Relays don't fail on their own, or it's relatively rare," says Helmut Brosz, director of the Institute of Forensic Electro-Pathology and former chair of the engineering sciences section of the American Academy of Forensic Sciences. He explains that there are alarm relays, which simply provide an alert if there is a problem, and protective relays, which actually trip breakers in the event of a problem, causing power to stop flowing.
One of the most notorious power outages traced to a relay was the 1965 blackout that plunged 30 million people in the Northeastern United States and Canada, which was caused by a protective relay outside of Toronto tripping a circuit. The U.S. regulators traced that problem to human error: System operators had input an incorrect setting into the backup protective relay, and when the load on the line exceeded that set point, the relay took a big 230-kilovolt line out of service, redistributing the flow on other lines, which tripped out in a cascading failure in a matter of seconds.
S&C, maker of the relay device used at the Superdome, markets a variety of "smart grid solutions." Although those are typically meant to improve efficiency and performance of large utility power systems, outside experts stress that the extent of an electrical system like that in Superdome should not be underestimated. "The stadium electrical system would be similar in size and complexity to a small town," said David Bassett, a retired senior staff engineer for PPL Electric Utilities of Pennsylvania and a senior member of the Institute of Electrical and Electronics Engineers.
In New Orleans this morning, some city council members expressed concern that there was not a more detailed explanation of what caused the relay device to fail at the showcase game before 108.4 million television viewers. "We've told the public we're going to have an investigation by a third party, and we have an obligation," said Council Vice President Jacquelyn Brechtel Clarkson. "What assurance do we have on this test? Was it an overload?"
Rice said Entergy would discuss the need for a third-party investigator with Superdome management, but said the utility was "certain" the cause of the outage was the device, which was taken out of service. New equipment is being evaluated, he said.
"Currently that piece is offline, and the Superdome is fully functional," Rice said. "What we will do is continue our analysis, and we are in the process of obtaining replacement equipment, and we are working very closely with the manufacturer of the equipment to ensure that never happens again."
At the end of the hearing, Clarkson said she still believed there should be a third-party investigation of the incident "to reassure the public... and defy the naysayers rising up around the country." She noted that there had been outages in football stadiums in the past. "This is not a one-of-a-kind event," she said. "But I think we have to take the attitude that we've handled it better than it's ever been handled before." In an unusual structure, the New Orleans City Council regulates the local subsidiary of parent company Entergy, while the Louisiana Public Service Commission regulates its operations outside of Orleans Parish.
The hearing revealed the extraordinary measures that had been undertaken in advance of the Super Bowl to upgrade, bolster, and maintain the electrical systems.
Entergy started evaluating the stadium electrical systems two years ago, and decided to add a protective relay system. "The purpose was to provide a newer, more advanced type of protection to the Superdome," said Dennis Dawsey, Louisiana vice president for transmission and distribution operations for Entergy. On the night of the Super Bowl, Entergy had nine people at the stadium to monitor the electrical systems, two at the substation and two at a distribution facility. "We were prepared for this event," he said. (See related photos: "Super Bowl Caps Banner Season in NFL Green Drive.")
The Superdome management, meanwhile, last year had replaced some aging electric feeder cables that bring power into the stadium, said Doug Thornton, senior vice president, Stadiums and Arenas for SMG, which manages the operations of the Superdome. "I never even heard of a relay device until Monday," he told the council. "But we did take steps to ensure that those feeder cables were 100 percent reliable."
Black Box Data
Just as investigators search for the "black box" that stores flight data after a plane crash, probes on electrical outages initially center on a similar device that records electrical flow and circuit trips, experts say. But the black box of electrical systems—known as the SCADA (supervisory control and data acquisition) system—"has its limits," says B. Don Russell, distinguished professor of electrical engineering at Texas A & M University, who has 30 years of experience as an electric power forensic engineer. "It's not very accurate, but will give a general idea of the time when the power went off—if it occurred as a disruption on the utility system. It will tell them if the breaker at the utility system went off, but it will tell them nothing about what happened inside the facility."
For that part of the puzzle, examiners would need to start at the main service panel, where the electricity supply enters the facility and look at the entire system between that point and the lights. The fact that the stadium lights came back on relatively quickly suggests "there's probably not anything broken," Russell says. "They will want to reconstruct what lights went off and why, asking questions and testing equipment tracing back the power" until they find the piece of equipment that was responsible.
Much like a medical examiner conducting an autopsy, a forensic electrical engineer is an expert in the field whose determination of the cause of an electrical event can withstand scrutiny in a court of law. "If you've got a problem that involved engineering and it's in the legal system, we probably have somebody you could hire," says Marvin Specter, Executive Director of the National Academy of Forensic Engineers. "The process is no different from the case of the car hit the pole hit the bicycle: gather information and analyze with competence."
In the past, forensic electrical engineers have investigated the causes of events such as the 1979 accident at the Three Mile Island nuclear power plant and the widespread blackouts across the Northeastern United States and Canada in the 1960s and in 2003. They also work on smaller scale incidents such as an individual being electrocuted by her toaster. They are usually called in when death or injury has occurred, or a large sum of money is at stake.
For New Orleans, which is preparing to host its annual Mardi Gras celebration next week, the city's reputation as an event venue is clearly on the line. "More than anything else we want to get rid of the speculation," said Councilmember Cynthia Hedge-Morrell, chair of the utility committee. "Let's get rid of the conspiracy theories and get down to nuts and bolts facts. Then we can make sure it won't happen again." Like other council colleagues, she mentioned New Orleans' bid to have the Super Bowl return for the city's tricentennial in 2018. "After all, we would love to welcome the NFL back for our 300th anniversary," she said.
This story is part of a special series that explores energy issues. For more, visit The Great Energy Challenge.
