News
The FCC has released findings from an investigation into a Feb. outage at AT&T that blocked thousands of 911 calls and disconnected millions of devices.
This article was originally released by our sister publication, Broadband Communities
The direct cause of an AT&T outage in February that impacted 125 million devices and prevented 25,000 calls to 911 call centers was an error by an employee who misconfigured a single network element, according to a newly released report from the Federal Communications Commission (FCC).
The report, released by the FCC today, concluded the misconfigured network element caused the AT&T network to respond by entering a “Protection Mode,” which then disconnected all wireless devices.
It was one of several findings made during the course of the FCC’s investigation into the outage, which impacted users across all 50 states, according to the FCC.
According to the FCC’s report, the outage lasted at least 12 hours and blocked a total of 92 million phone calls.
The report also found that AT&T’s lab testing “did not discover the improper configuration of the network element that caused the outage.”
Additionally, lab testing conducted by the provider “did not identify the potential impact to the network of that or similar misconfigurations,” according to the report.
In comments included with a summary accompanying the report, FCC Chairwoman Jessica Rosenworcel called the incident a ‘sunny day outage.’
“When you sign-up for wireless service, you expect it will be available when you need it – especially for emergencies,” Rosenworcel said. “This ‘sunny day’ outage prevented consumers across the country from communicating, including by blocking 911 calls, and stopped public safety personnel from using FirstNet.”
Rosenworcel’s comments continued.
“We take this incident seriously and are working to provide accountability for this lapse in service and prevent similar outages in the future,” the chairwoman stated.
The FCC summary of the report, which concludes with a statement saying the matter is being referred to the FCC’s enforcement bureau for potential violations, also noted that the outage “cut off service to devices operated by public safety users of the First Responder Network Authority (FirstNet).”
“AT&T prioritized the restoration of FirstNet before other services but did not notify FirstNet customers of the outage until three hours after it began, and nearly one hour after service was restored,” the FCC’s summary stated.
Another finding included in the report involved post-installation testing.
The FCC’s investigation surmised the company “either lacked sufficient oversight and controls” to ensure post-installation testing practices were followed, or that the processes used by the company itself was insufficient.
However, AT&T’s corrective action following the outage occurred quickly, according to the FCC’s findings.
“Within 48 hours of the outage, AT&T implemented additional technical controls in its network,” the report stated. “This included scanning the network for any network elements lacking the controls that would have prevented the outage, and promptly putting those controls in place.”
Other steps taken by the provider following the outage included the implementation of additional network enhancements to boost network resilience and extra steps for peer review. The reviews serve a purpose of ensuring that “maintenance work cannot take place without confirmation that required peer reviews have been completed,” the report stated.
To get content like this delivered to your inbox, subscribe to the Broadband Communities newsletter.