Facebook and its various apps and products suffered a serious outage yesterday for nearly six hours. Here's what we know.
If you were on your phone yesterday and tried to log into Facebook, Instagram, or WhatsApp and couldn’t, you weren’t alone. Virtually no one was able to access any of Facebook’s products and apps due to a serious outage the company suffered.
Here’s a basic timeline of what happened:
At around 11:30 am ET, users started reporting that both Instagram and Facebook were acting weird. Those users, naturally, reported the issues on Twitter.
Following these complaints, more came pouring in regarding WhatsApp, Oculus, and other services that used Facebook’s log-in method. This seemed to indicate a widespread outage that affected virtually every aspect of Facebook’s online presence.
At 12:22 pm ET, Facebook formally acknowledged that its products were experiencing problems. The company apologized to folks like advertisers and small businesses for any inconveniences caused by the issues.
Over the next ~6 hours, reports would begin to surface as to what may be the cause of the outage. Some speculated it was a mistake during normal maintenance of Facebook’s DNS servers. Others thought that it was an orchestrated attack on Facebook after last night’s 60 Minutes where a whistleblower claimed Facebook was using hate speech to propel its own wealth.
During this time, it was reported that the outage was so bad, it wasn’t capable of being fixed remotely. Facebook had to send a team of its employees out to one of its data centers in Santa Clara to try a “manual reset.” Meanwhile, other employees weren’t able to access their own offices. (In case you couldn’t already tell, Facebook’s services are very deep in the inner workings of the company as a whole.)
At around 6:20 pm ET, users began reporting that Facebook and its suite of apps were back online. Shortly after, I checked my own devices and found most of the services to be back online.
Facebook then confirmed it was bringing its products back online at 6:33 pm ET in an apology to anyone the outage affected.
The most reputable cause of the outage seems to be a malfunction during a routine BGP (or Border Gateway Protocol) upgrade, in which Facebook’s DNS routing information was accidentally scrapped so that no network could connect to the company’s servers. This then led to Facebook’s own servers going offline, leaving no communication between Facebook’s platform and the internet.
Facebook published a blog post late last night where the company blamed “configuration changes” as the culprit of today’s malfunction. They don’t go into detail on those changes, however, so there’s a solid chance a BGP update was to blame.
Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.
Did my data leak?
No. Currently, there is no evidence that any users’ data leaked or was hacked as a result of the outage. Facebook even touched on this in its post.
Our services are now back online and we’re actively working to fully return them to regular operations. We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime.
How bad did it affect Facebook?
According to Fortune, Facebook lost about $60 million in revenue between all of its apps due to the outage. The company generates around $13.3 million per hour, so an estimate like this isn’t completely dismissible.
Meanwhile, CEO Mark Zuckerberg is said to have lost $7 billion of his own net worth due to plummeting stocks all day. According to Bloomberg, the company’s stock sank 4.9 percent, enough for Zuckerberg’s worth to dip to $121.6 billion, making him the fifth richest person in the world behind Bill Gates.
In addition, Facebook’s shares dropped 5 percent in the midst of the chaos. An influx of share sell-offs led to the company’s cost per share to sink to $326.23.
How will Facebook safeguard against this happening in the future?
That remains unclear. Without a direct reason for what led to today’s outage, it isn’t currently known how Facebook will try to prevent a shutdown like this from happening again.