Facebook costliest outage caused $160 million loss: NetBlocks

Facebook insists that a configuration change, most likely done by an engineer, broke the connection on the backbone routers which connect the data centers to the internet

 

 

It wasn’t the small matter of a social networking website going offline for a short period. When Facebook went offline globally for almost 6 hours in the early hours of October 5, it resulted in a mad scramble to find a fix and the cause of the outage. The enormity of the problem was compounded by the fact that the most popular messaging app in the world, WhatsApp, along with Instagram and Messenger, as well as Facebook Workplace too stopped working. Facebook eventually said that a configuration change to its backbone routers was the reason for services to go offline. It wasn’t hacked, the social media giant insisted. Mark Zuckerberg and Facebook have since apologized for the inconvenience caused by the sudden disruption. However, the human or technical error has cost more than just inconvenience to the billions of Facebook app family users, globally.

The cost of this outage, according to cybersecurity watchdog NetBlocks, was to the tune of $160 million to the global economy. The Cost Of Shutdown Tool (COST) estimates the loss caused by an internet disruption using specific indicators including from World Bank and the International Telecommunication Union (ITU). Facebook’s share price tumbled, and data suggests that Mark Zuckerberg, founder and CEO of Facebook, lost as much as $7 billion and Facebook saw $40 billion in market capitalisation wiped out. “Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt,” explained Santosh Janardhan, vice president of Infrastructure at Facebook.

RELATED STORIES

Facebook has more than 2.89 billion monthly active users, of which more than 340 million are in India. WhatsApp has more than 2 billion monthly active users while Facebook’s Messenger app has 1.3 billion users and Instagram has more than 1.3 billion active users. The demography impacted by this configuration issue is massive, and on a global scale. And that is before we count the millions of advertisers and small as well as medium businesses that rely on Facebook’s family of apps for their livelihood. WhatsApp Payments app was unavailable as well. Facebook Workplace, the company’s enterprise communications platform with more than 7 million users, was offline as well. This meant organisations which use Workplace were restricted in internal communications.

Also Read: Windows 11 released; here’s how to upgrade your PC before others

Facebook insists that a configuration change, most likely done by an engineer, broke the connection on the backbone routers which connect the data centers to the internet. There are no further specifics, with the only takeaway being that Facebook’s systems were unable to communicate with each other or the internet. There was no attempted attack on its servers and none of the user data was compromised. While the configurations were being done remotely, Facebook engineers had to rush to one of its main US data centers in California to understand the magnitude of the problem and find a fix. With the outage completely taking down internal company services as well, access was also a challenge.

The Facebook and app family outage also meant services such as Sign In With Facebook that millions use on third party apps and websites, were also impacted. With no way for these services to authenticate your credentials with a Facebook server at the time, users were locked out on more than just the Facebook app or Instagram. All of Facebook apps, including WhatsApp and Instagram, share the infrastructure. If one goes down, everything else will likely be unavailable to users. Facebook has suffered an outage in June as well, though that wasn’t of the same magnitude.