Facebook, Insta, Whatsapp down in major outage

[Capricorn]

What's our network gurus think happened here?what does it say about how their services are arranged?

Seems the outage cascaded globally starting around 5pm

[Frosty]

I definitely won’t be losing any sleep

[Jurgens Smit]

No idea, our peering is still up. So BGP sessions have not dropped.
Assuming it's some internal routing issue or maybe equipment failure.
Could be the servers hosting the platforms that fell over.

Or maybe someone messed coffee over their keyboard and when wiping it down hit the reboot button by accident????

[Rocket-Boy]

All of them are owned by Facebook so Im pretty sure they all use the same CDN.

I know Whatsapp is down because my wife asked a few minutes ago if the internet was down while trying to use it.

 

Seems its down to MTN and Afrihost fibre. The standard down detectors seem to think its down too so this is a pretty big outage.

[fanievb]

The comments on this tweet and the one from Instagram makes for some hilarious reading. All the "creators" loosing their **** about not being able to "influence" the masses 

[Jurgens Smit]

I no longer use facebook, but somewhat enjoy instagram.

Been pretty eyeopening now that its been down for approx 2 hrs how easily one becomes "reliant" on social media. And I don't even post anything, just enjoy looking at cool ****.

[Vetplant]

 

 

[Jurgens Smit]

For the ones interested.
It was a network config change that broke their DNS. And, they got locked out of their remote access.

So they ended up having to go to their DC to physically restart the servers and fix the config change.

[100Tours]

From last night obviously. still funny.

 

Apparently Facebook is on the same BGP as the system that runs employee access cards for Facebook. lol. As a rough translation of the above, the security guard is locked inside the datacentre. The sysadmin is right outside but his keycard doesn't work, and the engineer lives in Mexico.. 

Edited October 5, 2021 by 100Tours

['Dale]

Not surprising and still shocking 

 

 

[Delgado]

22 hours ago, 'Dale said:

Not surprising and still shocking 

Interesting that she has the same lawyers as the famous Ukranian whistleblower that led to the Trump impeachment (same whistleblower who turned out to be VP Bidens top advisor on the Ukraine! Pure theatrics for "tighter tech laws".

 

 

[Jurgens Smit]

On 10/5/2021 at 6:23 PM, 100Tours said:

From last night obviously. still funny.

 

Apparently Facebook is on the same BGP as the system that runs employee access cards for Facebook. lol. As a rough translation of the above, the security guard is locked inside the datacentre. The sysadmin is right outside but his keycard doesn't work, and the engineer lives in Mexico.. 

Not to nitpick, but BGP is a routing protocol, their issue had something to do with DNS going down due to a disconnect between their datacentres.
pretty decent explanation here - https://www.theguardian.com/technology/2021/oct/05/what-caused-facebook-whatsapp-instagram-outage

If you want a easy to understand analogy, imagine that DNS is a post office, it is able to identify a persons name with a street address. BGP would be the delivery driver getting your package to the destination.

In realworld, DNS resolves hostnames to IP addresses, so when you type in google.com into your browser, DNS resolves the hostname to an ip address. Your ISP will then use BGP routes to see where it is learning the route from and steer the packet accordingly.

[100Tours]

8 hours ago, Jurgens Smit said:

Not to nitpick, but BGP is a routing protocol, their issue had something to do with DNS going down due to a disconnect between their datacentres.
pretty decent explanation here - https://www.theguardian.com/technology/2021/oct/05/what-caused-facebook-whatsapp-instagram-outage

If you want a easy to understand analogy, imagine that DNS is a post office, it is able to identify a persons name with a street address. BGP would be the delivery driver getting your package to the destination.

In realworld, DNS resolves hostnames to IP addresses, so when you type in google.com into your browser, DNS resolves the hostname to an ip address. Your ISP will then use BGP routes to see where it is learning the route from and steer the packet accordingly.

Yah I know the difference - we just read different versions of the news.

What took Facebook down | ZDNet

Why Facebook went down, and what's BGP routing | CyberNews

More details about the October 4 outage - Facebook Engineering (fb.com)