Talk:2021 Facebook outage

UTC times only, please.

It's convention to use UTC time only, when discussing global network events, since they affect everyone simultaneously at different relative local times. So, please be sure to convert any quoted times to UTC for the article. Thanks. Bill Woodcock (talk) 22:45, 4 October 2021 (UTC)

Start time

I think we should use the 15:39 that Doug Madory reported in the Kentik article... Doug is reliable, his data sources are accurate, he reported in UTC, which is what he collects in, so he wouldn't have made any timezone errors. That also coincides with the first reportage on the Outages list, and NANOG wasn't far behind. Bill Woodcock (talk) 01:04, 5 October 2021 (UTC)

That matches up with Cloudflare report as well, as they started seeing an uptick of BGP updates from Facebook just before that time, before peaking at 15.40 UTC. Sideswipe9th (talk) 02:46, 5 October 2021 (UTC)

End time

It was shortly after 0:00 CEST for me when i stumped into a facebookpage (unintentionally) , this is 22:00 UTC . --Itu (talk) 22:58, 4 October 2021 (UTC)

These things take a long time to settle down. I just watched the Facebook main page take four minutes trying to load, and then result in a "Sorry, something went wrong. We're working on it and we'll get it fixed as soon as we can" error. This is a gradual process. We can check the outages list in the morning to see what the consensus is on service restoration time. Bill Woodcock (talk) 23:04, 4 October 2021 (UTC)

FWIW, at 23:13 UTC, I just got the first successful load of a Facebook main page, albeit a stripped-down, mostly-static-content one, rather than the usual dynamically-built one. So my personal observation was 14:40 UTC to 23:13 UTC, or 8:33 elapsed time. Bill Woodcock (talk) 23:14, 4 October 2021 (UTC)

There's a difficulty in relying on personal reports for when the site became available again. WP:PRIMARY and WP:VERIFY being the major two. That said, I know I started receiving Facebook and Whatsapp messages and notifications shortly after 21.50 UTC, which maps on with the restoration of service from the Cloudflare analysis I cited previously. Sideswipe9th (talk) 01:02, 5 October 2021 (UTC)

With this being a DNS, BGP, redundancy, and so on issue availability can be different in different areas. --Kathy262 (talk) 05:29, 5 October 2021 (UTC)

Cause

Cloudflare have just released an analysis on the outage, based on how it impacted their services. It seems to be BGP related, as at 15.40UTC Facebook started announcing a significant number of BGP updates that isolated Facebook's servers from the wider internet. I'm skimming now to figure out how to add this into the article, but could use some help. Sideswipe9th (talk) 21:39, 4 October 2021 (UTC)

I've just added a summary from Cloudflare's writeup of the issue into the causes section. Some of the text might need altering for clarity, but it and the source article have a nice break down of the timeline of the outage. Sideswipe9th (talk) 22:01, 4 October 2021 (UTC)

@Bwoodcock: I was wondering if you could explain this diff please? The statement removed seems to be factually correct, at least as far as I can verify with other WP:RS. I was wondering if perhaps I'd missed something? Sideswipe9th (talk) 22:31, 4 October 2021 (UTC)

@Sideswipe9th: The DNS began to come back online at 21:05 UTC, but the actual application-layer services hadn't yet been restarted an hour later, much less thirteen minutes later. The DNS fixes hadn't even propagated to very many resolvers yet at that point. This is well-established on both the NANOG and Outages lists, from many points of view globally. You can trawl through the archives, or wait for the press to catch up, but we should avoid putting known-incorrect information into the article if we can avoid it. Just makes for more work later. Bill Woodcock (talk) 22:51, 4 October 2021 (UTC)

And Facebook's main page is, in fact, still down now, eight hours in. So we'll need to update the close and duration at whatever point the dust actually settles. Bill Woodcock (talk) 23:01, 4 October 2021 (UTC)

Good news. Facebook have now released their own explainer for what went wrong. It's a little lacking in terms of times in relation to the others, but there seems on first glance to be some good stuff in it. It's almost certainly WP:PRIMARY but hopefully this'll result in a couple of WP:SECONDARY sources we can use. Sideswipe9th (talk) 20:21, 5 October 2021 (UTC)

References

Background information

The current information in the background section regarding a whistleblower and impact of Instagram on girls body image has nothing to do with the outage to Facebook services.

In my opinion, this information is irrelevant and should not be included within this article. SkippyKR (talk) 21:53, 4 October 2021 (UTC)

Agreed - it completely changes the context of the article to one unrelated to the actual event. I've removed the section - please discuss here if you think the content was actually relevant. ──post by kenny2wiki ( Talk | Contribs ) 22:18, 4 October 2021 (UTC)

It's relevant because multiple reliable sources discuss it. cf: BBC News, "The disruption comes the day after an interview with a former Facebook employee who leaked documents about the company. Frances Haugen told CBS news on Sunday that the company had prioritised "growth over safety".", Indianapolis Star "The outage comes after a whistleblower said that the world's largest social network is prioritizing profits over users' safety. Facebook's stock was down more than 5% on Monday afternoon." Ritchie333 ^{(talk) (cont)} 22:20, 4 October 2021 (UTC)

Completely irrelevant. At most, link to the "Facebook Controversies" page. Bill Woodcock (talk) 22:21, 4 October 2021 (UTC)

The reliably sourced commentary of BBC News is preferred in Wikipedia over the random opinion of one editor, per the verifiability policy. Ritchie333 ^{(talk) (cont)} 22:23, 4 October 2021 (UTC)

I think it's tangentially relevant. Many of the reliable sources being cited for the incident, like the BBC News article, do mention this as part of the background. While it's not relevant to the technical details, it is relevant to the discussions surrounding the outage. As such I'd support it being included in the article. Sideswipe9th (talk) 22:25, 4 October 2021 (UTC)

I agree with Ritchie here - it is mentioned by media sources as pretty much an interesting coincidence and we should reflect that in some way. Elli (talk | contribs) 22:42, 4 October 2021 (UTC)

It's a coincidence, nothing more - it's mentioned by media sources as a way to get readers to read more about other vaguely Facebook-related stories, not to actually provide background information related to the event. ──post by kenny2wiki ( Talk | Contribs ) 22:45, 4 October 2021 (UTC)

Where is your evidence for that assertion? Do you honestly think the BBC, one of the most important and well-established news organisations in the last 100 years (and one which is significant for having no advertising) needs to publish clickbait? It has no rationale to. Ritchie333 ^{(talk) (cont)} 22:48, 4 October 2021 (UTC)

Can't prove a negative. There are an infinite number of unrelated topics which could be jammed into the article, but which have nothing to do with the topic of the article, which is the outage. If you want to talk about Facebook controversies, there's a whole separate long-ass article just for that. Bill Woodcock (talk) 22:56, 4 October 2021 (UTC)

Wouldn't the burden of proof to be on a source that is able to substantially correlate the two events? How can you ask for evidence proving something isn't correlated? SkippyKR (talk) 22:53, 4 October 2021 (UTC)

I've just given two above. And it doesn't matter if it correlates them, gives them reasonable doubt, or suspects them .... what matters is we report what reliable sources say. Ritchie333 ^{(talk) (cont)} 22:55, 4 October 2021 (UTC)

Right, but the article is about the outage, it's not a catalog of all the unrelated things that people said. Bill Woodcock (talk) 22:57, 4 October 2021 (UTC)

You keep asserting without evidence that is "unrelated". However, multiple independent reliable sources say that the outage occurred on the day after the whistleblowing (the most recent BBC News piece I added also includes this). We don't say that the outage was caused by the whistleblowing, because that's original research. Ritchie333 ^{(talk) (cont)} 22:59, 4 October 2021 (UTC)

You keep asking for evidence that the matter is irrelevant, when the burden of proof is on the person adding the information to show that it is relevant. So far, you are the only person that worships sources enough to say that their mere mention of it is sufficient evidence for its relevancy. I, for one, remain unconvinced. ──post by kenny2wiki ( Talk | Contribs ) 23:03, 4 October 2021 (UTC)

We don't have to report everything that reliable sources say; only that everything we report must be said by reliable sources. It's a unidirectional relationship. I don't care if the BBC is 100 years old or 1000, they still need to maintain a readerbase to continue to receive the government funding that they do. They don't need views for ad money, but they still need views ultimately. Or from a less cynical perspective, bringing up other Facebook-related events is a way to try to have the reader be more knowledged about them - but they should be mentioned in a "see also", not as direct context for the event. ──post by kenny2wiki ( Talk | Contribs ) 22:59, 4 October 2021 (UTC)

Correct. An infinite number of unrelated events happened on the same day. And continue to happen, as the outage is still ongoing. Temporal overlap does not create a relationship or association, except in the minds of conspiracy theorists. Bill Woodcock (talk) 23:02, 4 October 2021 (UTC)

All that being said, without the background, the article doesn't actually look that notable. Ritchie333 ^{(talk) (cont)} 23:13, 4 October 2021 (UTC)

Oh, it's notable all right. Outages are measured in person-minutes. This one was approximately 1.54 trillion person-minutes, and may have been the largest outage of any service ever. We'll need to wait for the media to dredge up previous examples and compare them, but that's a tera-lapse, which may be a first. Bill Woodcock (talk) 23:19, 4 October 2021 (UTC)

I'm in favor of removing the background section as it stands now because it doesn't actually provide background context. The whistleblower incident is unrelated to the actual cause of the outage and the only connections between the two events are proximity in time and them being tied to the same company. The whistleblower information should, instead, find its place in the main article on Facebook or one of its offshoots. ASpacemanFalls (talk) 09:12, 5 October 2021 (UTC)

It's so odd to me seeing this connection dismissed by the majority of editors here. It needs to be addressed, simply because pretty much every news article you read on the outage mentions Haugen, and vice versa since the outage. At the very least, it should be said that the outage coincided with Haugen's leak, but the two events are not likely to be connected, e.g. here [1]. In any case, it's great to see all the technical information, but this wasn't an exclusively "technical" event. Historical events always take place in social and political contexts. Cameron.coombe (talk) 23:00, 5 October 2021 (UTC)

Rename article to "2021 Internet Outage"

Clearly, as of 5:04 PM EST, it is more than Facebook. Several other websites are down via "DownDetector" GMail DownDetector

As well as numerous other sites: BBC News: Facebook, Whatsapp and Instagram down worldwide NorthCentralPA: Worldwide Internet Outage takes out... everything — Preceding unsigned comment added by Crazyeditor23 (talk • contribs) 21:08, 4 October 2021 (UTC)

Other websites may be down right now - it happens every day. However, there are no reliable sources suggesting downtime on e.g. Gmail is related to the outage affecting Facebook and its subsidiaries at present. Buttons0603 (talk) 21:30, 4 October 2021 (UTC)

Agree. There appears to be no connection between Facebook Inc. products and any other internet outage today. SkippyKR (talk) 22:06, 4 October 2021 (UTC)

This is a Facebook outage. People using other services may be a byproduct of that, but the outage is all Facebook's. No rename. Facebook is not the Internet, it's one dot-com. Bill Woodcock (talk) 22:21, 4 October 2021 (UTC)

Yeah, by and large this was a Facebook outage, caused by a misconfiguration pushed by their network team. There were knock on effects to other services, Discord for example reported higher than expected usage for the time of day which resulted in some performance issues, but the outage was primarily a Facebook and Facebook related services one. Sideswipe9th (talk) 22:28, 4 October 2021 (UTC)

Many sites include links, icons, etc. for Facebook but unwisely use Facebook supplied code for this which is dependent on Facebook being always available. That's their fault for making themselves dependent on Facebook availability. - Philh-591 (talk) 22:38, 4 October 2021 (UTC)

As an obvious example, Wikipedia wasn't affected by it :-) Ritchie333 ^{(talk) (cont)} 22:39, 4 October 2021 (UTC)

References

Curiosity about WP:NOTNEWS and how it applies here

I'm experienced as an editor, but I'm not experienced in content. Can someone explain how this passes WP:NOTNEWS, particularly the part where it says:

"Ensure that Wikipedia articles are not:[...]News reports. Wikipedia considers the enduring notability of persons and events[...]While including information on recent developments is sometimes appropriate, breaking news should not be emphasized or otherwise treated differently from other information. Timely news subjects not suitable for Wikipedia may be suitable for our sister project Wikinews, though that is not a particularly active project."

I dream of horses (Contribs) (Talk) 23:33, 4 October 2021 (UTC) (removed <nowiki> tags at 23:34, 4 October 2021 (UTC))

WP:RAPID wizzito | say hello! 23:37, 4 October 2021 (UTC)

@I dream of horses: see above comment by Bwoodcock in reply to Ritchie - this isn't just a routine outage but one of the largest ever. WP:NEVENT is what's relevant here, and I think this article meets it. Elli (talk | contribs) 00:36, 5 October 2021 (UTC)

Agreed. While it's hard to untangle the effects of this from the whistleblower leaks, Facebook's stock price dropped nearly 5% today, with the CEO's personal net worth dropping by $6.1 billion. The impact of this outage goes beyond the site being unavailable for 6 hours. Sideswipe9th (talk) 00:58, 5 October 2021 (UTC)

A Commons file used on this page or its Wikidata item has been nominated for deletion

The following Wikimedia Commons file used on this page or its Wikidata item has been nominated for deletion:

• Facebook-outage-mcauliffe.png

Participate in the deletion discussion at the nomination page. —Community Tech bot (talk) 00:53, 5 October 2021 (UTC)

Mapillary downtime impacts

With Mapillary down, obviously no one could load the site, but contributors also wouldn't have been able to upload images to the servers. Further, and not mentioned anywhere I could find, I observed that Mapillary downtime also prevented OpenStreetMap editors' access to Mapillary street-level image sequences, which contributors can use to ground-truth various satellite image sets. (OSM editing UI also has checkboxes for Bing Streetside and KartaView image sets.) Doug Grinbergs (talk) 05:46, 5 October 2021 (UTC)

Media coverage

Have there been examples of especially bad media coverage that would only serve to confuse the layperson? Just wondering. For that matter, did any source(s) offer exemplary coverage in easy-to-understand layperson's terms? Doug Grinbergs (talk) 05:46, 5 October 2021 (UTC)

Did you know nomination

The following is an archived discussion of the DYK nomination of the article below. Please do not modify this page. Subsequent comments should be made on the appropriate discussion page (such as this nomination's talk page, the article's talk page or Wikipedia talk:Did you know), unless there is consensus to re-open the discussion at this page. No further edits should be made to this page.

The result was: promoted by Theleekycauldron (talk) 06:08, 23 October 2021 (UTC)

(

• Comment or view
• Article history

)

• ... that an outage on October 4, 2021 caused Facebook and related services such as Instagram and WhatsApp to be inaccessible for around six hours? Source: Reuters

• Reviewed: Template:Did you know nominations/The Song of Iowa
• Comment: Many other authors here who aren't listed. Open to a better hook if anyone has suggestions.

Created by Elli (talk) and Ritchie333 (talk) et al. Nominated by Elli (talk) at 03:16, 6 October 2021 (UTC).

• Comment: It looks to me like Ritchie333 is a significant contributor to the creation of this article, so I've added a credit for them—Elli, let me know if there's anyone else who you think should have a credit. theleekycauldron (talk • contribs) (they/them) 18:33, 8 October 2021 (UTC)

General: Article is new enough and long enough
New enough: Long enough:

Policy: Article is sourced, neutral, and free of copyright problems
Adequate sourcing: Neutral: Free of copyright violations, plagiarism, and close paraphrasing:

Hook: Hook has been verified by provided inline citation
Cited: Interesting:

QPQ: Done.

Overall: Looks good! I'm sure there's potential for more hooks in the article – maybe something about Zuck's wealth falling, or the impact on people who rely on Free Basics – but ALT0 is approved. ezlev (user/tlk/ctrbs) 21:44, 8 October 2021 (UTC)

To T:DYK/P6

Frances Haugen Testimony

Should the fact that the outage was close to the scheduled date of Frances Haugen's testimony before Congress be mentioned in the article as it relates to reactions? Is it possible to mention this as affecting public opinion without sounding like lending credence to any conspiracy theories that some people may have? CessnaMan1989 (talk) 03:01, 7 October 2021 (UTC)

No, I think that's unrelated and off-topic to mention about Frances Haugen's testimony in this article. None of the sources relate Haugen's testimony with the outage. The closest they've gone is,

[...] critics argue that yesterday’s six-hour Facebook outage — unrelated to today’s hearing — showed the downside of one company [...]
— "Facebook whistleblower Frances Haugen testifies before the Senate". Techcrunch. 7 October 2021.

In her prepared remarks, Haugen referenced the outage, using it to illustrate not only Facebook’s flaws but its promise.
— "Outage and whistleblower testimony renew focus on dangers of Facebook's global reach". The Washington Post. 6 October 2021.

"[...] make young girls and women feel bad about their bodies," Haugen said, nodding to a five-hour long global Facebook outage on Monday.
— "Mark Zuckerberg says he's 'proud of everything' Facebook does hours after blockbuster whistleblower testimony". Fortune. 5 October 2021.

Given that, mentioning Haugen's testimony in this article is giving WP:UNDUE weight to those unrelated events which happen to take place within the same timeline. —Wiki Linuz ( 💬 ) 06:32, 7 October 2021 (UTC)

I think it merits a sentence in the "Response" section, given that so many media outlets noted it as an interesting coincidence. Like, it was unrelated, but coverage of the event mentioned it so we probably should too, or we're not accurately representing the coverage. Elli (talk | contribs) 06:58, 9 October 2021 (UTC)

It's tricky. We want to stay neutral, but when an unsubstantiated accusation becomes notable enough, does it merit mention? CessnaMan1989 (talk) 15:35, 10 October 2021 (UTC)

Could you tell me which exact sentence that merits the mention of Haugen's testimony, Elli? CessnaMan1989, I'm still critical of mentioning Haugen's testimony in this article since many reliable media outlets explicitly mentions those two as unrelated. WikiLinuz (talk) 15:44, 10 October 2021 (UTC)

See some of the quotes provided by Ritchie333 in the "Background information" section of this talkpage. Pretty much every media outlet mentioned the two events as coincident, we should make a mention of it too. Elli (talk | contribs) 17:45, 10 October 2021 (UTC)

The utmost we can include is, “Few media outlets highlighted the coincidence of Haugen's testimony with that of the outage, although those two events are unrelated to each other”. I carefully “few” and “unrelated”, because we need to highlight that fact only few outlets mentioned comes the day after an interview while others explicitly denote them as unrelated, like the sources I shared above. WikiLinuz (talk) 19:10, 10 October 2021 (UTC)

That is not the "utmost" we can include but I'd be fine with that. Elli (talk | contribs) 19:12, 10 October 2021 (UTC)

Looking at it further, I think "few" was misleading, so I replaced it with "some" - "few" implies that it was rare, when we don't have a source stating that. Elli (talk | contribs) 19:29, 10 October 2021 (UTC)

Thanks. WikiLinuz (talk) 19:29, 10 October 2021 (UTC)

Whistleblower

No mention that it coincided with the "Whistleblower" being interviewed on national television about the issues with the site? Looks like censorship to me, numerous reliable sources mention it. Even if it was completely unrelated I don't think it would be unreasonable to mention that a number of sources brought it up..† Encyclopædius 09:55, 18 October 2021 (UTC)

Read the article again, it was already included in this revision diff. WikiLinuz (talk) 13:16, 18 October 2021 (UTC)

That's fine then, I would hate to think that Wikipedia is being censored...† Encyclopædius 16:52, 18 October 2021 (UTC)

How was access restored?

It's well documented that the outage knocked out internal comms and things like door passes. However there is still a small gap in the details of the service restoration.

Facebook said:

Our primary and out-of-band network access was down, so we sent engineers onsite to the data centers to have them debug the issue and restart the systems. But this took time, because these facilities are designed with high levels of physical and system security in mind. They’re hard to get into, and once you’re inside, the hardware and routers are designed to be difficult to modify even when you have physical access to them. So it took extra time to activate the secure access protocols needed to get people onsite and able to work on the servers. Only then could we confirm the issue and bring our backbone back online.

This is contrast to rumours that the relevant cage needed to sawed into.

However 'physical barriers' needed to be overcome to access the facility.

This sounds to me like some of the main data center doors were stuck shut (as they were likely facebook-powered), rather than the rack cages.

Working from home was also blamed here.

Does anyone have a RS clarifying this point?

There is presumably a complex series of events whereby relevant engineers contacted one another on a non Facebook/Whatapp service to authenticate one another and identify the team who needed to be dispatched to the data center to do the manual roll back. However I don't believe there is anything public domain about this.

Deku-shrub (talk) 14:13, 2 January 2022 (UTC)