Wikipedia:Community health initiative on English Wikipedia/Research about Administrators' Noticeboard Incidents/Quantitative data analysis

To supplement the AN/I Survey, the Foundation's Support and Safety and Anti-Harassment Tools teams looked at several ways to gain insights on the AN/I process through an analysis of its archives. We attempted to answer several questions:

• What is the relative frequency of reports?
• How many cases are formally closed versus simply archived?
• Who is participating?
• How often do edit conflicts occur?
• How long are cases usually open?
  • Why are some cases open longer than others?
• What are the outcomes for people who report?

Gleaning useful data from the AN/I archives is a difficult task. Reports are not structured, and there are a wide variety of templates, formats, and practices used by participants. We are aware that this is new area of study, and have been refining the queries used to get this data. However, we recognize that this is just a start to understanding AN/I from a quantitative perspective, and welcome suggestions on both how to improve the current approach, and how to use new queries to reveal more insights. Here is the code/queries created by Daylann Maza, an Anti-Harassment Tools team developer, for parsing ANI and making these reports.

Technical documentation related to this research can be found at Dayllan Maza's GitHub repository.

Cases per month

Overall, 56.5% of cases within the sample were resolved using the {{atop}} template, which formally applies a resolution to each case as well as bordering the section in a blue box indicating that the section has been archived. This is a rough indication that the case has been resolved, though it is not required by any policy to invoke this template in order to "close" a case at AN/I.

The most active month for AN/I cases was April 2017, in which 304 cases were opened. However, there doesn't seem to be a significantly "busy" month for cases.

Historically, cases on AN/I are becoming gradually less frequent.

Note: Because the data was collected for a full year, only the first of October 2017 was not included in the sample. They are charted below for consistency.

Cases per month

1 – October 2016 (8.89%) 2 – November 2016 (8.50%) 3 – December 2016 (7.91%) 4 – January 2017 (8.98%) 5 – February 2017 (8.01%) 6 – March 2017 (8.86%) 7 – April 2017 (9.86%) 8 – May 2017 (7.49%) 9 – June 2017 (7.33%) 10 – July 2017 (8.24%) 11 – August 2017 (7.91%) 12 – September 2017 (7.98%) 13 – October 2017 (0.03%)

Of which resolved

1 – October 2016 (57.66%) 2 – November 2016 (54.58%) 3 – December 2016 (52.46%) 4 – January 2017 (53.07%) 5 – February 2017 (62.75%) 6 – March 2017 (59.71%) 7 – April 2017 (62.17%) 8 – May 2017 (54.98%) 9 – June 2017 (49.56%) 10 – July 2017 (64.17%) 11 – August 2017 (54.51%) 12 – September 2017 (49.59%) 13 – October 2017 (100%)

---

Historical cases per month

2012 – 318 2013 – 3,802 2014 – 3,459 2015 – 3,728 2016 – 3,332 2017 – 2,826

---

Statistics per case

Overall statistics

Data was collected from 1 October 2016 to 1 October 2017, inclusive.

Total cases: 3,083

	Case duration	Users involved	Administrators involved	Case length (bytes)	Diffs included
Sum	13.64 years (430,154,689 seconds)	17,526 (3,505 unique)	5,590 (322 unique)	22,453,153	15,625
Mean	1 day, 15 hours, 53 minutes, 44 seconds (143,624.27 seconds)	5.68	1.81	7,282.89	5.07
Maximum	61 days, 21 hours, 6 minutes (5,346,360 seconds)	58	22	265,535	150
Minimum	0	1	0	124	0
Median	14 hours, 2 minutes (50,520 seconds)	4	1	2,647	1
Mode	30 minutes (1,800 seconds)	3	1	1,117	0

Case size

There are a number of ways to work out case "size", whether that is in terms of physical size in bytes or duration in seconds. To work out physical length, signatures were stripped out from the cases and the resulting wikitext was analysed for a final case length in bytes. Since this bytes size includes links and other wikitext, it is very difficult to summarise this length in terms of word count.

The vast majority of cases were fewer than 5,000 bytes in total size, once these signatures had been stripped out. The median size for a case was 2,647 bytes, though the shortest was 124 bytes. The mean case size was 7,282.89 bytes, with the longest case reaching 265,535 bytes (longer than List of vegetarians, currently the 689th-longest article on the English Wikipedia).

Bytes per case

1 – 0–4,999 bytes (68.66%) 2 – 5,000–9,999 (14.76%) 3 – 10,000–14,999 (5.58%) 4 – 15,000–19,999 (3.08%) 5 – 20,000–24,999 (2.01%) 6 – 25,000–29,999 (1.49%) 7 – 30,000–49,999 (1.98%) 8 – 50,000+ (2.43%)

Cases fewer than 5,000 bytes

1 – 0–249 bytes (0.57%) 2 – 250–499 (5.20%) 3 – 500–749 (10.63%) 4 – 750–999 (10.78%) 5 – 1,000–1,249 (10.30%) 6 – 1,250–1,499 (8.32%) 7 – 1,500–1,749 (6.57%) 8 – 1,750–1,999 (6.47%) 9 – 2,000–2,249 (5.62%) 10 – 2,250–2,499 (5.43%) 11 – 2,500–2,749 (4.54%) 12 – 2,750–2,999 (4.11%) 13 – 3,000–3,249 (3.59%) 14 – 3,250–3,499 (3.45%) 15 – 3,500–3,749 (2.84%) 16 – 3,750–3,999 (3.45%) 17 – 4,000–4,249 (2.03%) 18 – 4,250–4,499 (2.32%) 19 – 4,500–4,749 (1.75%) 20 – 4,750–4,999 (2.03%)

---

Long cases analysis

Method

We took a random sample (10 cases) from a generated list of longest AN/I threads based on first and last timestamps. It was analyzed for commonalities and factors that lead to extended discussion and/or slow closure.

Conclusions

The reports in this sample remained open three to five weeks between the first report and final comment. Topic bans and long-term disputes between editors were a recurring focus. Two themes emerged: high participation and low participation.

In the high group, as many as 30–40 editors and admins participated. These threads were hampered by long posts, back and forth interactions between involved editors, conflicting proposals (e.g. topic ban versus a block), and multiple overlapping discussions in separate sections. These cases often have more than three subsections; these sections are sometimes used structurally (e.g. "Topic ban proposal") or are just arbitrary breaks designed for ease of editing and to reduce edit conflicts (e.g. "Convenience break"). Participants are sometimes unsure where and when to !vote on outcomes; as a result, support or opposition for a specific outcome is often mixed in with threaded discussion. This makes determining consensus more difficult.

In the low group, a different pattern emerges. Threads start out with few participants, often only two or three people directly involved in the dispute. Uninvolved editors often do not participate until a concrete action is proposed (if that step does not happen, participation remains low) or if the conversation gets heated. The thread is sometimes archived automatically, and brought back to the board by participants. These threads often suffer from long "wall of text" posts, and are more likely to be closed without a concrete result.

Difficulties in organizing and presenting evidence is a common factor to both groups. These cases involve long histories and multiple articles and talk pages. Approaches vary to presenting evidence; this can range from collections of unlabelled diffs with insufficient context contrasting with large copy/pastes of whole discussions from other pages.

A final common factor is unclear requests for actions. When the initial reports and analyses do not request a specific action, participants struggle to find common ground on what should be !voted on. Administrators often do not participate until they see a clear consensus emerging on an outcome. If that consensus fails to emerge, the threads continue, the amount of text grows, and resolution becomes less likely.

Diffs per case

Diffs per case A "diff" is a link to an edit made to a page. These are generally used as evidence of actions made by users and generally accompany reports made on AN/I, though they also feature in follow-up comments from other users. On average, a case on AN/I contained 5.07, but this is distorted by cases which featured many. The median number for each case was 1, and more than three in five cases did not cite any diffs at all.

1 – 0 diffs (60.99%) 2 – 1 (25.38%) 3 – 2 (11.96%) 4 – 3 (9.61%) 5 – 4 (7.42%) 6 – 5 (6.68%) 7 – 6–9 (15.04%) 8 – 10–14 (8.88%) 9 – 15–19 (5.17%) 10 – 20–29 (4.75%) 11 – 30–59 (3.81%) 12 – 60+ (1.31%)

---

Case participation

In this sample, the total number of users participating in each case was collected. At least one user will always be present in a case, even if this is just the reporter. There were 88 cases where this was true (2.85% of all cases in the sample).

3,505 unique users were involved with cases on AN/I in the sample. In the majority of cases, 3 unique users were involved (this was true in 23.35% of all cases). In the most-populated case, 58 users participated.

In terms of administrators – elected users which possess abilities like deletion and protection, and who in theory should be handling cases on this noticeboard – there were 322 unique participants on AN/I. More than one-third of cases saw participation from 1 administrator. In almost a fifth of cases, there were no administrators present. Peak participation was 22 in one case.

Users per case

1 – 1 user (3.80%) 2 – 2 (14.14%) 3 – 3 (23.35%) 4 – 4 (15.18%) 5 – 5 (9.80%) 6 – 6 (8.11%) 7 – 7 (5.25%) 8 – 8 (4.09%) 9 – 9 (3.86%) 10 – 10–14 (7.23%) 11 – 15–19 (2.66%) 12 – 20–24 (0.94%) 13 – 25–29 (0.65%) 14 – 30+ (0.94%)

Administrators per case

1 – 0 administrators (18.49%) 2 – 1 (37.79%) 3 – 2 (21.60%) 4 – 3 (10.22%) 5 – 4 (4.70%) 6 – 5+ (7.20%)

Account age statistics

	Reporters	Non-admin participants	Admin participants
Total number	1,256	2,413	325
Mean account age	6.29 years	6.32 years	11.34 years
Mean edit count	26,368	23,684	97,005

Edit conflicts

An "edit conflict" occurs when a user attempts to submit a change to a page after another has already saved an edit to the page. For example, if User A edits the page while User B is making their edit, but User A saves the page first, User B will receive notice that the edit was conflicted. This currently requires the edits to be manually untangled in order for User B to be able to save their edit to the page. In a fast-moving editing environment such as AN/I, this is (at least anecdotally) a fairly common occurrence. This query indicates that 7.9% of all edits, or approximately 1 out of 12, are impacted by edit conflicts.

The data below covers the period from November 22, 2017 at 11:02:23 (UTC) to February 27, 2018 at 06:01:59 (UTC).

Edit conflicts

1 – Successful edits (16,514) 2 – Edit conflicts (1,308)

Boomerang

• Total cases: 3,083
• In which reporter blocked: 483
• Blocks which occurred during a case: 65

Blocks during a case

1 – 0–10% of way through case (7.69%) 2 – 10–20% (6.15%) 3 – 20–30% (7.69%) 4 – 30–40% (12.31%) 5 – 40–50% (4.62%) 6 – 50–60% (7.69%) 7 – 60–70% (7.69%) 8 – 70–80% (7.69%) 9 – 80–90% (9.23%) 10 – 90–100% (29.23%)

Blocks following last signature

1 – < 1 day after last signature (7.87%) 2 – 1–2 days (2.28%) 3 – 2–4 days (3.11%) 4 – 4–7 days (4.35%) 5 – 1–2 weeks (5.18%) 6 – 2–4 weeks (5.80%) In 280 cases (57.97%), the author was blocked more than 4 weeks after the case concluded.