Open main menu
Researchers don't get their own way with the Wikipedia community

Lab rats revolt: Researchers don't get their way with the Wikipedia community

Press Barnstar Hires.png

A proposed research project which would have randomly awarded barnstars to Wikipedia editors was recently withdrawn by researchers at Carnegie Mellon University (CMU). Bending to concerns expressed by en.Wikipedians that the process was a social experiment, Ph.D. student Diyi Yang and Robert E. Kraut, Ph.D, Herbert A. Simon Professor of Human-Computer Interaction at Language Technologies Institute, CMU, withdrew their proposal. Initially approved by the institutional review board (IRB) at CMU, the proposed research entitled How role-specific rewards influence Wikipedia editors' contribution would have involved placing thousands of randomly assigned barnstars on unsuspecting editors' user pages in order to monitor their reactions.

Yang's research is supported by a Facebook Fellowship. Facebook's own research has been criticized in an article in The Guardian by Sam Levin on 1 May 2017 over research in which it sought to alter the emotions of users without their consent, and again by George Monbiot in his opinion piece in the same newspaper on 31 December 2018, stating that "universities are leading us into temptation, when they should be enlightening us". The CMU proposal came under fire at Meta from several leading Wikipedians including BrownHairedGirl, Deryck Chan, Risker, SlimVirgin, and WereSpielChequers when the discussion at Meta spilled over to the Wikipedia Village Pump in a long and heated thread.

Words used by Wikipedia editors to describe the project included:

"...Barnstars awarded among Wikipedia editors and the WikiLove messages I give and receive actually mean something. To use the Barnstars (and potentially the WikiLove system) in the researchers' proposed way devalues their meaning..." – Shearonink (diff)

"Diyiy, can you reply, please, to the part of SarahSV's question where she asks "in whose interests it's being done?" For my part, I want to know why Carnegie Mellon wants to know about Wikipedian behaviour. What benefits accrue to the university? And is the experiment to be of benefit to any of the great manipulators of public behaviour such as Facebook, Google, Twitter, or anyone who desires to sharpen their sophisticated tools even further? Does the university have corporate, government, academic, or other partners who seek to benefit from barnstar-motivation studies? Are you, yourself, a ripe candidate for recruitment by Facebook or similar, based on your current social experiment activity, or arising out of your Facebook fellowship? I am seeking full transparency about any hidden partners or researcher motivations. Cui bono? Thank you." — O'Dea

Halfaker, Aaron Sept 2013.jpg
Aaron Halfaker Photo: Myleen Hollero

In a 455-page paper partly funded by Google, Who Did What: Editor Role Identification in Wikipedia, delivered at the Tenth International AAAI Conference on Web and Social Media (ICWSM 2016), Aaron Halfaker (currently WMF Principal Research Scientist) in his capacity as WMF staff collaborated with CMU researchers Diyi Yang, Robert Kraut, and Eduard Hovy. From the abstract: "Understanding the social roles played by contributors to on-line communities can facilitate the process of task routing. In this work, we develop new techniques to find roles in Wikipedia based on editors' low-level edit types and investigate how work contributed by people from different roles affect the article quality."

"Diyiy and I should have been more precise when saying 'the proposed work has nothing to do with Facebook' and 'Facebook won't benefit at all from the research we've been describing'. We should have said that Facebook does not benefit directly from our research and does not benefit more from this knowledge than do other online platforms. We started this research on the influence of social roles in Wikipedia in collaboration with the WMF and our first paper[1] on the topic was published in 2016 before Diyiy received a Facebook fellowship. The proposed research should lead to generalizable knowledge about the consequences of bestowing recognition and the influence of social roles in online groups. This generalizable knowledge could be useful to many different types of online groups, including Wikipedia, open-source software development communities, online health support groups, peer-to-peer lending groups and many others, including Facebook's online groups."– Robert Kraut

"Every single barnstar I have came as the result of significant effort on my part. I don't understand why the researchers have decided to grant what is, essentially, one of the highest interpersonal symbols of respect on the project to people who have not made the level of contribution that the rest of the community would expect to see when a barnstar is granted. It's like throwing a parade in recognition of successfully emptying the trash baskets, very disproportionate."– Risker

"Sorry but I'm not happy about this. Please see "Wikipedia is not a laboratory". The proposal could be regarded as somewhat "disruptive to the community" in diluting the value of the barnstar, which we would hope is intended as a sincere expression of appreciation from one Wikipedia editor to another. [...] Wikipedia editors are not lab rats and should not be fed barnstars to see if they scurry round any faster afterwards! Feel free to disregard this if other contributors don't see it this way." – Noyster

Winding the clock back...

Seven years ago in April at ANI an attempt by Boing! said Zebedee to retain the dignity attached to the barnstar philosophy, by restricting its rampant willy-nilly use by IP users, a discussion on 'IP handing out random barnstars' was closed with: "Barnstar campaign and other forms of appreciation are not, other than exceptional cases, problematic or disruptive or actionable. This was not the droid you were looking for."

"If the barnstars are to have any meaning, it's probably wrong. However, the guidelines on when to hand out a barnstar are pretty liberal. I suppose you could request a change in who is allowed to give barnstars maybe. Beyond that, though it seems a tad excessive, it's not really uncivil or disruptive. – Avanu"

In April 2012 almost exactly 12 months later Softlavender filed a further ANI report on IP Barnstar spaming: 'I'm all for barnstars, but their value and purpose is diluted (could even say desecrated) when meaninglessly sprayed shotgun by a constantly changing and anonymous IP range for no good reason.'

"...there is far worse vandalism than this, and many more people should be praised for the work they do, but this is just random and devalues well-deserved recognition. The IP editor clearly knows how to edit, and the right sort of phrases etc. to use, so they are not a novice, and could make useful contributions." –Arjayay

The case was closed with: 'While some find random (and inappropriate) acts of Love annoying, no consensus exists for mass action at ANI and cases can be handled one at a time. Changing policy on barnstars is clearly outside of the scope of ANI...'

The phantom barnstar bomber

The wild Barnstarist turns out in both cases to be none other than Mike Restivo editing while logged out in the pursuit of an early research agenda covered in The Signpost column 'Recent Research' from the issue of 30 April 2012. His works are cited by Halfaker et al:

  1. Restivo, Michael, and Arnout van de Rijt. "No praise without effort: experimental evidence on how rewards affect Wikipedia's contributor community." Information, Communication & Society 17, no. 4 (2014): 451-462.
  2. Restivo, Michael, and Arnout Van De Rijt. "Experimental study of informal rewards in peer production." PloS one 7, no. 3 (2012): e34358.