Open main menu

Wikipedia:Wikipedia Signpost/2017-02-27/Special report

Peer review – a history and call for reviewers: English Wikipedia's peer review process is a relatively unknown yet fairly active venue for feedback. Editors can request a review of an article, regardless of its stage of development, in general form or highlighting specific areas they are looking to improve. One or more editors may weigh in. Here, I outline the history of peer review and my contributions to it, and detail future plans and a request for more reviewers.
Tom (LT910001) has been a Wikipedia editor since 2013. He is an active anatomy editor who also dabbles in good article reviewing, template discussions, and occasional edits to medical articles. He discovered peer review in 2013 and has become very active in the process since.

English Wikipedia's peer review process is a relatively unknown yet fairly active venue for feedback. Editors can request a review of an article, regardless of its stage of development, in general form or highlighting specific areas they are looking to improve. One or more editors may weigh in.

Here, I outline the history of peer review and my contributions to it, and detail future plans and a request for more reviewers.

Introduction

Peer review runs similarly to other processes such as good articles and featured articles: an editor may post a request, and one or more editors answer that request, critiquing the article. Editors who submit their work for review range from novices with their first article to those seeking feedback before nomination to the good article or featured article forums.

The peer-review venue was created in 2003 by Wapcaplet (now inactive); the first article reviewed was States of the United States. The process quickly gained popularity. Reviews were manually added to a list, until a 2005 redesign based on transclusion. Editors had to manually add and remove articles until 2008, when the processes for adding articles to the list and for closing old reviews were automated by VeblenBot (inactive), created by CBM (inactive), and reviews were split into categories.

Processes and procedures

After creation, peer review rapidly developed a set of instructions and advice. A "policy" page formalising review closure criteria had developed by 2005, created first by Bishonen. Ta bu shi da yu (inactive) began to archive old reviews from 2005. A list of volunteers by topic area was created in 2007 by Marskell (inactive). From 2008, Ruhrfisch began a backlog of unanswered reviews. In 2012 an "alert" template cautioning editors from directly posting reviews on the main peer review page was created by Ruhrfisch.

I first encountered peer review in 2013. In comparison to processes such as "good articles" and "featured articles" I found it messy and disorganised. These had developed friendlier interfaces, clearer instructions, had split into several pages for ease of use and load times (a crippling issue for me, using wireless internet in a rural area at the time), had a greater degree of automation, and more active users. These had changed with time; peer review appeared mostly unaltered for many years. Three things struck me:

  1. Repetitive, mind-numbing processes existed which could be eased or automated
  2. Processes had been intermittently stalled because of a bus factor of 1 (Emphasised by the poetic interludes of Ruhrfisch, styled on Geometry guy (inactive): "Still getting darker in the north, lighter in the south, such is the time of year" [1] [2] [3]
  3. There was significant duplication among pages. This made it confusing to work out the "definitive" version of any set of instructions and also made it harder to improve and discuss change.

Finding the actual peer reviews very useful, I have since striven to gradually improve the process, with discussion and consensus. My goal has been to make it easier to use, and therefore more widely used, as part of Wikipedia's growing infrastructure for supporting new editors, and another way of helping users non-confrontationally gather feedback about their articles.

I started by creating a tabbed header of commonly used pages, similar to good and featured articles; and by branding the peer review pages with a colour scheme. I created a sidebar with other main pages (identified via "what links here" and also "subpages") and put links to as many relevant templates into the "tools" subpage, where I tried to document the byzantine template- and bot-based technical underpinnings of the peer review process.

A sample:

The Category Format (CF) templates are a suite of templates to be used in conjunction with automatic category listing. An automatically listed category is a category which has an associated category list page where the items in the category are listed ... for maximum flexibility, the entries on the category list page are formatted using a subtemplate of CF named after the category. For instance each page in an automatically listed Category:foo has a corresponding entry.

I have centralised information, truncated text, simplified instructions, and made small changes to ease reviewing (such as including the date of creation and last edit on all displayed reviews; slipping in a request for reciprocity in instructions). I have simplified the technical side of things for future editors (this has included creating simpler and comprehensive documentation; centralising the 11 identical templates; simplifying the archive process.) We now link to categories of monthly reviews; previously, an editor had to copy and parse reviews each month, a time-consuming process that understandably was often months behind.

Problems

Reviews would not transclude to the main page during mid-2015, due to a technical cap on the amount of text that can be transcluded. When this is exceeded, nothing transcludes and, with great foresight, peer review receives a message from VeblenBot. With too many reviews (and delays in automatic closure), this limit had been exceeded. We quickly implemented a fix – reviews of a certain length now display their text on the main page. After this limit, a link is provided to the main review. I hope in the future that we continue to have this problem, because it indicates that the process is being used more and more.

We've had long-running issues with on-again–off-again bots over the last two to three years, which has impacted the displaying of new reviews, and the closing of old reviews. I hope this has been finally solved by AnomieBOT (thank you!!). The bot was designed in response to requests at the technical village pump and bot requests, and even a failed attempt I made to learn bot coding. This again illustrates the frustrating aspect of a small bus factor, particularly for such a critical bot as VeblenBot.

The "unanswered reviews" list was automated in 2016. This replaced a "backlog" template that was infrequently edited, with months often lapsing between edits due to the complexity and time-consuming nature of the task. An editor has to manually scan the list of reviews (daily if the template were to be kept up to date) and add reviews older than a certain duration to the template; and check all reviews on the template to see if they had been reviews. This required a lot of effort and months often passed without reviews. The current automatic method is imperfect and not entirely satisfactory, but at least provides a more up-to-date list. It displays only reviews that have never been edited. I should mention and thank on behalf of the community Cirt (inactive) and Brianboulton (active), who maintained the backlog list for some time.

Call for reviewers

Peer review is busier and busier than ever, and is attracting more and more reviews. This is great news and I'm pleased to see the diverse nature of requests posted. But we're still plagued by a lack of reviewers; there's nothing special needed to become a reviewer, and frankly not that much time needed for a review.

Most editors, in my experience, are seeking some general tips about the articles they are editing – general help about sourcing, structure, titles, images and so forth. Interested editors have the potential to help ease newer editors' introduction to Wikipedia – this may appeal particularly to those civic-minded people who help out at the teahouse.

Editors seeking review often do so for no other reason than a desire to improve articles and better write articles in the future. By helping these editors, who are often new, you can make them feel welcome, and make a positive contribution to their future edits and to the encyclopedia as a whole. I hope I've given you an idea of the evolving nature of peer review, and that you'll consider participating! Please drop by and have a look at peer review every so often, and pick up one or two reviews.