Wikipedia:BMJ/Expert review

This is a planning page for a collaboration with BMJ to help improve Wikipedia's medical content by involving medical experts. WMF helped initiate contact with BMJ leadership and Wiki Project Med Foundation is leading the process.

Timeline

• The idea to ask Dr Godlee and the BMJ for advice or help in establishing an independent expert review process for our best medical articles was first raised by Anthony Cole with Lila Tretikov in a meeting at Wikimania 2014, and they discussed it in more detail in a November 2014 meeting at WMF's San Francisco office, where Lila confirmed her strong support for the initiative. After a discussion among the Wiki Project Med Foundation's trustees was supportiive, Lila, Anthony and Jacob de Wolff (Chair of WPMEDF) sent the invitation; and an exploratory phone conference was held in June 2015 between WMF, WPMEDF and BMJ. (Minutes)

20 November 2015 update

• Anthony announced BMJ's offer to English WikiProject Medicine on 15 September 2015. [1]. Two articles were proposed for review: Parkinsons disease and Endometrial cancer.
• On 10 October, BMJ chose Parkinsons disease to start with.
• Last Friday, 13th November, they said they have their reviewers and are ready to roll:

• Andrew Lees
• David Burn
• Mark Stacy
• Anthony Lang
• Mark kuijf

• This week, we're sending a few newbie guinea pigs through the online reviewer tutorial (Wikipedia:BMJ/Reviewer tutorial) just to make sure it's clear and simple enough.
• We expect the review to start in the next week or two.

16 March 2016

It's taking ages. You can see the ongoing review here.

Background

• Wikipedia has already experimented with peer reviewed versions of its content, for example the Dengue Fever article links to a formally peer reviewed version published by the journal Open Medicine, and between 2012 and 2014 seven articles were peer reviewed and published in the journal, PLoS Computational Biology.
• Wikipedia has an existing relationship with BMJ: BMJ gives our top medical editors free access to its journals through The Wikipedia Library
• Wikipedia has previously organized a Wikipedian-in-Residence position with medical organizations:

  Cochrane, in which we helped find talented candidates whom Cochrane finally selected and paid

  Cancer Research UK, in which Wikimedia UK advised & helped select the WiR, who was mainly funded by Wellcome Trust

  Consumer Reports Choosing Wisely Campaign, in which from 2012-present user:bluerasberry has encouraged the integration of health information provided by these societies into Wikipedia.

Motivation

• Wikipedia's 30,000 medical articles are some of the most viewed, if not the most viewed, medical content online
• Patients, medical students, and even clinical professionals (doctors!) use and rely on Wikipedia
• Wikipedia's collaborative, crowd-sourced model introduces the possibility of inaccuracy, incompleteness, or outdatedness
• Peer review is an established process to produce stable versions of published knowledge

Why BMJ?

• BMJ has 175 years experience in publishing peer reviewed literature, high public recognition and a deserved reputation for rigor
• Editor in Chief Dr. Godlee has built her reputation on her commitment to reliability in medical information and has written extensively on the topic of peer review
• With their wide stable of journals, BMJ has a comprehensive list of reviewers
• BMJ's mission aligns with ours: "to lead the debate on health and to engage, inform, and stimulate doctors, researchers, and other health professionals in ways that will improve outcomes for patients"
• BMJ's values align with ours: "Knowledge for healthcare professionals and patients should be independent and unbiased; Evidence matters; Being transparent and open creates trust;"

Expert Review

Wikipedia's model is based on anonymous authorship, and while this produces ample content, it lacks the rigor of peer review. For Wikipedia, an expert review process would probably involve named reviewers, who would take on accountability for the article.

Reviewers might need to be given incentive for contributing their expertise. Compensation may be of several kinds: (i) financial payment (ii) academic credit (iii) public visibility. Due to perceived conflict of interest, any source for funding would have to be neutral and independent of the WMF.

We hope that an invitation to review a Wikipedia article becomes a credible and very public acknowledgement that one has great expertise in their field.

Peer review could involve:

• Detailed notes and suggestions left on the Wikipedia article's talk pages
• A prominent link to the reviewed version at the top of the Wikipedia article
• A link to a simple diff between the expert-reviewed version and the current editable version, so readers can easily see how the article/topic has evolved since the review
• A link to a published version of the peer reviewed article hosted on BMJ's website

Questions

• Would the reviewers be paid? If so, by whom?

Not paid

• Would the reviewers edit or only make suggestions and approve versions?

Not edit, only post a written review to the talk page

• Would the reviewer's names be mentioned on Wikipedia?

Yes, on the talk page where the review is linked and posted

• Which articles would be selected by which reviewers?

WikiProject Medicine would select up to 10 Featured quality articles for review, and BMJ would select 1-5 reviewers (Noting that BMJ is a leader in rheumatic diseases, gastroenterology, and sports medicine in particular)

• What aspects would the editor review and by what standards?

Accuracy, completeness, bias, and currency (up-to-date) [within the context of what's appropriate for an encyclopedia]

• What is the timeline?

We would have reviews posted when ready between 6-12 months

• Would Wikipedia make changes based on the reviews? What if the consensus of medical editors disagrees with a reviewer's version?

The review would be subject to the consensus like any editor's opinion. Ideally reviews would include supporting citations. Reviewers may engage in community discussion to help form consensus.

• Would the organizing journal, review, reviewed version, or reviewer's name me linked/mentioned on the main article previewed?

Yes, in the spirit of transparency, accountability, and visibility

• Would articles later be re-reviewed?

Depending on this pilot and the reviewer, hopefully, but the pointers to a particular review would refer to a specific version of Wikipedia

WIR

A Wikipedian in Residence (WIR) helps an organization better develop their capacity to contribute to Wikipedia. By liaising, teaching, and project managing, WIRs build a pipeline of content and human connections between the institution and the encyclopedia.

Role

A Wikipedia BMJ WIR could/would:

• Give talks and webinars to BMJ staff, reviewers, or authors about Wikipedia and its role in the public health information ecosystem
• Offer Wikipedia trainings and editing lessons to individuals, teams, or large groups
• Help surface recent and relevant publications by BMJ staff, reviewers, or authors on Wikipedia's medical article talk pages
• Counsel BMJ staff, reviewers, or authors on how to contribute directly to Wikipedia articles in line with our conflict of interest guideline and Terms of Use
• Help BMJ staff, reviewers, or authors leave critical feedback on Wikipedia articles in their area of expertise offering suggestions
• Help with a pilot formal peer review process led by BMJ to create vetted versions of some of Wikipedia's top medical articles
• Work with BMJ to improve citations
• Support the authoring of articles about medical publications

Selection process

A team from WMF, Wiki Project Med Foundation, and BMJ would work together to find an ideal candidate.

1. All 3 would collaboratively craft a position description and role qualifications
2. WMF and Wiki Project Med would promote the position through Wikipedia's community channels
3. BMJ would generate press around the open call for applicants
4. A panel comprised of at least 1 person from each group would help vet applicants and create a shortlist
5. A panel would conduct interviews of top candidates
6. BMJ would have the final decision about whom to hire

Projected timeline

None. Timing is completely subordinate to getting it right.

Who's involved?

BMJ

• Josie Breen
• Peter Ashman
• Fiona Godlee

Wiki Project Med Foundation

• Anthony Cole
• Jacob De Wolff

Wikimedia Foundation

• Jake Orlowitz
• Lila Tretikov

Wikimedia UK