Wikipedia talk:Scientific peer review/Archive 1

Archive of Wikipedia talk:Scientific peer review

Verifying credentials

It seems prudent to verify that people have the credentials they claim. Trust but verify. I don't think people should have to disclose their real identity to everyone if they don't want, but they should be willing to disclose it to one or two people that can maintain privacy if desired. Otherwise people should just place links to their personal webpages, or whatever would verify their credentials. - Taxman ^Talk 18:24, 10 March 2006 (UTC)

I'm not sure this is necessary. If the individual has a good record in the Wikipedia science community then I say we trust their claims. --Oldak Quill 18:32, 10 March 2006 (UTC)

Right, my vote to support would be based more on the WP history of a candidate than anything else. Awolf002 18:37, 10 March 2006 (UTC)

It is good that this concern was raised. I agree with Oldak and Awolf002 that a good Wikipedia record is sufficient, with special attention given to provision of published references. I hope my self-nomination hasn't been the cause of this query, but if it is, you can find a link to my website on my user page; my email address is also enabled.

Samsara (talk • contribs) 19:18, 10 March 2006 (UTC)

I quite believe you are who you say, and so would all other Wikipedians. In fact I would have explicitly supported you if I could immediately see how the best way to do it would be. But that's not the people we need to convince. If we want to be reliable, then domonstrable credentials of reviewers is key, and one person able to do that but still keep them private if desired would be a good thing. We need to be able to hold it up to outside critics, not just ourselves. - Taxman ^Talk 15:07, 11 March 2006 (UTC)

At a later date, as we push further to 1.0, we may need to review articles and acertain that the reviewers are qualified. But if/when that happends, they'd probably be outside experts anyway. I see this as a great stepping stone between that and the regular peer review where anyone, regardless of qualification, can review the articles. I think it's a good system to allow comments from users both on and off the board, but to separate them.--HereToHelp (talk • contribs) 15:15, 11 March 2006 (UTC)

I agree with Taxman. We need to find a few people who are willing to review the publication record of the referees. This is important, because it is easy for people with a scientific background like me to edit wiki science pages on subjects they are not an experts in. E.g. I've never started or edited a physics page on a similar topic I'm writing my Ph.D. thesis about. So, there is no way to tell from my wiki record what my expertise really is.Count Iblis 14:31, 13 March 2006 (UTC)

Well, I'll have to withdraw my nomination as I've only published in one of the fields that I feel qualified to talk about. - Samsara (talk • contribs) 15:56, 13 March 2006 (UTC)

A good scientific wiki article which shows your ability to write is all what is needed for me. As member of the board you do not do the job on your own. But you should have the ability to chose the right reviewers. The reviewer has to be the expert not the board member!So be a canidate for the board!--Stone 17:31, 13 March 2006 (UTC)

Support for what Stone said! What is needed most is a good grasp on what a WP science article should look like, and the ability to find and "groom" reviewers that can give good advice how to push a specific article to this level. Awolf002 19:30, 13 March 2006 (UTC)

Okay, my nomination stays. Best wishes, Samsara (talk • contribs) 10:13, 14 March 2006 (UTC)

Great idea!

I just wanted to express my support for this proposal, which I think is needed. All too often new proposals are met with a deluge of criticism from a handful of people, and I wanted to first get in my thanks and support for a great idea! I wish I could offer to help, but my commitments to things like WP:Chem and WP:1.0 already take up a lot of my time. Good luck! Walkerma 05:36, 11 March 2006 (UTC)

Let me add my support to that of Walkerma, another chemist, although I have to say that I need to think about this a bit more and it needs more time. In fact it needs much more time, so I am going to be bold and extend the deadline for nominations. The 17th of this month is far too soon. I would like to discuss this on the WP:Chem pages, and I hold up nominating myself until that discussion has taken place. --Bduke 06:28, 11 March 2006 (UTC)

I think the end of the month is too long. The project will lose momentum and people will drop out. How about 2 weeks? --Oldak Quill 07:08, 11 March 2006 (UTC)

I do not think that momentum will be lost. This idea is so important that I think we need time for momentum to gather. I am more convinced of this now than I was when I did the edit because I looked at the history and saw that you were the only person who has written this proposal. I was assuming there was a group of you. We need to build that group. I have asked the WP:Chem folks on their talk page and have already got one comment. Also you are trying to attract academics to this idea and they often want to mull over an idea for a while before committing themselves. --Bduke 07:16, 11 March 2006 (UTC)

Yes, perhaps you are right. We could stage a sort of PR campaign in the buildup before the 1 April. Get people talking about the factual validity of Wikipedia, our need for verification mechanisms, etc. Logically this longer nomination period will result in more nominees and wider voting by the Wikipedia population. --Oldak Quill 07:25, 11 March 2006 (UTC)

If we get this off the ground with lots of support and good PR, work on this project will be much easier. So, going a bit slower might benefit in the long run. Awolf002 13:22, 11 March 2006 (UTC)

I also think it's better to take more time to build up the support. I know there are more than a handfull of scientists on WP that would be interested in this project. Karol 15:05, 11 March 2006 (UTC)

I see some good arguments in both of those replies. How about three weeks?--HereToHelp (talk • contribs) 15:15, 11 March 2006 (UTC)

Sounds good to me. Awolf002 16:40, 11 March 2006 (UTC)

Three weeks is essentially what I changed it to - i.e. to 31 March. --Bduke 21:23, 11 March 2006 (UTC)

Target fields

The project page states: Preferably, the board will cover a number of scientific disciplines such that any scientific article will get an appropriately in-depth grilling. I wonder about the final range of disciplines to be included in "science" here. Are we talking only about natural science (physics, chemistry, biology, etc.), or social sciences (psychology, sociology, etc.) also? What about mathematics or computer science, are they also to be treated by this SPR? The question of scope must come up sooner or later, and could be answered in practice. It would be nice, however, if the board we're supposed to nominate and vote on reflected the range of fields to be covered, which requires we know aforehand the target fields.Karol 15:32, 11 March 2006 (UTC)

I had assumed that this would entail areas fitting under the 'Science and Mathematics' umbrella. Similar to the portal structure we have. Would that make sense? Awolf002 17:17, 11 March 2006 (UTC)

I can't say I had the social sciences in mind. The academics of the natural and social sciences differ greatly.--Oldak Quill 17:32, 11 March 2006 (UTC)

In principle I agree, but I think we should raise the issue. Karol 17:41, 11 March 2006 (UTC)

Yes, let's get scope sorted out sooner rather than later. Obviously, all the major fields (physics, biology, astronomy, etc.) are in. Also, everything that is not recognized widely by the scientific community is out. That line, however, can get blurred. That said, the social sciences are indeed sciences, they just work differently. Furthermore, while a panel trained in the "hard" sciences may be able to rate broad articles such as psychology, they would not fare well rating some obscure, rare disorder. This project is still young, and we could make some major changes that would not be available options later on. What about two tranches (a word I borrowed from ArbCom): those for the natural sciences and those for the social ones. But, to use the KISS principle, we could just srap the tranche idea and have a diverse board that covers all of the sciences. We'd encourage those who know less about a topic to say so in their evaluation.--HereToHelp (talk • contribs) 18:29, 11 March 2006 (UTC)

The main problem being that everyone so far involved in the creation of this project and its discussion have been from the natural sciences. I don't know how social science peer review works and I don't think a single board with 12 people could cover both sufficiently. I say we cover the natural and let the social sciences create a board (or seperate boards) as they determine. --Oldak Quill 08:50, 12 March 2006 (UTC)

The board of 12 is big if you get Leonardo Davinci, Beniamin Franklin and Albert Einstein, but if you get a average phd student you have to have at least two from each branche of science to get a balanced few on a topic. And if social sciences is also covered the 12 will be more likely to be 12x12. I will not qualify for the 12 as a chemist but for a larger group I will try to get on board of the project.--Stone 12:37, 12 March 2006 (UTC)

True, but we're going to have specialists. See #Multiple boards.--HereToHelp (talk • contribs) 12:40, 12 March 2006 (UTC)

With respect to psychology, I think many disciplines can probably be comfortably encompassed. For example, cognition, sensation & perception and the like should be fairly stable, and verifiable on facts. Even social psychology should be broadly fine. However, clinical psychology/psychiatry may pose greater problems (autism, adhd anyone?), along with some parts of clinical medicine. Some topics in evolutionary psychology may also pose problems. --Limegreen 08:47, 13 March 2006 (UTC)

Multiple boards

I humbly suggest that WP may need a heirarchy of multiple peer-review boards, associated with each of the major wikiprojects (for example, Wikipedia:WikiProject Mathematics and Wikipedia:WikiProject Physics). I believe that this will become necessary, as no single board will have the technical depth and breadth to review all science articles. As an example, both of the above wikiprojects are populated with many dozens of PhD's who are more active, and have a deeper WP editing experience than the current (self-) nominations for this board. In particular, over time, I have come to know those editors well, and to know their quirks, and when to trust their opinions above others. I suspect that the other wikiprojects, e.g. the projects for biology, may have similar concerns: that the science board be staffed with editors with whom they are familiar, and whom they can trust. linas 19:32, 11 March 2006 (UTC)

If each project gets its own board, why even have this as a separate project? Just leave the peer review to them! No, I suggest that we stick with the idea of one board (after thinking it over, I'm going to retract my idea about two tranches). What we can do is send out messages to the talk pages of the major science Wikiprojects and invite them to send their best. The board is going to be twelve people (at this point in time), so let's make sure that, instead of winding up with four, we get twelve—and the best twelve. We should also ask people to say which science Wikiprojects they are active in.--HereToHelp (talk • contribs) 19:40, 11 March 2006 (UTC)

With four major fields covered by WikiProjects (Mathematics, Physics, Chemistry, Biology) and twelve board members, each Project would "represented" by at least two members. Karol 20:05, 11 March 2006 (UTC)

I'm not sure we want to be that formal about this. Wikipedia is not a bureaucracy.Also, a quick scan through the Wikipedia:list of WikiProjects gives the following that (IMHO) deserve to be told about this, and have to voers decide who to elect:

• WikiProject Science
• WikiProject Space
• WikiProject Ecology
• WikiProject Chemistry
• WikiProject Clinical medicine
• WikiProject Mathematics
• WikiProject Physics

That's seven projects, with 12 board seats between them and the independants. How about we ask them to send their best two people or so, and then do this only by vote?--HereToHelp (talk • contribs) 22:16, 11 March 2006 (UTC)

If I need to compromise on this, how about 1 seat for each project, and then 6 other slots. WikiProject Science is pretty general, so they'll be informed but won't get a free seat. That leaves half the board free and half structured.--HereToHelp (talk • contribs) 22:27, 11 March 2006 (UTC)

I wasn't thinking about any kind of "seats" for the WikiProjects, I was just trying to outline how the structure of an "optimal" board might look like. Karol 02:14, 12 March 2006 (UTC)

I think we need to be rather clearer about the function of the Board. Is it just a panel of reviewers? Maybe we need a Board, who manages the project, and sums up the final conclusion on an article, and a much larger panel of reviewers. For example, I am a chemist and a theoretical computational chemist. I have taught physical chemistry, so I could probably review all articles in physical chemistry. Someone else should review organic chemistry, inorganic chemistry, analytical chemistry, environmental chemistry, medicinal chemistry, and so on. One or two chemists on the Board is not enough. I suggest we appoint a Board and then ask them to approve other nominations to be a reviewer. It is also possible that some people will be prepared to review but do not wish to be on the Board. --Bduke 23:22, 11 March 2006 (UTC)

I, uh...never knew chemistry had so many sub-divisions.

Here's an idea: we get, in addition to the twelve over-all people, one or two specialists sent in on an article-to article basis from the appropriate WikiProject. We get ourselves some people who know about each of those chemistry departments, and either the WikiProject selects them special for each article or they are pre-prepared for each sub-division. Maybe that means a total of 14 people per article, or maybe that means ten regulars and 2 specialists. We could also skew the ratio even further, but those with broad ranges of expertise could sort out a lot of the article (anyone could get spelling, grammer, style, etc.) In fact, having them saves our specialists from that sort of grunt work; they focus on what only they know. I think two specialists is enough when backed by a full board of competent scientists, even if not trained in that specific discipline. But forget that detail. One the concept of specialists in general, any comments?--HereToHelp (talk • contribs) 23:33, 11 March 2006 (UTC)

I don't like the idea of 'reserving' spots for people from certain wikiprojects, or setting up multiple boards. It all sounds a lot like running before learning to walk, and I think it would be best to see how the idea as it stands works in practice before discussing any of these ideas. Worldtraveller 23:41, 11 March 2006 (UTC)

That is much what I think. Appoint the Board, but give them the authority to appoint other people as specialist reviewers. They may not need to do so. However, I suspect the whole Board will not look at an article in real detail. They will leave it to the person who knows most about the topic. That is why we might need more reviewers. A scientific journal has an editorial board and a long list of reviewers to do peer review on submitted papers. --Bduke 23:47, 11 March 2006 (UTC)

We leave the review open to non-board comments; doesn't that work for a quantitatively more thourough review? Yeah, as I think about it, I further dislike that system of have positions open for people from the different WikiProjects. I see no harm in having specialists, though; they will do us a great deal of good as far as getting the nuances right. The others on the board can, as I said above still look at the article for sources (it takes someone in-the-know to find really good sources), style, images, and all that other stuff. This should be a hierchy: the specialists (fewest but most kowledgeable, on top), the rest of the board (very well educated but not specialized, more numerous, in the middle) and everyone else (untrained in many instances, but a large amount exist). I think that the WikiProjects should send candidates to be specialists, but reviewed by either the elected board or the general wikipublic.--HereToHelp (talk • contribs) 00:03, 12 March 2006 (UTC)

Chemistry has at least that many branches, and physics and math have considerably more. I think a lesson can be learned from how academic journals function today. The chief editor/editorial board of a given journal are rarely experts on any given subject; however, they do know who the experts are. The editorial board forwards the papers to the actual experts for "peer review", and makes their final decision based on what the reviewers reported back. Thus, the word "peer" in "peer review" is used in the literal sense, not the figurative sense: the review is not done by an uber-court of twelve supreme lords, but by actual, low-level hacks who actually understand the subject. They are literally "peers" of the author. I think WP needs to be very careful to preserve this kind of peer social structure. In particular, I suspect many writers will hate to have it "lorded over them" by some supreme court of appointed/elected judges.

(I can't help thinking that many of the folks having the conversation here have never actually published in a peer-reviewed journal, nor have been contacted by a journal to perform an actual "peer review". Does everyone here have actual experience in this topic?) linas 00:04, 12 March 2006 (UTC)

I've published, yes. I am currently in the process of publishing others as well. So I've dealt with peer-reviewers, though I've not yet acted as a reviewer. Semiconscious • talk 00:14, 12 March 2006 (UTC)

I have published about 90 peer reviewed papers in the scientific literature and I have been a reviewer for several journals over the years. I know the process. I agree with the above comment about peer review. Maybe the Board should be smaller, say six, and fully elected in the usual WP style. Then they appoint reviewers from nominations. the only criteria would be expertise and experience. A smaller Board could finalise things by e-mail more easily than a larger one. Board members should also be expert. Yes, lets just elect six. WikiProjects can nominate but not appoint. --Bduke 00:24, 12 March 2006 (UTC)

What about Wikipedia:WikiProject_Biology? Wikipedia:WikiProject_Computer_science? Portal:Earth_sciences? Wikipedia:WikiProject Neuroscience? Just to name a few. Picking "board members" before hand based upon an admtted "quick scan through the Wikipedia:list of WikiProjects". A few people seem to want to apply an awful lot of rules to this endeavor. I'm all for having some professional peer-reviewing process, but let's not make this more bureaucratic than it needs to be. Semiconscious • talk 00:06, 12 March 2006 (UTC)

"Let's not make this more bureaucratic than it needs to be." Here, here! Also, I was looking for Wikipedia:WikiProject_Biology, but didn't see it in the list, so I substituted ecology. If you all agree, we could cut the board down to three. They would be aimnly in charge of finding the experts, and getting them to look at articles. Then, we keep a large pool of specialists—no size limits if they're all qualified— who read the articles and become the acting "board" for that article. They have to be handpicked, though, by the board of three. Then, that top-level board (who actually have less power than you might think, they just facilitate the process) designates people from the pool (sub-divided into specific sub-fields) to work on. Or, they simply label the article's subject matter and get the respective experts. ("Okay, this is quantum mechanics! Get the quantum mechanics people!") But, since this system allows for a lot of people to be experts, we get a wide reviewing body. Only the designated experts for the field would have the authority of the board. So, any takers?--HereToHelp (talk • contribs) 00:44, 12 March 2006 (UTC)

I think that, as Bduke poitned out, the function of the board is central to the future of this project (talking about SPR, not WP in general). After all, the guys on the board are supposed to organize the actual review process. I really don't think drawing a strict analogy to scientific journals is the right way to go (as in forming an extra body of expert reviewers and/or multiple boards). We must remember that Wikipedia is not a journal, and that the review process here will not be used to decide about accepting or rejecting articles for publication (although it could feasibly supplement VfD somehow). Also, it certainly is better to keep things as simple as possible, at least at the beggining. Karol 02:29, 12 March 2006 (UTC)

So you're saying that we should have the 12 person board, and I frankly disagree. I think it would be simple enough to maintain a pool of experts, sub-divided into specific fields, to evaluate the articles. Once we establish this pool, we could even have the nominator identify which specific field the article fits into and contact the associated Wikipedians through their talk pages or even a template. Sounds simple, easy, and effective to me.--HereToHelp (talk • contribs) 02:50, 12 March 2006 (UTC)

HereToHelp, I have nominated Charles Matthews to serve on a board of reviewers for Scientific peer review of selected articles under review. Charles is on the ArbCom; as he necessarily cannot spend too much time on SPR, I believe it is most appropriate for Charles to be involved in approval of the reviews. In this role, Charles might vet or suggest selection of the reviewers for an article, for example, and then monitor the process of review. But I should state that once an article is under review, it only makes sense to admit that the selected article has potentially reached stability, and that changes to a reviewed article would then be under control of a process still to be determined. --Ancheta Wis 08:42, 12 March 2006 (UTC)

Re HereToHelp: I'm not saying that. I just think trying to take up the PR process used in scientific journals almost literally is not a good idea.Karol 09:57, 12 March 2006 (UTC)

Sorry I haven't been part of this discussion, I've been sleeping. I think the idea of correlating the major branches of science to their WikiProjects and then giving these WikiProjects status because of this correlation is foolish. Scientists tend to gravitate towards their specialism (so a doctor would likely work at Wikipedia:WikiProject Medicine and a neuroscientist at Wikipedia:WikiProject Neuroscience rather than Wikipedia:WikiProject Biology). Obviously, they do this as far as is feasable - when the WikiProjects devoted to subfields are too small, this is not the case. As most scientists are specialists, excluding these smaller WikiProjects would eliminate alot of very good candidates. There is no reason why WikiProjects can't nominate a couple of candidates from among them (to eliminate too much competition for the set number of positions), but this should be done within an open nomination session.

During this discussion the rôle of the board has also come up. Someone made the very good suggestion of allowing the board to delegate work to specialist Wikipedians who could really give the article a workthrough. I don't see how this is incompatable with the other suggestion that the members of the board itself reviews. The members could review the article themselves (along with everyone else), but at the same time seek specialists out who do not belong to the board to review. At the end of this, the board will summarise the critique and release their recommendations. If this recommendation of having two major functions is kept, I don't see how 12 members would be problematic. Fewer than this would put too much work on each individual member.

If this venture is highly successful (such that a strain is put on the board), I see no reason why subprojects cannot be created: "Medical Peer Review", for example. This should only be done if there is a strain on the board as I think exposure to a larger set of scientists will make for a more rigorous discussion of the article (particularly if the board is also seeking out specialists anyway). In the case that sub-projects are created, I think this board should still be the point-of-access. Wikipedians would submit their article here and the board would redirect it to the appropriate review or not, as they determine. They might decide to not redirect it if they think the article is important enough to recieve wide exposure. However, this is a discussion for the future - walk before we run.

Finally, I think it would be detrimental if we create a too bureaucratic system. The designated processes should be kept to a minimum. The workings of the board should be kept transparent, open and easy-to-understand. --Oldak Quill 10:58, 12 March 2006 (UTC)

The idea of pre-assigned seats for the WikiProjects is bad; I'm sorry I ever brought that up. I also think that we need specialists in some way, shape, or form. But who knows, those with broader expertise may bring something new to the table (knowing about a standard template or format, for instance). So we get the board of twelve and our specialists, but there are two major ideas for how to do that. Some suggest that they just come in and say they are specialists and evaluate the article. That's about as unbureaucratic as you can get, which is good, but there's another idea. We maintain a pool of specialists, sub-divided into specific categories, that we call in whenever we get an article on their topic. This is not incompatible with the broad board (though I did suggest that the size of the board could be cut; that still stands but it could also stay at 12). The problem is getting this pool ready; once prepared it will work pretty well. It's more bureaucratic than the other idea, but still pretty simple and it allows more coordination between the board and specialists. Also, it eliminates the process of finding specialists on an article-to-article basis. That pool can be pretty large if everyone is qualified. I like that idea better, but we'll probably wind up with a compromise of some sort. So, who likes which idea?--HereToHelp (talk • contribs) 12:32, 12 March 2006 (UTC)

The multiple board system is OK if the project is already big enough, or to big like the PR project. I see this future but for now only a few people are around and to build a strong group is better than to seperate to quickly. My suggestion would be not 12 but 50 or 200 people. Example: The article on AM1 methode for computational chemistry would only intersting for one person in a board of 12 but in a board of 50 or 100 the three or five people with the knowledge to PR the article will come forward and discuss it. May be we need a board of a few which steps in in case of dispute. I dont like hirarchy in science but the mediators or administrors are sometimes helpfull. The board of reviewers should be as big as possible. I would not mind if we get all graduted wikipediands on to the list.--Stone 13:08, 12 March 2006 (UTC)

The idea is to have a board of 12 that will look over everything, and a pool of "50 or 200 people" that would respond only to articles in their topic (the pool would be sub-divided into things like computational chem).--HereToHelp (talk • contribs) 13:37, 12 March 2006 (UTC)

Devolution

Once an article has been reviewed, then what? I deal with a variety of articles which are in reasonably good shape, with one problem: every few days or weeks, some well-intentioned but misguided newbie shows up, and adds a bit of mis-information (presumably due to their own misunderstanding of the topic). The constant patrolling needed to keep article from degenerating is exhausting, and one of the leading causes of wiki-vacations. Who is going to patrol the peer reviewed articles? How will articles be kept in good shape? linas 00:18, 12 March 2006 (UTC)

Wikipedia:Stable versions might help. --Rikurzhen 00:30, 12 March 2006 (UTC)

This is actually one way articles improve. If someone has the energy to figure out how they are misunderstanding the subject to get result X, and can put in a corrective [for a very minor example, see this edit), the net result helps WP. Septentrionalis 02:41, 12 March 2006 (UTC)

If an article were to pass Scientific peer review, then the reviewers and the contributors might jointly request that an admin protect the article with a suitable notice. This does highlight the need for a template which might state, for example that "the selected article has undergone Scientific Peer Review and is protected from editing". Once an article has undergone review, it would be unfair to waste the limited time of the Scientific peer reviewers. --Ancheta Wis 09:02, 12 March 2006 (UTC)

This is where I might begin to have problems with this peer review project. I think it is one thing to mark it as having been scientifically peer reviewed, so editors might be more inclined to revert changes to that version unless reasons for change were on the talk page, but it is quite another thing to protect it. A WP article is never finished. --Bduke 09:46, 12 March 2006 (UTC)

Regardless of how many PhDs, tenures and presidentships an individual has, they are still fallible. Their interpretation of "the perfect article" is still subjective and imperfect. An article like brain not only has the standard scientific elements but has a section "The brain as food". Such non-standard sections would not be noticed in their absense by this board, but do add too the article.

Wikipedia is based on the idea that open editing smooths out the rough just as the sea slowly erodes a piece of broken glass into a pretty, rounded pebble-like object which you find along beaches. Wikipedia will always be open for editing, no matter how good an article seems to a group of people. We are currently eroding the rights of editing (particularly anonymous editing) - this is exceptionally dangerous to our process. These erosions are easy implement but are largely indefensible. As User:Rikurzhen said, the stable versions will eventually allow for both open editing and "safe", vandal-free versions. --Oldak Quill 11:09, 12 March 2006 (UTC)

Board

I don't quite understand the purpose of setting up a board when the objectives are not quite so clear...would it not be better to establish a community of reviewers first, and then from there develop a board to coordinate this review process? Right now, the board positions seem to be more symbolic than functional - but I'd be willing to lend my support to the project as right now on Wikipedia, we have no "formal" means of peer review like this. It might be worthwhile to give it a try, or at least so that we know what sorts of issues might be raised and such. --HappyCamper 15:55, 12 March 2006 (UTC)

Wikipedia:Peer review currently employs a community of reviewers. The point of a board is to give the process some sort of framework. They have particular goals to achieve while the article is in their hands. The board members are more likely to review articles than they would if they weren't bound to the project. They also have a rôle of seeking out expert reviewers who will give the article a factual grilling. --Oldak Quill 17:15, 12 March 2006 (UTC)

Panel of reviewers

is it worth starting a separate sign-up list for people who would be willing to be asked to contribute reviews as non-board members? would a list of user names and areas of expertise be sufficient/helpful? --Rikurzhen 20:32, 12 March 2006 (UTC)

I think this is sensible, but I think that reviewers should, like Board members, give us some details that justify their claim to expertise, such as area of Ph D, experience, etc. --Bduke 21:17, 12 March 2006 (UTC)

I thought the idea was to have the non-board member reviews be like regular peer review.--HereToHelp (talk • contribs) 21:49, 12 March 2006 (UTC)

That is perhaps one idea, but another was to have truly expert reviewers who perhaps were too busy to be on the Board, or perhaps in future people who use it as stepping stone to serving on the Board. Nothing stops anyone commenting, but I think the Board will have to ask another expert to review an article if the members of the Board themselves do not have that particular expertise. --Bduke 22:13, 12 March 2006 (UTC)

This idea is good. What is the exact job of the board members in difference to the reviewers? A good list of reviewers with a history in science and a good history in the wikipedia to chose will be good. The board members should have no rights above the other reviewers during SPR. The beginning and the end can be done by board members. The other users should be invited to do also reviewing.--Stone 22:28, 12 March 2006 (UTC)

I had an idea about having the board and then a large pool of specialists, sorted into their speciality, that can be called upon when they are need. See #Multiple boards. Nobody really responded to that idea, and the topic died down—the last comment (currently) is mine. I think that that would be a great way of getting the general public, qualified scientists, and specialists in the topic to look at the articles.--HereToHelp (talk • contribs) 22:33, 12 March 2006 (UTC)

AFAIK, the job of the review board is to administer the review process and dedicate enough time to write a review themselves or seek help to make sure reviews get done, or as Stone put it: The beginning and the end can be done by board members. However, the question remains, what is the best way to solicit informed reviews? --Rikurzhen 23:10, 12 March 2006 (UTC)

Well, we either get a pool of experts put together now or make one as we go along, article-to-article (we keep the experts for that topic listed after they do an article, building a pool as we go along). We can't ask 12 people to know every topic in detail—that's the whole point of specialists, after all.--HereToHelp (talk • contribs) 01:25, 13 March 2006 (UTC)

Peer review with highly disputed articles

The question whether this proposal for scientific peer review would ever be used where there is a massive dispute needs to be raised now. I suspect that in future there will be requests for exactly that to happen. Of course massive dissagreement and massive revert wars do not occur very often with scientific articles, but they can. One good case is Depleted uranium and some related articles. This looks as if it is (or perhaps was - I'm not sure) a complete mess. User:Physchim62, who is an academic chemist, a WP Administrator and a possible member of the Board, has been trying to mediate this situation. I dropped in once and fled. The point is that the dispute was about detailed scientific claims, that looked nonsense, but were strongly referenced. It needed someone with real expertise in Uranium chemistry to sort it out. Even though I am a chemist, I do not have that expertise. Would this peer review process get involved in something like this, or would it run a mile? --Bduke 22:13, 12 March 2006 (UTC)

DU is currently subject to arbcomm. global warming is another (better?) example of a (settled) major dispute. I would have said that SPR should only be done on fairly stable articles. William M. Connolley 22:20, 12 March 2006 (UTC)

I agree, but it will be mighty tempting for someone to add the tags and throw it in our direction. We need to be clear what we would do now, not then. --Bduke 22:26, 12 March 2006 (UTC)

Someone should put that in, about the articles being stable. As for the uranium, I presented an idea to have specialists lok at the articles but never got a big response—see #Multiple boards.--HereToHelp (talk • contribs) 22:34, 12 March 2006 (UTC)

I've added it. Do we need to also mention it further down in the "how to nominate an article" part? --Bduke 01:37, 13 March 2006 (UTC)

That wouldn't be a bad idea.--HereToHelp (talk • contribs) 01:57, 13 March 2006 (UTC)

I'll try to do it. --Bduke 02:39, 13 March 2006 (UTC)

I don't think the role of peer review should even attempt to untangle these sorts of articles. The statements made by the review board would only throw more fire into the dispute. If anything, we can say, we offer peer review services if requested by arbcomm. Even in this situation, I would advocate that peer review not get involved - there is no reason why peer review needs to be entangled in controversy and offer an opinion - it should be completely neutral, and it should not take a stance on anything, other than the quality and factuality of references and ideas being presented. --HappyCamper 12:52, 13 March 2006 (UTC)

Board and Reviewers

In order to try to move things along, do we have consensus on? :-

1. There will be a Board, appointed as explained on the main page, but leaving the size of the Board to be determined closer to the time for voting.
2. There will be a panel of Reviewers that consists of the Board and others, with expertise and experience similar to that of Board members, approved by the Board from time to time, such as when there is nobody qualified to review an article.
3. That those who nominate for Board membership, but are not successfull, be the first extra Reviewers to be considered by the new Board.
4. That people who wish to be a Reviewer but not a member of the Board, be asked to add their details as if they were nominating for the Board, but to finish the nomination with "(For Reviewer only, not Board membership)". Note we have already had one person, who would make an excellent reviewer, say he has no time to be on the Board. These people are to be considered by the new Board similar to the unsuccessfull candidates under (3) above.

If there is consensus, we can implement (4) and then move to seek consensus on the size of the Board. Note that (3) and (4) may produce no extra reviewers if the number of nominations equals the size of the Board, or unsuccessfull candidates do not meet the criteria of expertise and experience, and if nobody nominates to be a reviewer only. This does not matter as the Baord can seek out extra reviewers as required. I think this process is both pragmatic and flexible. --Bduke 02:39, 13 March 2006 (UTC)

Works for me. linas 04:29, 13 March 2006 (UTC)

Sounds realy good for me!--Stone 07:08, 13 March 2006 (UTC)

Yes, this sounds like a good start. Would there be any point in creating a charter to define the roles of the board members and reviewers - change subject to a particular concensus? Such a set of rules may help arbitrate possible disputes that arise out of fuzzy interpretation. --Oldak Quill 09:40, 13 March 2006 (UTC)

Bduke, This sounds reasonable. Karol 11:00, 13 March 2006 (UTC)

No charter at the moment I think :-) There is enough enthusiasm for this project right now, with enough people thinking on the same wavelength, that I think we do not need a chater - yet. I like number 4, Bduke. Since this project is sort of small at the moment, why don't we have a board of precisely 2 Wikipedians to start off with? This would ensure that the collaboration between the board members is quite tight knit. We need to have this started off on the right foot so that we know how to build from it afterwards. Having more than 2 board members would complicate things I think. Reviewers, I think we can have as many as we like - probably classified into subject area and expertise. I'm feel a bit cautious about this project, but I think it's worthwhile to try and get it set up well. --HappyCamper 12:47, 13 March 2006 (UTC)

The only thing I would like to add is that the Reviewers be grouped by area of expertise so we can get specialists quickly. Also, we should send a notice out to ALL of the science-oriented WikiProjects to tell them to send their best to apply for positions on the Board and team of Reviewers. I have a draft typed up already; I can take ccare of this later today.--HereToHelp (talk • contribs) 21:34, 13 March 2006 (UTC)

Broad based representation

I wonder if there is a WikiProject where we can find experts in the humanities and arts? There does not seem to be too many of them active here at the moment. Any ideas? --HappyCamper 12:54, 13 March 2006 (UTC)

Are you suggesting that we expand to review articles in the Arts and Humanities? I do not think so. We do need to decide the scope. Some people have said "Natural Science" only. Others have thought of "Social Science". Does "Natural Science" include "Medicine"? It does under "Natural Science Wikipedia Projects". Does it include "Mathematics"? The list of "Natural Science Wikipedia Projects" does not. We should be clear about this long before nominations close for the Board so it is fully representative of the disciplines we are covering. Remember that we do not have to cover everything. If this project works, others will copy us. My 2c worth is that we should cover Biology, Chemistry, Geology and Physics only, but each in a broad sense. --Bduke 22:16, 13 March 2006 (UTC)

I forgot momentarily that this was the scientific peer review. One concern I have had perpetually while contributing to Wikipedia is that experts in the arts and humanities have been relatively slow to embrace this project - but you are right, let's stick to the science related articles first, hopefully demonstrate that this is viable, and then expand subject areas if necessary. --HappyCamper 04:23, 15 March 2006 (UTC)

Concentric Fusie

If he has experience in writting articles in the wikipedia he can get on the board. The phd always looks nice but nothing more. For me reputation comes from the work not from the title. King, emperor, professor are no more than job names like janitor. I learned this lesson in the university where people insisting on the title are mostly not worth the title. --Stone 13:28, 13 March 2006 (UTC)

Thank you Stone for your words of support. However, Mr. Connolley just proved my point.... to a capital "J" for Joke. Good luck. Concentric Fusie 13:40, 13 March 2006 (UTC)

Nature of the Board

Building consensus is slow. It is now clear that we do not have consensus about the nature of the Board. In the section #Board and Reviewers above, there appears to be a developing consensus, but my point (2) clearly makes the point that the Board members are reviewers and are selected on the basis of expertise and experience in the same way as extra reviewers are selected. The recent discussion in #Verifying credentials at the top of the page, however, argues that criteria for being on the Board is all about writing good WP articles and that reviewers are a separate group with expertise and experience. We need to resolve this. Please add you views below.

My own view is that it will be good in future if reviewers, after gaining some experience in this project, move up to be on the Board. I also think that the Board is pretty well bound to have people who recognise what it takes to have a good article and that review from non-expert people (note that open review from anyone is of course allowed - this is Wikipedia) will also concentrate on that point. For example I have not developed an article to featured article level yet but I know what it is needed and recognise a good article. In the end the purpose of this project is to make sure the science is right. --Bduke 20:56, 13 March 2006 (UTC)

I thought the consensus is already very high, but this is only my opinion.

1. Twelve or two or eleven or three -- lets decide near the end of the nomination! Another idea is: Three for the start would be enough with the aim, after the project is going well to have more. 2. Reviewers get notice from the board and do their job together with interested wikipedians. 3. and 4. Some get nominated the rest gets reviewers. (There should be also be a annual or monthly possibility to get rid of board members, for example by reviewers votes,)

The point with writing good articles is, that we do not write for scientist, but for normal people and because of this we have to have a few people with the ability to get us featured articles which everybody will read. The larger group should be the scientist who get the science right are able to read the articles and get out the fishy things I always hate.

The only thing I worry about is how to get all the reviewers out of their holes? They hide in different projects or even worse they edit only some articles and have never heard of projects and SPR at all. I have no idea how advertising can get us enough people for a full review team. The well established projects like chemistry and physics can be a source. To get some experts through participating from the PR already running or the featured article project. In this projects also experts are around which have the ability to help. As we have the ability to help them share the load. May be it would be good at first to use the table at the PR page and work there until we are big enough to stand on our own feet. Than we would be only a extension to the PR. A later separation would be not really easy and we have to start to get noticed again. Or we are powerful and strong and stand on our own feet from the beginning which would be a good idea also. --Stone 00:11, 14 March 2006 (UTC)

Power. Use the Force, Luke. What Power do you seek? -Splash^talk 00:21, 14 March 2006 (UTC)

Good idea. Physician?

I am worried about the medical articles. While the climate articles and various other pure science articles should be good and if corrupted reduce our chnaces of longterm survival, if the entryist clique of alt.med and anti-vaccination editors damage articles abotu real medicine they may cause a much more direct adverse outcome to someone who depends on them. I'm not likely to nominate myself, I am short of time and am more of a dilletante than an academic, but this is a problem that I think we solved in http://ganfyd.org with our licencing and edit access controls. I don';t think those are approproate for WP though, different problems, different solutions. THis is clearly a good project though. Midgley 01:42, 14 March 2006 (UTC)

For this purpose we have a Wikipedia:Medical disclaimer. At any rate I think the Darwin awards should be extended to people who trust the internet for information on medicine. JFW | T@lk 02:41, 14 March 2006 (UTC)

Why hierarchy?

I don't see any persuasive case for the hierarchical nature of this. It elevate editors formerly of equal standing to explicitly different standings for no actually clear reason. Why not simply make this a project page where those interested in reviewing science articles can find a comprehensive list of those that are interested in being reviewed? Cull it from WP:PR, and open it up like the Computer and Video games one, too. But there is no tangible benefit to saying "You are better than you and thusly shalt thou have a title that acclaims thine superiority, but largely shall thee not actually be able to do squat with it". A Review (capital R, there) doesn't actually change the editable status or editorial status of an article: it doesn't create a mandate for protection or for FAness. A review (lowercase r, here) can certainly help the FA business but I can't see what benefit is perceived in giving some people a capital R and some a lower case r. They will all be heard just the same. If people want to be a Reviewer, they can dive in right now. No elections needed, no bureaucracy needed, no Status needed. Go and find a scientific article, link it to the project page, write your review and cajole others into doing the same. -Splash^talk 00:19, 14 March 2006 (UTC)

You have a point. But if you have a person responsible for the project you tie them to the project and this helps. The board is only for starting the things and ending them. Even this is not fully debated yet. The we are all equal has a point which gets me more and more I think of it. --Stone 00:26, 14 March 2006 (UTC)

I think that what we are trying to do here is to get expert review of the science, while recognising that all are indeed equal. We hope we can attract more experts to review articles than currently happens. The expert review might just be respected, although of course Splash is quite right. There is no special privilege. However, it would be nice to know whether the science is reasonable. I also agree with Stone. The Board is to keep things moving. --Bduke 00:46, 14 March 2006 (UTC)

Splash, I think you're kind of answering your own concern. In a project where everyone's a volunteer, giving some people the taller paper hats has no meaning per se. Rather, I think the point is to attract a few communicators, whose responsibility it is to then attract other science-minded people to this project. So after the initial notion that the board members would actually be doing the peer review, I think this has now turned into an exercise of researching edit histories of closely related articles to find editors with expertise suitable to each requested peer review. Although notions of ownership are generally frowned upon in wikipedia, they do actually help to motivate people... - Samsara (talk • contribs) 01:01, 14 March 2006 (UTC)

Splash, We each have equal access to the articles; that is not the same as having equal expertise. It only makes sense to allow those with the expertise the ability to identify what makes the article correct or incorrect. If we were to identify the wikipedia:stable version ID for the wikipedia:scientific peer reviewed_articles, then we could start turning them into reliable articles for use by the public; that would contribute to the usefulness of the encyclopedia. --Ancheta Wis 01:14, 14 March 2006 (UTC)

I certainly agree that there is inequality of expertise. This has to be traded against the fact that the wiki process is the final authority on content. At least now I'm a little clearer on why the project would want the taller paper hats and scrolls and gowns. I do not really agree with the stable versions proposal since it forks the Wiki all over the place within itself and it becomes immediately unclear which article I should read: and it probably isn't the one I can edit, which is pretty disappointing to a Wikipedian.

An alternative way to run this would be as a corps of volunteers, paperhat-less, who can go and ping possibly-knowledgeable editors and pro-actively ask them to review an article. (This is actually a moderately original idea, I think and is very closely related to Samsara's suggestion but does away with the surely-redundant-in-that-case Board.) We'd need a few more Wikipedian categories to kick start that but the page histories of related topics and WikiProject memberships would be a good place to start the data-mining. What to do with the reviews so garnered...well...the project needs to work that out more clearly too, since it can't really rely on a presumption that stable articles are the norm (since they're not). I suppose they could be used by willing victims for making FAs (much more worth than making protected forks, imo). It occurs to me that academic Reviewing in the real world doesn't rely on election, either. -Splash^talk 01:28, 14 March 2006 (UTC)

I believe that what you're suggesting is similar to what I've suggested at the Version 1.0 Editorial Team talk page, so you might want to have a look at that. Titoxd^{(?!? - help us)} 03:35, 14 March 2006 (UTC)

• I don't like heirarchy and support Splash's first post in this section. Even though I've argued for stability before, I think there are better ways to handle it. Remember too that science is constantly changing. I'm going back to editing :-) Mccready 02:48, 14 March 2006 (UTC)

I think this so-called "hierarchical" system simply recognises the fact that in order to establish credible peer reviews in science, the scientific credentials of the reviewers need to be established. In chemistry we regularly have high school students editing incorrectly (typical comment here, even moving pages to silly names, and the views of a high school student on the validity of a science article should not count as equal to someone like Bduke, who is a PhD chemist with many decades in academia and a long list of publications. In the world of chemistry at least, doing a peer review of an article is seen as a duty to the community, nothing more. Being a board member gives no one any great powers or privileges, just responsibilities. I see nothing wrong in this system as long as the people are elected to the board openly just as is done with RfA - the only difference being that for this purpose, scientific knowledge & experience count rather than technical knowledge and wiki-contributions. We need such knowledgable people to guide the peer review and choose suitable reviewers, and I for one am glad that there are people willing to take on this onerous task. Walkerma 04:21, 14 March 2006 (UTC)

Mccready, you might view the pool of Scientific peers as a list of potential friends of the article. Doesn't it make sense to know that there are Wikipedians out there who can actually help, when someone does not know that the article is credible or not? That's the current problem for John Q. Public, who might not know that Gravitation is any good or not. But if JQP can ask, say Splash, is Gravitation any good, all he would have to do would be to refer to 'who reviewed Gravitation, and when'. That helps us all, because we would then not need so many articles on our watchlists. --Ancheta Wis 10:49, 14 March 2006 (UTC)

Yes it does, but it does not require (evil) elections and Boards (read: Committees) to do this. A corps of volunteers going and fetching willing reviewers and archiving their invited reviews somewhere is all that is needed. It is for the reader of the review to determine the review's worth. Although perhaps there is some scope for a system of ranking articles by their reviews: get the reviewers to score various aspects of the article and post a tag on the relevant talk page about it. (This would also complement the long-promised but still-distant article rating mechanism, anyway.) -Splash^talk 20:35, 14 March 2006 (UTC)

You can achieve the aims without establishing a board. We risk the logic fallacy of appeal to authority, not to mention the laziness of quick flick. As for friends of the article .... OK I've spent enought time here and I said I was off to edit. Mccready 11:24, 14 March 2006 (UTC)

All in all, after all the reviews, someone will still have to revert that high school know-it-all in wikireality. Making a board will not help us with this. Writing reviews might be a guidance to what is correct, and having a database of reviews and reviewers can assist in this, but making a board really won't, although it might help motivate some people (which I frankly find quite lame). You must admit that all the ideas concerning articles stated above can be done right now, without the hierarchy. Although recognizing the credibility of experts is a real issue for the future Wikipedia, I doubt something like this will help at all. The only way to raise this credibility is to build it into the Wikipedia editing system somehow. The Digital Universe is an example of such a hybrid wiki system that might appeal in this regard; it will be interesting to see how that works out. Karol 11:06, 14 March 2006 (UTC)

Well, I looked at the Digital Universe, and all I see is a bunch of buzzwords and slow-loading graphics, so you may have to expand on exactly what you mean. Regards, Samsara (talk • contribs) 11:18, 14 March 2006 (UTC)

The Stewards of Digital Universe are analogous to members of the mooted Board. Digital Universe is 'packaging' the Stewards as publicly named certifiers whom John Q. Public can trust. --Ancheta Wis 11:32, 14 March 2006 (UTC)

Exactly! I like the principle idea behind DU, but they've really overbloated the site. Anyway, it seems to me that the board intended for this SPR will not make any real difference, because it cannot make decisions. So in the long run it probably won't matter if there is a board or not, in terms of the quality of reviewed articles. As far as "expert rights" are concerned, it would be more fruitful to formulate a proposal to the developers (which has been done in the past, if I'm not mistaken). That doesn't mean I'm against this SPR - I am actually quite intent on it and will contribute to reviewing. Karol 12:21, 14 March 2006 (UTC)

Sad irony about Digital Universe: it seems that their chief scientific advisor is one Bernard Haisch. Haisch has a long history of writing papers on topics which could be characterized as constituting (at best) fringe science, such as Stochastic electrodynamics and UFOs. He and his wife are past editors of something called the Journal of Scientific Exploration, which has published many articles on topics many would characterize as pseudoscience, such as ESP and reincarnation. Furthermore, the Digital Universe project is allegedly associated with one Joe Firmage. According to the Skeptics Dictionary, Firmage has said, among other things:

• We have discovered that the Cosmos in which we live is a field of intense but mainly uniform energy - in other words, all that you are, touch, see, hear, taste, smell, feel, or think is a pattern of light - a suspension within a single Universal force.
• We have realized that all things have formed through the function of this light - all ideas, animals, plants, worlds, suns, and galaxies.
• We have leading physicists who are well-grounded in their belief that gravity is caused by this light - in my words, gravity is like a shadow cast in time by the patterns of space.

So I wouldn't hold out much hope for this project. ---CH 04:03, 16 March 2006 (UTC)

• Comment. Concerning the heirarchical issue: what is most important is that there be a structure for recognising the (future/ongoing) involvement of expert WPians in improving the quality of scientific articles. Having a 'board' is a good move in that direction, because it provides a structural reward that comes with the expectation of future effort.

Three matters concern me: (1) I'd like to see not too great a difference in status between Board members and appointed reviewers, and perhaps some formal mechanism for the regular acknowledgement of the efforts of contributors/editors in the science field. (2) How is 'science' delineated? Does it include the social sciences? Should the delineation be explicitly stated? (3) Are you aiming to cover all, or almost all of the major scientific areas in the dozen board members? Will you attempt to 'plug holes' in the Board's expertise by targeted 'appointments'? Tony 07:11, 15 March 2006 (UTC)

I agree with Tony's 1st point. There should be no great difference. The Board should be people who have time to keep it all 'on track' and chase up reviewers. Reviewers are similar people who are too busy to be on the Board, but might at some later point take on that role. How do you see a formal process of acknowledgement? On his 2nd point, I think Karol in the 'summary so far' section below has covered it well. I expanded what that meant there. On point 3, yes, certainly in the appointment of reviewers and in looking for nominees to fill vacancies. --Bduke 09:39, 15 March 2006 (UTC)

Apologies for archiving this section and thanks for retrieving it. I see I was the lasrt contributor. --Bduke 23:49, 15 March 2006 (UTC)

Friends of the article

I've seen articles on wikipedia where someone stuck their corporate logo on the article as a supposed indication of Quality Assurance. In reality, it was just an advertisement - the party concerned had never touched the article other than to stick on their logo. So I see some difficulties with the implementation of this idea.

What I have recently missed, however, is a table of editor names and number of edits for given articles. Presumably, at least some of the top contributors to an article will also be experts in the field, and could be approached with questions, although they may also have a biased perception of the quality of the information... - Samsara (talk • contribs) 11:38, 14 March 2006 (UTC)

If you go to Special:Cite for an article, its output will give you a link to this which does what you be after. -Splash^talk 15:25, 14 March 2006 (UTC)

Splash, thank you for the link. I have found that I have to capitalize the first letter of the article. But it shows objectively what I have suspected: when anonymous is the most frequent editor, the article is typically under the aegis of the inmates of Usenet (based on my small sample of articles which I watch over). But when a long-time editor is the most frequent contributor to an article, it is typically Not Too Bad. Thus a heuristic for a 1.0 Article, (in Jimbo's sense, a count of 1/3 where 3/3=Featured, 0/3=something obviously wrong) might be

1. Anonymous is not the top contributor
2. Top contributor has a good reputation among the peers of the encyclopedia.

Then the problem might be determining 'good reputation' --Ancheta Wis 15:53, 14 March 2006 (UTC)

You're welcome. It is an interesting place to start as a heuristic. It has some kinks in it though. For example, one of my 'pet' articles is phase-shift keying. I wrote 80%+ of it and all but two of the images and User:HappyCamper wrote the rest. It's an old article though, and it fairly regularly has residual grammar things fixed in it by passing anons. It's age and the fact that I do 'big' edits conceals the fact that, although 'Anonymous' is the either the second biggest (inc. minor edits)[1] or biggest contributor (exc. minor edits which anons rarely tick)[2], I know for a fact that the article is good. (It's not featureable because it's pretty technical, and they won't feature anything with an equals sign in it.) So there'd have to be some sort of fuzziness to the #1 in your list, but I feel like we should be able to find something. -Splash^talk 20:40, 14 March 2006 (UTC)

So in addition to the Talk channel, it appears that valuable comment is available on a 'Back channel'; what I mean is that you have identified the principal contributors to an important article, which offsets the heuristic, because you have found a balance between the risk of using anonymous contributors and your time, while still keeping the article whole. Karol, Banno and I am in the same position for scientific method and Cugel is in the same position for history of science; there are a handful of us who watch the articles, but even anons can contribute because enough of us watch the articles for major damage. Banno and Nate Ladd are in the same position for the truth article, although they had a major problem which lasted for months with one troll. I would venture a guess that each article which has reached 'dynamic stability' (i.e., its core remains intact and afloat) has a core of editors who are 'friends of the article' and who watch it regularly. --Ancheta Wis 21:23, 14 March 2006 (UTC)

That's a very nice feature, Splash, that I was unfamiliar with, thanks! As a heuristic, though, I think it could be problematical for pages that get a lot of traffic. Such pages get lots of minor anon edits, but are also often watched carefully by a large group. A page like sulfuric acid is very busy and also close to GA (we're considering A-Class for it at WP:Chem), but has 60% of the edits by "anon". Many of those will be minor vandalism that get reverted quickly, but many are also anons fixing minor typos and grammar. As with Splash, my non-rvv edits were paragraphs or whole sections, whereas the anon edits are mostly one word. The core is very stable. So the heuristic may say more about the traffic rather than the article quality. Very interesting stuff, thanks! Walkerma 22:24, 14 March 2006 (UTC)

Or, to summarize that (tell me if I misinterpret your comment), the breakdown of the number of edits made to an article by specific editors is a valuable tool, but not a silver bullet. (not unlike a lot of things...)--HereToHelp (talk • contribs) 00:27, 15 March 2006 (UTC)

my edits

Dear contributors

I'm pleased to see that this procedure is going ahead. Congratulations to those who've contructed it.

I've edited the project page and made one substantive change: the addition of 'normally' at the end of the statement on 'Tenure'. Here, I thought you might wish to leave room for negotiation in exceptional circumstances.

I thought the text was a little hard in a few places—I softened 'grilling' and added 'please' in one place. Tony 14:32, 14 March 2006 (UTC)

Two more points: (1) I'm unsure what 'and compliance to standards and manuals of style' means. What standards? And is it referring to WP's MoS? (2) 'any scientific article will receive an appropriately in-depth critique'—can 'appropriately' be removed? It doesn't appear to add anything. (3) There's usage of contractions, such as it's and you're, under 'How non-board members will respond to a request', but not elsewhere that I can see. Should this be consistent? Tony 14:45, 14 March 2006 (UTC)

The edits you did seem fine to me. I have just done the "it's" etc. However, I think 'appropriately' should stay. Some will require less 'in-depth' than others. --Bduke 01:25, 15 March 2006 (UTC)

So better to remove both 'appropriately' and 'in-depth'—these epithets beg questions, and it's fine without, don't you think? Tony 06:55, 15 March 2006 (UTC)

Summary so far

This page is getting long...can we at least try to summarize what we have established so far, and then archive some bits if possible? --HappyCamper 04:29, 15 March 2006 (UTC)

1. A community for Scientific peer review exists, both informally, and perhaps formally, with a bit more work. The need for reviewers is not in question. If a board were to be established, it would work with and recruit a far larger pool of reviewers; the size and rôle of such a board remains To Be Determined.
2. It remains To Be Determined whether a formal mechanism for identifying the versions of the reviewed articles should be maintained by this project.
3. There are extant tools for identifying the principal contributors to articles.
4. Credentials are both important and divisive in this project; there is a tension between the existing wiki culture (which has demonstrably succeeded in creating articles) and the perceived need of the readers of these articles (to be reassured that these articles are good ones).
5. A critical mass is forming, if not there already.
6. The project will only treat articles on topics from the natural sciences at first. (Karol 08:35, 15 March 2006 (UTC))

Strongly agree with Karol on point 6. For now we stay with the natural science as defined in that article. To be clear that is:-

• Astronomy, the study of celestial objects and phenomena that are outside the Earth's atmosphere, e.g. stars, the cosmos, etc.
• Biology, the study of life.
  • Ecology and Environmental science, the studies of the interrelationships of life and the environment.
• Chemistry, the study of the composition, structure, and properties of substances and with the transformations that they undergo.
• Earth science, the study of earth and specialties including:
  • Geology
  • Hydrology
  • Meteorology
  • Science-based or Physical Geography and Oceanography
  • Soil science
• Physics, the study of the fundamental constituents of the universe, the forces and interactions they exert on one another, and the results produced by these forces.

To be clear Specifically that excludes Mathematics, Medicine and Social Sciences, for now anyway. --Bduke 09:31, 15 March 2006 (UTC)

Bduke, I propose re-stating your point as "To be clear that excludes Mathematics, Medicine and Social Sciences, for now anyway." The link natural science#Natural sciences is identical with the rest of the text above. --Ancheta Wis 11:30, 15 March 2006 (UTC)

Sure. Done. I thought it worthwhile to list the disciplines clearly. I have nothing against expanding, but we need to be quite clear now so we attract nominees for reviewing and the Board from those disciplines. --Bduke 11:44, 15 March 2006 (UTC)

I'll agree with the above. These sections focusing on what we've agreed on (over what we haven't) are really useful—keep it up.--HereToHelp (talk • contribs) 21:28, 15 March 2006 (UTC)

Topical exclusions

Hmm...I don't think it would be so good to say precisely what sort of natual science we are looking at. Again, there is some contention between being exclusive and inclusive, etc...why don't we simply say that we're focusing on the sciences, with emphasis in areas X,Y,Z because at the moment, these are the people with the expertise who are available? --HappyCamper 12:48, 15 March 2006 (UTC)

Oppose. I agree with the focus on quantitative, empirical fields. I assume that we are also excluding engineering - if so, this should be mentioned. - Samsara (talk • contribs) 14:42, 15 March 2006 (UTC)

What is the benefit to exclusion of anything scientific if reviewers can be found to look after the process? If reviewers (or, for that matter, articles) can't be found, then there is de facto exclusion until such time as they are. I increasingly get the impression that this proposal as a whole is plunging towards a very constructed state of affairs that it has no clear idea on the reasons for. -Splash^talk 14:51, 15 March 2006 (UTC)

I share a similar inclination with Splash at the moment...There are advantages of being more inclusive with the wording. Perhaps "scientific" is too loaded of a term - perhaps "technical peer review" might be better - a place to find someone to double check technical references, rederive equations, make sure the technical interpretation of results presented is correct and precise, et cetera. I certainly wouldn't want to exclude engineering topics from this for example! I'm fine with restricting the scope of this project conceptually to "technical" fields at the moment (so for exmaple, you would not be able to get a review for an article on 14 century linguistics here) - but this project at minimum needs to be welcoming to all technical fields - however loosely defined. Once this project grows, it would be obvious that the word "technical" does not have to necessarily refer to scientific/engineering/mathematical competency, but rather specialized, precise knowledge associated with specific vocabularies, concepts, and fields.

I have been around long enough to see a number of WikiProjects set up which ultimately did not work out, because the initial goal focused too much on establishing a structure of organization (ie, pseudo top down), and not one on growing a community from within (pseudo bottom up). I'm sure the right people will jump on board as needed - and if that does not happen immediately, it would be simple to reach out. Using this approach does not have to fundamentally lower the integrity of the peer review process on Wikipedia. We just need to be a bit careful with our boldness - we don't want to take it too far with institutionalizing procedures or whatnot at the moment. Perhaps not verbalized quite so succintly, but something doesn't quite feel so right at the moment. --HappyCamper 16:15, 15 March 2006 (UTC)

Now I think you really are diluting it to the point where we might as well all go to Wikipedia:Peer review and exercise our zealotry there. :)

As devil's advocate, how about we just make this one of the aims of Wikiproject Science, rather than having an altogether separate project? - Samsara (talk • contribs) 16:26, 15 March 2006 (UTC)

Comments on the project so far

Hi. I’ve been watching the developments on this page with interest and have a couple of comments. First, while on the subject of the name of the project (brought up by HappyCamper, above) I would like to point out that what is apparently being proposed is not a peer review at all, but a review by a person who is distinctly not equivalent to most editors. In professional science, the “peer” who reviews a manuscript submitted to a journal is ostensibly on equal footing in terms of training, expertise, and sometimes career to the author of the manuscript (i.e. the work of a U-Pb radiogenic isotope geochronologist at a university is reviewed by another U-Pb radiogenic isotope geochronologist at another university). It appears, to me at least, that what is being proposed here is not a peer review (this system is already in place as a fundamental tenet of the Wikipedia, as anyone can contribute to it) but rather the opposite, where certain persons opinions are given more weight by virtue of either a storied edit history or real-world professional qualifications. These people are singled out because they are not the peers of the average editor, but because they have a specialized knowledge (of either technical issues or how to write a good science article). Although strictly speaking this is a niggling issue of nomenclature (and as scientists I know we all hate that) but it seems to be obscuring the fact that the end result of this project is to have either edits or suggestions for edits that are given more weight than other edits or suggestions for edits by virtue of the fact that they came from the SPR board.

My opinion is that having more weight to a set of edits based on the fact that they came from an "approved" person (whether it be from this board or a reviewer approved by this board) is a flawed idea. Why? Because edits should be able to stand on their own two feet. If a fact is wrong or a concept presented in a misleading manner in an article, I can show that it is wrong through literature references or by a logical argument. I do not need to appeal to "it’s wrong because I say so, or because the SPR board says I am an authority". Although I am relatively new here, my approach to editing has been in the former manner. I do not have the skill at article writing that some/most of you all do (or the time!), but I can pick out incorrect or misleading sections of articles in my discipline (igneous/isotope geochemistry) and do my best to correct them or indicate how they should be corrected. It has been pointed out that this project may provide a focal mechanism for others with my particular editorial bent, and although this is a laudable goal I find it hard to believe that there are so few technically minded wikipedians actively separating the wheat from the chaff (and this still suffers from the fact that reviews would be given undue weight, as pointed out above). It has also been suggested that this may help settle some problems with "highly disputed articles", however most of us probably know so-called experts in our respective fields who (while equally proficient in their discipline) have violently differing opinions.

I find it hard to believe that a project of this size and structure is required to simply "point" technical people towards articles with bad science in them. If someone sees an article in need of a review, simply seek out someone that seems to know what they are talking about…there is no need to appeal to a formally defined board. If I saw a problem with an article on the geology of Australia, for example, I have a couple of people in mind that I would refer it to. Thanks, Rickert 17:15, 15 March 2006 (UTC)

You hit the nail on the head. Although I beleive that many of the others here will agree with you. linas 17:24, 15 March 2006 (UTC)

Living in Australia, all the interesting discussions occur while I'm asleep. I want to comment only on scope, as I put up the summary of "natural sciences". I do not want the project to be "focused too much on establishing a structure of organization". I want to build up a community. I draw different conclusions from above. I want to start small and start from where we are, which is a proposal about natural science, not technical articles. I think we can get a team of good natural science Wikipedians to go with it and develop things. I do not want them to be bogged down in trying to find reviewers in areas such as medicine or engineering that are outside their own fields. It would lose all focus. I suspect that if we throw it open to pretty well any area, it will turn out to be no different than WP:PR and will therefore have failed. That is not to say that WP:PR is a failure. It is just doing something different from what this project is seeking to do.

I will also comment on whether we need a structure to point technical people towards articles with bad science in them. First, I think we can fix bad science, i.e errors, but it is not so easy to fix a lack of full appreciation of the whole topic. It is often a problem of what is not there, rather then what is there. Second, there are few enough Wikipedians with scientific experience. We need to use them to search out people from outside that group to help the WP project. To "simply seek out someone that seems to know what they are talking about" may not be that easy. It will need work and that needs committment. Without a structure, it will probably not happen, or as in my first paragraph, it will move to work just like WP:PR. --Bduke 21:11, 15 March 2006 (UTC)

Agree. There are two differences between this and regular PR: scientific articles and educated reviewers.--HereToHelp (talk • contribs) 21:38, 15 March 2006 (UTC)

If you have medics and engineers on the team, where is the problem? Why would anyone want people not in those fields to go work in them? But why say that you don't want people in those fields, even if they are willing to help? Are you after Pure Science with a Committee to run it? You're in the wrong place. -Splash^talk 22:09, 15 March 2006 (UTC)

We do not have medics and engineers nominating, because we have not made an effort to get them to look here. This was started as Science Peer Review. I am not trying to exclude them because I want to stick to pure science. I want to stick to what we can manage. A small Board that included medics and engineers would be unrepresentative of too many other areas. A large Board would not work well. If Science Peer Review works then maybe others will start Medical Peer Review, Engineering Peer Review and so on. Let us not try to do too much or we will fail. --Bduke 22:24, 15 March 2006 (UTC)

Engineering is a product of the stregnth of materials and the laws of physics. Medicine is no less scientific than bio as a whole (maybe not 100% predictable but still valid). Psychology (for instance), is where I draw the line.--HereToHelp (talk • contribs) 22:42, 15 March 2006 (UTC)

Why? What does it matter if you can find experts to review psychology articles? The only reason there would be a problem with board having an engineer, a medic or a pscyhologist on it, making it unrepresentative of an undefined something is if there is a board at all. This having not been usefully motivated, other than "oh but we should", I don't think the Board has a either a viable or a defined role. There is not one good reason why it can't simply a collection of all willing volunteers who will jointly review and invite reviews. Even if you make this Board, because bureaucracy is always fun for those creating it, it will have no power. It's not even clear to me why a Board was ever proposed. -Splash^talk 23:21, 15 March 2006 (UTC)

How about changing from board to cabal — personally I don't see a problem with a board, or a committee or a project or a noticeboard or council or whatever, so long as it gets the job done — and that job is basically getting people who know what they're talking about and are interested to check over the content of articles. - FrancisTyers 00:56, 16 March 2006 (UTC)

Well, whatever, the original motivation for establishing the board, it was a useful tool for making everyone say "hello", and maybe we should leave it at that. However, I do think that the advocacy idea has merit. In reponse to FrancisTyers (forgive my not following proper indenting here), I don't think changing the exact word used makes any difference, but my voice is not native (does that sound more or less awkward than a more complex construction? ;) ). - Samsara (talk • contribs) 01:14, 16 March 2006 (UTC)

I don't think changing the exact word used makes any difference &mdash' Precisely my point :)) - FrancisTyers 01:20, 16 March 2006 (UTC)

The problem with regular peer review...is that it's broken

Responding to the comments immediately above: The big problem with Wikipedia:Peer review, in WP, as it currently stands, it completely fails to recruit anyone who is actually knowledgeable in the topic that is being reviewed. The same criticism applies to the Wikipedia:Good articles process: one could submit, and have accepted, as a "good article", any utter hogwash, as long as it was well written, pleasent to read, and illustrated with pretty pictures. By contrast, "academic peer review" should be a mechanism whereby actual "domain experts" are recruited to do the reviewing. The job of the board members would be to find, qualify and recruit such domain experts. If the board knew how to function in this way, we could peer-review articles in history, geography, law, and pokemon. The reason we're all talking about "scientific peer review" is, in fact, because the current Wikipedia:Peer review process is fundamentally broken. linas 16:54, 15 March 2006 (UTC)

So how about we call this thing here "peer review outreach" or "peer review advocacy"? - Samsara (talk • contribs) 17:56, 15 March 2006 (UTC)

Mmhhh... If this helps the effort, why not. But I never thought of the "board" as a group with "power" (see earlier discussion) but as a group with "duties". I like "advocacy" in that context. The members should actively recruit and help people to perform "scientific peer review" on articles, where such a request is made. In this way they serve WP as a whole. Whether the created reviews find acceptance will solely depend on their quality and that's also the way this effort will gain credibility with all Wikipedians. Awolf002 18:24, 15 March 2006 (UTC)

The practical purpose of WP:PR is to act as a gateway to WP:FAC, preventing hopeless nominations, bruised egos and wasted time. It does that job pretty well, usually. It openly confesses to not being peer review in the traditional sense, so it's not entirely fair to declare it broken because it fails to do something it doesn't intend to do. -Splash^talk 18:31, 15 March 2006 (UTC)

Agree. It's not that it doesn't work, it was never designed to work by your standards.--HereToHelp (talk • contribs) 21:30, 15 March 2006 (UTC)

OK, I guess I failed to make my point. Samsara asked if there was any difference between what we are talking about here, and Wikipedia:Peer review. I was trying to suggest what that difference could be. Namely, this process should concentrate on a process whereby the reviewers are actually knowledgable on the topic being reviewed. Right? linas 23:39, 15 March 2006 (UTC)

Presumably SciPR will actually check references as a matter of course? - FrancisTyers 01:13, 16 March 2006 (UTC)

Restrictions on who can nominate themselves or the Board?

I've seen some nominations that look like jokes. People with no userpage at all (sorry, see that red link biases me against the person), people with a handful of edits, people who I could review better than (let's just say I hope to help work out the logistics of this project more than help review stuff). Should we have something like, 500+ edits or some other way of weeding out those who are obviously unqualified, or just let them lose the election? Also, let's take people's egos into account. I don't want to hurt the feelings of anyone (except vandals) when not necessary. On the other hand, the articles come first.--HereToHelp (talk • contribs) 21:49, 15 March 2006 (UTC)

The first "redlink" nominee has a mass of edits. I do not know why his User page is not there. He has a talk page. The latest "redlink" nominee has 7 edits, 3 of which were doing this nomination, and seems to have joined recently. His 1st edit is 10 days ago. His idea that a student nomination is a good idea might have merit. Earlier this year I attended a workshop that was concerned with review of laboratory experiments. They were subjected to actual test in the laboratory and detailed discussion. The attendees were 50% academics, who had submitted the experiments (they spent one day demonstrating their experiment and two days reviewing), and 50% students. The student input was great. Maybe they would have something to contribute. (Side comment to chemists - see, I have been in a laboratory with more than coffee in it!). --Bduke 22:12, 15 March 2006 (UTC)

I'd sooner vote for the first "redlinker", but JarahE is also fairly new: see this. Only 91 edits, but 80 are to the article namespace (when I wrote this). He looks promising, but I wouldn't want him on board quite yet. Most, if not all, of the "bluelinkers" look good, though.--HereToHelp (talk • contribs) 22:38, 15 March 2006 (UTC)

If there's no Board, there will be no joke nominations. -Splash^talk 23:22, 15 March 2006 (UTC)

About the self-nominee who doesn't even have a user page--- I had the same reaction. Actually, I'd like to see a board composed of nonanonymous registered users who have a strong publication record in mainstream science. But that's probably too much to ask, I wouldn't really want to exclude members who have strong records but prefer to do their WP edits behind a pseudonym to avoid awkward questions about how they choose to spend their free time from their academic colleagues (or to avoid being pestered by cranks); given my sad experience as a non-anon I'd even have to recommend against anyone identifying themselves by name (too late!, in my own case, obviously), although I hope anonymous registrants do not use socks. ---CH 04:13, 16 March 2006 (UTC)

Chris, You chose immortality. Those of us with non-literal account names reserved the right to renounce it. I personally respect your choice for yourself. --Ancheta Wis 17:42, 16 March 2006 (UTC)

• My inclination is not to be restrictive and to let the voting speak for itself. (I suppose that we'll have to trust that voting will not involve sockpuppets, just as we'll trust statements by nominees concerning their PhDs and publications.) Tony 04:46, 16 March 2006 (UTC)
  • I'd suggest that some basic restrictions would be a good idea - mainly because of the "bruised egos" problem mentioned above. It's a lot gentler to say that they don't yet qualify (# edits etc) rather than for them to see lots of bold "object"s next to their name. Walkerma 05:04, 16 March 2006 (UTC)

Walkerma, I have observed that the finest scientists are gentle (unless roused). My observations are based on conversations with Feynman and Francis Crick. (So why did Feynman work on the Bomb? That was due to the press of history.) --Ancheta Wis 18:10, 16 March 2006 (UTC)

• • • Nominees should be willing to put up with 'objections', as long as they're not impolite. I'm more concerned that voters be able to view the range of expertise (i.e., areas of science) covered by the nominees in a simple list rather than hunting through a swag of info. Tony 05:51, 16 March 2006 (UTC)