In December 2016 Peter Gallert wrote an op-ed (Operation successful, patient dead) about the Wikipedia workshops he ran in Namibia. He asserted that while the workshops are fun as events, and may even produce a bunch of good edits, they are not effective tools for recruiting new people who will keep editing after the workshop ends.
I have run dozens of workshops for new editors myself since 2009, and sadly, I have to agree with Mr Gallert: Though there are pleasant exceptions, I rarely see people sticking around as editors after the workshop. Research from 2013 arrived at a similar conclusion. I know some workshop organizers who will beg to differ and say that applying certain techniques will make workshops more predictable and measurably effective; but I think everyone will agree that there is no known recipe for running a workshop that is truly efficient at creating new long-term editors.
There is, however, one way in which new editor workshops and other similar real-life events, such as edit-a-thons, are consistently highly effective: Observing new users of Wikipedia and other projects and learning about the technical and social challenges they face, challenges that are too easy for experienced editors to forget. It is very sobering to watch…
- … people trying to create an account and struggling with CAPTCHAs and other oddities.
- … people searching in Spanish in the English Wikipedia's search box.
- … people clicking on footnotes in Visual Editor to try to edit them.
- … people trying to publish an edit and trying to decipher warnings from AbuseFilter.
- … people creating a pretty OK article, seeing it deleted after a few minutes, and then trying to decipher the notices from administrators.
- … people not seeing notifications about talk page messages at all (and yes, it happened in the days of the big orange bar, too)
- … people trying to copy text by right-clicking and failing (you may be wondering why would people do it by right-clicking instead of using a keyboard shortcut, but it's just unbelievable how often I see this).
- … people having no clue what to write in edit summaries.
- … people clicking the "History" tab expecting to learn stuff about the Napoleonic Wars.
- … people staring blankly at the instructor talking four-tilde signatures and templates.
This is just the tip of the iceberg. Over the years I saw many, many more issues of this kind. Whenever relevant, I reported them as software bugs or started discussions in appropriate talk pages or mailing lists. Some of the bugs were fixed, and it's great; this is, very clearly, a thing for which workshops are useful.
But it does raise a few issues to consider and to remember.
First, it must be thoroughly remembered that it's not these people's fault, and there isn't something they were "supposed" to know. In the vast majority of those instances, they were trying to do rather sensible things, and to do them in good faith. Lack of computer expertise must also not be blamed—people who contribute to wikis are supposed to be good at the subject about which they are writing, more than they are supposed to be IT professionals. Besides, Wikipedia is different in many ways from a lot of other modern websites, and even people who are IT professionals often find it surprising. These problems are caused by mistakes in software design, by software bugs, by sysops fighting perceived vandalism too eagerly, and by other things inside the project. You organized the workshop for these people, so complaining about them is not productive. Neither is saying that learning all these features is a filter that good Wikimedians will be able to come through—this filter is artificial to say the least.
Furthermore, if not watched, such people will probably remain silent and go away. Thanks to my being there, I was able to address their problems immediately, by explaining what to do, or at least by working around them. The point is not to create a chance that they will remain good editors; most likely, as Mr Gallert says, they won't anyway. But at least they saw a human face that explained them the problem, rather than something totally robotic. And in some cases there is a chance that this problem will be fixed, so that it won't happen to other people at all.
And this brings me to the last point, which sums up all of the above: During the workshop, do make quick notes about the problems people report, and pass them on. Don't ever think that when a user complains or is confused by something, your job is done when you explain why the user is wrong. The user isn't wrong. Neither is your job done when you help the user overcome the problem on the spot. The user might complete the edit, and that is nice, but if the user is unlikely to stick around as an editor anyway, no significant impact will be made. If it confused this user, it may confuse thousands of others. Unless, that is, you report it. Reporting will create a chance that it will be fixed, and this will be an actual, undeniable positive outcome from your workshop.
How should you report it? For posting software bug reports, suggestions for changes and new features, and ideas for better design or workflow, Phabricator is usually the best place. When in doubt whether to report a bug or not to report it, do report it. For issues that are more about community, culture or local templates on your wiki, the appropriate talk pages and mailing lists are the right forum.
Should you treat your editing workshops as just user testing in disguise? No, you absolutely shouldn't. I don't. You should keep treating them as editing workshops, and it won't hurt to keep thinking of ways to make them better recruiting tools. But you should start thinking about them not just as opportunities to change people into being Wikimedians, but also as an opportunity to change your own project into one that is easier for good people to join—and this is in your hands.