We want to do a thorough analysis of the Pune pilot program to derive learnings and trends across schools, batches, classes etc. For this we'll be conducting an in depth qualitative and quantitative analysis. This analysis will not only help us assess the level of success or failure of the pilot but will also help us pave the way forward. This page outlines the broad structure of how this analysis can be conducted. We encourage discussion of this plan on the talk page, especially around the questions of how best to gauge the impact this program had on the Wikipedia community and what other points we are missing in our analysis.
Step 1: Identify data points to be analyzed edit
- Classroom performance
- Identify the courses most poised for success
- Engaged professors
- Engaged Campus Ambassadors
- Good CA:student ratio
- Determine amount of good content added by students in those courses (i.e., content that did not get reverted due to copyvio, poor English, wrong tone, etc.)
- See if we can find a correlation between engaged professors, engaged CAs, and low CA:student ratio and better performing classes, to determine if that can be a useful metric in predicting success.
- Impact on the community
Note: in this case, we want to compare Pune pilot students to normal Wikipedia newbies; we know newbies always impact community workload, but we want to determine how different the Pune pilot students were than traditional Wikipedia newbies.
- What was the impact on the New Page Patrollers?
- What was the impact on the Copyright Cleanup Investigators?
- How many hours did community members who were not Ambassadors for the IEP put into assisting students and coordinating cleanup efforts?
- Programmatic results
- What should we have done differently in programmatic decisions?
- What worked well that we should keep doing?
- What should we change for the next pilot?
- Cultural differences
- What are the differences between India and the United States in terms of cultural understandings of copyright?
Step 2: Collect and analyze data edit
- Quantitative
- Nitika is compiling an evaluation of which courses were most poised for success in terms of engaged professors, engaged CAs, and low CA:student ratios. We need to devote staff time to evaluating article quality of students in these courses and compare it to article quality of other courses' contributions to see if there are significant differences.
- We need to determine a way to estimate the volunteer hours impact of the pilot program on NPP, CCI, and the general community.
- Outstanding question: What is the best way to do this? Join the conversation at Wikipedia_talk:India_Education_Program/Analysis.
- UPDATE, 19 JAN 2012: Quantitative Analysis available here: /Quantitative_Analysis
- Qualitative
- Nitika is compiling learning points for the programmatic side in terms of changes we should make next term. These are based on feedback from the community on talk pages, staff discussions, and mailing list discussions.
- WMF staff interviewed several people to answer the cultural differences in terms of copyright. This analysis is here: /WMF_interviews.
- WMF has hired Tory Read to conduct an independent analysis of what went wrong with the Pune pilot. Tory was in India for the kickoff of the Pune pilot, and her observations are available in the the India Chronicles. She is interviewing Campus Ambassadors, professors, students, English Wikipedians, and WMF staff as part of her analysis. Her report is available here: /Independent_Report_from_Tory_Read.
- A new Wikipedia:India_Education_Program/Documentation page has been created to document various emails and messages that were private but should have been added to the program talk page as well.
Step 3: Evaluate data edit
We will use the results of all of these analyses to determine the plan of action for our next India Education Program pilot, which will be launched in mid-2012.