Wikipedia:Wikipedia Signpost/2013-11-13/Special report
FDC staff raise the benchmarks for activities, impact, planning, and governance
The supporting staff of the Wikimedia Foundation’s powerful volunteer Funds Dissemination Committee (FDC) have released their assessments for the third half-yearly round of funding applications. Staff assessments are a key preparatory stage before the Committee itself sits in San Francisco, which for this round will be from 17 to 22 November. The applications for the newly named annual plan grants were submitted by affiliated entities on 1 October, and comprise a total of more than US$5 million in bids—most of them by returning applicants that had gained one-year funding from the very first FDC round in October 2012.
Two nation-based chapters have joined the bidding for the first time—India and Serbia—as has Amical Wikimedia, the newly affiliated thematic organisation, which focuses on the Catalan language and culture in coastal regions of the western Mediterranean. The Hungarian and Australian chapters did not re-apply, and the French chapter successfully moved into the March round for each year after funding was declined in the initial October round. The Wikimedia Foundation did not make an application in this round.
As the grantmaking process evolves, it is clear that the Foundation's assessment of applicants is becoming more sophisticated. The assessments now include a detailed statement of methodology, a financial summary table, and an overview of each application's strengths, feasibility, risks, and concerns. Summaries of expert opinions and community commentary are presented for each application, although only Wikimedia Germany's application garnered one of the former—by Erik Möller, the WMF's Vice President of Engineering and Product Development. This time, the staff's nearly-final draft assessments were shared with all applicants a day ahead of their release, to provide lead-time for initial responses, which were then taken into consideration for any required factual changes and clarifications.
Summary of focus
In summary, the staff looked "for direct and indirect impact ... on Wikimedia projects; for example, growth in contributors and content donation. ... and board, staff and volunteer leadership that is committed, effective and engaged with these communities and practising movement values of openness and transparency." A frequent concern was that "in this round, only two returning applicants asked for funds within the recommended maximum growth rate in movement resources (annual plan grants/FDC allocations) of 20%, which is itself a significant rate in many contexts. In addition, many organizations are underspending on previous grant allocations ...". Other common concerns were budget and staffing growth, lack of demonstrable impact on WMF sites, and governance.
Numerical scoring
The staff's numerical scoring is based on 13 criteria that score from 1 (low) to 5 (high). The totals for each applicant are shown in Figure 1 in descending order, comparing this year's round (red) with the round 12 months ago (blue). The minimum possible total is 12 (1 for every criterion), and the maximum 65 (5 for every criterion). Last year the average total was 43.2 (standard deviation 6.2), excluding the WMF's ratings; this year's average was broadly the same, at 42.1 (SD 6.6). These criteria are grouped into five thematic areas: impact, ability to execute, funding efficiency, the applicant's construction of measures of success, and benefits to the movement. Figure 2 at the bottom shows the breakdown of these areas for each applicant in this round. They show averages of 9.5 out of a possible 15 for "impact" (last year 10.7), 10.3 out of 15 for "ability to execute" (10.2), 5.5 out of 10 for "funding efficiency" (6.6), 9.4 out of 15 for "measures of success" (8.8), and 7.3 out of 10 for "benefit to movement" (7.1).
Applicant by applicant
Argentina came first, rising slightly from last year to 49 despite concerns about high administration costs and overheads (the planned increase in staffing from 1.5 to 3.5 FTE was specified), too-rapid growth, and difficulty of tracking the impact of some aspects, such as "outreach with printed materials". The chapter's strengths are that it is "well-positioned for impact on a global scale" and has a good profile for regional cooperation. Sweden was equal-second, having risen slightly to 48; while there was praise for its clear metrics, innovations, and good management, the chapter was criticised for high growth relative to underspending, limited global impact, and poor levels of cohesion in its plan, which "lacks focus and detail".
The Netherlands was also equal-second, having dropped from a score of 51 to 48. The chapter has "strong community involvement, steady and effective leadership," an engaged board, good reporting practices, and a "thoughtful planning process that includes pilots, needs assessments, and community surveys", and "has improved in its willingness to reflect and share". The staff highlighted that "WMNL is beginning to think about improving and strengthening its relationship with the Dutch Wikipedia community, including facilitating discussion on the social climate of the Dutch Wikipedia, and measures to welcome new editors". However, the Dutch proposal was "not focused enough on programs that will have strong online impact; more than 50% of its budget is dedicated to administration and travel". There was concern about the small number of active volunteers and the small size of the community compared with the size of the chapter's request.
Austria, at sixth, increased its score from 42 to 45, with improved scores for "ability to execute" and "benefits to movement"; but its "impact" rating went down. The chapter was praised for its joint volunteer–staff leadership and "clear focus on a limited number of programs", although there was disappointment that "not all programs have a strong focus on impact on Wikimedia projects", and particular concern that some "funds allocated for travel for volunteers and staff [have] no clear outcomes related to the Wikimedia projects." Overall, "the potential impact of programs is not high enough relative to the cost of this proposal."
Common to the next four applicants was a crash in scores since last year—particularly by the two biggest chapters, Germany and the UK. Germany went from an enviable 53 last year down to 44, falling significantly in ratings of "impact", "ability to execute", and "measures of success". There was praise for its four priority areas, including software development, and the significant potential of its Wikidata program for global impact on Wikimedia projects. However, the proposed 2014 budget of more than US$7.5M "may be unsustainable", and the request for $2.4M from the FDC needed "strong direct impact" on quality and participation in WMF online projects. The chapter's committed and resourceful staff was seen as a positive, although "relationships among board, staff, and community have been challenging". Indeed, staffing was a serious concern in the assessment:
| “ | WMDE is planning to hire nine new positions in addition to increasing two positions to full-time status, bringing total staff to 51.45 part- and full-time staff. Adding US$742,500 in staffing cost seems neither prudent nor sustainable, given its current revenue plan. Only one of nine new positions is for its new focus area of software development, and five are devoted to evaluation, communications, and administration. This raises questions about how this staff growth corresponds to WMDE’s new focus areas. ... / ... Nine employees and more than [$1M] are to be allocated to the Volunteer Support program, but this significant investment may not have commensurate impact on the Wikimedia projects, especially considering WMDE’s mixed record in past community support work (for example, Fact Check and the community budget have had uneven results and response). ... / ... WMDE is not always forthcoming with sharing its challenges and lessons learned in its reports. | ” |
The Swiss chapter's ratings dropped from 42 to 37. The staff reacted positively to "useful and innovative work like Wikipedia in Jail and thematic programs like The Alps" and the support of a diverse language community within and outside the country. But the "proposed funding is too high, especially when the active community is very small [and the] potential impact of these programs on global Wikimedia projects is low or unclear". Some targets, the staff believed, "seem extremely low with respect to the funds requested. For example, 5 new active editors in the Community Support program or 10 participants in Edit Togethers in the Community Support program". The chapter came in for strong criticism about its poor record of reporting and its lack of coordination with other affiliated WMF organisations. There were complaints about "conflict of Interest issues, with community members asking about the relationship between Board and staff members".
Israel scored well last year at 41, but sagged to 36 with reductions in thematic assessments almost across the board. The staff noted "strong commitment of board and volunteers and the onboarding of a new ED", past programmatic successes rather than "institutionalization" (singling out for praise the partnership with the National Library of Israel and a related Wikipedian in Residence program), and improved reporting. Two local donors' positive opinions were mentioned. But there was concern at the proposed growth in budget and staffing ("ill-considered"), the number of activities planned at this stage, and the perennial complaint that "the impact on Wikimedia projects is not proportionate to the resources being requested." In addition, "Wikimedia Israel is increasing its operational costs and proposing a large amount for international travel and events, diluting its strong emphasis on effective programs (including editor recruitment, retention and content donation) [and] has a mixed track record of budgeting effectively and spending consistently."
Wikimedia UK's score plunged from a respectable 43 last year to just 33.5, falling significantly in all thematic areas but "ability to execute". The pluses were success in addressing governance challenges, "not growing its current staff of nine", and that its events appear to be popular and well-attended. But the complaints were many, from the all too typical "potential for impact on Wikimedia projects is too low in proportion to funds requested", to a diffuse and unfocused programmatic strategy and a plan that "lacks clear metrics or a feasible plan to evaluate work". There were complaints of significant underspending. Opportunities for global and further UK impact were suggested, given perceptions that "the impact of the proposed activities on Wikimedia projects is unclear." In particular, "WMUK has not yet been able to effectively set and measure long-term goals around outcomes such as editor retention."
| “ | WMUK has not yet been able to effectively set and measure long-term goals around outcomes such as editor retention. Given the amount of movement resources requested and WMUK’s size and potential, a continued focus on outputs or process indicators rather than outcomes is a serious concern. ... / /... WMUK has been working on some of the programs presented in this proposal for several years, without clear results or adequate reflection. WMUK sometimes pursues programs without sufficiently evaluating its impact or adapting its strategies to enable future success; examples include its edit-a-thons and microgrants program. | ” |
The Indian chapter's first foray into a centralised funding application yielded a score of 30, and while the plan used "good contextual data to highlight opportunities", there was valuable advice for future planning. The staff pointed to success in the education (OER) program and the high potential for "significant global impact on Wikimedia projects", including "high potential for global impact", and said that "an environment with many potential new contributors and readers as well as currently active volunteers could enable programmatic success." There was reference to "past successes in contests (Wiki Loves Monuments work) and content donation (NCERT work)". In several respects the grantmaking assessment showed just how valuable the goal of success on the subcontinent is to the Wikimedia movement. "Language communities in India", for example, "are gaining momentum and have some committed volunteers." However, the staff believe that the requested funding is too high and the plan too ambitious for the context of the chapter at this stage. The focus on conferences and other events "may not yield high impact, [with] too large a portion of the budget for operational costs." There were complaints that financial approval was sought on a number of counts after the fact. Of significant concern was that:
| “ | governance and leadership challenges could impact decision-making and execution of plan. [Governance challenges] have implications for its own internal decision-making as well as its relationship with the community, which has been seeking transparency and openness. This may have impact on the overall execution of the plan. | ” |
The FDC will make its recommendations to the WMF’s Board of Trustees on 1 December; the Board will make the final decisions on 1 January.
Discuss this story
I thought the summary chart was particularly useful and raised it for Wikimedia UK member's attention at wmuk:Water cooler#WMF assessment of WMUK's $707,000 bid.
For Wikimedia UK, the FDC process clarified that less than 48% of donated funds go to projects with outcomes in line with the charity's mission, the rest of the funds going on administration such lawyers, employment costs, rent, and local fund-raising.‡. Despite asking several times for the cost of administration to be published by chapters, very few publish anything like these figures, or make them impossibly complex to work out and strongly resist the proposal that this become a hard measurement; so Wikimedia UK was not alone in making this tough to calculate.
Speaking as a past Chapters Association Chair and Wikimedia UK Chair, with a good awareness of the value and organization of chapters, if there is one key improvement I would like to see the FDC put in place, it would be to firmly insist that all chapters submitting proposals make public an easy to understand ratio of "administration : fund-raising : programmes" so that anyone can compare the efficiency of one Chapter with another. There would be obvious benefits to learning from each other, and to judge if our "maturity model" inevitably means increasing employee numbers and indefinitely increasing demands for more funding, or whether organic and (unpaid) volunteer-centric ways of growing Chapters can become a strategic goal.
Competitiveness is good, but sometimes the rating given by staff who doesn't know much is like asking a fish to climb a tree... — Preceding unsigned comment added by 118.97.61.64 (talk) 12:56, 17 November 2013 (UTC)[reply]