Wikipedia talk:Wikipedia Signpost/2012-10-01/Technology report

• Possibly useful links:
  • m:Future of Toolserver
  • Current public understanding summarised by Carl (CBM): «plans for WMF Labs don't seem to include database database replication in a form that makes it a useful, direct replacement for toolserver».

  --Nemo 05:06, 1 October 2012 (UTC)

• Quick note -- saying that "DaB is the volunteer who administers the Toolserver, and who in the process has acquired unique expertise for running the system" might be a little misleading, as Nosy (Marlen Caemmerer) also administers the Toolserver, and is paid by WMDE. DaB certainly has tons of experience and expertise and I do not mean to dispute that! Perhaps you could clarify to say that he co-administers the Toolserver? Sumana Harihareswara, Wikimedia Foundation Engineering Community Manager (talk) 15:36, 2 October 2012 (UTC)

I've added a mention of Marlen. However, she's not mentioned anywhere on the WMF's site, although does come up on a google search in association with "Toolserver". I hope this addresses your concern. Thanks. Tony (talk) 16:03, 2 October 2012 (UTC)

Tony, thank you for your response. Clarification: Marlen works for WMDE, not WMF. I am going to make that correction in the article; hope the Signpost does not mind! Sumana Harihareswara, Wikimedia Foundation Engineering Community Manager (talk) 17:44, 2 October 2012 (UTC)

(Personal comment:) The current version is still somewhat misleading, as it implies that the Toolserver had no paid staff before October 2011. See e.g. http://journal.toolserver.org/ ("River becomes the first paid toolserver admin", February 5, 2010), Wikipedia:Wikipedia_Signpost/2010-02-08/News_and_notes#Briefly or Jarry1250 April 2011 overview article "What is: the Toolserver?".

Also, a casual reader unfamiliar with the topic might take away the mistaken impression that Toolserver performance problems are an entirely new phenomenon. (As a small example from the Signpost itself, encountering Toolserver failure has long been a routine part of the Signpost's publication process.)

Regards, HaeB (talk) 21:06, 2 October 2012 (UTC)

It's not a new phenomenon, but it certainly has become more frequent in recent months. Hersfold non-admin^(t/a/c) 21:58, 2 October 2012 (UTC)

• To respond to Hersfold regarding the estimated time of arrival for database replication on Wikimedia Labs: The WMF goals for Wikimedia Labs specify database replication by the end of March 2013 and Labs lead Ryan Lane aims to get it done sooner, by the end of November or December 2012. Sumana Harihareswara, Wikimedia Foundation Engineering Community Manager (talk) 15:36, 2 October 2012 (UTC)

My impression from the emails on the TS mailing list were that those features wouldn't be functional until December, or mid-2013 at the earliest. This still doesn't address much of the concerns expressed by toolserver users, which I believe hinged on the ability to join the replicated databases with their own user databases. Hersfold non-admin^(t/a/c) 16:16, 2 October 2012 (UTC)

As Sumana said, stated goal for DB replication is Q1 of 2013, and we're looking into whether an earlier roll-out is feasible. So I'm not sure what your impression is based on. :-) Eloquence* 20:15, 2 October 2012 (UTC)

I must have mis-read something or missed an email then. Apologies to Tony and Jarry for the bad info. Hersfold non-admin^(t/a/c) 21:58, 2 October 2012 (UTC)

I have corrected this in the article. Hope that's okay with you, Hersfold, and no worries, I'm sure worse mistakes have been made! :-) Ed ^{[talk] [majestic titan]} 05:28, 3 October 2012 (UTC)

In general, as I noted on toolserver-l, I agree with Carl that we should find ways to support projects like the WP 1.0 assessment DB in Labs. The feature set of the Labs DB replication isn't final, and it's likely going to be iterative.

We'll host an IRC meeting soon that we'll broadcast to toolserver-l@ as well to allow for more discussion of requirements for tool labs (the phase of the labs project dedicated to supporting tools development) and to answer questions about how folks can use Labs today. In the meantime, there are usually folks hanging out on #wikimedia-labs on irc.freenode.net as well in case you have immediate questions.--Eloquence* 20:18, 2 October 2012 (UTC)

Two comments as someone who uses some Toolserver tools heavily in dealing with spam:

1. Whatever the outcome of all this, thank you Wikimedia Deutschland for subsidizing this great capability for the rest of us for so long!
2. I believe the Foundation should fund and support the existing toolserver as long as necessary until Wikilabs is ready to replace it. (I'm also open to not replacing the toolserver -- whatever makes the best sense, I just want the tools)

Thanks also to all the tool developers around the world who've developed these useful tools, too.

--A. B. ^{(talk • contribs)} 20:32, 2 October 2012 (UTC)

I've written several tools that aid maintenance work on Wikipedia, most notably in identifying uncategorized articles and extensive work with disambiguation. If I lose (1) Wikipedia database replication or (2) the ability to join my user database to the replicated database, all of that work is lost. All of it. I know that maintenance work is not glamorous or interesting to most Wikipedians, but it is nevertheless important. I hope that those who are making the decisions about keeping Toolserver viable during the interim and how to set up Wikimedia Labs take into account the role Toolserver plays in maintaining Wikipedia infrastructure. --JaGa^talk 22:37, 2 October 2012 (UTC)

Yep, we hear you, including on the user-DB-to-production-DB join issue. Our main concern is in coming up with an architecture that's reliable and performant, even when users do crazy things ;-). We'll post more details on the DB replication strategy in coming weeks, and as I noted above, will also organize open IRC sessions to dig more into some of the current use cases for tool developers. We'll post updates to toolserver-l.--Eloquence* 23:09, 2 October 2012 (UTC)

• On a Toolserver related question, whatever happened to the Articles Created tool? The one where you could put in a user's name and it would list the articles (with or without redirects) that the user had created? It was really useful and I can't seem to find it anymore. Silverseren^C 23:38, 2 October 2012 (UTC)
  • It works for me (the link is "Articles created" under a user's contributions). Probably the link broke when TParis took over the tool from whoever. David1217 ^{What I've done} 02:21, 3 October 2012 (UTC)

• Possibly the old Toolserver reports will be archived by Wayback Machine.—Wavelength (talk) 02:04, 3 October 2012 (UTC)

The thing which i have little bit hard to understand is that why the Toolserver need to be shutted down at all. The reasoning behind why to create the Labs is pretty solid, but answer for the question why The Labs and the Toolserver cant coexists is not. The key question in this is seems to be the SQL replication to the outside world. If WMF takes it away then there is no future for anything like Toolserver at all. Period. Alternative vision could be that in the future besides the Labs there could be multiple instances of independent [tool]servers working with replicated data. The current TS could be used as prototype for this. Reasoning for independent systems would be that even when the Labs system is fully operational it can't ever be used for everything. One limiting thing is licence policy, one cannot use the closed source in the labs, second is that even the Labs horsepower is considerable it is not unlimited and suitable for everything. One can prefer to use specialized computing for him/her own needs. --Zache (talk) 08:13, 3 October 2012 (UTC)

I have direct, personal experience of the utility of the Toolserver for creating content on the projects (Wikisource, in particular). Whatever the engineering considerations, I'm certainly concerned that the approach taken doesn't seem driven by free content. Does seem "more of the same" with the "cool" stuff. I.e. the cart gets put before the horse. Charles Matthews (talk) 16:30, 3 October 2012 (UTC)

• The Toolserver is an essential tool. If it is underfunded by either WMDE or WMF, I would say those organisations have dropped the ball. This is especially true because those organisations have plenty of money, a lot of which they spend for more and more bureaucratic overhead. They should think hard about maintaining the Toolserver in an adequate way. I would like to point out that the Toolserver and the tools on it are also powerful projects that give reason to participate as donor in the donation campaign. Longbow4u (talk) 04:55, 5 October 2012 (UTC)

• Setting aside any question of how this mess arose or how to fix it, it is clear that once Labs is able to support the tools, there will still be a significant time needed (presumably in some part by the tool owners) to migrate them from Toolserver to Labs (or perhaps some other infrastructure), to test them on the new infrastructure, verify they work at least as well as before, then bring them into general use. It seems braindead obvious that we need the Toolserver to continue functioning until this is accomplished. Killing Toolserver at the end of 2012, months in advance of even starting the port to Labs is absolutely ass-backwards. It should get the necessary support to keep it going until all the tools are moved and working on Labs. Where are the discussions on scheduling these moves? All I've seen so far are vague pronouncements about when Labs will be running, not when the tools will be. So, WMF, be prepared to have a large number of supremely upset users if this doesn't get sorted out. Do what it takes so that WMDE have the means and motivation to keep TS going until it is no longer needed.LeadSongDog come howl! 15:54, 5 October 2012 (UTC)
  • I don't think anyone's suggested killing the Toolserver in 2 months' time. Perhaps you meant the end of 2013 instead? I guess the assumption is that there if things like DB replication arrive on time, most tool owners will have multiple months to migrate (I'm not suggesting that's a long time, just being factual; you can make up your own mind :) ) - Jarry1250 ^{[Deliberation needed]} 16:05, 5 October 2012 (UTC)
    • Well, the thread here seems to say the active support will end 30 December 2012, and realistically forecasts some TBD thing will fail not long after that. LeadSongDog come howl! 16:19, 5 October 2012 (UTC)

Can you explain why you are interpreting the statement "Toolserver will not end early next year. Period. Wikimedia Deutschland will make all necessary investments to keep the Toolserver up and running" as "the active support will end 30 December 2012"? Regards, HaeB (talk) 21:05, 5 October 2012 (UTC)

DaB (main admin of toolserver) said that if toolserver doesn't get proper support (new hardware so that they can handle the growth) then he will resign at end of the year. Pavel answered that Wikimedia Deutschland will make all necessary investments to keep the Toolserver up and running, but it seems that means something like the replacement parts because toolserver is going to be replaced by Labs. This however is not enough to handle current situation of the toolserver. --Zache (talk) 10:20, 8 October 2012 (UTC)