Wikipedia:Bots/Requests for approval/HBC Archive Indexerbot

The following discussion is an archived debate. Please do not modify it. Subsequent comments should be made in a new section. The result of the discussion was

Approved.

HighInBCBot edit

Operator: HighInBC

Automatic or Manually Assisted:

Automatic

Programming Language(s):

Perl with the MediaWiki module

Function Summary:

It will go through all of the archives of a particular user, take all of the section headings, and index them on one page.

Edit period(s) (e.g. Continuous, daily, one time run):

Once daily, non peak times

Edit rate requested: X edits per TIME

For each user benefiting from this bot, this bot will use special:export to read all archives at once, and write one time to the index page. I consider this a negligible use of bandwidth for the service provided.

No more than 3 writes per minute, reading potentially more, most likely through the special:export command.

This bot will use it's watchlist to coordinate a cache, checking all of the archives it serves for changes in a single http request. It will only read altered archives after the initial reading. see User:HighInBCBot#Method_of_reading for details.

Already has a bot flag (Y/N):

No

Function Details:

Will go through all of a users archives, take all of the section headings, and index them on one page.

The idea being that somebody looking for a specific section from an old link(linking to a section no longer on my userapge) will see a notice saying

If you are looking for a section that has already been archived you can look for it in the index of archived sections

This index will have every section title linked to it's location in it's archive. If the same section title is on more than one archive title then it will be given a link for each archive.

Example

Atomic power - Archive 1
Irish - Archive 2
RfA thanks - Archive 1 (2 sections with this name in Archive 1)
RfA thanks - Archive 2
thanks - Archive 3
Welcome! - Archive 1

This can also create a second index by username on the same page. This will not effect how many times it must read or write.

Any user can benefit from this. Parameters would include a mask for the archives in printf format, and the target location for the result of the indexing. The user can add themselves to a category and define their parameters in HTML comments like so:

{{User:HighInBCBot/Archive_index_opt_in}}<!-- ARCHIVES=User:HighInBC/Archive%d TARGET=User:HighInBC/Archive_index -->

This bot would only write to a page with  or something similiar somewhere on it as it will be blanking and replacing the target page each write, this will prevent abuse as ordering the bot to blank the page involves making an edit people on that page can see.

I for one am always looking through the Wikipedia:AN archives trying to find a specific item, an index of that would be usefull. More detials and progress reports on User:HighInBCBot. HighInBC ^{(Need help? Ask me)} 15:30, 9 December 2006 (UTC)[reply]

Discussion edit

Note I have not written this yet, but I feel I can in within a couple of days. I am currently working on the read only portion. HighInBC ^{(Need help? Ask me)} 16:35, 9 December 2006 (UTC)[reply]
Oh yes, I am willing to release the source to GFDL for all to see. HighInBC ^{(Need help? Ask me)} 16:48, 9 December 2006 (UTC)[reply]

I'm not sure what I think about this bot. Would it be possible to create a mockup? It doesn't have to be done with a bot, but I'd like to see an example diff or two of this. -- RM 23:57, 11 December 2006 (UTC)[reply]

Sure, I will create a sample of what is will do later today. HighInBC ^{(Need help? Ask me)} 15:39, 12 December 2006 (UTC)[reply]

Here is a rough example of what I had in mind: User talk:HighInBC/archive_index(diff). I will of course format it better. This is all done in one posting. HighInBC ^{(Need help? Ask me)} 19:27, 12 December 2006 (UTC)[reply]

this looks like a wonderfull idea it has my full support. I too have come across the same annoying issue of searching the archives. But how would it handle achives with differnt nameing schemes? I look forward to seeing the trials of this bot. Betacommand ^{(talk • contribs • Bot)} 19:58, 13 December 2006 (UTC)[reply]

As for archives with different naming schemes it will just need a different mask. As long as it uses numbers then it can be described in the printf format. Here is another, larger, implementation of this idea: User talk:HighInBC/ANI_archive_index. For archive sets this large it may split the index into alphabetical sections.

Some advice is requested. I need for convert headings(what is between the =='s) into link anchors. I have made a set of regex's that succeed 90% of the time but fails in special cases. Does anyone know the correct way to convert headings to links? I am not downloading the HTML, but instead am downloading several pages at once in XML format from the special:export command, so I only have source to work with. Also, is there a way to add many pages to your watchlist at once instead of one at a time? HighInBC ^{(Need help? Ask me)} 20:23, 13 December 2006 (UTC)[reply]

one of to ways can create links to titles:

<code>
[[Pagename#section title]]

Betacommand ^{(talk • contribs • Bot)} 22:23, 13 December 2006 (UTC)[reply]

I have explained myself poorly, when taking a section heading it often must be modified before being used as a link. For example: ==Blocked IP still editing?!==

Becomes: #Blocked_IP_still_editing.3F.21

and ==Blocked [[User:Rms125a@hotmail.com]] back in sockpuppet form==

Becomes: #Blocked_User:Rms125a.40hotmail.com_back_in_sockpuppet_form

I can keep adding rules as I find improperly constructed links, but if there is a place where the rules for this type of parsing are already described, that would help. HighInBC ^{(Need help? Ask me)} 22:33, 13 December 2006 (UTC)[reply]

Just FYI, we have recently created a (hopefully) standardized bot opt-in/opt-out interface. See the Template:bots for how this works. -- RM 01:01, 20 December 2006 (UTC)[reply]

What i'm assuming the anchor encoding scheme is: Urlencode everything but colon, forward slash, [[ and ]]. Encode all spaces to underscores. Replace the %(percent) from the urlencode to a .(dot), then replace all [[foo|bar]] and [[bar]] with bar, which a simple regex can take care of. GeorgeMoney (talk) 18:56, 25 December 2006 (UTC)[reply]

Will use that standard, handy. HighInBC ^{(Need help? Ask me)} 01:06, 20 December 2006 (UTC)[reply]

Pretty much done edit

The bot is pretty much done. Examples of it's output can be seen here User talk:HighInBC/Archive index and here User:HighInBCBot/AN index. The bot reads from here. Details are passed to the bot by adding {{User:HighInBCBot/OptIn|target=User talk:HighInBC/Archive index|mask=User_talk:HighInBC/Archive_<#>|leading_zeros=0}} to a page.

The bot will not write to a page that does not contain  near the top of the page. It currently writes to a text file for me to add manually, but once I am approved, for testing at least I can have it actually write to the page. HighInBC ^{(Need help? Ask me)} 01:41, 29 December 2006 (UTC)[reply]

So, the bots been done for a few days now, you folks need any more info? How do we proceed? HighInBC ^{(Need help? Ask me)} 16:39, 1 January 2007 (UTC)[reply]

Bot trial run approved for the duration of one week. Betacommand ^{(talk • contribs • Bot)} 17:26, 1 January 2007 (UTC)[reply]

Results of testing edit

The script ran as it should have for the full week. Logs are here: [1] and contributions here: [2].

Caching works very well, it only downloads what it needs to, and only uploads report when there has been a change. Any further changes I plan to make will be aesthetic ones such as reformatting the indexes and fixing badly parsed links. HighInBC ^{(Need help? Ask me)} 00:01, 8 January 2007 (UTC)[reply]

Approved. Sorry for the delay, although I think that you pretty much just went ahead running the bot :-) This bot does not need a flag. —Mets501 (talk) 21:16, 25 January 2007 (UTC)[reply]

Note: Flagged following later discussion about the increasing number of edits being made by this Bot [3]. WjBscribe 17:47, 10 January 2008 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. Subsequent comments should be made in a new section.