User:PerfektesChaos/js/WikiSyntaxTextMod/flow/char

WikiSyntaxTextModSyntax polishing → Step 1

Single characters

The first step on the syntax polishing lane standardizes on single character level. This is prerequisite for later analysis by this script, by bots and other gadgets, in order to detect strings in expected manner.

Current implementation

edit

1–31 ASCII control codes

edit
10 – newline
The only appreciated control code is newline.
All other control codes should be removed.
9 – horizontal tab
The horizontal tabulator is replaced by exactly one space character.
This has the same effect when rendering wikitext.
Tabs are often introduced by copy&paste, when existing rendered texts (mainly enumerations and tables) are inserted into Wikipedia. Strange effects occur when editing in TEXTAREA.
The only eception is within syntaxhighlight where the tabs remain. Therefore the occurrence of a tab is only prebooked. If any tab was seen, cleanup is postponed until finished tag analysis and syntaxhighlight areas have been protected.
11 – ver tab, 12 – form feed
It seems that the database rejects these codes if attempting to save. It is not possible to enter them by normal keystrokes, but they might result from copy&paste.
These characters indicate a separation between two sections. If really found in edited text, they are replaced by newline.
Any other code ≤ 31
As before; neither insertable, nor stored in database.
If occurring nevertheless they will be removed.

Unicode control codes

edit
Writing left-to-right
The invisible l-t-r and r-t-l (200E16 and 200F16) might be very meaningful for interlanguage and pages with content in other languages. Basically they are kept.
All are consistently made visible as ‎ and ‏ entities.
If these codes are found
they will be removed programmatically.
Templates for support of foreign languages are supposed to handle the bi-directional issues internally. In the plain article text the entities may be removed without any consequence then.
Zero width characters
These are currently 200B16 (ZERO WIDTH SPACE), 200C16 (ZERO WIDTH NON-JOINER) and 200D16 (ZERO WIDTH JOINER) and they are made visible as ​, ‌ and ‍ homogeneously.
In non latin scripts they have a certain meaning. In latin text they are pointless; usually resulting from copy&paste insertions, perhaps database queries, and may be deleted. They are regularly occurring in interlanguage links and are mandatory there; within links they remain unchanged. For the sake of undisturbed diffpage all entities are turned into character codes within link targets later.
Line Separator
In Unicode there is a “line separator” 202816, which is not visible in normal browser displays. This particular line break is not meaningful for a wiki and will be replaced by a regular newline character. Within link targets the code is ignored by the parser anyway and will be removed. The character is generated by copy&paste insertions, when a mostly “soft” line break has been inserted into the displayed text by the rendering system.

Unicode typography

edit
Space
Simple spaces (ASCII) are replacing:
  • 200216 – N-SPACE
  • 200316 – M-SPACE
Non-breaking hyphen
This character (201116 = 820910 – NON-BREAKING HYPHEN) might be helpful for future rendering of web sites with improved automatic syllabification sinnvoll sein; but searching for strings is obstructed.
Directly before or after a (regular) space or a line break (\n as well as <br />) it is always pointless. There it is always turned into by ASCII hyphen.
In some cases where no syllabification is possible the non-breaking hyphen is removed, if never coming into effect.
A user defined change into ASCII hyphen or removal is left to the user.

Exotic characters

edit
  • Instead of regular characters sometimes decorative variants from dingbats are used. Typically the author is not aware of the difference and was not heading for that. Those codes will break search processes and scripts/bots, and they are not defined in many fonts yet and won’t be rendered.
    • Instead of dagger † (822410, dagger, might mean “died”) one of three representations 10013–1001510 (271D–271F16); those will be replaced.
    • Instead of times cross × (21510, times) one of 10005/1000610 (2715/271616); those will be replaced.
  • The (male) number sign º \xBA &ordm; looks very similar to the degree sign ° \xB0 &deg;. Digits ahead indicate more a degree sign, and a temperature (°C) clearly requires a degree sign. If unambiguous the character ist corrected automatically.
  • Some authors use roman numbers 8544–857510 (216016 –217F16) rather than a sequence of ASCII letters. However, this is often unavailable to user fonts and not expected on string search. Out of protected regions this will be replaced by common ASCII letters. In context of CJK that Unicode solution gives better typographic results; if such CJK character is next replacement is inhibited.
  • CJK glyphs with latin equivalent between latin characters (mostly ASCII) are reduced to ASCII characters. This affects the ranges U+3000-300D and U+FF01-U+FF5E.

Invisible characters

edit
  • Some source texts contain hidden characters from copy&paste activities in normal TEXTAREA. They are made visible as entities by the script as &lrm; or &zwj; and could be deleted now manually (and should be discarded[1] – see also bugzilla:29794). If the inserted entity remains in source code, that does no harm since the source in effect has not been changed and rendering keeps the same appearance. If that character is placed at end or begin of alink target and known to be redundant, it will be removed.
  • In general invisible characters for control of writing direction (left/right) are made visible as hexadecimal entities, in order to remove superfluous ones. The same goes for the “zero width space” U+200B (ZERO WIDTH SPACE) etc.:
Invisible characters, transformed into entities
Hexcode decimal named meaning remark
00A0 160 nbsp non-breaking space &#160; if found as code in text; see below.
034F 847 COMBINING GRAPHEME JOINER
2009 8201 thinsp thin space
200A 8202 HAIR SPACE
200B 8203 ZERO WIDTH SPACE
200C 8204 zwnj ZERO WIDTH NON-JOINER see above
200D 8205 zwj ZERO WIDTH JOINER
200E 8206 lrm LEFT-TO-RIGHT MARK see above
200F 8207 rlm RIGHT-TO-LEFT MARK
202A 8234 LEFT-TO-RIGHT EMBEDDING
202B 8235 RIGHT-TO-LEFT EMBEDDING
202C 8236 POP DIRECTIONAL FORMATTING
202D 8237 LEFT-TO-RIGHT OVERRIDE
202E 8238 RIGHT-TO-LEFT OVERRIDE
202F 8239 NARROW NO-BREAK SPACE
2060 8288 WORD JOINER

Zero width characters

edit

Some non-latin scripts use invisible “zero width” characters. Within link targets (mostly interlanguage) or recognized text sequences (especially the entire page) they areencoded invisible; else visualized as entity.

Entity Lanaguages
&zwj; kn ml sa si
&zwnj; bn fa kn ml mr mzn te
&#x200A; bo km

Spaces at line end

edit
  1. If a line consists of spaces only, all these spaces are removed always.
    This avoids that searching expressions for \n\n fail.
  2. If for any other reason it happens to change the text, on termination of automatic processing all spaces invisible at end of line will be removed.
    • If this would be the only change, that is dispensed if control diffpage is active as common, otherwise the user would be confused by solely invisible modifications.
    • If the line end is terminated by one single equal sign, exactly one from existing space characters is kept. Some people prefer this for not yet assigned template parameters.

Character entities

edit
  • Since a character entity could be protected by nowiki or syntaxhighlight (source) or may be used within a comment as plain text, entities are replaced by UCS not during this first step, but after identification of protected areas (syntax readability).

Kept characters

edit

It may be discussed whether some character codes are meaningful for a wiki project. The script does not replace them by default, but they might be changed or removed by user defined rules.

U+00A0 – non-breaking space
Since a few years this code (not only as entity &nbsp;) is possible in wikitext. In the first years the database changed it into a normal space.
Currently they are made visible as &#160; and might be judged for intended usage.
U+00AD – soft hyphen
Sometimes AD16 = 17310 is present in wikitext, mainly if entire text has been imported from other text systems. In future it might improve the layout of websites with automatic syllabification.
Searching for strings is obstructed. If no wikilink adheres, the character can be removed without any consequences by user defined rules or replaced by &shy; entity.
U+2010 – hyphen
The Unicode hyphen 820810 is generated by text systems when automatic syllabification breaks a line. It is unlikely that they result from manual input, probably they result from copy&paste of an entire word which has been separated. In wikitext it might be assumed that the parts are to be joined again into the single word. Unicode hyphen and an adjacent space are to remove or replace by soft hyphen, but this requires human control.

127–159 (Windows code pages)

edit
  • This range is not defined in ANSI/UCS.
  • Such character codes appear sometimes in wikitext, but are invisible for many authors.
  • The script makes them visible as decimal entities.
  • In latin based wikipedia it might be assumed that codes 128–159 shall be interpreted according to codepage 1252. User defined replacements might help if a frequent problem.

Remarks

edit
  1. ^ In common TEXTAREA without WikEd the browser could miscount the character position. If such a character »|« is present in source text, e.g. between A and B: A|BCEF, and the cursor between C and E, than it might happen that the intended ‘d’ is inserted at wrong position: A|BdCEF or A|BCEdF instead of ABCdEF.

[ German page ]