Character Inspector and Code Table

A reader sees a letter. A computer stores a number. Most of what goes strangely wrong with text begins in the gap between those two facts: the password that fails after a copy and paste, the search that cannot find a word plainly on the screen, the file name that seems to exist twice.

The Unicode Standard calls the number a code point. Its codespace runs from 0 to 10FFFF in hexadecimal, which is 1,114,112 code points, and each is written as U+ followed by hexadecimal digits: U+0041 is the capital letter A, U+20AC is the euro sign. A code point is not a shape and it is not a byte. The shape belongs to a font. The bytes belong to an encoding, a rule for writing the number down. Keeping those three ideas apart is most of the subject.

The older answer was ASCII. In October 1969 Vint Cerf’s RFC 20 proposed it for network interchange as a 7-bit code carried in an 8-bit byte whose high bit is always zero. Seven bits give 128 positions. Thirty-three of them are control codes with names like NUL, ESC and DEL, and several describe the mechanics of a printer: backspace moves the printing position one space backward, and BEL is a character for use when there is a need to call for human attention. The other ninety-five are the space, the digits, and the letters and punctuation of American English.

That is enough for English and for very little else.

Unicode was the attempt to give every script one shared numbering. According to the consortium’s own account it was first designed as a pure 16-bit encoding, and it became clear over time that sixteen bits were not sufficient. The codespace grew to its present size, and a block from U+D800 to U+DFFF was set aside for a trick described below. Those 2,048 surrogate code points can never be characters. What remains is the set of scalar values: 1,112,064 numbers that can each stand for a character and be written in any of the standard encodings.

How UTF-8 packs a number into bytes

UTF-8 writes a scalar value in one to four bytes, and RFC 3629 gives the whole scheme in four lines. Values up to 7F take one byte of the form 0xxxxxxx. Values up to 7FF take two: 110xxxxx 10xxxxxx. Values up to FFFF take three: 1110xxxx 10xxxxxx 10xxxxxx. Everything above takes four, beginning 11110xxx. The x positions carry the number. The fixed bits in front are markers, and they are the clever part: the count of leading ones in the first byte says how long the sequence is, and every later byte announces itself as a continuation by starting with 10.

Take the euro sign. U+20AC in binary is 10000010101100, fourteen bits, too many for the eleven that two bytes can carry. The three-byte form has sixteen payload positions, so the number is padded to 0010000010101100 and cut into groups of four, six and six: 0010, 000010, 101100. Put the markers in front and the bytes are 11100010, 10000010 and 10101100. In hexadecimal that is E2 82 AC. Paste a euro sign into the inspector above and the 3D view shows these three rows, with the marker bits as round blocks and the payload bits running down into the code point.

The design has consequences that explain its success. A plain ASCII string is already a valid UTF-8 string, byte for byte. A byte with an ASCII value can only stand for that ASCII character and nothing else, so software that searches for a slash or a quotation mark keeps working on text it does not otherwise understand. Character boundaries are easily found from anywhere in a stream of bytes, because a continuation byte can never be mistaken for a first byte.

That last property was the reason UTF-8 exists in its present form. Wikipedia’s history of the format records that Ken Thompson outlined the design on a placemat in a New Jersey diner, with Rob Pike, on September 2, 1992, as a change that made an earlier proposal self-synchronizing. In the following days the two built it into their operating system, Plan 9, and used it throughout. Three decades later W3Techs reports that UTF-8 is used by 99.1% of all the websites whose character encoding it knows, and the WHATWG Encoding Standard requires it for new protocols and formats.

A decoder has to be as strict as the table. The bytes C0 80 follow the two-byte pattern but spell the value zero, which already has a one-byte form. RFC 3629 warns that a naive implementation may decode such an overlong sequence, with security consequences, and the Unicode Standard lists every such sequence as ill-formed.

UTF-16, surrogates and the four lengths

UTF-16 uses 16-bit units instead of bytes. A value below 10000 is one unit. For anything larger, RFC 2781 says to subtract 10000, which leaves twenty bits, and to add the high ten to D800 and the low ten to DC00. That is the purpose of the reserved block: the grinning face U+1F600 becomes the pair D83D DE00, and neither half can be confused with an ordinary character. In UTF-8 the same face is F0 9F 98 80.

JavaScript strings are sequences of these units, and the length of a string is the number of units, not the number of characters. A unit left without its partner is an unpaired surrogate. It has no UTF-8 form at all, and the Encoding Standard has browsers replace it with U+FFFD.

So how long is a string? There are four honest answers. A reader counts what Unicode’s segmentation annex calls grapheme clusters, the user-perceived characters. A program may count code points, UTF-16 units or UTF-8 bytes. A family emoji of a man, a woman and a girl is three pictures joined by two copies of the zero width joiner, U+200D. It is one cluster, 5 code points, 8 UTF-16 units and 18 bytes. A flag is a pair of regional indicator characters: one cluster, 2 code points, 4 units, 8 bytes.

Which answer is right depends on who asks. A storage limit is in bytes. NIST’s password guidance counts each code point as a single character. A person counts clusters and is surprised by everything else.

One accent, two spellings

The letter e with an acute accent can be written as the single code point U+00E9, or as a plain e followed by the combining acute accent U+0301. Unicode calls the two canonically equivalent: they represent the same abstract character and should always look and behave the same. Their bytes differ. The first is C3 A9; the second is 65 CC 81. Type the word cafe with its accent both ways and one version is 4 code points and 5 bytes, the other 5 code points and 6 bytes. Both are four clusters. A byte comparison says they are different words.

Normalization exists to settle this. NFC composes wherever it can; NFD decomposes; NFKC and NFKD go further and also replace compatibility characters, so that the single-character fi ligature becomes an f and an i. The Unicode FAQ calls NFC the best form for general text and NFKC the preferred form for identifiers. It also notes that Apple’s old HFS+ file system stored file names in a variant of NFD and that the macOS Finder still decomposes them, which is one way decomposed text arrives on machines that expected the other kind.

Passwords are where this bites. NIST recommends that a verifier accepting Unicode normalize with NFKC or NFKD before hashing, and warns that some characters may be represented differently by some endpoints, which can affect the ability to authenticate.

The characters nobody sees

Invisible characters were not put into Unicode to cause trouble. The zero width space, U+200B, marks a place where a line may break in languages written without spaces between words, such as Thai, Khmer and Japanese. The no-break space, U+00A0, has the width of a space and forbids a break. The zero width joiner and non-joiner ask for more or less connection between neighboring letters, which the standard says is sometimes required in writing Persian, and the joiner also holds emoji sequences together. U+FEFF at the start of a UTF-8 file is an encoding signature, the three bytes EF BB BF.

They travel because ordinary tools produce them. The LibreOffice help gives one key combination for a non-breaking space and another for a soft hyphen. Every family emoji sent in a chat carries its joiners. Copy and paste then moves code points faithfully, seen and unseen alike. The word password with a zero width space in the middle looks like the word password; it is 9 code points and 11 bytes, and to a login form, a search index or a compiler it is a different string.

Direction controls are the sharpest case. The bidirectional algorithm defines U+202E, the right-to-left override, as forcing the characters after it to be treated as strong right-to-left characters. A name stored as report_, the override, then fdp.txt is displayed as report_txt.pdf. Boucher and Anderson’s Trojan Source paper applied the same idea to program code that appears different to a compiler and to the human eye; two CVE numbers were issued, one for the bidirectional attack and one for a homoglyph attack.

Homoglyphs need no invisible character at all. The Cyrillic letter U+0430 is drawn like the Latin letter a and encodes as D0 B0 instead of 61. Unicode’s security mechanisms define a string as mixed-script when no single script covers all of its characters, with allowances for writing systems such as Japanese that legitimately combine several.

What an inspector can and cannot tell you

Reading the code points settles what a text contains. It does not settle what the text means. A non-joiner inside a Persian word is correct spelling, and a no-break space before a unit is good typography; removing either makes a text worse, which is why a cleaning step should produce a copy and say what it changed. A mixed-script check finds a Cyrillic letter inside a Latin word but not a whole word written in lookalike letters. Category and script answers come from the Unicode data built into the browser, so a character newer than that data is reported as unassigned. And no list of numbers can show how another device’s fonts will draw them.

The habit worth keeping is small. When text misbehaves, stop looking at it and count it.