8.3 Unicode Storage Encoding Forms

UTF-8: Universal Transformation Format, 8 bits

A 1-6 bytes per character scheme

0xxxxxxx
110xxxxx	10xxxxxx
1110xxxx	10xxxxxx	10xxxxxx
11110xxx	10xxxxxx	10xxxxxx	10xxxxxx
111110xx	10xxxxxx	10xxxxxx	10xxxxxx	10xxxxxx
1111110x	10xxxxxx	10xxxxxx	10xxxxxx	10xxxxxx	10xxxxxx

• Reduces to ASCII when only the first 128 characters are in use.

UTF-16

• Each character one of the following forms.
  • One 16-bit value: 0000-FFFF, except for D800-DBFF, DC00-DFFF, FEFF, FEFF, FFFF
  • Two 16-bit values: high surrogate D800-DBFF and low surrogate DC00-DFFF

    UTF-16	ordinal
    (x,y)	((x - D800) * 400 + (y - DC00)) + 00010000
    (x,y)	v
    x=(v - 0001 0000) / 400) + D800
    y=(v - 0001 0000) % 400) + DC00

• Big-endian encoding: most significant byte is accessed first
• Little-endian encoding: least significant byte is accessed first
• A byte order mark (BOM) is offered for indicating the order: big-endian FEFF, little-edian FFFF.
• The BOM should appear at the start of the data (before the first encoded character)
• The encoding schemes UTF-16LE and UTF-16BE assume implicit BOMs
• The high and low surrogate fields each covers 1024 combinations.

0079 007A D800 DC00	big edian
7900 7A00 00D8 00DC	little edian
FEFF 0079 007A D800 DC00	big edian with BOM
FFFF 7900 7A00 00D8 00DC	little edian with BOM

UTF-32

A variant of UTF-16 uniformly employing 32 bits per character

Universal Character Sets: UCS-2, USC-4

A sequence of 2 or 4 bytes per character, holding the character code point.

• UTF-16 reduces to UCS-2 when the high surrogate is not in use
• UTF-32 provides a subset of UCS-2