Unicode -- 从code point到UTF16的计算方法

UTF16，即是通常所说的Unicode。其实把UTF16叫成Unicode不太合适，容易给人造成混乱。因为Unicode是字符集，而不是实际的存储编码方案。

UTF16是 变长编码方案。

比如Unicode code point为2F92B的字，把它保存成UTF16（也就是Windows XP记事本中的Unicode），就变成了FC D8 2B DD，如果是 Big endian的话就应该是D8 FC DD 2B。这个值是怎么来的？

对于 0-FFFF的Unicode字符，UTF16中用一个两个字节的Unicode code point直接表示。对于 10000-10FFFF的Unicode字符，UTF16中用surrogate pair表示，既用两个字符表示，它们之间的转换过程是：

下面把code point为U+64321(十六进制)的Unicode字符编码成UTF-16，由于它大于U+FFFF，所以它要编码成surrogate pair：

v  = 0x64321
v′ = v - 0x10000
   = 0x54321
   = 0101 0100 0011 0010 0001

vh = 0101010000 // higher 10 bits of v′
vl = 1100100001 // lower  10 bits of v′
w1 = 0xD800 // the resulting 1st word is initialized with the high bits
w2 = 0xDC00 // the resulting 2nd word is initialized with the low bits

w1 = w1 | vh
   = 1101 1000 0000 0000 |
            01 0101 0000
   = 1101 1001 0101 0000
   = 0xD950

w2 = w2 | vl
   = 1101 1100 0000 0000 |
            11 0010 0001
   = 1101 1111 0010 0001
   = 0xDF21

详细描述：

The improvement that UTF-16 made over UCS-2 is its ability to encode characters in planes 1–16, not just those in plane 0 (BMP).

UTF-16 represents non-BMP characters (those from U+10000 through U+10FFFF) using a pair of 16-bit words, known as a surrogate pair. First 10000 ₁₆ is subtracted from the code point to give a 20-bit value. This is then split into two separate 10-bit values each of which is represented as a surrogate with the most significant half placed in the first surrogate. To allow safe use of simple word-oriented string processing, separate ranges of values are used for the two surrogates: 0xD800–0xDBFF for the first, most significant surrogate and 0xDC00-0xDFFF for the second, least significant surrogate.

For example, the character at code point U+10000 becomes the code unit sequence 0xD800 0xDC00, and the character at U+10FFFD, the upper limit of Unicode, becomes the sequence 0xDBFF 0xDFFD. Unicode and ISO/IEC 10646 do not, and will never, assign characters to any of the code points in the U+D800–U+DFFF range, so an individual code value from a surrogate pair does not ever represent a character.



我们可以用Windows自带的计算器的科学计算模式完成上述计算，当然也可以自己写个小程序：）
要输入10000-10FFFF的字符，可以使用微软拼音输入法。它有一项以Unicode码输入字符的功能。
要显示这些字符中的汉字部分，可以安装Unifont，参见海峰五笔的网站。

关于编码知识，可以google一下这一系列文章，写的非常精彩：“Java中的字符集编码入门”

一点问题：
.net framework平台下，string类型变量name包含两个字符，一个是0-FFFF的字符，另一个是10000-10FFFF的字符，那么name的长度将是3而不是2，因为name有6个字节。

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;

namespace CodePoint2UTF16
{
         public partial class Form1 : Form
        {
                 public Form1()
                {
                        InitializeComponent();
                }

                 private void btnConvert_Click( object sender, EventArgs e)
                {
                        String cp = tbUnicodeCodePoint.Text.Trim();

                         try
                        {
                                 int n = Convert.ToInt32(cp, 16);
                                 if (n < 0 || n > 0x10FFFF)
                                {
                                        MessageBox.Show(cp + " is not in 0x0 - 0x10FFFF");
                                         return;
                                }
                                 if (n < 0x10000)
                                {
                                        tbUTF16Code.Text = Convert.ToString(n, 16);
                                         return;
                                }
                                 else
                                {
                                        n -= 0x10000;
                                         int h = n >> 10;
                                         int l = n & 0x3FF;
                                        h |= 0xD800;
                                        l |= 0xDC00;
                                        tbUTF16Code.Text = Convert.ToString(h, 16) + " " + Convert.ToString(l, 16);
                                }
                        }
                         catch (Exception ex)
                        {
                                MessageBox.Show( "Invalid text: " + cp + Environment.NewLine + ex.Message);
                        }
                }
        }
}

转载于:https://blog.51cto.com/h2appy/144639