asfman
android developer
posts - 90,  comments - 213,  trackbacks - 0

1. Codepage的定义和历史

字符内码(charcter code)指的是用来代表字符的内码.读者在输入和存储文档时都要使用内码,内码分为

  • 单字节内码 -- Single-Byte character sets (SBCS),可以支持256个字符编码.
  • 双字节内码 -- Double-Byte character sets)(DBCS),可以支持65000个字符编码.主要用来对大字符集的东方文字进行编码.
codepage 指的是一个经过挑选的以特定顺序排列的字符内码列表,对于早期的单字节内码的语种,codepage中的内码顺序使得系统可以按照此列表来根据键盘的输入值给出一个对应的内码.对于双字节内码,则给出的是MultiByte到Unicode的对应表,这样就可以把以Unicode形式存放的字符转化为相应的字符内码,或者反之,在Linux核心中对应的函数就是utf8_mbtowc和utf8_wctomb.

在1980年前,仍然没有任何国际标准如ISO-8859或Unicode来定义如何扩展US-ASCII编码以便非英语国家的用户使用.很多IT 厂商发明了他们自己的编码,并且使用了难以记忆的数目来标识:

例如936代表简体中文. 950代表繁体中文.

1.1 CJK Codepage

同 Extended Unix Coding ( EUC )编码大不一样的是,下面所有的远东 codepage 都利用了C1控制码 { =80..=9F } 做为首字节, 使用ASCII值 { =40..=7E { 做为第二字节,这样才能包含多达数万个双字节字符,这表明在这种编码之中小于3F的ASCII值不一定代表ASCII字符.

CP932

Shift-JIS包含日本语 charset JIS X 0201 (每个字符一个字节) 和 JIS X 0208 (每个字符两个字节),所以 JIS X 0201平假名包含一个字节半宽的字符,其剩馀的60个字节被用做7076个汉字以及648个其他全宽字符的首字节.同EUC-JP编码区别的是, Shift-JIS没有包含JIS X 202中定义的5802个汉字.

CP936

GBK 扩展了 EUC-CN 编码( GB 2312-80编码,包含 6763 个汉字)到Unicode (GB13000.1-93)中定义的20902个汉字,中国大陆使用的是简体中文zh_CN.

CP949

UnifiedHangul (UHC) 是韩文 EUC-KR 编码(KS C 5601-1992 编码,包括2350 韩文音节和 4888 个汉字a)的超集,包含 8822个附加的韩文音节( 在C1中 )

CP950

是代替EUC-TW (CNS 11643-1992)的 Big5 编码(13072 繁体 zh_TW 中文字) 繁体中文,这些定义都在Ken Lunde的 CJK.INF中或者 Unicode 编码表中找到.

注意: Microsoft采用以上四种Codepage,因此要访问Microsoft的文件系统时必需采用上面的Codepage .

1.2 IBM的远东语言Codepage

IBM的Codepage分为SBCS和DBCS两种:

IBM SBCS Codepage

  • 37 (英文) *
  • 290 (日文) *
  • 833 (韩文) *
  • 836 (简体中文) *
  • 891 (韩文)
  • 897 (日文)
  • 903 (简体中文)
  • 904 (繁体中文)

    IBM DBCS Codepage

  • 300 (日文) *
  • 301 (日文)
  • 834 (韩文) *
  • 835 (繁体中文) *
  • 837 (简体中文) *
  • 926 (韩文)
  • 927 (繁体中文)
  • 928 (简体中文)

    将SBCS的Codepage和DBCS的Codepage混合起来就成为: IBM MBCS Codepage

  • 930 (日文) (Codepage 300 加 290) *
  • 932 (日文) (Codepage 301 加 897)
  • 933 (韩文) (Codepage 834 加 833) *
  • 934 (韩文) (Codepage 926 加 891)
  • 938 (繁体中文) (Codepage 927 加 904)
  • 936 (简体中文) (Codepage 928 加 903)
  • 5031 (简体中文) (Codepage 837 加 836) *
  • 5033 (繁体中文) (Codepage 835 加 37) *

    *代表采用EBCDIC编码格式

    由此可见,Mircosoft的CJK Codepage来源于IBM的Codepage.

    2. Linux下Codepage的作用

    在Linux下引入对Codepage的支持主要是为了访问FAT/VFAT/FAT32/NTFS/NCPFS等文件系统下的多语种文件名的问题,目前在NTFS和FAT32/VFAT下的文件系统上都使用了Unicode,这就需要系统在读取这些文件名时动态将其转换为相应的语言编码.因此引入了NLS支持.其相应的程序文件在/usr/src/linux/fs/nls下:

    • Config.in
    • Makefile
    • nls_base.c
    • nls_cp437.c
    • nls_cp737.c
    • nls_cp775.c
    • nls_cp850.c
    • nls_cp852.c
    • nls_cp855.c
    • nls_cp857.c
    • nls_cp860.c
    • nls_cp861.c
    • nls_cp862.c
    • nls_cp863.c
    • nls_cp864.c
    • nls_cp865.c
    • nls_cp866.c
    • nls_cp869.c
    • nls_cp874.c
    • nls_cp936.c
    • nls_cp950.c
    • nls_iso8859-1.c
    • nls_iso8859-15.c
    • nls_iso8859-2.c
    • nls_iso8859-3.c
    • nls_iso8859-4.c
    • nls_iso8859-5.c
    • nls_iso8859-6.c
    • nls_iso8859-7.c
    • nls_iso8859-8.c
    • nls_iso8859-9.c
    • nls_koi8-r.c

    实现了下列函数:

    • extern int utf8_mbtowc(__u16 *, const __u8 *, int);
    • extern int utf8_mbstowcs(__u16 *, const __u8 *, int);
    • extern int utf8_wctomb(__u8 *, __u16, int);
    • extern int utf8_wcstombs(__u8 *, const __u16 *, int);

    这样在加载相应的文件系统时就可以用下面的参数来设置Codepage:

    对于Codepage 437 来说

    mount -t vfat /dev/hda1 /mnt/1 -o codepage=437,iocharset=cp437

    这样在Linux下就可以正常访问不同语种的长文件名了.

    3. Linux下支持的Codepage

  • nls codepage 437 -- 美国/加拿大英语

  • nls codepage 737 -- 希腊语

  • nls codepage 775 -- 波罗的海语

  • nls codepage 850 -- 包括西欧语种(德语,西班牙语,意大利语)中的一些字符

  • nls codepage 852 -- Latin 2 包括中东欧语种(阿尔巴尼亚语,克罗地亚语,捷克语,英语,芬兰语,匈牙利语,爱尔兰语,德语,波兰语,罗马利亚语,塞尔维亚语,斯洛伐克语,斯洛文尼亚语,Sorbian语)

  • nls codepage 855 -- 斯拉夫语

  • nls codepage 857 -- 土耳其语

  • nls codepage 860 -- 葡萄牙语

  • nls codepage 861 -- 冰岛语

  • nls codepage 862 -- 希伯来语

  • nls codepage 863 -- 加拿大语

  • nls codepage 864 -- 阿拉伯语

  • nls codepage 865 -- 日尔曼语系

  • nls codepage 866 -- 斯拉夫语/俄语

  • nls codepage 869 -- 希腊语(2)

  • nls codepage 874 -- 泰语

  • nls codepage 936 -- 简体中文GBK

  • nls codepage 950 -- 繁体中文Big5

  • nls iso8859-1 -- 西欧语系(阿尔巴尼亚语,西班牙加泰罗尼亚语,丹麦语,荷兰语,英语,Faeroese语,芬兰语,法语,德语,加里西亚语,爱尔兰语,冰岛语,意大利语,挪威语,葡萄牙语,瑞士语.)这同时适用于美国英语.

  • nls iso8859-2 -- Latin 2 字符集,斯拉夫/中欧语系(捷克语,德语,匈牙利语,波兰语,罗马尼亚语,克罗地亚语,斯洛伐克语,斯洛文尼亚语)

  • nls iso8859-3 -- Latin 3 字符集, (世界语,加里西亚语,马耳他语,土耳其语)

  • nls iso8859-4 -- Latin 4 字符集, (爱莎尼亚语,拉脱维亚语,立陶宛语),是Latin 6 字符集的前序标准

  • nls iso8859-5 -- 斯拉夫语系(保加利亚语,Byelorussian语,马其顿语,俄语,塞尔维亚语,乌克兰语) 一般推荐使用 KOI8-R codepage

  • nls iso8859-6 -- 阿拉伯语.

  • nls iso8859-7 -- 现代希腊语

  • nls iso8859-8 -- 希伯来语

  • nls iso8859-9 -- Latin 5 字符集, (去掉了 Latin 1中不经常使用的一些冰岛语字符而代以土耳其语字符)

  • nls iso8859-10 -- Latin 6 字符集, (因纽特(格陵兰)语,萨摩斯岛语等Latin 4 中没有包括的北欧语种)

  • nls iso8859-15 -- Latin 9 字符集, 是Latin 1字符集的更新版本,去掉一些不常用的字符,增加了对爱莎尼亚语的支持,修正了法语和芬兰语部份,增加了欧元字符)

  • nls koi8-r -- 俄语的缺省支持

    4. 简体中文GBK/繁体中文Big5的Codepage

    如何制作简体中文GBK/繁体中文Big5的Codepage?

    1. Unicode 组织取得GBK/Big5的Unicode的定义.

      由于GBK是基于ISO 10646-1:1993标准的,而相应的日文是JIS X 0221-1995,韩文是KS C 5700-1995,他们被提交到Unicode标准的时间表为:
      Unicode Version 1.0
      Unicode Version 1.1 <-> ISO 10646-1:1993, JIS X 0221-1995, GB 13000.1-93
      Unicode Version 2.0 <-> KS C 5700-1995

      从Windows 95开始均采用GBK编码. 您需要的是 CP936.TXTBIG5.TXT

    2. 然后使用下面的程序就可以将其转化为Linux核心需要的Unicode<->GBK码表
      ./genmap BIG5.txt | perl uni2big5.pl
      ./genmap CP936.txt | perl uni2gbk.pl
    3. 再修改fat/vfat/ntfs的相关函数就可以完成对核心的修改工作. 具体使用时可以使用下面的命令:

  • 简体中文: mount -t vfat /dev/hda1 /mnt/1 -o codepage=936,iocharset=cp936
  • 繁体中文: mount -t vfat /dev/hda1 /mnt/1 -o codepage=950,iocharset=cp936

    有趣的是,由于GBK包含了全部的GB2312/Big5/JIS的内码,所以使用936的Codepage也可以显示Big5的文件名.

    5. 附录

    5.1 作者和相关文档

    制作codepage950支持的是台湾的 cosmos先生, 主页为 http://www.cis.nctu.edu.tw:8080/~is84086/Project/kernel_cp950/

    制作GBK的cp936支持的是TurboLinux的中文研发小组的 方汉陈向阳

    5.2 genmap

    #!/bin/sh
    cat $1 | awk '{if(index($1,"#")==0)print $0}' | awk 'BEGIN{FS="0x"}{print $2 $3}' | awk '{if(length($1)==length($2))print $1,$2}'

    5.3 uni2big5.pl

    #!/usr/bin/perl

    @code = (
    "00", "01", "02", "03", "04", "05", "06", "07",
    "08", "09", "0A", "0B", "0C", "0D", "0E", "0F",
    "10", "11", "12", "13", "14", "15", "16", "17",
    "18", "19", "1A", "1B", "1C", "1D", "1E", "1F",
    "20", "21", "22", "23", "24", "25", "26", "27",
    "28", "29", "2A", "2B", "2C", "2D", "2E", "2F",
    "30", "31", "32", "33", "34", "35", "36", "37",
    "38", "39", "3A", "3B", "3C", "3D", "3E", "3F",
    "40", "41", "42", "43", "44", "45", "46", "47",
    "48", "49", "4A", "4B", "4C", "4D", "4E", "4F",
    "50", "51", "52", "53", "54", "55", "56", "57",
    "58", "59", "5A", "5B", "5C", "5D", "5E", "5F",
    "60", "61", "62", "63", "64", "65", "66", "67",
    "68", "69", "6A", "6B", "6C", "6D", "6E", "6F",
    "70", "71", "72", "73", "74", "75", "76", "77",
    "78", "79", "7A", "7B", "7C", "7D", "7E", "7F",
    "80", "81", "82", "83", "84", "85", "86", "87",
    "88", "89", "8A", "8B", "8C", "8D", "8E", "8F",
    "90", "91", "92", "93", "94", "95", "96", "97",
    "98", "99", "9A", "9B", "9C", "9D", "9E", "9F",
    "A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7",
    "A8", "A9", "AA", "AB", "AC", "AD", "AE", "AF",
    "B0", "B1", "B2", "B3", "B4", "B5", "B6", "B7",
    "B8", "B9", "BA", "BB", "BC", "BD", "BE", "BF",
    "C0", "C1", "C2", "C3", "C4", "C5", "C6", "C7",
    "C8", "C9", "CA", "CB", "CC", "CD", "CE", "CF",
    "D0", "D1", "D2", "D3", "D4", "D5", "D6", "D7",
    "D8", "D9", "DA", "DB", "DC", "DD", "DE", "DF",
    "E0", "E1", "E2", "E3", "E4", "E5", "E6", "E7",
    "E8", "E9", "EA", "EB", "EC", "ED", "EE", "EF",
    "F0", "F1", "F2", "F3", "F4", "F5", "F6", "F7",
    "F8", "F9", "FA", "FB", "FC", "FD", "FE", "FF");

    while (<STDIN>){
    ($unicode, $big5) = split;
    ($high, $low) = $unicode =~ /(..)(..)/;
    $table2{$high}{$low} = $big5;
    ($high, $low) = $big5 =~ /(..)(..)/;
    $table{$high}{$low} = $unicode;
    }

    print <<EOF;
    /*
    * linux/fs/nls_cp874.c
    *
    * Charset cp874 translation tables.
    * Generated automatically from the Unicode and charset
    * tables from the Unicode Organization (www.unicode.org).
    * The Unicode to charset table has only exact mappings.
    */

    #include <linux/module.h>
    #include <linux/kernel.h>
    #include <linux/string.h>
    #include <linux/nls.h>

    /* A1 - F9*/
    static struct nls_unicode charset2uni[(0xF9-0xA1+1)*(0x100-0x60)] = {
    EOF

    for ($high=0xA1; $high <= 0xF9; $high++){
    for ($low=0x40; $low <= 0x7F; $low++){
    $unicode = $table2{$code[$high]}{$code[$low]};
    $unicode = "0000" if (!(defined $unicode));
    print "\n\t" if ($low%4 == 0);
    print "/* $code[$high]$code[$low]*/\n\t" if ($low%0x10 == 0);
    ($uhigh, $ulow) = $unicode =~ /(..)(..)/;
    printf("{0x%2s, 0x%2s}, ", $ulow, $uhigh);
    }
    for ($low=0xA0; $low <= 0xFF; $low++){
    $unicode = $table2{$code[$high]}{$code[$low]};
    $unicode = "0000" if (!(defined $unicode));
    print "\n\t" if ($low%4 == 0);
    print "/* $code[$high]$code[$low]*/\n\t" if ($low%0x10 == 0);
    ($uhigh, $ulow) = $unicode =~ /(..)(..)/;
    printf("{0x%2s, 0x%2s}, ", $ulow, $uhigh);
    }
    }

    print "\n};\n\n";
    for ($high=1; $high <= 255;$high++){
    if (defined $table{$code[$high]}){
    print "static unsigned char page$code[$high]\[512\] = {\n\t";
    for ($low=0; $low<=255;$low++){
    $big5 = $table{$code[$high]}{$code[$low]};
    $big5 = "3F3F" if (!(defined $big5));
    if ($low > 0 && $low%4 == 0){
    printf("/* 0x%02X-0x%02X */\n\t", $low-4, $low-1);
    }
    print "\n\t" if ($low == 0x80);
    ($bhigh, $blow) = $big5 =~ /(..)(..)/;
    printf("0x%2s, 0x%2s, ", $bhigh, $blow);
    }
    print "/* 0xFC-0xFF */\n};\n\n";
    }
    }

    print "static unsigned char *page_uni2charset[256] = {";
    for ($high=0; $high<=255;$high++){
    print "\n\t" if ($high%8 == 0);
    if ($high>0 && defined $table{$code[$high]}){
    print "page$code[$high], ";
    }
    else{
    print "NULL, ";
    }
    }
    print <<EOF;

    };

    static unsigned char charset2upper[256] = {
    0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, /* 0x00-0x07 */
    0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, /* 0x08-0x0f */
    0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, /* 0x10-0x17 */
    0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, /* 0x18-0x1f */
    0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, /* 0x20-0x27 */
    0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, /* 0x28-0x2f */
    0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, /* 0x30-0x37 */
    0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, /* 0x38-0x3f */
    0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, /* 0x40-0x47 */
    0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, /* 0x48-0x4f */
    0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, /* 0x50-0x57 */
    0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, /* 0x58-0x5f */
    0x60, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x60-0x67 */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x68-0x6f */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x70-0x77 */
    0x00, 0x00, 0x00, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, /* 0x78-0x7f */
    0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, /* 0x80-0x87 */
    0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, /* 0x88-0x8f */
    0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, /* 0x90-0x97 */
    0x98, 0x99, 0x9a, 0x00, 0x9c, 0x00, 0x00, 0x00, /* 0x98-0x9f */
    0x00, 0x00, 0x00, 0x00, 0xa4, 0xa5, 0xa6, 0xa7, /* 0xa0-0xa7 */
    0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, /* 0xa8-0xaf */
    0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, /* 0xb0-0xb7 */
    0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, /* 0xb8-0xbf */
    0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, /* 0xc0-0xc7 */
    0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, /* 0xc8-0xcf */
    0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0x00, 0x00, /* 0xd0-0xd7 */
    0x00, 0xd9, 0xda, 0xdb, 0xdc, 0x00, 0x00, 0xdf, /* 0xd8-0xdf */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xe0-0xe7 */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xef, /* 0xe8-0xef */
    0xf0, 0xf1, 0x00, 0x00, 0x00, 0xf5, 0x00, 0xf7, /* 0xf0-0xf7 */
    0xf8, 0xf9, 0x00, 0x00, 0x00, 0x00, 0xfe, 0xff, /* 0xf8-0xff */
    };


    static void inc_use_count(void)
    {
    MOD_INC_USE_COUNT;
    }

    static void dec_use_count(void)
    {
    MOD_DEC_USE_COUNT;
    }

    static struct nls_table table = {
    "cp950",
    page_uni2charset,
    charset2uni,
    inc_use_count,
    dec_use_count,
    NULL
    };

    int init_nls_cp950(void)
    {
    return register_nls();
    }

    #ifdef MODULE
    int init_module(void)
    {
    return init_nls_cp950();
    }


    void cleanup_module(void)
    {
    unregister_nls();
    return;
    }
    #endif

    /*
    * Overrides for Emacs so that we follow Linus's tabbing style.
    * Emacs will notice this stuff at the end of the file and automatically
    * adjust the settings for this buffer only. This must remain at the end
    * of the file.
    *
    ---------------------------------------------------------------------------
    * Local variables:
    * c-indent-level: 8
    * c-brace-imaginary-offset: 0
    * c-brace-offset: -8
    * c-argdecl-indent: 8
    * c-label-offset: -8
    * c-continued-statement-offset: 8
    * c-continued-brace-offset: 0
    * End:
    */
    EOF

    5.4 uni2gbk.pl

    #!/usr/bin/perl

    @code = (
    "00", "01", "02", "03", "04", "05", "06", "07",
    "08", "09", "0A", "0B", "0C", "0D", "0E", "0F",
    "10", "11", "12", "13", "14", "15", "16", "17",
    "18", "19", "1A", "1B", "1C", "1D", "1E", "1F",
    "20", "21", "22", "23", "24", "25", "26", "27",
    "28", "29", "2A", "2B", "2C", "2D", "2E", "2F",
    "30", "31", "32", "33", "34", "35", "36", "37",
    "38", "39", "3A", "3B", "3C", "3D", "3E", "3F",
    "40", "41", "42", "43", "44", "45", "46", "47",
    "48", "49", "4A", "4B", "4C", "4D", "4E", "4F",
    "50", "51", "52", "53", "54", "55", "56", "57",
    "58", "59", "5A", "5B", "5C", "5D", "5E", "5F",
    "60", "61", "62", "63", "64", "65", "66", "67",
    "68", "69", "6A", "6B", "6C", "6D", "6E", "6F",
    "70", "71", "72", "73", "74", "75", "76", "77",
    "78", "79", "7A", "7B", "7C", "7D", "7E", "7F",
    "80", "81", "82", "83", "84", "85", "86", "87",
    "88", "89", "8A", "8B", "8C", "8D", "8E", "8F",
    "90", "91", "92", "93", "94", "95", "96", "97",
    "98", "99", "9A", "9B", "9C", "9D", "9E", "9F",
    "A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7",
    "A8", "A9", "AA", "AB", "AC", "AD", "AE", "AF",
    "B0", "B1", "B2", "B3", "B4", "B5", "B6", "B7",
    "B8", "B9", "BA", "BB", "BC", "BD", "BE", "BF",
    "C0", "C1", "C2", "C3", "C4", "C5", "C6", "C7",
    "C8", "C9", "CA", "CB", "CC", "CD", "CE", "CF",
    "D0", "D1", "D2", "D3", "D4", "D5", "D6", "D7",
    "D8", "D9", "DA", "DB", "DC", "DD", "DE", "DF",
    "E0", "E1", "E2", "E3", "E4", "E5", "E6", "E7",
    "E8", "E9", "EA", "EB", "EC", "ED", "EE", "EF",
    "F0", "F1", "F2", "F3", "F4", "F5", "F6", "F7",
    "F8", "F9", "FA", "FB", "FC", "FD", "FE", "FF");

    while (<STDIN>){
    ($unicode, $big5) = split;
    ($high, $low) = $unicode =~ /(..)(..)/;
    $table2{$high}{$low} = $big5;
    ($high, $low) = $big5 =~ /(..)(..)/;
    $table{$high}{$low} = $unicode;
    }

    print <<EOF;
    /*
    * linux/fs/nls_cp936.c
    *
    * Charset cp936 translation tables.
    * Generated automatically from the Unicode and charset
    * tables from the Unicode Organization (www.unicode.org).
    * The Unicode to charset table has only exact mappings.
    */

    #include <linux/module.h>
    #include <linux/kernel.h>
    #include <linux/string.h>
    #include <linux/nls.h>

    /* 81 - FE*/
    static struct nls_unicode charset2uni[(0xFE-0x81+1)*(0x100-0x40)] = {
    EOF

    for ($high=0x81; $high <= 0xFE; $high++){
    for ($low=0x40; $low <= 0x7F; $low++){
    $unicode = $table2{$code[$high]}{$code[$low]};
    $unicode = "0000" if (!(defined $unicode));
    print "\n\t" if ($low%4 == 0);
    print "/* $code[$high]$code[$low]*/\n\t" if ($low%0x10 == 0);
    ($uhigh, $ulow) = $unicode =~ /(..)(..)/;
    printf("{0x%2s, 0x%2s}, ", $ulow, $uhigh);
    }
    for ($low=0x80; $low <= 0xFF; $low++){
    $unicode = $table2{$code[$high]}{$code[$low]};
    $unicode = "0000" if (!(defined $unicode));
    print "\n\t" if ($low%4 == 0);
    print "/* $code[$high]$code[$low]*/\n\t" if ($low%0x10 == 0);
    ($uhigh, $ulow) = $unicode =~ /(..)(..)/;
    printf("{0x%2s, 0x%2s}, ", $ulow, $uhigh);
    }
    }

    print "\n};\n\n";
    for ($high=1; $high <= 255;$high++){
    if (defined $table{$code[$high]}){
    print "static unsigned char page$code[$high]\[512\] = {\n\t";
    for ($low=0; $low<=255;$low++){
    $big5 = $table{$code[$high]}{$code[$low]};
    $big5 = "3F3F" if (!(defined $big5));
    if ($low > 0 && $low%4 == 0){
    printf("/* 0x%02X-0x%02X */\n\t", $low-4, $low-1);
    }
    print "\n\t" if ($low == 0x80);
    ($bhigh, $blow) = $big5 =~ /(..)(..)/;
    printf("0x%2s, 0x%2s, ", $bhigh, $blow);
    }
    print "/* 0xFC-0xFF */\n};\n\n";
    }
    }

    print "static unsigned char *page_uni2charset[256] = {";
    for ($high=0; $high<=255;$high++){
    print "\n\t" if ($high%8 == 0);
    if ($high>0 && defined $table{$code[$high]}){
    print "page$code[$high], ";
    }
    else{
    print "NULL, ";
    }
    }
    print <<EOF;

    };

    static unsigned char charset2upper[256] = {
    0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, /* 0x00-0x07 */
    0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, /* 0x08-0x0f */
    0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, /* 0x10-0x17 */
    0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, /* 0x18-0x1f */
    0x20, 0x21, 0x22, 0x23, 0x24, 0x25, 0x26, 0x27, /* 0x20-0x27 */
    0x28, 0x29, 0x2a, 0x2b, 0x2c, 0x2d, 0x2e, 0x2f, /* 0x28-0x2f */
    0x30, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, /* 0x30-0x37 */
    0x38, 0x39, 0x3a, 0x3b, 0x3c, 0x3d, 0x3e, 0x3f, /* 0x38-0x3f */
    0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, 0x47, /* 0x40-0x47 */
    0x48, 0x49, 0x4a, 0x4b, 0x4c, 0x4d, 0x4e, 0x4f, /* 0x48-0x4f */
    0x50, 0x51, 0x52, 0x53, 0x54, 0x55, 0x56, 0x57, /* 0x50-0x57 */
    0x58, 0x59, 0x5a, 0x5b, 0x5c, 0x5d, 0x5e, 0x5f, /* 0x58-0x5f */
    0x60, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x60-0x67 */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x68-0x6f */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0x70-0x77 */
    0x00, 0x00, 0x00, 0x7b, 0x7c, 0x7d, 0x7e, 0x7f, /* 0x78-0x7f */
    0x80, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87, /* 0x80-0x87 */
    0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f, /* 0x88-0x8f */
    0x90, 0x91, 0x92, 0x93, 0x94, 0x95, 0x96, 0x97, /* 0x90-0x97 */
    0x98, 0x99, 0x9a, 0x00, 0x9c, 0x00, 0x00, 0x00, /* 0x98-0x9f */
    0x00, 0x00, 0x00, 0x00, 0xa4, 0xa5, 0xa6, 0xa7, /* 0xa0-0xa7 */
    0xa8, 0xa9, 0xaa, 0xab, 0xac, 0xad, 0xae, 0xaf, /* 0xa8-0xaf */
    0xb0, 0xb1, 0xb2, 0xb3, 0xb4, 0xb5, 0xb6, 0xb7, /* 0xb0-0xb7 */
    0xb8, 0xb9, 0xba, 0xbb, 0xbc, 0xbd, 0xbe, 0xbf, /* 0xb8-0xbf */
    0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7, /* 0xc0-0xc7 */
    0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf, /* 0xc8-0xcf */
    0xd0, 0xd1, 0xd2, 0xd3, 0xd4, 0xd5, 0x00, 0x00, /* 0xd0-0xd7 */
    0x00, 0xd9, 0xda, 0xdb, 0xdc, 0x00, 0x00, 0xdf, /* 0xd8-0xdf */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, /* 0xe0-0xe7 */
    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xef, /* 0xe8-0xef */
    0xf0, 0xf1, 0x00, 0x00, 0x00, 0xf5, 0x00, 0xf7, /* 0xf0-0xf7 */
    0xf8, 0xf9, 0x00, 0x00, 0x00, 0x00, 0xfe, 0xff, /* 0xf8-0xff */
    };


    static void inc_use_count(void)
    {
    MOD_INC_USE_COUNT;
    }

    static void dec_use_count(void)
    {
    MOD_DEC_USE_COUNT;
    }

    static struct nls_table table = {
    "cp936",
    page_uni2charset,
    charset2uni,
    inc_use_count,
    dec_use_count,
    NULL
    };

    int init_nls_cp936(void)
    {
    return register_nls();
    }

    #ifdef MODULE
    int init_module(void)
    {
    return init_nls_cp936();
    }


    void cleanup_module(void)
    {
    unregister_nls();
    return;
    }
    #endif

    /*
    * Overrides for Emacs so that we follow Linus's tabbing style.
    * Emacs will notice this stuff at the end of the file and automatically
    * adjust the settings for this buffer only. This must remain at the end
    * of the file.
    *
    ---------------------------------------------------------------------------
    * Local variables:
    * c-indent-level: 8
    * c-brace-imaginary-offset: 0
    * c-brace-offset: -8
    * c-argdecl-indent: 8
    * c-label-offset: -8
    * c-continued-statement-offset: 8
    * c-continued-brace-offset: 0
    * End:
    */
    EOF

    5.5 转换CODEPAGE的工具

    /*
    * CPI.C: A program to examine MSDOS codepage files (*.cpi)
    * and extract specific codepages.
    * Compiles under Linux & DOS (using BC++ 3.1).
    *
    * Compile: gcc -o cpi cpi.c
    * Call: codepage file.cpi [-a|-l|nnn]
    *
    * Author: Ahmed M. Naas (ahmed@oea.xs4all.nl)
    * Many changes: aeb@cwi.nl [changed until it would handle all
    * *.cpi files people have sent me; I have no documentation,
    * so all this is experimental]
    * Remains to do: DRDOS fonts.
    *
    * Copyright: Public domain.
    */

    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <unistd.h>

    int handle_codepage(int);
    void handle_fontfile(void);

    #define PACKED __attribute__ ((packed))
    /* Use this (instead of the above) to compile under MSDOS */
    /*#define PACKED */

    struct {
    unsigned char id[8] PACKED;
    unsigned char res[8] PACKED;
    unsigned short num_pointers PACKED;
    unsigned char p_type PACKED;
    unsigned long offset PACKED;
    } FontFileHeader;

    struct {
    unsigned short num_codepages PACKED;
    } FontInfoHeader;

    struct {
    unsigned short size PACKED;
    unsigned long off_nexthdr PACKED;
    unsigned short device_type PACKED; /* screen=1; printer=2 */
    unsigned char device_name[8] PACKED;
    unsigned short codepage PACKED;
    unsigned char res[6] PACKED;
    unsigned long off_font PACKED;
    } CPEntryHeader;

    struct {
    unsigned short reserved PACKED;
    unsigned short num_fonts PACKED;
    unsigned short size PACKED;
    } CPInfoHeader;

    struct {
    unsigned char height PACKED;
    unsigned char width PACKED;
    unsigned short reserved PACKED;
    unsigned short num_chard PACKED;
    } ScreenFontHeader;

    struct {
    unsigned short p1 PACKED;
    unsigned short p2 PACKED;
    } PrinterFontHeader;

    FILE *in, *out;
    void usage(void);

    int opta, optc, optl, optL, optx;
    extern int optind;
    extern char *optarg;

    unsigned short codepage;

    int main (int argc, char *argv[])
    {
    if (argc < 2)
    usage();

    if ((in = fopen(argv[1], "r")) == NULL) {
    printf("\nUnable to open file %s.\n", argv[1]);
    exit(0);
    }

    opta = optc = optl = optL = optx = 0;
    optind = 2;
    if (argc == 2)
    optl = 1;
    else
    while(1) {
    switch(getopt(argc, argv, "alLc")) {
    case 'a':
    opta = 1;
    continue;
    case 'c':
    optc = 1;
    continue;
    case 'L':
    optL = 1;
    continue;
    case 'l':
    optl = 1;
    continue;
    case '?':
    default:
    usage();
    case -1:
    break;
    }
    break;
    }
    if (optind != argc) {
    if (optind != argc-1 || opta)
    usage();
    codepage = atoi(argv[optind]);
    optx = 1;
    }

    if (optc)
    handle_codepage(0);
    else
    handle_fontfile();

    if (optx) {
    printf("no page %d found\n", codepage);
    exit(1);
    }

    fclose(in);
    return (0);
    }

    void
    handle_fontfile(){
    int i, j;

    j = fread(, 1, sizeof(FontFileHeader), in);
    if (j != sizeof(FontFileHeader)) {
    printf("error reading FontFileHeader - got %d chars\n", j);
    exit (1);
    }
    if (!strcmp(FontFileHeader.id + 1, "DRFONT ")) {
    printf("this program cannot handle DRDOS font files\n");
    exit(1);
    }
    if (optL)
    printf("FontFileHeader: id=%8.8s res=%8.8s num=%d typ=%c offset=%ld\n\n",
    FontFileHeader.id, FontFileHeader.res,
    FontFileHeader.num_pointers,
    FontFileHeader.p_type,
    FontFileHeader.offset);

    j = fread(, 1, sizeof(FontInfoHeader), in);
    if (j != sizeof(FontInfoHeader)) {
    printf("error reading FontInfoHeader - got %d chars\n", j);
    exit (1);
    }
    if (optL)
    printf("FontInfoHeader: num_codepages=%d\n\n",
    FontInfoHeader.num_codepages);

    for (i = FontInfoHeader.num_codepages; i; i--)
    if (handle_codepage(i-1))
    break;
    }

    int
    handle_codepage(int more_to_come) {
    int j;
    char outfile[20];
    unsigned char *fonts;
    long inpos, nexthdr;

    j = fread(, 1, sizeof(CPEntryHeader), in);
    if (j != sizeof(CPEntryHeader)) {
    printf("error reading CPEntryHeader - got %d chars\n", j);
    exit(1);
    }
    if (optL) {
    int t = CPEntryHeader.device_type;
    printf("CPEntryHeader: size=%d dev=%d [%s] name=%8.8s \
    codepage=%d\n\t\tres=%6.6s nxt=%ld off_font=%ld\n\n",
    CPEntryHeader.size,
    t, (t==1) ? "screen" : (t==2) ? "printer" : "?",
    CPEntryHeader.device_name,
    CPEntryHeader.codepage,
    CPEntryHeader.res,
    CPEntryHeader.off_nexthdr, CPEntryHeader.off_font);
    } else if (optl) {
    printf("\nCodepage = %d\n", CPEntryHeader.codepage);
    printf("Device = %.8s\n", CPEntryHeader.device_name);
    }
    #if 0
    if (CPEntryHeader.size != sizeof(CPEntryHeader)) {
    /* seen 26 and 28, so that the difference below is -2 or 0 */
    if (optl)
    printf("Skipping %d bytes of garbage\n",
    CPEntryHeader.size - sizeof(CPEntryHeader));
    fseek(in, CPEntryHeader.size - sizeof(CPEntryHeader),
    SEEK_CUR);
    }
    #endif
    if (!opta && (!optx || CPEntryHeader.codepage != codepage) && !optc)
    goto next;

    inpos = ftell(in);
    if (inpos != CPEntryHeader.off_font && !optc) {
    if (optL)
    printf("pos=%ld font at %ld\n", inpos, CPEntryHeader.off_font);
    fseek(in, CPEntryHeader.off_font, SEEK_SET);
    }

    j = fread(, 1, sizeof(CPInfoHeader), in);
    if (j != sizeof(CPInfoHeader)) {
    printf("error reading CPInfoHeader - got %d chars\n", j);
    exit(1);
    }
    if (optl) {
    printf("Number of Fonts = %d\n", CPInfoHeader.num_fonts);
    printf("Size of Bitmap = %d\n", CPInfoHeader.size);
    }
    if (CPInfoHeader.num_fonts == 0)
    goto next;
    if (optc)
    return 0;

    sprintf(outfile, "%d.cp", CPEntryHeader.codepage);
    if ((out = fopen(outfile, "w")) == NULL) {
    printf("\nUnable to open file %s.\n", outfile);
    exit(1);
    } else printf("\nWriting %s\n", outfile);

    fonts = (unsigned char *) malloc(CPInfoHeader.size);

    fread(fonts, CPInfoHeader.size, 1, in);
    fwrite(, sizeof(CPEntryHeader), 1, out);
    fwrite(, sizeof(CPInfoHeader), 1, out);
    j = fwrite(fonts, 1, CPInfoHeader.size, out);
    if (j != CPInfoHeader.size) {
    printf("error writing %s - wrote %d chars\n", outfile, j);
    exit(1);
    }
    fclose(out);
    free(fonts);
    if (optx) exit(0);
    next:
    /*
    * It seems that if entry headers and fonts are interspersed,
    * then nexthdr will point past the font, regardless of
    * whether more entries follow.
    * Otherwise, first all entry headers are given, and then
    * all fonts; in this case nexthdr will be 0 in the last entry.
    */
    nexthdr = CPEntryHeader.off_nexthdr;
    if (nexthdr == 0 || nexthdr == -1) {
    if (more_to_come) {
    printf("mode codepages expected, but nexthdr=%ld\n",
    nexthdr);
    exit(1);
    } else
    return 1;
    }

    inpos = ftell(in);
    if (inpos != CPEntryHeader.off_nexthdr) {
    if (optL)
    printf("pos=%ld nexthdr at %ld\n", inpos, nexthdr);
    if (opta && !more_to_come) {
    printf("no more code pages, but nexthdr != 0\n");
    return 1;
    }

    fseek(in, CPEntryHeader.off_nexthdr, SEEK_SET);
    }

    return 0;
    }

    void usage(void)
    {
    printf("\nUsage: cpi code_page_file [-c] [-L] [-l] [-a|nnn]\n");
    printf(" -c: input file is a single codepage\n");
    printf(" -L: print header info (you don't want to see this)\n");
    printf(" -l or no option: list all codepages contained in the file\n");
    printf(" -a: extract all codepages from the file\n");
    printf(" nnn (3 digits): extract codepage nnn from the file\n");
    printf("Example: cpi ega.cpi 850 \n");
    printf(" will create a file 850.cp containing the requested codepage.\n\n");
    exit(1);
    }


    作者Blog:http://blog.csdn.net/findsun/
  • posted on 2006-07-20 14:31 汪杰 阅读(3809) 评论(1)  编辑 收藏 引用

    FeedBack:
    # re: Codepage的定义和历史
    2006-07-20 14:32 | 汪杰
    这是一篇程序员写给程序员的趣味读物。所谓趣味是指可以比较轻松地了解一些原来不清楚的概念,增进知识,类似于打RPG游戏的升级。整理这篇文章的动机是两个问题:

    问题一:
    使用Windows记事本的“另存为”,可以在GBK、Unicode、Unicode big endian和UTF-8这几种编码方式间相互转换。同样是txt文件,Windows是怎样识别编码方式的呢?

    我 很早前就发现Unicode、Unicode big endian和UTF-8编码的txt文件的开头会多出几个字节,分别是FF、FE(Unicode),FE、FF(Unicode big endian),EF、BB、BF(UTF-8)。但这些标记是基于什么标准呢?

    问题二:
    最近在网上看到 一个ConvertUTF.c,实现了UTF-32、UTF-16和UTF-8这三种编码方式的相互转换。对于Unicode(UCS2)、GBK、 UTF-8这些编码方式,我原来就了解。但这个程序让我有些糊涂,想不起来UTF-16和UCS2有什么关系。
    查了查相关资料,总算将这些问题弄清楚了,顺带也了解了一些Unicode的细节。写成一篇文章,送给有过类似疑问的朋友。本文在写作时尽量做到通俗易懂,但要求读者知道什么是字节,什么是十六进制。

    0、big endian和little endian
    big endian和little endian是CPU处理多字节数的不同方式。例如“汉”字的Unicode编码是6C49。那么写到文件里时,究竟是将6C写在前面,还是将49写在前 面?如果将6C写在前面,就是big endian。还是将49写在前面,就是little endian。

    “endian”这个词出自《格列佛游记》。小人国的内战就源于吃鸡蛋时是究竟从大头(Big-Endian)敲开还是从小头(Little-Endian)敲开,由此曾发生过六次叛乱,其中一个皇帝送了命,另一个丢了王位。

    我们一般将endian翻译成“字节序”,将big endian和little endian称作“大尾”和“小尾”。

    1、字符编码、内码,顺带介绍汉字编码
    字符必须编码后才能被计算机处理。计算机使用的缺省编码方式就是计算机的内码。早期的计算机使用7位的ASCII编码,为了处理汉字,程序员设计了用于简体中文的GB2312和用于繁体中文的big5。

    GB2312(1980年)一共收录了7445个字符,包括6763个汉字和682个其它符号。汉字区的内码范围高字节从B0-F7,低字节从A1-FE,占用的码位是72*94=6768。其中有5个空位是D7FA-D7FE。

    GB2312 支持的汉字太少。1995年的汉字扩展规范GBK1.0收录了21886个符号,它分为汉字区和图形符号区。汉字区包括21003个字符。2000年的 GB18030是取代GBK1.0的正式国家标准。该标准收录了27484个汉字,同时还收录了藏文、蒙文、维吾尔文等主要的少数民族文字。现在的PC平 台必须支持GB18030,对嵌入式产品暂不作要求。所以手机、MP3一般只支持GB2312。

    从ASCII、GB2312、GBK到 GB18030,这些编码方法是向下兼容的,即同一个字符在这些方案中总是有相同的编码,后面的标准支持更多的字符。在这些编码中,英文和中文可以统一地 处理。区分中文编码的方法是高字节的最高位不为0。按照程序员的称呼,GB2312、GBK到GB18030都属于双字节字符集 (DBCS)。

    有的中文Windows的缺省内码还是GBK,可以通过GB18030升级包升级到GB18030。不过GB18030相对GBK增加的字符,普通人是很难用到的,通常我们还是用GBK指代中文Windows内码。

    这里还有一些细节:

    GB2312的原文还是区位码,从区位码到内码,需要在高字节和低字节上分别加上A0。

    在DBCS中,GB内码的存储格式始终是big endian,即高位在前。

    GB2312 的两个字节的最高位都是1。但符合这个条件的码位只有128*128=16384个。所以GBK和GB18030的低字节最高位都可能不是1。不过这不影 响DBCS字符流的解析:在读取DBCS字符流时,只要遇到高位为1的字节,就可以将下两个字节作为一个双字节编码,而不用管低字节的高位是什么。

    2、Unicode、UCS和UTF
    前面提到从ASCII、GB2312、GBK到GB18030的编码方法是向下兼容的。而Unicode只与ASCII兼容(更准确地说,是与ISO-8859-1兼容),与GB码不兼容。例如“汉”字的Unicode编码是6C49,而GB码是BABA。

    Unicode 也是一种字符编码方法,不过它是由国际组织设计,可以容纳全世界所有语言文字的编码方案。Unicode的学名是"Universal Multiple-Octet Coded Character Set",简称为UCS。UCS可以看作是"Unicode Character Set"的缩写。

    根据维基百科全书(http://zh.wikipedia.org/wiki/)的记载:历史上存在两个试图独立设计 Unicode的组织,即国际标准化组织(ISO)和一个软件制造商的协会(unicode.org)。ISO开发了ISO 10646项目,Unicode协会开发了Unicode项目。

    在1991年前后,双方都认识到世界不需要两个不兼容的字符集。于是它们开始合并双方的工作成果,并为创立一个单一编码表而协同工作。从Unicode2.0开始,Unicode项目采用了与ISO 10646-1相同的字库和字码。

    目前两个项目仍都存在,并独立地公布各自的标准。Unicode协会现在的最新版本是2005年的Unicode 4.1.0。ISO的最新标准是10646-3:2003。

    UCS规定了怎么用多个字节表示各种文字。怎样传输这些编码,是由UTF(UCS Transformation Format)规范规定的,常见的UTF规范包括UTF-8、UTF-7、UTF-16。

    IETF 的RFC2781和RFC3629以RFC的一贯风格,清晰、明快又不失严谨地描述了UTF-16和UTF-8的编码方法。我总是记不得IETF是 Internet Engineering Task Force的缩写。但IETF负责维护的RFC是Internet上一切规范的基础。

    3、UCS-2、UCS-4、BMP
    UCS有两种格式:UCS-2和UCS-4。顾名思义,UCS-2就是用两个字节编码,UCS-4就是用4个字节(实际上只用了31位,最高位必须为0)编码。下面让我们做一些简单的数学游戏:

    UCS-2有2^16=65536个码位,UCS-4有2^31=2147483648个码位。

    UCS -4根据最高位为0的最高字节分成2^7=128个group。每个group再根据次高字节分为256个plane。每个plane根据第3个字节分为 256行 (rows),每行包含256个cells。当然同一行的cells只是最后一个字节不同,其余都相同。

    group 0的plane 0被称作Basic Multilingual Plane, 即BMP。或者说UCS-4中,高两个字节为0的码位被称作BMP。

    将UCS-4的BMP去掉前面的两个零字节就得到了UCS-2。在UCS-2的两个字节前加上两个零字节,就得到了UCS-4的BMP。而目前的UCS-4规范中还没有任何字符被分配在BMP之外。

    4、UTF编码
    UTF-8就是以8位为单元对UCS进行编码。从UCS-2到UTF-8的编码方式如下:

    UCS-2编码(16进制) UTF-8 字节流(二进制)
    0000 - 007F 0xxxxxxx
    0080 - 07FF 110xxxxx 10xxxxxx
    0800 - FFFF 1110xxxx 10xxxxxx 10xxxxxx

    例如“汉”字的Unicode编码是6C49。6C49在0800-FFFF之间,所以肯定要用3字节模板了:1110xxxx 10xxxxxx 10xxxxxx。将6C49写成二进制是:0110 110001 001001, 用这个比特流依次代替模板中的x,得到:11100110 10110001 10001001,即E6 B1 89。

    读者可以用记事本测试一下我们的编码是否正确。

    UTF -16以16位为单元对UCS进行编码。对于小于0x10000的UCS码,UTF-16编码就等于UCS码对应的16位无符号整数。对于不小于 0x10000的UCS码,定义了一个算法。不过由于实际使用的UCS2,或者UCS4的BMP必然小于0x10000,所以就目前而言,可以认为UTF -16和UCS-2基本相同。但UCS-2只是一个编码方案,UTF-16却要用于实际的传输,所以就不得不考虑字节序的问题。

    5、UTF的字节序和BOM
    UTF -8以字节为编码单元,没有字节序的问题。UTF-16以两个字节为编码单元,在解释一个UTF-16文本前,首先要弄清楚每个编码单元的字节序。例如收 到一个“奎”的Unicode编码是594E,“乙”的Unicode编码是4E59。如果我们收到UTF-16字节流“594E”,那么这是“奎”还是 “乙”?

    Unicode规范中推荐的标记字节顺序的方法是BOM。BOM不是“Bill Of Material”的BOM表,而是Byte Order Mark。BOM是一个有点小聪明的想法:

    在UCS 编码中有一个叫做"ZERO WIDTH NO-BREAK SPACE"的字符,它的编码是FEFF。而FFFE在UCS中是不存在的字符,所以不应该出现在实际传输中。UCS规范建议我们在传输字节流前,先传输 字符"ZERO WIDTH NO-BREAK SPACE"。

    这样如果接收者收到FEFF,就表明这个字节流是Big-Endian的;如果收到FFFE,就表明这个字节流是Little-Endian的。因此字符"ZERO WIDTH NO-BREAK SPACE"又被称作BOM。

    UTF -8不需要BOM来表明字节顺序,但可以用BOM来表明编码方式。字符"ZERO WIDTH NO-BREAK SPACE"的UTF-8编码是EF BB BF(读者可以用我们前面介绍的编码方法验证一下)。所以如果接收者收到以EF BB BF开头的字节流,就知道这是UTF-8编码了。

    Windows就是使用BOM来标记文本文件的编码方式的。

    6、进一步的参考资料
    本文主要参考的资料是 "Short overview of ISO-IEC 10646 and Unicode" (http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html)。

    我还找了两篇看上去不错的资料,不过因为我开始的疑问都找到了答案,所以就没有看:

    "Understanding Unicode A general introduction to the Unicode Standard" (http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&item_id=IWS-Chapter04a)
    "Character set encoding basics Understanding character set encodings and legacy encodings" (http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&item_id=IWS-Chapter03)
    我写过UTF-8、UCS-2、GBK相互转换的软件包,包括使用Windows API和不使用Windows API的版本。以后有时间的话,我会整理一下放到我的个人主页上(http://fmddlmyy.home4u.china.com)。

    我是想清楚所有问题后才开始写这篇文章的,原以为一会儿就能写好。没想到考虑措辞和查证细节花费了很长时间,竟然从下午1:30写到9:00。希望有读者能从中受益。


    作者Blog:http://blog.csdn.net/fmddlmyy/

      回复  更多评论
      
    只有注册用户登录后才能发表评论。

    <2024年3月>
    252627282912
    3456789
    10111213141516
    17181920212223
    24252627282930
    31123456

    常用链接

    留言簿(15)

    随笔分类(1)

    随笔档案(90)

    文章分类(727)

    文章档案(712)

    相册

    收藏夹

    http://blog.csdn.net/prodigynonsense

    友情链接

    最新随笔

    搜索

    •  

    积分与排名

    • 积分 - 456283
    • 排名 - 6

    最新随笔

    最新评论

    阅读排行榜

    评论排行榜