unicodedata 字符处理例子

>>> import unicodedata
>>> import sys
>>> cmb_chrs = dict.fromkeys(c for c in range(sys.maxunicode)
...                         if unicodedata.combining(chr(c)))
...
>>> b = unicodedata.normalize('NFD', a)
>>> b
'pýtĥöñ is awesome\n'
>>> b.translate(cmb_chrs)
'python is awesome\n'
>>>

import unicodedata
def unicode_to_ascii(s):
    ## 统一使用一种编码格式 NFD表示字符应该分解为多个组合字符表示。
    return ''.join(c for c in unicodedata.normalize('NFD', s) if unicodedata.category(c) != 'Mn')

## 选取 
en_sentence = 'Then what?'
sp_sentence = '¿Entonces qué?'

print(unicode_to_ascii(en_sentence))
print(unicode_to_ascii(sp_sentence))

2.12 审查清理文本字符串

奇葩字符 “a๎๎๎๎๎๎๎๎๎๎๎๎๎๎๎๎๎๎๎” 的简单分析