python3: UnicodeEncodeError:’utf-8’/'gbk’编解码器无法编码:不允许代理
问题描述:
(1)例如下代码
for root, dirs, files in os.walk('.'):
print(root)
会出现错误:
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 27: surrogates not allowed
如何在没有像这样的有毒字符串的情况下浏览文件树?
(2)例如下代码
print('Best hypothesis segments: ',[seg.word for seg in decoder.seg()]))
会出现以下结果(pycharm project encoding—utf-8):
Best hypothesis segments: ['<s>', '\udce4\udcaf\udcc0\udcc0\udcc6\udcf7', '\udcb9ر\udcd5', '</s>']
出现以下错误(pycharm----gbk):
UnicodeEncodeError: 'gbk' codec can't encode character '\u0631' in position 55: illegal multibyte sequence
在Linux上,文件名只是“一堆字节”,并不一定用特定的编码进行编码. Python 3尝试将所有内容都转换为Unicode字符串.在这样做的过程中,开发人员提出了一种方案,将字节字符串转换为Unicode字符串,并且不会丢失,并且不知道原始编码.他们使用部分代理来编码’坏’字节,但普通的UTF8编码器在打印到终端时无法处理它们.
例如,这是一个非UTF8字节字符串:
>>> b'C\xc3N'.decode('utf8','surrogateescape')
'C\udcc3N'
它可以转换为Unicode而不会丢失:
>>> b'C\xc3N'.decode('utf8','surrogateescape').encode('utf8','surrogateescape')
b'C\xc3N'
但它无法打印:
>>> print(b'C\xc3N'.decode('utf8','surrogateescape'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 1: surrogates not allowed
你必须弄清楚你想用非默认编码对文件名做什么.也许只是将它们编码回原始字节并用未知替换解码它们.使用此选项进行显示,但保留原始名称以访问该文件.
>>> b'C\xc3N'.decode('utf8','replace')
C�N
os.walk也可以使用字节字符串,并返回字节字符串而不是Unicode字符串:
for p,d,f in os.walk(b'.'):
然后你可以随意解码.
根据以上分析将代码:
print ('Best hypothesis segments: ',[seg.word for seg in decoder.seg()])
修改为
# print ('Best hypothesis segments: ',[str(seg.word.encode('utf-8','surrogateescape'),'gbk','ignore') for seg in decoder.seg()])
使用str(words,‘gbk’,‘ignore’)是其在windows下能够打印出中文(bytes to str),
word.encode(‘utf-8’,‘surrogateescape’)将其转换为Unicode而不会丢失不能编码的部分。
有关‘surrogateescape’可以参看文章:
http://timothyqiu.com/archives/surrogateescape-in-python-3/
本文参考了:
https://codeday.me/bug/20180901/237634.html
https://www.crifan.com/unicodeencodeerror_gbk_codec_can_not_encode_character_in_position_illegal_multibyte_sequence/