Python用Scrapy爬虫报错UnicodeEncodeError: 'gbk' codec can't encode character '\u2022' ，解决方案-优快云博客

错误：UnicodeEncodeError: 'gbk' codec can't encode character '\u2022' in position 7: illegal multibyte sequence
解决：import io
     import sys
     sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')

从网上抓取网站写下面这段代码时，发现报UnicodeEncodeError: 'gbk' codec can't encode character '\xXX' in position XX 错误

from urllib import request
req=request.Request("https://www.baidu.com")
req.add_header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0")
resp=request.urlopen(req)
print(resp.read().decode('utf-8'))

查了一下发现了解决办法原来是print()函数自身有限制，不能完全打印所有的unicode字符。

其实print()函数的局限就是Python默认编码的局限，因为系统是win7的，python的默认编码不是'utf-8',改一下python的默认编码成'utf-8'就行了

import io  
import sys 
from urllib import request
sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf8') #改变标准输出的默认编码
req=request.Request("https://www.baidu.com")
req.add_header("User-Agent","Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:51.0) Gecko/20100101 Firefox/51.0")
resp=request.urlopen(req)
print(resp.read().decode('utf-8'))

虽然可以解决了报错，但发现中文乱码，原来是cmd编码的不兼容utf-8，若要解决这问题，改一下python的默认编码成'gb18030'就行了

sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')         #改变标准输出的默认编码

转载自：https://blog.youkuaiyun.com/qq_28359387/article/details/54974578