UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026' in position 30: ordinal not in r

最新推荐文章于 2023-06-13 12:01:50 发布

原创最新推荐文章于 2023-06-13 12:01:50 发布 · 4.9k 阅读

3 ·

CC 4.0 BY-SA版权

错误总结专栏收录该内容

4 篇文章

订阅专栏

本文介绍了一种常见的Python爬虫错误，即在设置请求头部时使用了不正确的字符，如省略号，导致UnicodeEncodeError。通过调整User-Agent字符串，移除非法字符，成功解决了爬虫无法正确获取网页数据的问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

运行淘宝图片爬取，经常出错`

import urllib.request
import re
keyname = "秋上新"
key = urllib.request.quote(keyname)
headers = ("User-Agent" , "Mozilla/5.0 (Windows NT 10.0; …) Gecko/20100101 Firefox/62.0")
openers = urllib.request.build_opener()
openers.addheaders = [headers]
urllib.request.install_opener(openers)

for i in range(1,4):
    url = "https://s.taobao.com/list?spm=a21bo.2017.201867-links-0.3.5af911d9qdrblU&q="+key+"&cat=16&seller_type=taobao&oetag=6745&source=qiangdiao&bcoffset=12&s="+str(i*60)
    data = urllib.request.urlopen(url).read().decode("utf-8","ignore")
    pat = '"pic_url":"//(.*?)"'
    imagelist = re.compile(pat).findall(data)
    for j in range(0,len(imagelist)):
        this_img = imagelist[j]
        this_img_url = "http://" + this_img
        file = "E:/Test/result/img/"+str(i)+str(j)+".jpg"
        urllib.request.urlretrieve(this_img_url,filename=file)

一直报错

Traceback (most recent call last):
  File "D:/Users/jiang/workspaceForPycharm/Test.py", line 12, in <module>
    data = urllib.request.urlopen(url).read().decode("utf-8","ignore")
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\urllib\request.py", line 526, in open
    response = self._open(req, data)
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\urllib\request.py", line 544, in _open
    '_open', req)
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\urllib\request.py", line 504, in _call_chain
    result = func(*args)
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\urllib\request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\urllib\request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\http\client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\http\client.py", line 1280, in _send_request
    self.putheader(hdr, value)
  File "D:\Users\jiang\AppData\Local\Continuum\anaconda3\lib\http\client.py", line 1212, in putheader
    values[i] = one_value.encode('latin-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026' in position 30: ordinal not in range(256)

尝试了很多解决方案，后来发现在浏览器伪装的时候，报头写错了。中间不能有省略号。
headers = (“User-Agent” , “Mozilla/5.0 (Windows NT 10.0; …) Gecko/20100101 Firefox/62.0”)
把10.0；后面的省略号删除后运行成功