scrapy [boto] ERROR: Caught exception reading instance data URLError: <urlopen error [Errno 10051] >

最新推荐文章于 2023-03-30 19:10:56 发布

原创最新推荐文章于 2023-03-30 19:10:56 发布 · 3.4k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#python #scrapy

爬虫专栏收录该内容

16 篇文章

订阅专栏

本文介绍了解决Scrapy项目中因boto库导致的S3下载异常问题的方法。通过在settings.py文件中禁用S3下载处理器，避免了Scrapy与boto之间的冲突。此外，还提供了设置虚拟环境的建议。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

执行过程中出现错误：

2015-09-09 11:13:26 [boto] DEBUG: Retrieving credentials from metadata server.
2015-09-09 11:13:27 [boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
  File "D:\anzhuang\Anaconda\lib\site-packages\boto\utils.py", line 210, in retry_url
    r = opener.open(req, timeout=timeout)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 431, in open
    response = self._open(req, data)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 449, in _open
    '_open', req)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error timed out>
2015-09-09 11:13:27 [boto] ERROR: Unable to read instance data, giving up

在setting.py中禁用s3 download就可以了
解决方法：

DOWNLOAD_HANDLERS = {'S3': None,}

在stackoverflow上有些解释,有时间可以看看这篇错误分析错误
下面是我贴的部分内容。
That particular error message is being generated by boto (boto 2.38.0 py27_0), which is used to connect to Amazon S3. Scrapy doesn’t have this enabled by default.

EDIT: In reply to the comments, this appears to be a bug with Scrapy when boto is present (bug here).

In response “how to disable the Download handler”, add the following to your settings.py file:

DOWNLOAD_HANDLERS = {
        'S3':None,
}

Your settings.py file should be in the root of your Scrapy project folder, (one level deeper than your scrapy.cfg file).

If you’ve already got DOWNLOAD_HANDLERS in your settings.py file, just add a new entry for ‘s3’ with a None value.

EDIT 2: I’d highly recommend looking at setting up virtual environments for your projects. Look into virtualenv, and it’s usage. I’d make this recommendation regardless of packages used for this project, but doubly so with your extreme number of packages.