安装pyspider遇到Traceback (most recent call last)问题

项目场景:

Python爬虫,安装pyspider遇到的问题。


问题描述

提示:这里描述项目中遇到的问题:

一、安装步骤(为了好复现问题)

  1. 打开https://www.lfd.uci.edu/~gohlke/pythonlibs ,按下Ctrl+f调出搜索框,搜索pycur,点击查找到项会跳到pycurl对应的位置
  2. 版本选择:选择你与以安装python版本相同位的,比如我安装的是python3.8的,我就选择pycurl-7.43.0.4-cp38-cp38-win32.whl
  3. 点击下载完成后,使用“pip install 文件路径”
    pip install C:\Users\14340\Downloads\pycurl-7.43.0.4-cp38-cp38-win32.whl

二、报错信息

Traceback (most recent call last):
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\run.py", line 236, in fetcher
    Fetcher = load_cls(None, None, fetcher_cls)
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\run.py", line 48, in load_cls
    return utils.load_object(value)
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\libs\utils.py", line 369, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\fetcher\__init__.py", line 1, in <module>
    from .tornado_fetcher import Fetcher
  File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\fetcher\tornado_fetcher.py", line 81
    def __init__(self, inqueue, outqueue, poolsize=100, proxy=None, async=True):
	}

解决方案:

async在3.7中是关键字不能作为参数了
两种解决方案
1、替换关键字
2、降低python版本

async 分别在

1 ...site-packages/pyspider/run.py
2 .../site-packages/pyspider/fetcher/tornado_fetcher.py
3.   ...site-packages/pyspider/webui/app.py
将async替换成_async
...site-packages/pyspider/webui/webdav.py,修改第209行即可。
将'domaincontroller': NeedAuthController(app),改为
'http_authenticator':{
        'HTTPAuthenticator':NeedAuthController(app),
}

报此错误DispatcherMiddleware’ from 'werkzeug.wsgi

python -m pip install -U --force-reinstall pip
python -m pip  uninstall wsgidav 
pip install pyspider

参考资源
pyspider安装问题

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值