项目场景:
Python爬虫,安装pyspider遇到的问题。
问题描述
提示:这里描述项目中遇到的问题:
一、安装步骤(为了好复现问题)
- 打开https://www.lfd.uci.edu/~gohlke/pythonlibs ,按下Ctrl+f调出搜索框,搜索pycur,点击查找到项会跳到pycurl对应的位置
- 版本选择:选择你与以安装python版本相同位的,比如我安装的是python3.8的,我就选择pycurl-7.43.0.4-cp38-cp38-win32.whl
- 点击下载完成后,使用“pip install 文件路径”
pip install C:\Users\14340\Downloads\pycurl-7.43.0.4-cp38-cp38-win32.whl
二、报错信息
Traceback (most recent call last):
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\run.py", line 236, in fetcher
Fetcher = load_cls(None, None, fetcher_cls)
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\run.py", line 48, in load_cls
return utils.load_object(value)
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\libs\utils.py", line 369, in load_object
module = __import__(module_name, globals(), locals(), [object_name])
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\fetcher\__init__.py", line 1, in <module>
from .tornado_fetcher import Fetcher
File "d:\ruanjian\pycharm 2019.2.5\py3.8.3\lib\site-packages\pyspider\fetcher\tornado_fetcher.py", line 81
def __init__(self, inqueue, outqueue, poolsize=100, proxy=None, async=True):
}
解决方案:
async在3.7中是关键字不能作为参数了
两种解决方案
1、替换关键字
2、降低python版本
async 分别在
1 ...site-packages/pyspider/run.py
2 .../site-packages/pyspider/fetcher/tornado_fetcher.py
3. ...site-packages/pyspider/webui/app.py
将async替换成_async
...site-packages/pyspider/webui/webdav.py,修改第209行即可。
将'domaincontroller': NeedAuthController(app),改为
'http_authenticator':{
'HTTPAuthenticator':NeedAuthController(app),
}
报此错误DispatcherMiddleware’ from 'werkzeug.wsgi
python -m pip install -U --force-reinstall pip
python -m pip uninstall wsgidav
pip install pyspider
参考资源
pyspider安装问题