u014775723的博客

运行scrapy spider时，出现Forbidden by robots.txt错误原因：在settings.py文件里有个ROBOTSTXT_OBEY参数，默认为True。False为拒绝遵守robot协议，True为遵守robot协议解决：将settings.py文件里的ROBOTSTXT_OBEY参数，设为False.。即ROBOTSTXT_OBEY = False...

pip install scrapy时报错：Could not find a version that satisfies the requirement Twisted>=13.1.0 (from Scrapy) (from versions: ) No matching distribution found for Twisted>=13.1.0 (from Scrapy) ...

pip install scrapy时报错：Could not find a version that satisfies the requirement Twisted>=13.1.0 (from Scrapy) (from versions: ) No matching distribution found for Twisted>=13.1.0 (from Scrapy) ...

报错如图：原因：twisted版本比较低，而python的版本比较高1.查看pip版本2.查看twisted版本pythonimport twistedtwisted.version 因此更新twisted版本 wget https://pypi.python.org/packages/source/T/Twisted/Twisted-17.1.0....

最开始遇到的问题是：使用pip安装scrapy报错：unbuntu pkg_resources.DistributionNotFound: The 'pip==1.5.4' distribution was not found and is required by the application因此参考博客https://blog.youkuaiyun.com/allyli0022/article...

第一步：在腾讯云购买服务器 第二步：在本机下载scrapyd-client：对于windows系统，建议不要用pip install scrapyd-client去安装scrapyd-client，会出现，scrapyd-deploy不是内部或外部命令，因为scrapyd-deploy不能被windows执行。应当直接去github上下载并解压安装包后，进入解压后的目录下，执行py...

第一部分：服务器端安装mongodb参考：参考文章下载安装包wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.2.9.tgz解压tar -zxvf mongodb-linux-x86_64-3.2.9.tgz解压后文件夹重命名mv mongodb-linux-x86_64-3.2.9 mong...