谈谈Python爬虫框架Scrapy安装过程中的那些坑

Scrapy的安装有一些依赖包，没有这些包是会安装失败的：

安装指南：http://scrapy-chs.readthedocs.io/zh_CN/0.24/intro/install.html

需要的依赖包都安装好了，pip install Scrapy还是报错，报错提示如下：

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-x1OMoL/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-d1qbSa-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-x1OMoL/cryptography/

解决方案：

sudo apt-get install build-essential libssl-dev libffi-dev python-dev

pip install cryptography

pip install Scrapy成功

参考来自：https://www.digitalocean.com/community/questions/pip-install-scrapy-error-with-python-3-on-ubuntu-16-04