Kali 安装scrapy爬虫框架

最新推荐文章于 2024-12-11 20:07:10 发布

月流霜

最新推荐文章于 2024-12-11 20:07:10 发布

阅读量1.7k

点赞数

CC 4.0 BY-SA版权

分类专栏：自阅

本文链接：https://blog.youkuaiyun.com/abcd51685168/article/details/44206409

自阅专栏收录该内容

20 篇文章

订阅专栏

本文介绍了Scrapy爬虫环境的搭建过程，包括Python、Twisted、w3lib等依赖库的安装步骤，并验证了安装的成功。

参考http://www.linuxidc.com/Linux/2012-07/66236.htm

准备工作
Requirements
Python 2.5, 2.6, 2.7 (3.x is not yet supported)
Twisted 2.5.0, 8.0 or above (Windows users: you’ll need to install Zope.Interface and maybe pywin32 because of this Twisted bug)
w3lib
lxml or libxml2 (if using libxml2, version 2.6.28 or above is highly recommended)
simplejson (not required if using Python 2.6 or above)
pyopenssl (for HTTPS support. Optional, but highly recommended)
---------------------------------------------
Twisted安装过程
sudo apt-get install python-twisted python-libxml2 python-simplejson
安装完成后进入python，测试Twisted是否安装成功

pyOpenSSL
wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz#md5=767bca18a71178ca353dff9e10941929
tar -zxvf pyOpenSSL-0.13.tar.gz
cd pyOpenSSL-0.13
sudo python setup.py install

pycrypto
wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz#md5=783e45d4a1a309e03ab378b00f97b291
tar -zxvf pycrypto-2.5.tar.gz
cd pycrypto-2.5
sudo python setup.py install

测试是否安装成功
$python
>>> import Crypto
>>> import twisted.conch.ssh.transport
>>> print Crypto.PublicKey.RSA
<module 'Crypto.PublicKey.RSA' from '/usr/python/lib/python2.5/site-packages/Crypto/PublicKey/RSA.pyc'>
>>> import OpenSSL
>>> import twisted.internet.ssl
>>> twisted.internet.ssl
<module 'twisted.internet.ssl' from '/usr/python/lib/python2.5/site-packages/Twisted-10.1.0-py2.5-linux-i686.egg/twisted/internet/ssl.pyc'>
如果出现类似提示，说明pyOpenSSL模块已经安装成功了，否则，请检查上面的安装过程(OpenSSL需要pycrypto)。

w3lib
sudo easy_install -U w3lib

Scrapy
wget http://pypi.python.org/packages/source/S/Scrapy/Scrapy-0.14.3.tar.gz#md5=59f1225f7692f28fa0f78db3d34b3850
tar -zxvf Scrapy-0.14.3.tar.gz
cd Scrapy-0.14.3
sudo python setup.py install

Scrapy安装验证
经过上面的安装和配置过程，已经完成了Scrapy的安装，我们可以通过如下命令行来验证一下：
$ scrapy
Scrapy 0.14.3 - no active project

Usage:
scrapy <command> [options] [args]

Available commands:
fetch         Fetch a URL using the Scrapy downloader
runspider     Run a self-contained spider (without creating a project)
settings      Get settings values
shell         Interactive scraping console
startproject Create new project
version       Print Scrapy version
view          Open URL in browser, as seen by Scrapy