python爬虫周学习___赵俊杰

Scrapy安装与使用
打开命令提示符下载安装Scrapy所必须的环境：

优先下载python下载更新文件：python -m pip install --upgrade pip

然后下载：

pip install wheel

pip install lxml

pip install twisted

pip install pywin32

pip install scrapy

下载之后输入：pip list查询是否下载成功

创建项目：

scrapy start project TXmovies

cd TXmovies

scrapy genspider txms v.qq.com

修改setting:

ROBOTSTXT_OBEY = False

DOWNLOAD_DELAY=1

DEFAULT_REQUEST_HEADERS{

        'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',

        'Accept-Language':'en',

        'UserAgent':'Mozilla/5.0(WindowsNT6.2;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/27.0.1453.94Safari/537.36'

}

ITEM_PIPELINES={

        'TXmovies.pipelines.TxmoviesPipeline':300,

}

创建一个run项

from scrapy import cmdline
cmdline.exectute('scrapy crawl txms',sp;it())