Scrapy学习

1. 如何创建一个Scrapy的HelloWorld项目程序

1.1 通过Anaconda下载scrapy框架

- activate tensorflow
-  conda install scrapy

下载之后可以通过输入命令：scrapy查看

Scrapy 1.6.0 - project: scPro1

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  check         Check spider contracts
  crawl         Run a spider
  edit          Edit spider
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  list          List available spiders
  parse         Parse URL (using its spider) and print the results
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

Use "scrapy <command> -h" to see more info about a command

2. 创建爬虫项目

scrapy startproject 项目名
创建好之后在当前路径下会出现一个文件夹

如图所示：

3. 右键该文件夹用Pycharm打开

4. 创建爬虫文件

5. 创建之后在Pycharm中效果如下：

6. 启动爬虫程序

	scrapy crawl baidu
	这里的baidu是爬虫文件名，也就是baidu.py中name的值