创建Scrapy项目,Ubuntu 16.04 Python3.5 pip3

本文介绍了如何使用Python虚拟环境创建并配置Scrapy爬虫项目的过程。从搭建虚拟环境开始,到安装Scrapy库,再到启动新项目及生成首个爬虫,提供了详细的步骤指导。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

jhczz@vaio:~$ cd work/Python/Vietualenv/
jhczz@vaio:~/work/Python/Vietualenv$ virtualenv -p /usr/bin/python3.5 ENV3.5
Running virtualenv with interpreter /usr/bin/python3.5
Using base prefix '/usr'
New python executable in /home/jhczz/work/Python/Vietualenv/ENV3.5/bin/python3.5
Also creating executable in /home/jhczz/work/Python/Vietualenv/ENV3.5/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
jhczz@vaio:~/work/Python/Vietualenv$ cd ENV3.5/bin/
jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/bin$ source activate
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/bin$ cd ..
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5$ ll
total 36
drwxrwxr-x 6 jhczz jhczz 4096 Sep 12 09:07 ./
drwxrwxr-x 3 jhczz jhczz 4096 Sep 12 09:07 ../
drwxrwxr-x 2 jhczz jhczz 4096 Sep 12 09:07 bin/
drwxrwxr-x 2 jhczz jhczz 4096 Sep 12 09:07 include/
drwxrwxr-x 3 jhczz jhczz 4096 Sep 12 09:07 lib/
-rw-rw-r-- 1 jhczz jhczz   60 Sep 12 09:07 pip-selfcheck.json
drwxrwxr-x 3 jhczz jhczz 4096 Sep 12 09:07 share/
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5$ sudo pip3 install scrapy
[sudo] password for jhczz: 
The directory '/home/jhczz/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/jhczz/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): scrapy in /usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): queuelib in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.17.0 in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): Twisted>=13.1.0 in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): service-identity in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): parsel>=1.1 in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): PyDispatcher>=2.0.5 in /home/jhczz/.local/lib/python3.5/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=1.9 in /home/jhczz/.local/lib/python3.5/site-packages (from pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): hyperlink>=17.1.1 in /home/jhczz/.local/lib/python3.5/site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=4.0.2 in /home/jhczz/.local/lib/python3.5/site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): incremental>=16.10.1 in /home/jhczz/.local/lib/python3.5/site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): constantly>=15.1 in /home/jhczz/.local/lib/python3.5/site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): Automat>=0.3.0 in /home/jhczz/.local/lib/python3.5/site-packages (from Twisted>=13.1.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /home/jhczz/.local/lib/python3.5/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): attrs in /home/jhczz/.local/lib/python3.5/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /home/jhczz/.local/lib/python3.5/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.7 in /home/jhczz/.local/lib/python3.5/site-packages (from cryptography>=1.9->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.1 in /home/jhczz/.local/lib/python3.5/site-packages (from cryptography>=1.9->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): asn1crypto>=0.21.0 in /home/jhczz/.local/lib/python3.5/site-packages (from cryptography>=1.9->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/jhczz/.local/lib/python3.5/site-packages (from zope.interface>=4.0.2->Twisted>=13.1.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /home/jhczz/.local/lib/python3.5/site-packages (from cffi>=1.7->cryptography>=1.9->pyOpenSSL->scrapy)
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5$ scrapy startproject ArticleSpider
New Scrapy project 'ArticleSpider', using template directory '/usr/local/lib/python3.5/dist-packages/scrapy/templates/project', created in:
    /home/jhczz/work/Python/Vietualenv/ENV3.5/ArticleSpider

You can start your first spider with:
    cd ArticleSpider
    scrapy genspider example example.com
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5$ cd ArticleSpider/
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/ArticleSpider$ scrapy genspider jobbole blog.jobbole.com
Created spider 'jobbole' using template 'basic' in module:
  ArticleSpider.spiders.jobbole
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/ArticleSpider$ ll
total 24
drwxr-sr-x 3 jhczz jhczz 4096 Sep 12 09:10 ./
drwxrwxr-x 7 jhczz jhczz 4096 Sep 12 09:10 ../
drwxr-sr-x 4 jhczz jhczz 4096 Sep 12 09:10 ArticleSpider/
-rw-r--r-- 1 jhczz jhczz  270 Sep 12 09:10 scrapy.cfg
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/ArticleSpider$ cd ArticleSpider/
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/ArticleSpider/ArticleSpider$ ll
total 72
drwxr-sr-x 4 jhczz jhczz 4096 Sep 12 09:10 ./
drwxr-sr-x 3 jhczz jhczz 4096 Sep 12 09:10 ../
-rw-r--r-- 1 jhczz jhczz    0 Sep 11 15:38 __init__.py
-rw-rw-r-- 1 jhczz jhczz  292 Sep 12 09:10 items.py
-rw-rw-r-- 1 jhczz jhczz 1911 Sep 12 09:10 middlewares.py
-rw-rw-r-- 1 jhczz jhczz  293 Sep 12 09:10 pipelines.py
drwxr-sr-x 2 jhczz jhczz 4096 Sep 12 09:11 __pycache__/
-rw-rw-r-- 1 jhczz jhczz 3198 Sep 12 09:10 settings.py
drwxr-sr-x 3 jhczz jhczz 4096 Sep 12 09:11 spiders/
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/ArticleSpider/ArticleSpider$ cd spiders/
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/ArticleSpider/ArticleSpider/spiders$ ll
total 36
drwxr-sr-x 3 jhczz jhczz 4096 Sep 12 09:11 ./
drwxr-sr-x 4 jhczz jhczz 4096 Sep 12 09:10 ../
-rw-r--r-- 1 jhczz jhczz  161 Sep 11 15:38 __init__.py
-rw-rw-r-- 1 jhczz jhczz  231 Sep 12 09:11 jobbole.py
drwxr-sr-x 2 jhczz jhczz 4096 Sep 12 09:11 __pycache__/
(ENV3.5) jhczz@vaio:~/work/Python/Vietualenv/ENV3.5/ArticleSpider/ArticleSpider/spiders$ 



评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值