win10, python3.6, pycharm IDE, 采用pip安装scrapy库遇到的坑

在win10, python3.6, pycharm IDE下, 采用pip安装scrapy库遇到了一些坑，来记录一下，能给大家节省一些时间。

 

1.pip 升级的坑

Package    Version
---------- -------
pip        10.0.1
setuptools 39.1.0
You are using pip version 10.0.1, however version 19.0.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

采用pycharm的虚拟环境，基于python3.6，库很干净。打算安装提示升级的，但是出现以下信息：

AttributeError: 'NoneType' object has no attribute 'bytes' 

考虑到主要是来安装scrapy的，并没有细追究，直接

easy_install -U pip 

直接搞定！

2. pip安装的坑

pip install scrapy

前面的依赖库都很好安装，但到了安装Twisted会出现问题，最终会报错。

  Running setup.py install for Twisted ... -

  building 'twisted.test.raiser' extension
    error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

    ----------------------------------------
Command "D:\CODELan\test\venv\Scripts\python.exe -u -c "import setuptools, tokenize;__file__='D:\\temp\\AppData\\Local\\Temp\\pip-install-icdvcm6n\\Twisted\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.
close();exec(compile(code, __file__, 'exec'))" install --record D:\temp\AppData\Local\Temp\pip-record-00v8rouy\install-record.txt --single-version-externally-managed --compile --install-headers D:\CODELan\test\venv\include\site\python3.6\Twisted" fa
iled with error code 1 in D:\temp\AppData\Local\Temp\pip-install-icdvcm6n\Twisted\

 解决方法：采用whl的方式安装

1.下载twisted的whl文件：

https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted

2.打开文件所在文件夹，点击属性，选择安全，复制对象名称。然后终端输入：

(venv) D:\CODELan\test>pip install E:\Twisted-18.9.0-cp36-cp36m-win_amd64.whl

稍等会就看到以下提示： 

Installing collected packages: Twisted
Successfully installed Twisted-18.9.0

安装成功了，再次安装scrapy

(venv) D:\CODELan\test>pip install scrapy

稍等会就会看到

Installing collected packages: lxml, w3lib, parsel, scrapy
Successfully installed lxml-4.3.1 parsel-1.5.1 scrapy-1.6.0 w3lib-1.20.0

3.终端输入scrapy，查看是否安装成功。

(venv) D:\CODELan\test>scrapy
Scrapy 1.6.0 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

3.创建demo试运行

(venv) D:\CODELan\test>scrapy startproject hello
New Scrapy project 'hello', using template directory 'd:\codelan\test\venv\lib\site-packages\scrapy\templates\project', created in:
    D:\CODELan\test\hello

You can start your first spider with:
    cd hello
    scrapy genspider example example.com

cd hello

(venv) D:\CODELan\test\hello>scrapy genspider baidu www.baidu.com
Created spider 'baidu' using template 'basic' in module:
  hello.spiders.baidu

scrapy crawl baidu

本以为可以运行成功的，结果提示没有win32api。

ModuleNotFoundError: No module named 'win32api'

解决方法：

(venv) D:\CODELan\test\hello>pip install Pywin32
Collecting Pywin32
  Using cached https://files.pythonhosted.org/packages/b2/1a/7727b406391b0178b6ccb7e447e963df5ebf1ce9e0f615fc6ce23b6f6753/pywin32-224-cp36-cp36m-win_amd64.whl
Installing collected packages: Pywin32
Successfully installed Pywin32-224

安装成功后，再次运行应该就不会出错了。

>scrapy crawl baidu

 

 

以上，就是自己安装scrapy中出现的一些问题，希望遇到相同问题的，可以帮助到你们。