OS X 10.11.5 Scrapy安装指南

最新推荐文章于 2022-05-08 22:57:51 发布

转载最新推荐文章于 2022-05-08 22:57:51 发布 · 64 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：https://yq.aliyun.com/articles/577215

文章标签：

#python #shell

本文详细记录了在Mac OS X 10.11.5系统上安装Scrapy的过程，包括解决安装过程中遇到的libxml2和lxml依赖问题，并最终成功安装Scrapy。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

今天终于在安装成功了scrapy，之前试过很多次都安装失败，系统重装之后也没有成功，在mac系统上的安装不像windows那么容易，今天再次试了一遍成功了。
我的系统：OS X 10.11.5
python：2.7.10

安装pip

pip是python 的包管理器，按照这里的安装指南https://pip.pypa.io/en/stable/installing/，先下载get-pip.py,之后运行

python get-pip.py

查看pip是否安装成功：

  ~ pip -V
pip 8.1.2 from /Library/Python/2.7/site-packages (python 2.7)

安装scrapy

使用pip安装scrapy只需要一条命令：

sudo pip install scrapy

但是现实并不是那么顺利，报错信息：

 #include "libxml/xpath.h"
           ^
  1 error generated.
  *********************************************************************************
  Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
  Perhaps try: xcode-select --install
  *********************************************************************************
  error: command 'cc' failed with exit status 1

提示libxml安装失败，执行命令：

xcode-select --install

之后安装libxml：

sudo  pip install lxml

提示信息：

  ~ sudo pip install lxml 

Collecting lxml
  Downloading lxml-3.6.0.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 323kB/s 
Installing collected packages: lxml
  Running setup.py install for lxml ... done
Successfully installed lxml-3.6.0

lxml安装成功以后再次执行安装scrapy命令：

  ~ sudo pip install scrapy

提示scrapy安装成功，执行一下测试：

  ~ scrapy
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 34, in <module>
    from scrapy.spiders import Spider
  File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line 12, in <module>
    from scrapy.http.request.rpc import XmlRpcRequest
  File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py", line 7, in <module>
    from six.moves import xmlrpc_client as xmlrpclib
ImportError: cannot import name xmlrpc_client

好事多磨，离成功还有一步，依次执行以下命令：

sudo rm -rf /Library/Python/2.7/site-packages/six*
sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six*
sudo pip install six

再次测试：

  ~ scrapy
Scrapy 1.1.0 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  commands      
  fetch         Fetch a URL using the Scrapy downloader
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

成功！