爬虫服务器环境部署

 安装python

首先要升级python版本到2.7.12

1.下载安装包

wget  https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tar.xz

xz -d  Python-2.7.12.tar.xz

tar -xvf Python-2.7.12.tar.xz

cd Python-2.7.12

./configure

make

make install

2.备份老python,建立软链接

mv /usr/bin/python /usr/bin/python2.7.5

ln -s /usr/local/bin/python2.7 /usr/bin/python

 

3.升级python以后yum的使用可能会受到影响，需要进行一下设置

vi /usr/libexec/urlgrabber-ext-down

vi /usr/bin/yum  

找到第一行#!/usr/bin/python 改为 #!/usr/bin/python2.7.5

 

安装 scrapy

1.首先准备环境
yum install gcc libffi-devel openssl-devel libxml2 libxslt-devel libxml2-devel python-devel -y

 

2.安装easy_install
yum install python-setuptools

 

3.安装pip
easy_install pip

 

4.安装 lxml
easy_install lxml

 

5.安装scrapy
pip install scrapy

 

安装selenium

pip install -U selenium

 

 安装 fontconfig

yum install fontconfig