centos 部署 SparkR

---恢复内容开始---

环境配置——

• 操作系统：CentOS 6.5
• JDK版本：1.7.0_67
• Hadoop集群版本：CDH 5.3.0

 

安装过程——

1、（1）安装R

yum install -y R

（2）出现问题：

yum 安装软件时，报错：No package XXX available.

[root@localhost ~]# yum -y install redis
Loaded plugins: fastestmirror
No package redis available.
Nothing to do

（3）解决方案

配置一台CentOS 7.x或RHEL 7.x的系统使用Fedora EPEL  repos和第三方软件包安装各种软件包。请注意，这些软件包没有正式获得任何的CentOS或Red Hat的支持，但提供了许多流行的包和应用程序。

显示以上错误，有以下2种原因：

1、dns问题；

2、镜像连接错误；

DNS问题

首先测试下网络连接，如果ping不通，需要设置下DNS。

[root@localhost ~]# ping www.baidu.com
XXXXXXXXXXXXXXXXX
[root@localhost ~]# echo "nameserver 8.8.8.8">>/etc/resolv.conf

再次ping即可ping通了。

镜像连接错误问题

具体原因：使用yum搜索某些rpm包，找不到包是因为CentOS是RedHat企业版编译过来的，去掉了所有关于版权问题的东西。安装EPEL后可以很好的解决这个问题。EPEL(Extra Packages for Enterprise Linux )即企业版Linux的扩展包，提供了很多可共Centos使用的组件，安装完这个以后基本常用的rpm都可以找到。

Method #1 » 从企业版Linux库配置包中安装扩展包。（推荐 ）

 

[root@localhost ~]# sudo yum install epel-release
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-5 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                Arch             Version         Repository        Size
================================================================================
Installing:
 epel-release           noarch           7-5             extras            14 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 14 k
Installed size: 24 k
Is this ok [y/d/N]: y
Downloading packages:
epel-release-7-5.noarch.rpm                                |  14 kB   00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : epel-release-7-5.noarch                                      1/1 
  Verifying  : epel-release-7-5.noarch                                      1/1 

Installed:
  epel-release.noarch 0:7-5                                                     

Complete!

(4)最后运行下列代码 即可完成安装

yum install -y R

 

2、安装curl-devel（非常重要！否则无法进行RCurl包的安装，进而无法安装devtools）

yum install -y curl-devel

3.在R shell中安装devtools

 

install.packages("RCurl")
install.packages("devtools")

install.packages("devtools")

出现错误：

[plain] view plain copy
------------------------- ANTICONF ERROR ---------------------------  
Configuration failed because libcurl was not found. Try installing:  
 * deb: libcurl4-openssl-dev (Debian, Ubuntu, etc)  
 * rpm: libcurl-devel (Fedora, CentOS, RHEL)  
 * csw: libcurl_dev (Solaris)  
If libcurl is already installed, check that 'pkg-config' is in your  
PATH and PKG_CONFIG_PATH contains a libcurl.pc file. If pkg-config  
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:  
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'  
--------------------------------------------------------------------  
ERROR: configuration failed for package ‘curl’  
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/curl’  
* installing *source* package ‘whisker’ ...  
** package ‘whisker’ successfully unpacked and MD5 sums checked  
** R  
** inst  
** preparing package for lazy loading  
** help  
*** installing help indices  
** building package indices  
** testing if installed package can be loaded  
* DONE (whisker)  
* installing *source* package ‘digest’ ...  
** package ‘digest’ successfully unpacked and MD5 sums checked  
** libs  

[plain] view plain copy
configure: error: in `/tmp/RtmpkQMD9t/R.INSTALL6c1166a0db0e/git2r':  
configure: error:   
  ---------------------------------------------  
   OpenSSL library required  
  
   Please install:  
   libssl-dev (deb) or openssl-devel (rpm)  
  ---------------------------------------------  
See `config.log' for more details  
ERROR: configuration failed for package ‘git2r’  
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/git2r’  
* installing *source* package ‘stringr’ ...  
** package ‘stringr’ successfully unpacked and MD5 sums checked  
** R  
** inst  
** preparing package for lazy loading  
** help  
*** installing help indices  
** building package indices  
** installing vignettes  
** testing if installed package can be loaded  
* DONE (stringr)  
* installing *source* package ‘xml2’ ...  
** package ‘xml2’ successfully unpacked and MD5 sums checked  
Package libxml-2.0 was not found in the pkg-config search path.  
Perhaps you should add the directory containing `libxml-2.0.pc'  
to the PKG_CONFIG_PATH environment variable  
No package 'libxml-2.0' found  
Package libxml-2.0 was not found in the pkg-config search path.  
Perhaps you should add the directory containing `libxml-2.0.pc'  
to the PKG_CONFIG_PATH environment variable  
No package 'libxml-2.0' found  
Using PKG_CFLAGS=  
Using PKG_LIBS=-lxml2  
------------------------- ANTICONF ERROR ---------------------------  
Configuration failed because libxml-2.0 was not found. Try installing:  
 * deb: libxml2-dev (Debian, Ubuntu, etc)  
 * rpm: libxml2-devel (Fedora, CentOS, RHEL)  
 * csw: libxml2_dev (Solaris)  
If libxml-2.0 is already installed, check that 'pkg-config' is in your  
PATH and PKG_CONFIG_PATH contains a libxml-2.0.pc file. If pkg-config  
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:  
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'  
--------------------------------------------------------------------  
ERROR: configuration failed for package ‘xml2’  
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/xml2’  
* installing *source* package ‘memoise’ ...  
** package ‘memoise’ successfully unpacked and MD5 sums checked  
** R  
** preparing package for lazy loading  
** help  
*** installing help indices  
** building package indices  
** testing if installed package can be loaded  
* DONE (memoise)  
ERROR: dependency ‘curl’ is not available for package ‘httr’  
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/httr’  
* installing *source* package ‘evaluate’ ...  
** package ‘evaluate’ successfully unpacked and MD5 sums checked  
** R  
** preparing package for lazy loading  
** help  
*** installing help indices  
** building package indices  
** testing if installed package can be loaded  
* DONE (evaluate)  
* installing *source* package ‘roxygen2’ ...  
** package ‘roxygen2’ successfully unpacked and MD5 sums checked  
** libs  
g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c RcppExports.cpp -o RcppExports.o  
g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c isComplete.cpp -o isComplete.o  
g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c leadingSpaces.cpp -o leadingSpaces.o  
g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c parser.cpp -o parser.o  
g++ -I/usr/local/lib64/R/include -DNDEBUG  -I/usr/local/include -I"/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/Rcpp/include"   -fpic  -g -O2  -c wrapString.cpp -o wrapString.o  
g++ -shared -L/usr/local/lib64 -o roxygen2.so RcppExports.o isComplete.o leadingSpaces.o parser.o wrapString.o -L/usr/local/lib64/R/lib -lR  
installing to /home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/roxygen2/libs  
** R  
** inst  
** preparing package for lazy loading  
** help  
*** installing help indices  
** building package indices  
** installing vignettes  
** testing if installed package can be loaded  
* DONE (roxygen2)  
ERROR: dependencies ‘curl’, ‘xml2’ are not available for package ‘rversions’  
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/rversions’  
ERROR: dependencies ‘httr’, ‘curl’, ‘rversions’, ‘git2r’ are not available for package ‘devtools’  
* removing ‘/home/jifeng/R/x86_64-unknown-linux-gnu-library/3.1/devtools’  
  
The downloaded source packages are in  
        ‘/tmp/Rtmp1A16li/downloaded_packages’  
Warning messages:  
1: In install.packages("devtools") :  
  installation of package ‘curl’ had non-zero exit status  
2: In install.packages("devtools") :  
  installation of package ‘git2r’ had non-zero exit status  
3: In install.packages("devtools") :  
  installation of package ‘xml2’ had non-zero exit status  
4: In install.packages("devtools") :  
  installation of package ‘httr’ had non-zero exit status  
5: In install.packages("devtools") :  
  installation of package ‘rversions’ had non-zero exit status  
6: In install.packages("devtools") :  
  installation of package ‘devtools’ had non-zero exit status  

根据错误安装，我这里安装了

[plain] view plain copy
[root@feng03 ~]# yum install libcurl-devel  
[root@feng03 ~]# yum install openssl-devel  
[root@feng03 ~]# yum install libxml2-devel  

进入R shell
>install.packages("git2r")
>install.packages("xml2")
>install.packages("rversions")

　　

步骤4: 安装好rJava及devtools，接下来安装SparkR

library(devtools)
install_github("amplab-extras/SparkR-pkg", subdir="pkg")

出现错误

Launching sbt from sbt/sbt-launch-0.13.6.jar

Error: Invalid or corrupt jarfile sbt/sbt-launch-0.13.6.jar

make: *** [target/scala-2.10/sparkr-assembly-0.1.jar] 错误 1

ERROR: compilation failed for package ‘SparkR’

* removing ‘/home/ndscbigdata/R/x86_64-pc-linux-gnu-library/3.2/SparkR’

错误: Command failed (1)

 

解决：

进入spark安装包中的R目录，直接输入如下命令，直接就DONE了。

 

cd进入目录

/data1/bigdata/spark-1.6.1/R/lib/SparkR

 R -e "devtools::install('.')"

 

---恢复内容结束---