spark安装和编译

最新推荐文章于 2022-10-10 14:55:11 发布

cyuyan2014

最新推荐文章于 2022-10-10 14:55:11 发布

阅读量664

点赞数

分类专栏： spark

本文链接：https://blog.youkuaiyun.com/u013239621/article/details/78448126

版权

spark 专栏收录该内容

1 篇文章

订阅专栏

1、去官网下载spark,如果找不到你要的版本，随便点击下载一个。把地址复制出来，在浏览器地址栏中粘贴。去掉后面的具体位置，打开，进入父目录，找一个你要的下载。如果没有对应的hadoop版本，比如说2.5.。就要下载without for hadoop,然后自己编译吧。

2、机器上要安装java,hadoop,scala,如果要编译，还需要安装maven。具体方法，解压，修改配置文件，可以百度。

3、修改scala目录下conf\spark_evn.sh. SPARK_DIST_CLASSPATH=$(hadoop classpath)。是小括号不是大括号。

maven编译

[root@hadoop1 conf]# export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
[root@hadoop1 conf]#  mvn -Pyarn -Phadoop-2.5 -Dhadoop.version=2.5.1 -DskipTests clean package

4、完成。（可惜我的报错了）

[WARNING] The requested profile "yarn" could not be activated because it does not exist.
[WARNING] The requested profile "hadoop-2.5" could not be activated because it does not exist.
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/root/spark-1.6.1-bin-without-hadoop/conf). Please verify you invoked Maven from the correct directory. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MissingProjectException
[root@hadoop1 conf]#
5、找了半天好像不用编译，直接去spark/bin/spark-shell下运行就可以


报错的可能是需要源码编译。或者直接启动shell有问题。

下次再想把。