compile spark2.0.0 with maven

最新推荐文章于 2026-01-03 08:52:33 发布

原创最新推荐文章于 2026-01-03 08:52:33 发布 · 310 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark

spark 专栏收录该内容

2 篇文章

订阅专栏

本文介绍如何安装配置 Spark 2.0.0 版本，并提供详细的步骤说明，包括 Maven 的配置及 Spark 相关插件的启用。此外还提供了针对 Debian GNU/Linux 特别是 8/jessie 版本的解决方案。

spark version : spark-2.0.0
maven : Apache Maven 3.5.0 

cd /usr/local/spark-2.0.0
1
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

Note:
For Java 8 and above this step is not required.
If using build/mvn with no MAVEN_OPTS set, the script will automate this for you.

2 
mvn -Pyarn -Phadoop-2.7 -Pspark-ganglia-lgpl -Pkinesis-asl -Phive -DskipTests clean package


3
./dev/make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.7 -Phive -Phive-thriftserver -Pyarn

Note:
if you come across spark-core_2.11: Command execution failed
may be you should try some ways the following 
On Debian GNU/Linux (in particular 8/jessie) you would need to run 
apt-get install r-base r-base-dev
or 
aptitude install r-base r-base-dev