CDH+apache spark集群

Spark 1.6.3 集群部署与配置

最新推荐文章于 2022-08-08 11:25:36 发布

原创最新推荐文章于 2022-08-08 11:25:36 发布 · 372 阅读

1 ·

CC 4.0 BY-SA版权

Spark&Spark streaming 专栏收录该内容

4 篇文章

订阅专栏

本文详细介绍Spark 1.6.3版本在Hadoop 2.6环境下的集群部署步骤，包括环境变量配置、spark-defaults.conf、slaves和spark-env.sh文件的设置，以及如何通过yarn cluster模式提交Job，并提供了Spark HistoryServer的访问路径。

spark-1.6.3-bin-hadoop2.6.tgz 官网下载，环境变量配置：

/etc/profile bdp-40 节点

81 # set java env
82 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
83 export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
84 export PATH=$PATH:$JAVA_HOME/bin
85
86 # set CDH env
87 export CDH_BIN=/disk4/bd/opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/bin
88 export PATH=$PATH:$JAVA_HOME/bin:$CDH_BIN
89
90 # set zk env
91 export ZOOKEEPER_HOME=/disk4/bd/opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/zookeeper
92 export PATH=$ZOOKEEPER_HOME/bin:$PATH
93
94 # set spark env
95 export SPARK_HOME=/disk4/bd/spark-1.6.3-bin-hadoop2.6
96 export PATH=$SPARK_HOME/bin:$PATH

spark-defaults.conf 配置：

21 spark.eventLog.enabled true
22 spark.eventLog.dir hdfs://finai/tmp/spark_1.6.3/sparkHistoryLogs
23 spark.serializer org.apache.spark.serializer.KryoSerializer
24 spark.eventLog.compress true
25 #spark.history.updateInterval 5
26 spark.history.ui.port 18080
27 spark.history.fs.logDirectory hdfs://finai/tmp/spark_1.6.3/sparkHistoryLogs

slaves 配置：

bdp-40
bdp-41
bdp-42
bdp-43
bdp-44
bdp-45
bdp-46
bdp-47
bdp-48
bdp-49
bdp-50
bdp-51

spark-env.sh 配置：
37 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
38 export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
39 # export PATH=$PATH:$JAVA_HOME/bin
40
41 export HADOOP_HOME=/disk4/bd/opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hadoop
42 export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
43 export YARN_HOME=$HADOOP_HOME
44 export YARN_CONF_DIR=$YARN_HOME/etc/hadoop
45
46 export SPARK_HOME=/disk4/bd/spark-1.6.3-bin-hadoop2.6
47 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
48 export SPARK_LIBARY_PATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME/lib/native
49
50 #export SPARK_MASTER_IP=finai-bdp-40
51 export SPARK_LOCAL_IP=finai-bdp-40
52 export SPARK_DRIVER_MEMORY=1G

job 提交测试：

yarn cluster模式提交job：
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1G --executor-memory 1G --executor-cores 1 /disk4/bd/spark-1.6.3-bin-hadoop2.6/lib/spark-examples-1.6.3-hadoop2.6.0.jar 40

spark History Server 查看:
http://10.*.3.40:18080/