spark-1.2.0 集群环境搭建

最新推荐文章于 2022-03-09 20:27:17 发布

转载最新推荐文章于 2022-03-09 20:27:17 发布 · 541 阅读

文章标签：

#spark-1.2.0 集群环境搭建

hadoop大数据同时被 2 个专栏收录

19 篇文章

订阅专栏

spark

3 篇文章

订阅专栏

本文详细介绍了如何从零开始搭建Spark1.2.0集群环境，包括Scala和Spark的下载与安装、环境变量配置、Spark集群配置及测试等关键步骤。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、下载scala2.11.4版本下载地址为：http://www.scala-lang.org/download/2.11.4.html ，也可以使用wget http://downloads.typesafe.com/scala/2.11.4/scala-2.11.4.tgz?_ga=1.248348352.61371242.1418807768

2、解压和安装：解压：[spark@S1PA11 scala]$ tar -xvf scala-2.11.4.tgz ,安装：[spark@S1PA11 scala]$ mv scala-2.11.4 ~/opt/

3、编辑 ~/.bash_profile文件增加SCALA_HOME环境变量配置，

export JAVA_HOME=/home/spark/opt/java/jdk1.6.0_37
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
export SCALA_HOME=/home/spark/opt/scala-2.11.4
export HADOOP_HOME=/home/spark/opt/hadoop-2.6.0
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:${SCALA_HOME}/bin

立即生效 bash_profile ，[spark@S1PA11 scala]$ source ~/.bash_profile

4、验证scala：[spark@S1PA11 scala]$ scala -version
Scala code runner version 2.11.4 -- Copyright 2002-2013, LAMP/EPFL

[spark@S1PA11 scala]$ scala
Welcome to Scala version 2.11.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_37).
Type in expressions to have them evaluated.
Type :help for more information.

scala> var str = "SB is"+"SB"
str: String = SB isSB

scala>

5、copy到slave机器，[spark@S1PA11 scala]$ scp ~/.bash_profile spark@10.126.45.56:~/.bash_profile

6、下载spark，wget http://d3kbcqa49mib13.cloudfront.net/spark-1.2.0-bin-hadoop2.4.tgz

7、在master主机配置spark ：

将下载的spark-1.2.0-bin-hadoop2.4.tgz 解压到 ~/opt/目前了即 ~/opt/spark-1.2.0-bin-hadoop2.4，配置环境变量SPARK_HOME

# set java env
export JAVA_HOME=/home/spark/opt/java/jdk1.6.0_37
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
export SCALA_HOME=/home/spark/opt/scala-2.11.4
export HADOOP_HOME=/home/spark/opt/hadoop-2.6.0
export SPARK_HOME=/home/spark/opt/spark-1.2.0-bin-hadoop2.4
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:${HADOOP_HOME}/bin

配置完成后使用source命令使配置生效

进入 spark conf目录：

[spark@S1PA11 opt]$ cd spark-1.2.0-bin-hadoop2.4/
[spark@S1PA11 spark-1.2.0-bin-hadoop2.4]$ ls
bin conf data ec2 examples lib LICENSE logs NOTICE python README.md RELEASE sbin work
[spark@S1PA11 spark-1.2.0-bin-hadoop2.4]$ cd conf/
[spark@S1PA11 conf]$ ls
fairscheduler.xml.template metrics.properties.template slaves.template spark-env.sh
log4j.properties.template slaves spark-defaults.conf.template spark-env.sh.template

first ：修改slaves文件，增加两个slave节点S1PA11、S1PA222

[spark@S1PA11 conf]$ vi slaves
S1PA11
S1PA222

second：配置spark-env.sh

首先把spark-env.sh.template copy spark-env.sh

vi spark-env.sh文件在最下面增加：

export JAVA_HOME=/home/spark/opt/java/jdk1.6.0_37
export SCALA_HOME=/home/spark/opt/scala-2.11.4
export SPARK_MASTER_IP=10.58.44.47
export SPARK_WORKER_MEMORY=2g
export HADOOP_CONF_DIR=/home/spark/opt/hadoop-2.6.0/etc/hadoop

HADOOP_CONF_DIR是Hadoop配置文件目录，SPARK_MASTER_IP主机IP地址，SPARK_WORKER_MEMORY是worker使用的最大内存

完成配置后，将spark目录copy slave机器 scp -r ~/opt/spark-1.2.0-bin-hadoop2.4 spark@10.126.45.56:~/opt/

8、启动spark分布式集群并查看信息

[spark@S1PA11 sbin]$ ./start-all.sh

查看：

[spark@S1PA11 sbin]$ jps
31233 ResourceManager
27201 Jps
30498 NameNode
30733 SecondaryNameNode
5648 Worker
5399 Master
15888 JobHistoryServer
如果HDFS没有启动，请启动起来，参考hadoop集群搭建点击打开链接

查看slave节点：

[spark@S1PA222 scala]$ jps
20352 Bootstrap
30737 NodeManager
7219 Jps
30482 DataNode
29500 Bootstrap
757 Worker

9、页面查看集群状况：

进去spark集群的web管理页面，访问