【环境安装】Spark2.4.4_spark-2.4.4-bin-hadoop2.6.tgz-优快云博客

本文档详细介绍了如何在集群环境下安装和配置Spark2.4.4，包括从下载源码到启动集群的每一步骤，以及验证安装成功的实践操作。通过这个指南，你可以学会在Master和Slave节点上安装Spark，并使用网页监控面板进行集群状态监控。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

0. Spark源码包下载

mirror.bit.edu.cn/apache/spark/

1. 集群环境

Master 172.16.71.10

Slave1 172.16.71.11

Slave2 172.16.71.12

2. 下载软件包

#Master

wget http://mirror.bit.edu.cn/apache/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.6.tgz

tar zxvf spark-2.4.4-bin-hadoop2.6.tgz

wget https://downloads.lightbend.com/scala/2.12.4/scala-2.12.4.tgz

tar zxvf scala-2.12.4.tgz

3. 修改Spark配置文件

cd /usr/local/src/spark-2.4.4-bin-hadoop2.6/conf

mv spark-env.sh.template spark-env.sh

vim spark-env.sh

export SCALA_HOME=/usr/local/src/scala-2.11.12

export JAVA_HOME=/usr/local/src/jdk1.8.0_272

export HADOOP_HOME=/usr/local/src/hadoop-2.6.5

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

SPARK_MASTER_IP=master

SPARK_LOCAL_DIRS=/usr/local/src/spark-2.4.4-bin-hadoop2.6

SPARK_DRIVER_MEMORY=1G

mv slaves.template slaves

vim slaves

slave1

slave2

4. 配置环境变量

vim ~/.bashrc

export SPARK_HOME=/usr/local/src/spark-2.4.4-bin-hadoop2.6

export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

source ~/.bashrc

5. 拷贝安装包及环境变量

scp -r /usr/local/src/spark-2.4.4-bin-hadoop2.6 root@slave1:/usr/local/src/spark-2.4.4-bin-hadoop2.6

scp -r /usr/local/src/spark-2.4.4-bin-hadoop2.6 root@slave2:/usr/local/src/spark-2.4.4-bin-hadoop2.6

scp -r ~/.bashrc root@slave1:~/

scp -r ~/.bashrc root@slave2:~/

source ~/.bashrc

在slave1,slave2 执行

6. 启动集群

在master

cd /usr/local/src/spark-2.4.4-bin-hadoop2.6/sbin

./start-all.sh