spark搭建

1.准备tools包

2.ftp上传到~目录下

3.然后把文件全部解压

tar -zxvf [文件名]

rm -f[文件]

3.配置环境文件

先echo $HOME 看一下当前路径

vi ~/.bash_profile

export JAVA_HOME=$HOME/jdk1.7.0_79

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export SCALA_HOME=$HOME/scala-2.10.5

export SPARK_HOME=$HOME/spark-1.4.0-bin-hadoop2.6

export HADOOP_HOME=$HOME/hadoop-2.6.0

export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop

export MAVEN_HOME=$HOME/apache-maven-3.2.5

export SBT_HOME=$HOME/sbt

export PATH=$PATH:$SCALA_HOME/bin:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MAVEN_HOME/bin:$SBT_HOME/bin

source ~/.bash_profile

最后：X保存

4.每个测试一下

java -version

scala -version

mvn --version

sbt --version

5.配置Spark

[root@bigdata ~]# cd /root/spark-1.4.0-bin-hadoop2.6/conf

--复制配置文件

[root@bigdata conf]# cp spark-env.sh.template spark-env.sh

在spark-env.sh最后添加下面

[root@bigdata conf]#vi spark-env.sh

export SCALA_HOME=/root/scala-2.10.5

export SPARK_MASTER_IP=bigdata

export SPARK_WORKER_MEMORY=2G

export JAVA_HOME=/root/jdk1.7.0_79

source spark-env.sh

6.配置slaves

[root@bigdata conf]#vi slaves

在slaves最后添加下面

bigdata

bigdata2

bigdata3

7.启动节点

cd spark-1.4.0-bin-hadoop2.6/

启劢master ./sbin/start-master.sh

启劢worker ./sbin/start-slaves.sh spark://localhost:7077

启劢Shell ./bin/spark-shell --master spark://localhost:7077

因为系统在虚拟机下面，如果在windows下浏览器访问需要如下操作：

1.另外要在windows访问

修改Linux系统防火墙配置需要修改 /etc/sysconfig/iptables 这个文件，如果要开放哪个端口，在里面添加一条

 -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT

  service iptables stop --停止

  service iptables start --启动

或者直接关闭防火墙：

service iptables stop

service iptables start

--------------------------------------重复---------------------------------------

1.解压包

tar zxf jdk-7u79-linux-x64.gz

tar zxf sbt-0.13.7.tgz

tar zxf spark-1.4.0-bin-hadoop2.6.tgz

tar zxf apache-maven-3.2.5-bin.tar.gz

tar zxf hadoop-2.6.0.tar.gz

tar zxf scala-2.10.5.tgz

tar zxf spark-1.4.0.tgz

2.配置环境

vi ~/.bash_profile

export JAVA_HOME=$HOME/jdk1.7.0_79

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export SCALA_HOME=$HOME/scala-2.10.5

export SPARK_HOME=$HOME/spark-1.4.0-bin-hadoop2.6

export HADOOP_HOME=$HOME/hadoop-2.6.0

export HADOOP_CONF_DIR=$HOME/hadoop-2.6.0/etc/hadoop

export MAVEN_HOME=$HOME/apache-maven-3.2.5

export SBT_HOME=$HOME/sbt

export PATH=$PATH:$SCALA_HOME/bin:$SPARK_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MAVEN_HOME/bin:$SBT_HOME/bin

source ~/.bash_profile

3.配置vi spark-env.sh文件

cd spark-1.4.0-bin-hadoop2.6/conf/

cp spark-env.sh.template spark-env.sh

vi spark-env.sh

export SCALA_HOME=/root/scala-2.10.5

export SPARK_MASTER_IP=bigdata

export SPARK_WORKER_MEMORY=2G

export JAVA_HOME=/root/jdk1.7.0_79

source spark-env.sh

4.配置vi slaves文件

vi slaves

bigdata

bigdata2

bigdata3

3.启动spark服务

./sbin/start-master.sh

关闭：./sbin/stop-master.sh

./sbin/start-slaves.sh spark://localhost:7077

关闭：./sbin/stop-slaves.sh

./bin/spark-shell --master spark://localhost:7077

4.配置防火墙

vi /etc/sysconfig/iptables

service iptables stop

service iptables start

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 8081 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 7077 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 4040 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 6066 -j ACCEPT

5.Maven编译

./make-distribution.sh --tgz -Phadoop-2.6 -Pyarn -DskipTests -Dhadoop.version=2.6.0 -Phive -Phivethriftserver clean package

dataDir=/root/zookeeper-3.4.6/data

dataLogDir=/root/zookeeper-3.4.6/logs

6.配置主机名(优先做这个)

vim    /etc/sysconfig/network 

vim /etc/hosts

service network restart 

reboot    //重启

9.SSH免密登陆

a：生成密钥

ssh-keygen -t rsa

b：把本机的公钥追到bigdata2的 .ssh/authorized_keys 里

ssh-copy-id -i ~/.ssh/id_rsa.pub root@bigdata2

c：测试 ssh feng02

Zookeeper 安装

1 .下载

tar zxf zookeeper-3.4.6.tar.gz

cd zookeeper-3.4.6

2 .配置

cd conf

cp zoo_sample.cfg zoo.cfg

vi zoo.cfg

（注意：默认的包是没有data和logs目录的，需要新建）

mkdir --新建目录

mkdir logs

mkdir data

最后面加上

dataDir=/root/zookeeper-3.4.6/data

dataLogDir=/root/zookeeper-3.4.6/logs

server.3=bigdata3:2888:3888

server.2=bigdata2:2888:3888

server.1=bigdata:2888:3888

4.复制到其它节点并设置serverid

scp -r ./zookeeper-3.4.6 bigdata2:/root/

scp -r ./zookeeper-3.4.6 bigdata3:/root/

echo "3">myid

(不同的机器设置不同的数字)

cd bin

启动

./zkServer.sh start

测试

./zkCli.sh

Spark配置

1 .修改配置

export SCALA_HOME=/root/scala-2.10.5

#export SPARK_MASTER_IP=bigdata

export SPARK_WORKER_MEMORY=2G

export JAVA_HOME=/root/jdk1.7.0_79

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -

Dspark.deploy.zookeeper.url=bigdata:2181,bigdata2:2181,bigdata3:2181 -Dspark.deploy.zookeeper.dir=/spark"

复制到其它2个节点

scp -r ./spark-env.sh bigdata2:/root/spark-1.4.0-bin-hadoop2.6/conf

scp -r ./spark-env.sh bigdata3:/root/spark-1.4.0-bin-hadoop2.6/conf