Hadoop+Spark集群安装步骤详解

一、环境：

操作系统版本：SUSE Linux Enterprise Server 11 (x86_64) SP3

主机名：

192.168.0.10    node1

192.168.0.11    node2

192.168.0.12    node3

192.168.0.13    node4

 

软件路径：/data/install

Hadoop 集群路径：/data

JAVA_HOME 路径：/usr/jdk1.8.0_66

 

版本

组件名	版本	说明
JRE	jdk-8u66-linux-x64.tar.gz
zookeeper	zookeeper-3.4.6.tar.gz
Hadoop	hadoop-2.7.3.tar.gz	主程序包
spark	spark-2.0.2-bin-hadoop2.7.tgz
hbase	hbase-1.2.5-bin.tar.gz

 

 

一、        常用命令

1.         查看系统版本：

linux-n4ga:~ # uname –a                            # 内核版本

Linux node1 3.0.76-0.11-default #1 SMP Fri Jun 14 08:21:43 UTC 2013 (ccab990) x86_64 x86_64 x86_64 GNU/Linux

linux-n4ga:~ # lsb_release                         # 发行版本

LSB Version:    core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch

linux-n4ga:~ # cat /etc/SuSE-release           # 补丁版本

SUSE Linux Enterprise Server 11 (x86_64)

VERSION = 11

PATCHLEVEL = 3

node1:~ # cat /etc/issue

Welcome to SUSE Linux Enterprise Server 11 SP3  (x86_64) - Kernel \r (\l).

node1:~ #

2.         启动集群

start-dfs.sh

start-yarn.sh

3.         关闭集群

stop-yarn.sh

stop-dfs.sh

4.         监控集群

hdfs dfsadmin -report

5.         单个进程启动 / 关闭

hadoop-daemon.sh start|stop namenode|datanode| journalnode

yarn-daemon.sh start |stop resourcemanager|nodemanager

http://blog.chinaunix.net/uid-25723371-id-4943894.html

 

 

 

二、        环境准备（所有服务器）

6.         关闭防火墙并禁止开机自启动

linux-n4ga:~ # rcSuSEfirewall2 stop

Shutting down the Firewall                                done

 

linux-n4ga:~ # chkconfig SuSEfirewall2_setup off

linux-n4ga:~ # chkconfig SuSEfirewall2_init off

linux-n4ga:~ # chkconfig --list|grep fire

SuSEfirewall2_init        0:off  1:off  2:off  3:off  4:off  5:off  6:off

SuSEfirewall2_setup       0:off  1:off  2:off  3:off  4:off  5:off  6:off

7.         设置主机名（其它类似）

linux-n4ga:~ # hostname node1

linux-n4ga:~ # vim /etc/HOSTNAME

node1.site

8.         ssh 免密登陆

node1:~ # ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

node1:~ # cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

node1:~ # ll -d .ssh/

drwx------ 2 root root 4096 Jun  5 08:50 .ssh/

node1:~ # ll .ssh/  

total 12

-rw-r--r-- 1 root root 599 Jun  5 08:50 authorized_keys

-rw------- 1 root root 672 Jun  5 08:50 id_dsa

-rw-r--r-- 1 root root 599 Jun  5 08:50 id_dsa.pub

把其它服务器的~/.ssh/id_dsa.pub 内容也追加到node1 服务器的~/.ssh/authorized_keys 文件中，然后分发

scp –rp ~/.ssh/authorized_keys root@192.168.0.11: ~/.ssh/

scp –rp ~/.ssh/authorized_keys root@192.168.0.12: ~/.ssh/

scp –rp ~/.ssh/authorized_keys root@192.168.0.13: ~/.ssh/

 

9.         修改 hosts 文件

node1:~ # vim /etc/hosts

… …

ff02::2         ipv6-allrouters

ff02::3         ipv6-allhosts

192.168.0.10    node1

192.168.0.11    node2

192.168.0.12    node3

192.168.0.13    node4

分发：

scp -rp /etc/hosts root@192.168.0.11:/etc/

scp -rp /etc/hosts root@192.168.0.12:/etc/

scp -rp /etc/hosts root@192.168.0.13:/etc/

10.     修改文件句柄数

node1:~ # vim /etc/security/limits.conf

*           soft   nofile       24000

*           hard  nofile       65535

*           soft  nproc        24000

*           hard  nproc       65535

node1:~ # source /etc/security/limits.conf

node1:~ # ulimit -n

24000

11.     时间同步

测试（举例）

node1 :~ # /usr/sbin/ntpdate 192.168.0.10

13 Jun 13:49:41 ntpdate[8370]: adjust time server 192.168.0.10 offset -0.007294 sec

添加定时任务

node1 :~ # crontab –e

*/10 * * * * /usr/sbin/ntpdate 192.168.0.10 > /dev/null 2>&1;/sbin/hwclock -w

node1:~ # service cron restart

Shutting down CRON daemon                                                          done

Starting CRON daemon                                                                             done

node1:~ # date

Tue Jun 13 05:32:49 CST 2017

node1:~ #

12.     上传安装包到 node1 服务器