java学习讨论群:725562382 欢迎加入学习
先从别人那里复制了个思维导图给大家参考
一.主机环境准备
1>.操作系统环境
[root@backup01 hadoop-3.1.2]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@backup01 hadoop-3.1.2]# uname -r
3.10.0-862.el7.x86_64
[root@backup01 hadoop-3.1.2]# uname -m
x86_64
[root@backup01 hadoop-3.1.2]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
devtmpfs 471M 0 471M 0% /dev
tmpfs 488M 0 488M 0% /dev/shm
tmpfs 488M 15M 473M 4% /run
tmpfs 488M 0 488M 0% /sys/fs/cgroup
/dev/sda1 197M 135M 62M 69% /boot
tmpfs 98M 4.0K 98M 1% /run/user/42
tmpfs 98M 44K 98M 1% /run/user/1000
/dev/sr0 4.2G 4.2G 0 100% /run/media/admin/CentOS 7 x86_64
tmpfs 98M 0 98M 0% /run/user/0
[root@backup01 hadoop-3.1.2]# df -h *
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
/dev/mapper/centos-root 10G 8.3G 1.8G 83% /
[root@backup01 hadoop-3.1.2]# free -h
total used free shared buff/cache available
Mem: 974M 600M 105M 4.3M 268M 155M
Swap: 2.0G 261M 1.7G
我这边有 1 个 master 主机(主节点),2 个 slave 主机(从节点),
3 台主机的 hosts 配置文件如下
master的hostname
slave1的hostname
slave2的hostname
2> ssh免密登录
[root@backup01software]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:r7urq4AdFvcCOq7dBafbRbNe61FA8Yvu+/SEHJZ9uFM root@backup01
The key's randomart image is:
+---[RSA 2048]----+
| o. |
| . . |
| o . . . |
| . + . o + . |
|o o o o S . * o E|
|.= . = . = + o + |
|..o . . o = + + |
|.. o + o + + o . |
|. . +.+o*==o. . |
+----[SHA256]-----+
[root@backup01 software]# cd /root/
[root@backup01 ~]# ll
total 8
-rw-------. 1 root root 2044 May 4 18:14 anaconda-ks.cfg
-rw-r--r--. 1 root root 2075 May 4 18:23 initial-setup-ks.cfg
[root@backup01 ~]# ls -a
. anaconda-ks.cfg .bash_logout .bashrc .config .dbus .ssh .Xauthority
.. .bash_history .bash_profile .cache .cshrc initial-setup-ks.cfg .tcshrc
[root@backup01 ~]# cd .ssh/
[root@backup01 .ssh]# ll
total 12
-rw-------. 1 root root 1679 May 19 19:29 id_rsa
-rw-r--r--. 1 root root 398 May 19 19:29 id_rsa.pub
-rw-r--r--. 1 root root 175 May 19 17:01 known_hosts
[root@backup01 .ssh]# scp ./id_rsa.pub root@192.168.0.121:/root/.ssh/authorized_keys
root@192.168.0.121's password:
id_rsa.pub 100% 398 309.1KB/s 00:00
[root@backup01 .ssh]# ll
total 16
-rw-r--r--. 1 root root 398 May 19 19:40 authorized_keys
-rw-------. 1 root root 1679 May 19 19:29 id_rsa
-rw-r--r--. 1 root root 398 May 19 19:29 id_rsa.pub
-rw-r--r--. 1 root root 175 May 19 17:01 known_hosts
[root@backup01 .ssh]# ssh 192.168.0.121
Last login: Sun May 19 17:21:07 2019
[root@backup01 ~]#
[root@backup02 software]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:W1wq1Uf9Z725ZHpWm4G/3CUjOCRxFjZsGCl3xzR+/EI root@backup02
The key's randomart image is:
+---[RSA 2048]----+
| .=+oo .. |
| . +.+=o+ .|
| o.o+.+ E o|
| * o + .=|
| S = o.=|
| * . . B.|
| . o . B B|
| . o.Xo|
| ooo|
+----[SHA256]-----+
[root@backup02 software]# cd /root/
[root@backup02 ~]# ls
anaconda-ks.cfg initial-setup-ks.cfg
[root@backup02 ~]# ls -a
. anaconda-ks.cfg .bash_profile .cache .cshrc initial-setup-ks.cfg .tcshrc
.. .bash_logout .bashrc .config .dbus .ssh .Xauthority
[root@backup02 ~]# cd .ssh/
[root@backup02 .ssh]# ll
total 12
-rw-------. 1 root root 1675 May 19 19:33 id_rsa
-rw-r--r--. 1 root root 398 May 19 19:33 id_rsa.pub
-rw-r--r--. 1 root root 175 May 19 19:16 known_hosts
[root@backup02 .ssh]# ll
total 16
-rw-r--r--. 1 root root 398 May 19 19:37 authorized_keys
-rw-------. 1 root root 1675 May 19 19:33 id_rsa
-rw-r--r--. 1 root root 398 May 19 19:33 id_rsa.pub
-rw-r--r--. 1 root root 175 May 19 19:16 known_hosts
[root@backup02 .ssh]# scp ./id_rsa.pub root@192.168.0.120:/root/.ssh/authorized_keys
root@192.168.0.120's password:
id_rsa.pub 100% 398 431.7KB/s 00:00
[root@backup02 .ssh]# ssh 192.168.0.120
Last login: Sun May 19 19:16:54 2019 from 192.168.0.121
[root@backup02 ~]#
3>.配置安装jdk(三台主机都要配置)
3.1 首先确认删除 centos 系统自带的 jdk
$ rpm -qa | grep java
python-javapackages-3.4.1-11.el7.noarch
tzdata-java-2018e-3.el7.noarch
javapackages-tools-3.4.1-11.el7.noarch
java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.102-4.b14.el7.x86_64
- rpm 是一种用于打包及安装工具
- -q 代表 query,a 代表 all
- grep: 用于文本搜索
名称里有 openjdk
的要删除
$ rpm -e --nodeps java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64
$ rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.102-4.b14.el7.x86_64
3.2 安装 jdk
这里安装的是 jdk1.8
1)创建 jdk 环境的路径
$ mkdir /usr/local/java
2)将下载好的jdk压缩包解压到指定路径
设置坏境变量
vim /etc/profile
export JAVA_HOME=/usr/local/java/jdk1.8.0_172
export PATH=$PATH:$JAVA_HOME/bin
使配置生效
source /etc/profile
查看jdk是否配置成功
4>.关闭防火墙
$ su firewall-cmd --state 查看防火墙状态
$ su systemctl stop firewalld.service 关闭防火墙
$ su systemctl disable firewalld.service 禁止开机时防火墙自启
[root@backup01 hadoop-3.1.2]# systemctl stop firewalld
[root@backup01 hadoop-3.1.2]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@backup01 hadoop-3.1.2]#
5. 下载、解压 Hadoop(三台主机都要安装)
到 这里 下载 Hadoop 3.1.2 版本
5.1 创建 hadoop 目录
$ mkdir /usr/local/hadoop
5.2 解压 hadoop 压缩包到指定目录
6. 修改配置文件(三台主机都要配置)
配置文件的路径在 /usr/local/hadoop/hadoop-3.1.2/etc/hadoop
1)hadoop-env.sh
2)core-site.xml
3)hdfs-site.xml
4)mapred-site.xml
5)yarn-site.xml
6)workers
7)start-dfs.sh 和 stop-dfs.sh
8)start-yarn.sh 和 stop-yarn.sh
9)配置 hadoop 环境变量
注意具体配置参考:https://blog.youkuaiyun.com/ywl470812087/article/details/100065160
7. 启动 hadoop(在主节点上操作)
7.1 启动 hadoop 之前在 master 上 format 名称节点
cd /usr/local/hadoop/hadoop-3.1.2/bin
hdfs namenode -format
出现以下信息则说明 namenode 格式化成功
INFO common.Storage: Storage directory /usr/local/hadoop/hadoop-3.1.2/namenode has been successfully formatted.
7.2 在 master 节点上启动 hadoop 服务
cd /usr/local/hadoop/hadoop-3.1.2
7.3 查看 hadoop 服务是否启动成功
查看 master
节点
查看 slave1
节点
查看 slave2
节点
当 3 台节点分别出现这些 java 进程,则 hadoop 启动成功
java学习讨论群:725562382 欢迎加入学习
既然我们都已经hadoop启动成功:我们访问下
hadoop默认端口:9870
yarn默认端口: 8088
访问hadoop地址:http://192.168.0.120:9870/
yarn访问地址:http://192.168.0.120:8088/