hadoop伪分布式安装

一、安装jdk

检查卸载已经安装的默认jdk

[root@localhost ~]# rpm -qa | grep jdk
copy-jdk-configs-3.3-2.el7.noarch
java-1.8.0-openjdk-headless-1.8.0.161-2.b14.el7.x86_64
java-1.7.0-openjdk-1.7.0.171-2.6.13.2.el7.x86_64
java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64
java-1.7.0-openjdk-headless-1.7.0.171-2.6.13.2.el7.x86_64
[root@localhost ~]# yum -y remove copy-jdk-configs-3.3-2.el7.noarch

官网获取jdk，本文使用的版本为 jdk-8u161-linux-x64.tar.gz

解压jdk(这里解压到/soft/目录下):

[root@localhost ~]# tar -zxvf jdk-8u161-linux-x64.tar.gz -C /soft/

进入etc目录配置profile文件

[root@localhost ~]# vim /etc/profile

将以下内容粘贴到profile最后

export JAVA_HOME=/soft/jdk1.8.0_161
export JAVA_BIN=/soft/jdk1.8.0_161/bin
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATHSPH

执行配置文件

[root@localhost ~]# source /etc/profile

查看jdk是否安装成功

[root@localhost ~]# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

二、安装ssh

[root@localhost hadoop]# sudo yum install ssh
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.zju.edu.cn
 * extras: mirrors.zju.edu.cn
 * updates: mirrors.zju.edu.cn
没有可用软件包 ssh。
错误：无须任何处理

[root@localhost hadoop]# ssh-keygen -t rsa     // 回车之后需要输入东西，直接回车就可以了
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:gRf5SPrUNlrAV1TGxCIwE4ukiX69Xquz4lBFTqt/6Oo root@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
|      +.*+ oo=+  |
|   . B +==o ..o  |
|  . o *o+* . .   |
| .   +..o.*      |
|  . + .oS+ .     |
|   o . oo        |
|  .   + o        |
|   ..o.o .       |
|   oEo=+.        |
+----[SHA256]-----+

进入.ssh目录

[root@localhost ~]# cd /root/.ssh
[root@localhost .ssh]# cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
// 验证是否安装成功
[root@localhost .ssh]# ssh localhost

三、安装hadoop

获取hadoop:

wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz

解压并指定安装目录

tar -zxvf hadoop-2.6.0-cdh5.7.0.tar.gz -C /soft/

配置hadoop配置文件，官网参考链接

编辑安装hadoop目录下etc/hadoop/hadoop-env.sh文件，配置jdk路径

[root@localhost hadoop-2.6.0-cdh5.7.0]# vim etc/hadoop/hadoop-env.sh

注释掉： export JAVA_HOME=${JAVA_HOME}，添加: export JAVA_HOME=/soft/jdk1.8.0_161，如下所示

#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/soft/jdk1.8.0_161

修改hadoop配置文件

修改hadoop安装目录下etc/hadoop/core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:8020</value>
        </property>
        <!--hadoop临时文件目录-->
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/home/hadoop/tmp</value>
        </property>
	</configuration>

修改hadoop安装目录下etc/hadoop/hdfs-site.xml

<configuration>
		<!--hadoop默认副本是3，这里使用的是伪分布式，所以是1-->
	    <property>
	        <name>dfs.replication</name>
	        <value>1</value>
	    </property>
	</configuration>

四、启动和停止hdfs

（1）启动hdfs

格式化文件系统（仅第一次执行即可，不要重复执行），进入hadoop安装目录的bin目录下执行：./hdfs namenode -format

[root@localhost bin]# ./hdfs namenode -format

启动hdfs，进入hadoop安装目录的sbin目录下执行：./start-dfs.sh

[root@localhost sbin]# ./start-dfs.sh

验证是否启动成功：

[root@localhost sbin]# jps
2660 Jps
2390 DataNode
2265 NameNode
2553 SecondaryNameNode

也可以在浏览器中输入http://localhost:50070验证是否启动成功      

停止hdfs
进入hadoop安装目录的sbin目录下执行./stop-dfs.sh

[root@localhost sbin]# ./stop-dfs.sh

五、配置hadoop环境变量

为了方便使用hadoop的一些命令，将hadoop的bin目录配置到系统的环境变量

[root@localhost /]# vim /etc/profile

将以下内容粘贴到配置文件的后面

export HADOOP_HOME=/soft/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin:$PATH

YARN环境搭建

https://blog.youkuaiyun.com/lizc_lizc/article/details/81839910