hadoop hdfs单节点简易部署（二）

最新推荐文章于 2023-05-16 22:04:35 发布

kaloKu

最新推荐文章于 2023-05-16 22:04:35 发布

阅读量279

点赞数

本文链接：https://blog.youkuaiyun.com/qq_31622585/article/details/87893946

版权

本文详细介绍了如何进行Hadoop HDFS的单节点部署，包括配置SSH localhost无密码信任、HDFS的格式化、系统启动以及相关操作命令的使用，适合初学者入门。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

5.配置ssh localhost无密码信任关系

[hadoop@hadoop002 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
ba:48:3d:ff:af:4d:da:74:67:31:d6:98:ad:a0:b3:76 hadoop@hadoop002
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|               +.|
|        S   . o+o|
|     . .   . ...o|
|    . +   o o o o|
|   . . +  .OE. o |
|    . . .o=++    |
+-----------------+
[hadoop@hadoop002 ~]$ cd .ssh
[hadoop@hadoop002 .ssh]$ ll
total 8
-rw------- 1 hadoop hadoop 1675 Feb 13 22:36 id_rsa     私钥
-rw-r--r-- 1 hadoop hadoop  398 Feb 13 22:36 id_rsa.pub 公钥
[hadoop@hadoop002 .ssh]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop002 .ssh]$ 

[hadoop@hadoop002 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop  398 Feb 13 22:37 authorized_keys
-rw------- 1 hadoop hadoop 1675 Feb 13 22:36 id_rsa
-rw-r--r-- 1 hadoop hadoop  398 Feb 13 22:36 id_rsa.pub
-rw-r--r-- 1 hadoop hadoop    0 Feb 13 22:39 known_hosts

ssh localhost date 是需要输入密码，但是这个用户是没有配置密码。
我们应该在没有配置密码情况下去完成无密码信任呢？

改权限
[hadoop@hadoop002 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop002 .ssh]$ ssh localhost date
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Wed Feb 13 22:41:17 CST 2019
[hadoop@hadoop002 .ssh]$ 
[hadoop@hadoop002 .ssh]$ 
[hadoop@hadoop002 .ssh]$ ssh localhost date
Wed Feb 13 22:41:22 CST 2019
[hadoop@hadoop002 .ssh]$

6.格式化

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format

7.启动

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
19/02/13 22:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out

ssh 信任关系 是配置localhost

Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
19/02/13 22:49:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 

ssh localhost date
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/stop-dfs.sh
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
19/02/13 22:57:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
19/02/13 22:57:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ jps
15059 Jps
14948 SecondaryNameNode 第二名称节点 老二
14783 DataNode  数据节点  小弟
14655 NameNode  名称节点  老大 读写
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 

open http://ip:50070

8.环境变量

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ cat ~/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_PREFIX/bin:$PATH
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$

9.相关操作命令

 hdfs dfs操作命令和Linux命令极其相似

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
19/02/13 23:08:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
19/02/13 23:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /
bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -mkdir /ruozedata
19/02/13 23:11:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs dfs -ls /
19/02/13 23:11:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-02-13 23:11 /ruozedata
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ ls /
bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$