Hadoop HA测试及动态添加节点和副本数

最新推荐文章于 2023-03-11 10:15:30 发布

原创最新推荐文章于 2023-03-11 10:15:30 发布 · 435 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Hadoop HA测试及动态添加节点和副本数

hadoop 专栏收录该内容

17 篇文章

订阅专栏

本文主要介绍了Hadoop HA的故障切换测试，包括主动节点失效后的自动切换，断电情况下的集群恢复，以及在运行过程中切换Active节点且不影响文件上传。此外，还详细说明了如何通过haadmin命令管理和添加新的DataNode节点到集群。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

测试1、
将活跃的hadoop01进程杀死后，发现hadoop02状态被切换到avtive状态，集群正常工作；
测试2、
将hadoop01断电，hadoop02向hadoop01发送ssh kill -9 namenode进程，三十秒内无ssh相应，则执行shell脚本，其中无namenode工作。三十秒后集群正常工作；
测试3
进程中hadoop02为active，在namenode01上上传文件中，杀死hadoop02namenode进程，文件正常上传hdfs；
测试4

管理HA的客户端：

命令行hdfs haadmin进行查看帮助文档

[root@hadoop02 ~]# hdfs haadmin
Usage: haadmin
    [-transitionToActive [--forceactive] <serviceId>]
    [-transitionToStandby <serviceId>]
    [-failover [--forcefence] [--forceactive] <serviceId> <serviceId>]
    [-getServiceState <serviceId>] 
    [-checkHealth <serviceId>]
    [-help <command>]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

获取nn1的服务状态

[root@hadoop02 ~]# hdfs haadmin -getServiceState nn1
active

切换nn1服务的状态：active–>standby

[root@hadoop02 ~]# hdfs haadmin -transitionToStandby nn1
Automatic failover is enabled for NameNode at hadoop01/192.168.100.12:9000
Refusing to manually manage HA state, since it may cause
a split-brain scenario or other incorrect state.
If you are very sure you know what you are doing, please 
specify the --forcemanual flag.

[root@hadoop02 ~]# hdfs haadmin -transitionToStandby nn1 --forcemanual
You have specified the --forcemanual flag. This flag is dangerous, as it can induce a split-brain scenario that WILL CORRUPT your HDFS namespace, possibly irrecoverably.

It is recommended not to use this flag, but instead to shut down the cluster and disable automatic failover if you prefer to manually manage your HA state.

You may abort safely by answering 'n' or hitting ^C now.

Are you sure you want to continue? (Y or N)    y
19/05/27 08:59:55 WARN ha.HAAdmin: Proceeding with manual HA state management even though
automatic failover is enabled for NameNode at hadoop01/192.168.100.12:9000
[root@hadoop02 ~]# hdfs haadmin -getServiceState nn1
standby

添加datanode节点

启动一台机器，安装一份hadoop文件，修改其配置文件后手动启动datanode进程即可；

[root@hadoop02 ~]# hadoop-daemon.sh start datanode