linux环境安装hadoop集群

I小屁孩T

已于 2024-11-17 02:37:34 修改

阅读量978

点赞数 31

文章标签： linux hadoop 运维 java

于 2024-11-17 02:32:28 首次发布

本文链接：https://blog.youkuaiyun.com/qq_28317295/article/details/143824495

版权

准备linux环境

准备三台Linux服务器，这里使用centos7，如下图所示：

设置静态IP

为三台服务器设置静态ip,分别为：hadoop101的ip为192.168.79.100，hadoop102的ip为192.168.79.101，hadoop103的ip为192.168.79.102，静态ip设置见如下链接：

centos7下设置静态ip

设置主机名称

把ip为192.168.79.100、192.168.79.101、192.168.79.102的主机分别设置主机名为hadoop101、hadoop102、hadoop103，设置主机名称见如下链接：

centos7下设置主机名称

配置主机名映射

修改主机hadoop101、hadoop102、hadoop103的/etc/hosts文件，添加内容如下：

[hdp@hadoop101 profile.d]$ sudo vi /etc/hosts
[hdp@hadoop101 profile.d]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6



192.168.79.100 hadoop101
192.168.79.101 hadoop102
192.168.79.102 hadoop103
[hdp@hadoop101 profile.d]$

设置ssh免密登入

设置hadoop101、hadoop102、hadoop103之间实现免密登入，具体操作见如下链接：

linux设置ssh免密登入

安装JDK

linux下安装JDK见如下链接：

linux下安装JDK

添加用户

为hadoop101、hadoop102、hadoop103服务器添加hdp用户，并设置该用户的密码，添加用户见如下链接：

linux添加用户

配置hdp具有root的权限，方便hdp有sudo执行root权限，具体修改如下：

编辑/etc/sudoers文件

[root@hadoop101 opt]# vi /etc/sudoers

添加如下内容：

hdp    ALL=(ALL)      NOPASSWD: ALL

安装hadoop

在hadoop101、hadoop102、hadoop103的/opt目录下创建soft目录，并把权限赋予给hdp,具体操作如下：

[hdp@hadoop101 opt]$ sudo mkdir soft
[hdp@hadoop101 opt]$ ls -l
total 187260
drwxr-xr-x. 7   10  143       245 Oct  6  2018 jdk1.8.0_191
-rwxr--r--. 1 root root 191753373 Nov 16 21:02 jdk-8u191-linux-x64.tar.gz
drwxr-xr-x. 2 root root         6 Oct 31  2018 rh
drwxr-xr-x. 2 root root         6 Nov 16 23:27 soft
[hdp@hadoop101 opt]$ sudo chown hdp:hdp soft/
[hdp@hadoop101 opt]$

把hadoop安装包上传到hadoop101、hadoop102、hadoop103的/opt/soft/目录下：

hadoop101：

hadoop102：

hadoop103：

在hadoop101、hadoop102、hadoop103的/opt/soft/目录下，对hadoop-3.3.6.tar.gz安装包执行解压操作：

[hdp@hadoop101 soft]$ tar -xzvf hadoop-3.3.6.tar.gz

[hdp@hadoop101 soft]$ ls -l
total 712996
drwxr-xr-x. 10 hdp hdp       215 Jun 18  2023 hadoop-3.3.6
-rw-rw-r--.  1 hdp hdp 730107476 Oct 10 10:10 hadoop-3.3.6.tar.gz
[hdp@hadoop101 soft]$

对hadoop101、hadoop102、hadoop103服务器设置环境变量，并使其生效：

[hdp@hadoop101 ~]$ cd /etc/profile.d/
[hdp@hadoop101 profile.d]$ 
[hdp@hadoop101 profile.d]$ sudo vi hdp.sh
[hdp@hadoop101 profile.d]$ 
[hdp@hadoop101 profile.d]$ cat hdp.sh 
export HADOOP_HOME=/opt/soft/hadoop-3.3.6
export PATH=$PATH:$HADOOP/bin
export PATH=$PATH:$HADOOP/sbin
[hdp@hadoop101 profile.d]$ 
[hdp@hadoop101 profile.d]$ source /etc/profile
[hdp@hadoop101 profile.d]$

验证hadoop是否配置正确，如下所示表示配置正确：

[hdp@hadoop101 profile.d]$ hadoop
Usage: hadoop [OPTIONS] SUBCOMMAND [SUBCOMMAND OPTIONS]
 or    hadoop [OPTIONS] CLASSNAME [CLASSNAME OPTIONS]
  where CLASSNAME is a user-provided Java class

  OPTIONS is none or any of:

buildpaths                       attempt to add class files from build tree
--config dir                     Hadoop config directory
--debug                          turn on shell script debug mode
--help                           usage information
hostnames list[,of,host,names]   hosts to use in worker mode
hosts filename                   list of hosts to use in worker mode
loglevel level                   set the log4j level for this command
workers                          turn on worker mode

  SUBCOMMAND is one of:


    Admin Commands:

daemonlog     get/set the log level for each daemon

    Client Commands:

archive       create a Hadoop archive
checknative   check native Hadoop and compression libraries availability
classpath     prints the class path needed to get the Hadoop jar and the required libraries
conftest      validate configuration XML files
credential    interact with credential providers
distch        distributed metadata changer
distcp        copy file or directories recursively
dtutil        operations related to delegation tokens
envvars       display computed Hadoop environment variables
fs            run a generic filesystem user client
gridmix       submit a mix of synthetic job, modeling a profiled from production load
jar <jar>     run a jar file. NOTE: please use "yarn jar" to launch YARN applications, not this command.
jnipath       prints the java.library.path
kdiag         Diagnose Kerberos Problems
kerbname      show auth_to_local principal conversion
key           manage keys via the KeyProvider
rumenfolder   scale a rumen input trace
rumentrace    convert logs into a rumen trace
s3guard       S3 Commands
trace         view and modify Hadoop tracing settings
version       print the version

    Daemon Commands:

kms           run KMS, the Key Management Server
registrydns   run the registry DNS server

SUBCOMMAND may print help when invoked w/o parameters or with -h.
[hdp@hadoop101 profile.d]$

集群配置规划

hadoop101

hadoop102

hadoop103

HDFS

NameNode

DataNode

SecondaryNameNode

DataNode

YARN

NodeManager

ResourceManager

NodeManager

集群配置

在hadoop101、Hadoop102、Hadoop103中修改如下配置文件

编辑/opt/soft/hadoop-3.3.6/etc/hadoop/core-site.xml文件，修改内容如下：

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hadoop101:8020</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/soft/hadoop-3.3.6/data</value>
  </property>
</configuration>

编辑/opt/soft/hadoop-3.3.6/etc/hadoop/hdfs-site.xml文件，修改内容如下：

<configuration>
  <property>
    <name>dfs.namenode.http-address</name>
    <value>hadoop101:9870</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop103:9868</value>
  </property>
</configuration>

编辑/opt/soft/hadoop-3.3.6/etc/hadoop/yarn-site.xml文件，修改内容如下：

<configuration>

<!-- Site specific YARN configuration properties -->
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop102</value>
  </property>
</configuration>

编辑/opt/soft/hadoop-3.3.6/etc/hadoop/mapred-site.xml文件，修改内容如下：

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

编辑/opt/soft/hadoop-3.3.6/etc/hadoop/workers文件，修改内容如下：

hadoop101
hadoop102
hadoop103

在hadoop101执行如下命令：

[hdp@hadoop101 hadoop]$ hdfs namenode -format
WARNING: /opt/soft/hadoop-3.3.6/logs does not exist. Creating.
2024-11-17 01:27:48,093 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop101/192.168.79.100
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.3.6
STARTUP_MSG:   classpath = /opt/soft/hadoop-3.3.6/etc/hadoop:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-aarch_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-socks-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-xml-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-xml-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-sctp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-codec-1.15.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/reload4j-1.2.22.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jersey-server-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jsch-0.1.55.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerby-util-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-io-2.8.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/token-provider-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/stax2-api-4.2.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-dns-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/avro-1.7.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-text-1.10.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-kqueue-4.1.89.Final-osx-aarch_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/hadoop-auth-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerby-config-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-lang3-3.12.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-server-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jersey-servlet-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-common-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/guava-27.0-jre.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-rxtx-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jersey-core-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/curator-recipes-5.2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-classes-macos-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-handler-ssl-ocsp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-epoll-4.1.89.Final-linux-aarch_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/curator-framework-5.2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jul-to-slf4j-1.7.36.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-http-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-http2-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-haproxy-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/checker-qual-2.5.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jsr305-3.0.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-server-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-util-ajax-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-client-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/slf4j-api-1.7.36.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-handler-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/zookeeper-3.6.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-util-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-configuration2-2.8.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jersey-json-1.20.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-udt-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-compress-1.21.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-memcache-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jettison-1.5.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-smtp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-epoll-4.1.89.Final-linux-x86_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-http-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-common-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-daemon-1.0.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/httpcore-4.4.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/failureaccess-1.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jakarta.activation-api-1.2.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-stomp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-all-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/metrics-core-3.2.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-unix-common-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-x86_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jackson-core-2.12.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-servlet-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/dnsjava-2.1.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/zookeeper-jute-3.6.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-redis-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-webapp-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-util-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jackson-annotations-2.12.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/hadoop-annotations-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jackson-databind-2.12.7.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-handler-proxy-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/paranamer-2.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-core-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/gson-2.9.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-security-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-native-kqueue-4.1.89.Final-osx-x86_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/re2j-1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/httpclient-4.5.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/curator-client-5.2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-classes-epoll-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-dns-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-resolver-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jetty-io-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/commons-net-3.9.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/snappy-java-1.1.8.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/woodstox-core-5.4.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-transport-classes-kqueue-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-codec-mqtt-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/nimbus-jose-jwt-9.8.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/lib/netty-buffer-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/hadoop-common-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/hadoop-nfs-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/hadoop-common-3.3.6-tests.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/hadoop-registry-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/common/hadoop-kms-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-aarch_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-socks-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/HikariCP-java7-2.4.12.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-xml-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-xml-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-sctp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-codec-1.15.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/reload4j-1.2.22.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-server-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-io-2.8.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kotlin-stdlib-1.4.10.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/stax2-api-4.2.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-dns-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/avro-1.7.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-text-1.10.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.89.Final-osx-aarch_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-auth-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-lang3-3.12.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-server-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-servlet-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-rxtx-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-core-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/curator-recipes-5.2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-classes-macos-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-handler-ssl-ocsp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/okio-2.8.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.89.Final-linux-aarch_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/curator-framework-5.2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-http-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-http2-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-haproxy-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-handler-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/zookeeper-3.6.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-util-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-configuration2-2.8.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jersey-json-1.20.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-udt-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-compress-1.21.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-memcache-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jettison-1.5.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-smtp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-epoll-4.1.89.Final-linux-x86_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-http-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-common-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jakarta.activation-api-1.2.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-stomp-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-all-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/metrics-core-3.2.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-unix-common-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-native-macos-4.1.89.Final-osx-x86_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-core-2.12.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-servlet-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/re2j-1.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/zookeeper-jute-3.6.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-redis-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-webapp-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-annotations-2.12.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/hadoop-annotations-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-databind-2.12.7.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-handler-proxy-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/paranamer-2.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/gson-2.9.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-security-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-native-kqueue-4.1.89.Final-osx-x86_64.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/httpclient-4.5.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/curator-client-5.2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-classes-epoll-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-dns-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-resolver-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jetty-io-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/okhttp-4.9.3.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/commons-net-3.9.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/snappy-java-1.1.8.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/woodstox-core-5.4.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-transport-classes-kqueue-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-codec-mqtt-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/nimbus-jose-jwt-9.8.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/netty-buffer-4.1.89.Final.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/lib/kotlin-stdlib-common-1.4.10.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.6-tests.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.6-tests.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-3.3.6-tests.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-native-client-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-client-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-nfs-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-rbf-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-client-3.3.6-tests.jar:/opt/soft/hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.6-tests.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/java-util-1.9.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/objenesis-2.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/fst-2.50.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-client-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jersey-client-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/bcpkix-jdk15on-1.68.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.12.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/asm-tree-9.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/snakeyaml-2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/javax-websocket-server-impl-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-api-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/asm-commons-9.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/bcprov-jdk15on-1.68.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jackson-jaxrs-base-2.12.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/javax-websocket-client-impl-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jline-3.9.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-server-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/javax.websocket-api-1.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/javax.websocket-client-api-1.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-plus-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-client-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-annotations-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-servlet-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/json-io-2.5.1.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jetty-jndi-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/websocket-common-9.4.51.v20230217.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jersey-guice-1.19.4.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jna-5.2.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.12.7.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/lib/guice-4.0.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-registry-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-applications-mawo-core-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-client-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-router-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-services-core-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-services-api-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-common-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-server-common-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.3.6.jar:/opt/soft/hadoop-3.3.6/share/hadoop/yarn/hadoop-yarn-api-3.3.6.jar
STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r 1be78238728da9266a4f88195058f08fd012bf9c; compiled by 'ubuntu' on 2023-06-18T08:22Z
STARTUP_MSG:   java = 1.8.0_191
************************************************************/
2024-11-17 01:27:48,111 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2024-11-17 01:27:48,275 INFO namenode.NameNode: createNameNode [-format]
2024-11-17 01:27:49,414 INFO namenode.NameNode: Formatting using clusterid: CID-d50e952e-692f-4ac4-bcdc-8b38bb9ce122
2024-11-17 01:27:49,541 INFO namenode.FSEditLog: Edit logging is async:true
2024-11-17 01:27:49,592 INFO namenode.FSNamesystem: KeyProvider: null
2024-11-17 01:27:49,594 INFO namenode.FSNamesystem: fsLock is fair: true
2024-11-17 01:27:49,594 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2024-11-17 01:27:49,610 INFO namenode.FSNamesystem: fsOwner                = hdp (auth:SIMPLE)
2024-11-17 01:27:49,610 INFO namenode.FSNamesystem: supergroup             = supergroup
2024-11-17 01:27:49,610 INFO namenode.FSNamesystem: isPermissionEnabled    = true
2024-11-17 01:27:49,610 INFO namenode.FSNamesystem: isStoragePolicyEnabled = true
2024-11-17 01:27:49,610 INFO namenode.FSNamesystem: HA Enabled: false
2024-11-17 01:27:49,699 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2024-11-17 01:27:50,054 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit : configured=1000, counted=60, effected=1000
2024-11-17 01:27:50,054 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2024-11-17 01:27:50,058 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2024-11-17 01:27:50,058 INFO blockmanagement.BlockManager: The block deletion will start around 2024 Nov 17 01:27:50
2024-11-17 01:27:50,061 INFO util.GSet: Computing capacity for map BlocksMap
2024-11-17 01:27:50,061 INFO util.GSet: VM type       = 64-bit
2024-11-17 01:27:50,067 INFO util.GSet: 2.0% max memory 405.5 MB = 8.1 MB
2024-11-17 01:27:50,067 INFO util.GSet: capacity      = 2^20 = 1048576 entries
2024-11-17 01:27:50,080 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2024-11-17 01:27:50,080 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2024-11-17 01:27:50,087 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.999
2024-11-17 01:27:50,087 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2024-11-17 01:27:50,087 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2024-11-17 01:27:50,088 INFO blockmanagement.BlockManager: defaultReplication         = 3
2024-11-17 01:27:50,088 INFO blockmanagement.BlockManager: maxReplication             = 512
2024-11-17 01:27:50,088 INFO blockmanagement.BlockManager: minReplication             = 1
2024-11-17 01:27:50,088 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2024-11-17 01:27:50,088 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2024-11-17 01:27:50,088 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2024-11-17 01:27:50,088 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2024-11-17 01:27:50,279 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2024-11-17 01:27:50,279 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2024-11-17 01:27:50,279 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2024-11-17 01:27:50,279 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2024-11-17 01:27:50,313 INFO util.GSet: Computing capacity for map INodeMap
2024-11-17 01:27:50,313 INFO util.GSet: VM type       = 64-bit
2024-11-17 01:27:50,313 INFO util.GSet: 1.0% max memory 405.5 MB = 4.1 MB
2024-11-17 01:27:50,313 INFO util.GSet: capacity      = 2^19 = 524288 entries
2024-11-17 01:27:50,321 INFO namenode.FSDirectory: ACLs enabled? true
2024-11-17 01:27:50,321 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2024-11-17 01:27:50,321 INFO namenode.FSDirectory: XAttrs enabled? true
2024-11-17 01:27:50,321 INFO namenode.NameNode: Caching file names occurring more than 10 times
2024-11-17 01:27:50,356 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2024-11-17 01:27:50,363 INFO snapshot.SnapshotManager: SkipList is disabled
2024-11-17 01:27:50,374 INFO util.GSet: Computing capacity for map cachedBlocks
2024-11-17 01:27:50,374 INFO util.GSet: VM type       = 64-bit
2024-11-17 01:27:50,375 INFO util.GSet: 0.25% max memory 405.5 MB = 1.0 MB
2024-11-17 01:27:50,375 INFO util.GSet: capacity      = 2^17 = 131072 entries
2024-11-17 01:27:50,442 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2024-11-17 01:27:50,442 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2024-11-17 01:27:50,442 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2024-11-17 01:27:50,472 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2024-11-17 01:27:50,472 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2024-11-17 01:27:50,485 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2024-11-17 01:27:50,486 INFO util.GSet: VM type       = 64-bit
2024-11-17 01:27:50,486 INFO util.GSet: 0.029999999329447746% max memory 405.5 MB = 124.6 KB
2024-11-17 01:27:50,486 INFO util.GSet: capacity      = 2^14 = 16384 entries
2024-11-17 01:27:50,629 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1754539893-192.168.79.100-1731778070574
2024-11-17 01:27:50,671 INFO common.Storage: Storage directory /opt/soft/hadoop-3.3.6/data/dfs/name has been successfully formatted.
2024-11-17 01:27:50,844 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/soft/hadoop-3.3.6/data/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2024-11-17 01:27:51,445 INFO namenode.FSImageFormatProtobuf: Image file /opt/soft/hadoop-3.3.6/data/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 398 bytes saved in 0 seconds .
2024-11-17 01:27:51,470 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2024-11-17 01:27:51,487 INFO namenode.FSNamesystem: Stopping services started for active state
2024-11-17 01:27:51,487 INFO namenode.FSNamesystem: Stopping services started for standby state
2024-11-17 01:27:51,500 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=0 when meet shutdown.
2024-11-17 01:27:51,500 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop101/192.168.79.100
************************************************************/
[hdp@hadoop101 hadoop]$ ls

在hadoop101下执行/opt/soft/hadoop-3.3.6/sbin/start-dfs.sh脚本

[hdp@hadoop101 hadoop-3.3.6]$ /opt/soft/hadoop-3.3.6/sbin/start-dfs.sh 
Starting namenodes on [hadoop101]
Starting datanodes
hadoop103: WARNING: /opt/soft/hadoop-3.3.6/logs does not exist. Creating.
hadoop102: WARNING: /opt/soft/hadoop-3.3.6/logs does not exist. Creating.
Starting secondary namenodes [hadoop103]
[hdp@hadoop101 hadoop-3.3.6]$

在hadoop102下执行/opt/soft/hadoop-3.3.6/sbin/start-yarn.sh脚本

[hdp@hadoop102 sbin]$ /opt/soft/hadoop-3.3.6/sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers
[hdp@hadoop102 sbin]$

验证所有进程是否都启动

hadoop101:

[hdp@hadoop101 wcinput]$ jps
12788 Jps
10309 DataNode
10153 NameNode
11754 NodeManager
[hdp@hadoop101 wcinput]$

hadoop102:

[hdp@hadoop102 sbin]$ jps
3938 DataNode
8708 Jps
7766 NodeManager
7631 ResourceManager
[hdp@hadoop102 sbin]$

hadoop103

[hdp@hadoop103 hadoop]$ jps
5092 NodeManager
4213 SecondaryNameNode
5686 Jps
4088 DataNode
[hdp@hadoop103 hadoop]$

访问http://192.168.79.100:9870/

访问http://192.168.79.101:8088/

验证创建目录：

[hdp@hadoop101 hadoop-3.3.6]$ hadoop fs -mkdir /input
[hdp@hadoop101 hadoop-3.3.6]$

验证上传文件：

[hdp@hadoop101 hadoop-3.3.6]$ hadoop fs -put input/d.txt /input
[hdp@hadoop101 hadoop-3.3.6]$

异常处理

上传文件时有警告，如下所示：

[hdp@hadoop101 input]$ hadoop fs -put a.txt /input
2024-11-17 01:49:26,378 WARN hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073741825_1001
java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 192.168.79.102:9866
	at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:128)
	at org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:104)
	at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1827)
	at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1728)
	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:713)
2024-11-17 01:49:26,390 WARN hdfs.DataStreamer: Abandoning BP-1754539893-192.168.79.100-1731778070574:blk_1073741825_1001
2024-11-17 01:49:26,528 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[192.168.79.102:9866,DS-400ef360-e39c-4c08-889b-69d656cc1307,DISK]
[hdp@hadoop101 input]$

解决办法

hadoop101、hadoop102、hadoop103关闭防火墙：

[hdp@hadoop102 sbin]$ sudo systemctl stop firewalld
[hdp@hadoop102 sbin]$ sudo systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[hdp@hadoop102 sbin]$