Hadoop 第一次无法启动

本文介绍了在启动Hadoop集群过程中遇到的问题,包括警告信息和无法解析主机名等错误,并提供了解决方案,即通过设置环境变量来修复这些问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

开启Hadoop时,出现如下信息:

[root@hd-m1 /]# ./hadoop/hadoop-2.6.0/sbin/start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/01/23 20:23:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [Java HotSpot(TM) Client VM warning: You have loaded library /hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
hd-m1]
sed: -e expression #1, char 6: unknown option to `s'
-c: Unknown cipher type 'cd'

hd-m1: starting namenode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-root-namenode-hd-m1.out
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution
Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
Client: ssh: Could not resolve hostname Client: Temporary failure in name resolution
You: ssh: Could not resolve hostname You: Temporary failure in name resolution
warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
might: ssh: Could not resolve hostname might: Temporary failure in name resolution
which: ssh: Could not resolve hostname which: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
The: ssh: Could not resolve hostname The: Temporary failure in name resolution
try: ssh: Could not resolve hostname try: Temporary failure in name resolution
will: ssh: Could not resolve hostname will: Temporary failure in name resolution
to: ssh: Could not resolve hostname to: Temporary failure in name resolution
fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
the: ssh: Could not resolve hostname the: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
It's: ssh: Could not resolve hostname It's: Temporary failure in name resolution
now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
that: ssh: Could not resolve hostname that: Temporary failure in name resolution
you: ssh: Could not resolve hostname you: Temporary failure in name resolution
with: ssh: Could not resolve hostname with: Temporary failure in name resolution
'execstack: ssh: Could not resolve hostname 'execstack: Temporary failure in name resolution
the: ssh: Could not resolve hostname the: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
< libfile>',: ssh: Could not resolve hostname <libfile>',: Temporary failure in name resolution
or: ssh: Could not resolve hostname or: Temporary failure in name resolution
link: ssh: Could not resolve hostname link: Temporary failure in name resolution
it: ssh: Could not resolve hostname it: Temporary failure in name resolution
'-z: ssh: Could not resolve hostname '-z: Temporary failure in name resolution

with: ssh: Could not resolve hostname with: Temporary failure in name resolution
noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Temporary failure in name resolution
hd-s1: starting datanode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-hd-s1.out
hd-s2: starting datanode, logging to /hadoop/hadoop-2.6.0/logs/hadoop-root-datanode-hd-s2.out
Starting secondary namenodes [Java HotSpot(TM) Client VM warning: You have loaded library /hadoop/hadoop-2.6.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
SecondaryNameNode]
sed: -e expression #1, char 6: unknown option to `s'
-c: Unknown cipher type 'cd'
Client: ssh: Could not resolve hostname Client: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
You: ssh: Could not resolve hostname You: Temporary failure in name resolution
Java: ssh: Could not resolve hostname Java: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
loaded: ssh: Could not resolve hostname loaded: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
might: ssh: Could not resolve hostname might: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution
have: ssh: Could not resolve hostname have: Temporary failure in name resolution
VM: ssh: Could not resolve hostname VM: Temporary failure in name resolution
fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution
to: ssh: Could not resolve hostname to: Temporary failure in name resolution
the: ssh: Could not resolve hostname the: Temporary failure in name resolution
guard: ssh: Could not resolve hostname guard: Temporary failure in name resolution
now.: ssh: Could not resolve hostname now.: Temporary failure in name resolution
It's: ssh: Could not resolve hostname It's: Temporary failure in name resolution
disabled: ssh: Could not resolve hostname disabled: Temporary failure in name resolution
highly: ssh: Could not resolve hostname highly: Temporary failure in name resolution
that: ssh: Could not resolve hostname that: Temporary failure in name resolution

recommended: ssh: Could not resolve hostname recommended: Temporary failure in name resolution
stack: ssh: Could not resolve hostname stack: Temporary failure in name resolution

try: ssh: Could not resolve hostname try: Temporary failure in name resolution
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Temporary failure in name resolution

fix: ssh: Could not resolve hostname fix: Temporary failure in name resolution

the: ssh: Could not resolve hostname the: Temporary failure in name resolution
library: ssh: Could not resolve hostname library: Temporary failure in name resolution
'execstack: ssh: Could not resolve hostname 'execstack: Temporary failure in name resolution
warning:: ssh: Could not resolve hostname warning:: Temporary failure in name resolution
with: ssh: Could not resolve hostname with: Temporary failure in name resolution
or: ssh: Could not resolve hostname or: Temporary failure in name resolution
< libfile>',: ssh: Could not resolve hostname <libfile>',: Temporary failure in name resolution
you: ssh: Could not resolve hostname you: Temporary failure in name resolution
link: ssh: Could not resolve hostname link: Temporary failure in name resolution
it: ssh: Could not resolve hostname it: Temporary failure in name resolution
which: ssh: Could not resolve hostname which: Temporary failure in name resolution
with: ssh: Could not resolve hostname with: Temporary failure in name resolution
The: ssh: Could not resolve hostname The: Temporary failure in name resolution
noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Temporary failure in name resolution
'-z: ssh: Could not resolve hostname '-z: Temporary failure in name resolution
will: ssh: Could not resolve hostname will: Temporary failure in name resolution
SecondaryNameNode: ssh: Could not resolve hostname SecondaryNameNode: Temporary failure in name resolution
guard.: ssh: Could not resolve hostname guard.: Temporary failure in name resolution
15/01/23 20:24:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

starting yarn daemons
starting resourcemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-root-resourcemanager-hd-m1.out
hd-s1: starting nodemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-hd-s1.out
hd-s2: starting nodemanager, logging to /hadoop/hadoop-2.6.0/logs/yarn-root-nodemanager-hd-s2.out


解决办法:

出现上述问题主要是环境变量没设置好,在~/.bash_profile或者/etc/profile中加入以下语句就没问题了。

  #vi /etc/profile或者vi ~/.bash_profile
    export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

然后用source重新编译使之生效即可!
  #source /etc/profile或者source ~/.bash_profile


### Hadoop 初次查询性能优化 对于Hadoop初次查询速度较慢的问题,可以从多个方面进行分析并采取相应的措施来提升性能。 #### 1. 数据预加载与缓存机制 为了改善第一次访问的速度,可以在启动MapReduce作业之前预先加载部分常用数据到内存中。通过配置`dfs.client.read.shortcircuit`参数为true启用短路读取功能[^1]。这使得客户端可以直接从本地磁盘读取文件而不需要经过DataNode服务端转发,减少了网络传输开销。 另外,利用Linux操作系统自带的页面缓存(Page Cache),可以显著减少I/O延迟。适当调整JVM堆外内存大小(-XX:MaxDirectMemorySize)以便更好地支持PageCache的有效利用[^2]。 #### 2. 文件系统元数据管理 由于NameNode负责维护整个文件系统的命名空间及块映射信息,在大规模集群环境下其重启时间较长会影响整体响应效率。针对这一点可以通过优化Checkpoint流程、增加Secondary NameNode数量或者引入HA(High Availability)架构下的Standby NameNodes等方式缩短恢复周期。 同时建议定期运行FsImage合并操作以保持较小的日志长度,并合理设置安全检查点间隔(`dfs.namenode.checkpoint.txns`)防止过多的小事务累积影响性能。 #### 3. 数据布局策略 考虑到HDFS不适合频繁随机写入的特点,应尽量采用批量导入模式而非逐条记录追加的方式向目标路径上传新数据集。如果业务场景允许的话还可以考虑使用BulkLoad工具绕过Write-Ahead Log(WAL)直接将外部表格转换成内部格式存储于RegionServer之上,以此规避因日志刷盘带来的额外负载[^3]。 此外,合理的分区设计有助于分散热点区域集中度高的情况发生概率,降低单个节点的压力水平。比如按照日期维度拆分目录结构或将大表按照行键范围切分成若干子表等方法均能有效缓解此类瓶颈现象[^5]。 ```bash # 设置 dfs.client.read.shortcircuit 参数 hdfs-site.xml: <property> <name>dfs.client.read.shortcircuit</name> <value>true</value> </property> # 调整 JVM 堆外内存大小 export JAVA_OPTS="-XX:MaxDirectMemorySize=8G" ```
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值