[转]Increasing the handler count can improve the performance of NameNode significantly

最新推荐文章于 2024-04-26 12:00:00 发布
最新推荐文章于 2024-04-26 12:00:00 发布
阅读量178 收藏
点赞数
分类专栏： 大数据
原文链接：https://blog.youkuaiyun.com/houzhizhen/article/details/78520100
版权
本文通过调整Hadoop NameNode的handler数量，测试了不同线程数下创建目录的性能，发现handler数量与性能提升密切相关。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

 

In this test, use NNThroughputBenchmark to create directories, the dirs parameter is always 10000, and dirsPerDir is 128. Cpu cores is 8.

 

set handler count to 2

 

  <property>
    <name>dfs.namenode.handler.count</name>
    <value>2</value>
  </property>

 

 

test with 2 threads, Elapsed Time: 102022

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 2 -dirs 10000 -dirsPerDir 128
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:45:10 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:45:10 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrThreads = 2
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: Elapsed Time: 102022
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark:  Ops per sec: 98.01807453294387
17/11/13 11:46:52 INFO namenode.NNThroughputBenchmark: Average Time: 20

 

 

test with 20 threads, Elapsed Time: 102154

 

This indicates that increase the client threads will not result in the performance increase.

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:47:33 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:47:33 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: Elapsed Time: 102154
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark:  Ops per sec: 97.89141883822464
17/11/13 11:49:15 INFO namenode.NNThroughputBenchmark: Average Time: 204

 

 

set handler count to 20, and restart the namenode.

 

  <property>
    <name>dfs.namenode.handler.count</name>
    <value>20</value>
  </property>

 

 

test with 20 threads, Elapsed Time: 9799

 

This indicates that performance improved 10 times.

 

hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:50:38 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:50:38 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: Elapsed Time: 9799
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark:  Ops per sec: 1020.512297173181
17/11/13 11:50:47 INFO namenode.NNThroughputBenchmark: Average Time: 19

 

 

test with 30 threads, Elapsed Time: 9695

 

This indicates that performance is the same as test with 20 threads.

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 30 -dirs 10000 -dirsPerDir 128
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 11:52:32 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 11:52:32 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: 
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrThreads = 30
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: Elapsed Time: 9695
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark:  Ops per sec: 1031.4595152140278
17/11/13 11:52:42 INFO namenode.NNThroughputBenchmark: Average Time: 28

 

 

set handler count to 200, and restart the namenode.

 

  <property>
    <name>dfs.namenode.handler.count</name>
    <value>200</value>
  </property>

 

 

test with 20 threads, Elapsed Time: 11288

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 20 -dirs 10000 -dirsPerDir 128
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 14:17:39 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 14:17:39 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: 
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrThreads = 20
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: Elapsed Time: 11288
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark:  Ops per sec: 885.896527285613
17/11/13 14:17:50 INFO namenode.NNThroughputBenchmark: Average Time: 22

 

 

test with 200 threads, Elapsed Time: 1285

 

Indicates that performance improved 10 times.

 

[houzhizhen@localhost hadoop]$ hadoop jar share/hadoop/hdfs/hadoop-hdfs-2.8.2-tests.jar org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op mkdirs -threads 200 -dirs 10000 -dirsPerDir 128
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Generate 10000 inputs for mkdirs
17/11/13 14:23:27 FATAL namenode.NNThroughputBenchmark: Log level = ERROR
17/11/13 14:23:27 INFO namenode.NNThroughputBenchmark: Starting 10000 mkdirs(s).
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: 
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: --- mkdirs inputs ---
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrDirs = 10000
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrThreads = 200
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: nrDirsPerDir = 128
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: --- mkdirs stats  ---
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: # operations: 10000
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: Elapsed Time: 1285
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark:  Ops per sec: 7782.101167315175
17/11/13 14:23:28 INFO namenode.NNThroughputBenchmark: Average Time: 23

 

 

set handler count to 2000, and restart the namenode, but there is no performance improvement. So it is safe to set handler to about 25 times of cores.


---------------------
作者：houzhizhen
来源：优快云
原文：https://blog.youkuaiyun.com/houzhizhen/article/details/78520100
版权声明：本文为作者原创文章，转载请附上博文链接！

sklearn 逻辑回归Increase the number of iterations (max_iter) or scale the data as shown in解决方案
weixin_43178406的博客
10-17 4万+
本文主要介绍了使用sklearn逻辑回归出现Increase the number of iterations (max_iter) or scale the data as shown in的解决方案，希望能对使用sklearn的同学们有所帮助。 文章目录 1. 问题描述 2. 解决方案
177.Hadoop（九）：生产调优手册（一）：NameNode内存生产、心跳并发配置，回收站配置，hdfs集群压测配置，多目录配置，扩容缩容，存储优化
鹏哥哥Aaa
09-28 782
NameNode内存生产、心跳并发配置，回收站配置，hdfs集群压测配置，多目录配置，扩容缩容，存储优化
如何有效提高namenode性能
lvhdbb的博客
07-15 832
In this test, use NNThroughputBenchmark to create directories, the dirs parameter is always 10000, and dirsPerDir is 128. Cpu cores is 8. set handler count to 2 <property> <name...
namenode 性能优化（1）RPC队列拆分
weixin_30363981的博客
07-17 570
一、Service RPC port NameNode默认使用8020端口侦听所有的RPC请求（HDP版本），包括客户端数据请求，DataNode心跳和block上报，ZKFC模块监控检查和切换控制。当集群规模越和RPC请求来越大时，RPC请求响应时间也变得越来越长。NameNode也给DataNode提供了专用的Service RPC port用于上报数据块和心跳状态，这样可以减...
Hadoop Namenode性能诊断及优化
01-22
Hadoop Namenode性能诊断及优化
NameNode优化笔记 (一)
coderplay的专栏
01-12 318
很久没有发博客了, 最近这段时间工作上、生活上杂事比较多。最近经常有人问我在学校还是在公司。其实之前在学校读研, 入研之前工作过几年。那时候在学校研究MapReduce, 部署了10台的PC机做些Hadoop与机器学习的研究。08年末觉得学校限制我的发展, 就联系了几家公司实习。最后我到了淘宝实习了一年半, 那时候因为身份还是学生, 前期主要维护淘宝的Hadoop集群, 后期主要研发Hive, 同...
dfs.namenode.handler.count
Faded1573606285的博客
01-20 3255
dfs.namenode.handler.count=20 * log2(Cluster Size)，比如集群规模为8台时，此参数设置为60 The number of Namenode RPC server threads that listen to requests from clients. If dfs.namenode.servicerpc-address is not config...
Increasing the hole energy by grading the alloy composition of the p-type electron blocking layer for very high-performance deep ultraviolet light-emitting diodes
01-26
It is well known that the p-type AlGaN electron blocking layer (p-EBL) can block hole injection for deep ultraviolet light-emitting diodes (DUV LEDs). The polarization induced electric field in the p-...
Increasing the bandwidth of slow light in fishbone-like grating waveguides
01-26
a technology to control the optical signal by reducing the group velocity, has been widely studied to obtain enhanced nonlinearities and increased phase shifts owing to its promoting of the light–...
Increasing endothelial cell permeability improves the efficiency of myocyte adenoviral vector infection
06-29
Increasing endothelial cell permeability improves the efficiency of myocyte adenoviral vector infection RESEARCH ARTICLE Increasing endothelial cell permeability improves the ef:registered:ciency ...
NameNode锁细粒度优化在B站的实践
最新发布
bilibili_TC的博客
04-26 1138
随着业务的高速发展，针对HDFS元数据的访问请求量呈指数级上升。
namenode性能问题
qq_25418883的博客
04-13 2639
影响namenode内存主要因素：NameNode常驻内存主要被Namespace和BlockManager使用，二者使用占比分别接近50%。其它部分内存开销较小且相对固定，与Namespace和BlockManager相比基本可以忽略。Namespace：维护整个文件系统的目录树结构及目录树上的状态变化；BlockManager：维护整个文件系统中与数据块相关的信息及数据块的状态变化；数据增长n...
提高namenode容错的两种方式
Kruskual的博客
09-08 246
点个赞，交个朋友把。
hadoop中NameNode节点CPU使用量高 解决方案（亲测）
qq_30498935的博客
08-23 5059
集群中任务执行超慢，查原因 top一下 查看19118进程  发现这个节点是NameNode节点 然后 ps aux |grep 19118 jstat -gcutil 19118 1000 1000查看jvm GC GC超高 问题就是namenode负载太高，内存不够用导致CPU飙升 解决方案调整hadoop_namenode_heapsize参数调到16G（按照集群情...
hadoop源码解读namenode高可靠：HA;web方式查看namenode下信息；dfs/data决定datanode存储位置
needkane
05-22 1812
点击browserFilesystem，和命令查看结果一样 当我们查看hadoop源码时，我们看到hdfs下的hdfs-default.xml文件信息 我们查找${hadoop.tmp.dir}这是引用变量，肯定在其他文件有定义，在core-default.xml中查看到，这两个配置文件有个共同点： 就是不要修改此文件，但可以复制信息到core-site.xml和hdfs-sit
"Namenode瓶颈解决方案"分享总结
smallnetvisitor的博客
06-08 686
概述： 此分享是关于Namenode中Blocksmap太大等引起的瓶颈问题解决方案 技术点： 1.用行级锁缩小Namespace锁粒度 2.用Cache + FusionIO解决吃内存的问题 3.持久化Blockmap, 加速重启过程 4.无状态化Namenode, 支持热部署 5.简单主备策略保证可靠性 详情参见附件 ...
Hadoop参数调优
shyming's blog
08-17 266
Hadoop参数调优HDFS参数调优hdfs-site.xmlYARN参数调优yarn-site.xmlHadoop宕机 HDFS参数调优hdfs-site.xml （1）dfs.namenode.handler.count=20 * log2 n(Cluster Size)，比如集群规模为n=8台时，此参数设置为20 * log2 8=60 The number of Namenode RPC server threads that listen to requests from clients. If
企业经验---参数调优---尚硅谷
weixin_52996387的博客
05-21 731
但注意，存储小文件所需要的磁盘容量和数据块的大小无关。XOR-2-1-1024k：使用XOR编码（速度比RS编码快），每2个数据单元，生成1个校验单元，共3个单元，也就是说：这三个单元中，只要有任意的2个单元存在（不管是数据单元还是校验单元，只要总数是2）就可以得到原始数据。RS-10-4-1024k：使用RS编码，每10个数据单元生成4个校验单元，共14个单元，也就是说：这14个单元中，只要有任意的10个单元存在（不管是数据单元还是校验单元，只要总数=10）就可以得到原始数据。
ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.
05-12
这个警告是说你的模型在迭代过程中没有收敛到一个稳定的状态，达到了最大迭代次数而停止了训练。这可能意味着你的模型过于复杂，或者你的数据集不够大，或者你的学习率设置不合适。你可以尝试增加最大迭代次数，或者...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值