Hadoop-9710问题

最新推荐文章于 2025-12-20 18:17:43 发布

原创最新推荐文章于 2025-12-20 18:17:43 发布 · 194 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #hdfs #datanode ibr #IBR #增量块汇报

CDH和大数据组件专栏收录该内容

21 篇文章

订阅专栏

针对Hadoop中IBR(增量块汇报)机制存在的问题，即当DataNode接收多个数据块时会向NameNode发送大量独立的IBR RPC请求，导致NameNode负载过高。本文探讨了Jira HDFS-9710提议的解决方案，即让DataNode批量发送IBR报告，以减少NameNode的处理负担。

1. 概述

IBR:IBR(增量块汇报)

jira-9710描述

Description

When a DN has received a block, it immediately sends a block receipt IBR RPC to NN for reporting the block. Even if a DN has received multiple blocks at the same time, it still sends multiple RPCs. It does not scale well since NN has to process a huge number of RPCs when many DNs receiving many blocks at the same time.

2. 参考资料

[1].https://issues.apache.org/jira/browse/HDFS-9198 . Coalesce IBR processing in the NN
[2].https://issues.apache.org/jira/browse/HDFS-9710 . Change DN to send block receipt IBRs in batches

https://blog.youkuaiyun.com/androidlushangderen/article/details/101643921

趣头条百PB规模 Hadoop实践(HDFS篇) https://blog.youkuaiyun.com/weixin_35792948/article/details/104997862

https://issues.apache.org/jira/browse/HDFS-9198

https://issues.apache.org/jira/browse/HDFS-9710

https://issues.apache.org/jira/plugins/servlet/mobile#issue/HDFS-9710