is dfsclient caches the data block to local disk before writing?

转载于 2014-04-02 17:29:23 发布 · 885 阅读

hadoop安装&运行专栏收录该内容

74 篇文章

订阅专栏

本文解答了关于HDFS写入过程中是否使用缓存阶段的问题。确认了HDFS客户端不将数据块缓存到本地文件系统，而是直接通过内存中的数据队列和确认队列将64KB的数据包流式传输到数据节点。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Question:

Is staging still used in hdfs when writing the data? This doubt arose when I was going through the hdfs documents. 
ref :http://hadoop.apache.org/hdfs/docs/current/hdfs_design.html#Staging

I believe dfsclient does not cache the datablock to local fs (as the document says) but it does streaming of 64KB packets to the datanode and caches the packets of current block only in memory via dataqueue and ackqueue.Is the document needs to be corrected or my understanding is wrong?

Answer:

DFS client does not write the data to local disk first. Instead, it streams data directly to the datanodes in the write pipeline. I will update the document.