在使用Hadoop进行MR任务时，客户端在读取数据后关闭socket，但并未读取完整块数据。这导致数据节点继续写入数据直到连接关闭或块结束，而非客户端完成读取。此行为会产生错误日志记录，尽管这不是真正的错误，但会污染日志并混淆用户。
最近在跑mr任务的时候 ,查看datanode节点的log日志,总是会看到这个错误,仔细的查了下发现有人提过这个bug,如下：
When a client reads data using read(), it closes the sockets after it is done. Often it might not read till the end of a block. The datanode on the other side keeps writing data until the client connection is closed or end of the block is reached. If the client does not read till the end of the block, Datanode writes an error message and stack trace to the datanode log. It should not. This is not an error and it just pollutes the log and confuses the user.
下面是这个bug的修改地址
https://issues.apache.org/jira/browse/HADOOP-3678
When a client reads data using read(), it closes the sockets after it is done. Often it might not read till the end of a block. The datanode on the other side keeps writing data until the client connection is closed or end of the block is reached. If the client does not read till the end of the block, Datanode writes an error message and stack trace to the datanode log. It should not. This is not an error and it just pollutes the log and confuses the user.
下面是这个bug的修改地址
https://issues.apache.org/jira/browse/HADOOP-3678
扫一扫
1677
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
