kafka报错ERROR Failed to clean up log for __consumer_offsets-49 in dir /tmp/kafka-logs

Kafka服务异常与数据丢失

最新推荐文章于 2023-09-13 12:32:02 发布

原创

最新推荐文章于 2023-09-13 12:32:02 发布 · 4k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#kafka

探讨Kafka服务运行中遇到的数据文件丢失问题，分析可能原因包括临时目录清理及Zookeeper同步机制，分享通过更改数据目录配置并重启服务解决异常的经验。

kafka运行了一段时间，有一天突然发现kafka服务停了。看日志more logs/server.log.2020-04-20-11如下：

./server.log.2020-04-20-11:java.nio.file.NoSuchFileException: /tmp/kafka-logs/__consumer_offsets-49/00000000000000000000.log.swap
./server.log.2020-04-20-11:     Suppressed: java.nio.file.NoSuchFileException: /tmp/kafka-logs/__consumer_offsets-49/00000000000000000000.log.swap -> /tmp/kafka-logs/__consumer_offsets-49/00000000000000000000.log
[kduser@master logs]$ more server.log.2020-04-20-11
[2020-04-20 11:06:05,736] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2020-04-20 11:16:05,736] INFO [GroupMetadataManager brokerId=0] Group testGroup transitioned to Dead in generation 0 (kafka.coordinator.group.GroupMetadataManager)
[2020-04-20 11:16:05,744] INFO [ProducerStateManager partition=__consumer_offsets-49] Writing producer snapshot at offset 19 (kafka.log.ProducerStateManager)
[2020-04-20 11:16:05,745] INFO [

最低0.47元/天解锁文章

4 条评论

bee_yyy 2021.07.15
2、为什么本地文件丢失会报错？建议看这一段代码 [code=java] private def cleanLog(cleanable: LogToClean): Unit = { val startOffset = cleanable.firstDirtyOffset var endOffset = startOffset try { val (nextDirtyOffset, cleanerStats) = cleaner.clean(cleanable) endOffset = nextDirtyOffset recordStats(cleaner.id, cleanable.log.name, startOffset, endOffset, cleanerStats) } catch { case _: LogCleaningAbortedException => // task can be aborted, let it go. case _: KafkaStorageException => // partition is already offline. let it go. case e: IOException => val logDirectory = cleanable.log.parentDir val msg = s"Failed to clean up log for ${cleanable.topicPartition} in dir $logDirectory due to IOException" logDirFailureChannel.maybeAddOfflineLogDir(logDirectory, msg, e) } finally { cleanerManager.doneCleaning(cleanable.topicPartition, cleanable.log.parentDirFile, endOffset) } } [/code]