ELK学习9_ELK数据流传输过程_问题总结2_elk message中文乱码-优快云博客

本文链接：https://blog.youkuaiyun.com/wang_zhenwei/article/details/50285345

本文介绍了一个使用Java程序将.csv文件数据发送至Kafka，然后转发给ELK的过程。在处理较大文件时遇到了中文乱码及不稳定的错误报告问题。文中详细记录了错误信息，包括Kafka在获取主题元数据时遇到的ClosedChannelException。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

目的：使用Java程序把一个.csv文件发送给Kafka，再发送给ELK

实验步骤：见以前的博客

遇到的问题：

一、

直接发送比较大的.csv文件，ELK接到的汉字内容是乱码的

发送比较小的.csv文件，ELK接到的汉字内容正常

猜测：是否是Kafka出错

二、

把大的.csv文件拆分成多个小.csv文件发送，发送过程中，会不定期报错

报错内容如下

[2015-12-13 16:02:14,273] ERROR fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.utils.Utils$:106)
kafka.common.KafkaException: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
	at kafka.utils.Utils$.swallow(Utils.scala:172)
	at kafka.utils.Logging$class.swallowError(Logging.scala:106)
	at kafka.utils.Utils$.swallowError(Utils.scala:45)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
	at kafka.producer.Producer.send(Producer.scala:77)
	at kafka.javaapi.producer.Producer.send(Producer.scala:33)
	at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)
	at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)
	at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)
Caused by: java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	... 11 more
[2015-12-13 16:02:14,283] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.producer.async.DefaultEventHandler:97)
[2015-12-13 16:02:14,394] ERROR fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.utils.Utils$:106)
kafka.common.KafkaException: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)
	at kafka.utils.Utils$.swallow(Utils.scala:172)
	at kafka.utils.Logging$class.swallowError(Logging.scala:106)
	at kafka.utils.Utils$.swallowError(Utils.scala:45)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
	at kafka.producer.Producer.send(Producer.scala:77)
	at kafka.javaapi.producer.Producer.send(Producer.scala:33)
	at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)
	at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)
	at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)
Caused by: java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	... 11 more
[2015-12-13 16:02:14,404] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.producer.async.DefaultEventHandler:97)
[2015-12-13 16:02:14,514] ERROR fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.utils.Utils$:106)
kafka.common.KafkaException: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)
	at kafka.utils.Utils$.swallow(Utils.scala:172)
	at kafka.utils.Logging$class.swallowError(Logging.scala:106)
	at kafka.utils.Utils$.swallowError(Utils.scala:45)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
	at kafka.producer.Producer.send(Producer.scala:77)
	at kafka.javaapi.producer.Producer.send(Producer.scala:33)
	at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)
	at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)
	at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)
Caused by: java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	... 11 more
[2015-12-13 16:02:14,524] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.producer.async.DefaultEventHandler:97)
[2015-12-13 16:02:14,644] ERROR fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.utils.Utils$:106)
kafka.common.KafkaException: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)
	at kafka.utils.Utils$.swallow(Utils.scala:172)
	at kafka.utils.Logging$class.swallowError(Logging.scala:106)
	at kafka.utils.Utils$.swallowError(Utils.scala:45)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
	at kafka.producer.Producer.send(Producer.scala:77)
	at kafka.javaapi.producer.Producer.send(Producer.scala:33)
	at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)
	at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)
	at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)
Caused by: java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	... 11 more
[2015-12-13 16:02:14,654] ERROR Failed to collate messages by topic, partition due to: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.producer.async.DefaultEventHandler:97)
[2015-12-13 16:02:14,764] ERROR fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed (kafka.utils.Utils$:106)
kafka.common.KafkaException: fetching topic metadata for topics [Set(MyPattern2)] from broker [ArrayBuffer(id:0,host:10.10.16.14,port:9092)] failed
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
	at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
	at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)
	at kafka.utils.Utils$.swallow(Utils.scala:172)
	at kafka.utils.Logging$class.swallowError(Logging.scala:106)
	at kafka.utils.Utils$.swallowError(Utils.scala:45)
	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
	at kafka.producer.Producer.send(Producer.scala:77)
	at kafka.javaapi.producer.Producer.send(Producer.scala:33)
	at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)
	at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)
	at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)
Caused by: java.nio.channels.ClosedChannelException
	at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
	at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
	at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
	at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
	at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
	... 11 more
[2015-12-13 16:02:14,764] ERROR Failed to send requests for topics MyPattern2 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler:97)
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.error

	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
	at kafka.producer.Producer.send(Producer.scala:77)
	at kafka.javaapi.producer.Producer.send(Producer.scala:33)
	at com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)
	at com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)
	at com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)

没能找到出错原因

三、使用服务器本地安装的Eclipse执行程序，中间不会报错，但是，接收到的中文内容还是乱码