本文详细记录了一次解决Flume监听数据发送给Kafka时出现的重复消费和发布失败问题的过程，通过调整Kafka版本至0.9.0.1，成功解决了Flume与Kafka之间的兼容性问题。
flume监听数据发送给kafka的消费者，出现情况，kafka重复消费flume的数据，查看了flume的日志报错如下
(org.apache.flume.sink.kafka.KafkaSink.process:145) - Failed to publish events
kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:93)
at kafka.producer.Producer.send(Producer.scala:77)
at kafka.javaapi.producer.Producer.send(Producer.scala:42)
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:135)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner
P
o
l
l
i
n
g
R
u
n
n
e
r
.
r
u
n
(
S
i
n
k
R
u
n
n
e
r
.
j
a
v
a
:
147
)
a
t
j
a
v
a
.
l
a
n
g
.
T
h
r
e
a
d
.
r
u
n
(
T
h
r
e
a
d
.
j
a
v
a
:
745
)
07
J
a
n
201915
:
52
:
42
,
142
E
R
R
O
R
[
S
i
n
k
R
u
n
n
e
r
−
P
o
l
l
i
n
g
R
u
n
n
e
r
−
D
e
f
a
u
l
t
S
i
n
k
P
r
o
c
e
s
s
o
r
]
(
o
r
g
.
a
p
a
c
h
e
.
f
l
u
m
e
.
S
i
n
k
R
u
n
n
e
r
PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745) 07 Jan 2019 15:52:42,142 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner
PollingRunner.run(SinkRunner.java:147)atjava.lang.Thread.run(Thread.java:745)07Jan201915:52:42,142ERROR[SinkRunner−PollingRunner−DefaultSinkProcessor](org.apache.flume.SinkRunnerPollingRunner.run:160) - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to publish events
at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:156)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner
P
o
l
l
i
n
g
R
u
n
n
e
r
.
r
u
n
(
S
i
n
k
R
u
n
n
e
r
.
j
a
v
a
:
147
)
a
t
j
a
v
a
.
l
a
n
g
.
T
h
r
e
a
d
.
r
u
n
(
T
h
r
e
a
d
.
j
a
v
a
:
745
)
C
a
u
s
e
d
b
y
:
k
a
f
k
a
.
c
o
m
m
o
n
.
F
a
i
l
e
d
T
o
S
e
n
d
M
e
s
s
a
g
e
E
x
c
e
p
t
i
o
n
:
F
a
i
l
e
d
t
o
s
e
n
d
m
e
s
s
a
g
e
s
a
f
t
e
r
3
t
r
i
e
s
.
a
t
k
a
f
k
a
.
p
r
o
d
u
c
e
r
.
a
s
y
n
c
.
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
h
a
n
d
l
e
(
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
s
c
a
l
a
:
93
)
a
t
k
a
f
k
a
.
p
r
o
d
u
c
e
r
.
P
r
o
d
u
c
e
r
.
s
e
n
d
(
P
r
o
d
u
c
e
r
.
s
c
a
l
a
:
77
)
a
t
k
a
f
k
a
.
j
a
v
a
a
p
i
.
p
r
o
d
u
c
e
r
.
P
r
o
d
u
c
e
r
.
s
e
n
d
(
P
r
o
d
u
c
e
r
.
s
c
a
l
a
:
42
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
u
m
e
.
s
i
n
k
.
k
a
f
k
a
.
K
a
f
k
a
S
i
n
k
.
p
r
o
c
e
s
s
(
K
a
f
k
a
S
i
n
k
.
j
a
v
a
:
135
)
.
.
.
3
m
o
r
e
07
J
a
n
201915
:
52
:
50
,
151
I
N
F
O
[
S
i
n
k
R
u
n
n
e
r
−
P
o
l
l
i
n
g
R
u
n
n
e
r
−
D
e
f
a
u
l
t
S
i
n
k
P
r
o
c
e
s
s
o
r
]
(
k
a
f
k
a
.
u
t
i
l
s
.
L
o
g
g
i
n
g
PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745) Caused by: kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:93) at kafka.producer.Producer.send(Producer.scala:77) at kafka.javaapi.producer.Producer.send(Producer.scala:42) at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:135) ... 3 more 07 Jan 2019 15:52:50,151 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (kafka.utils.Logging
PollingRunner.run(SinkRunner.java:147)atjava.lang.Thread.run(Thread.java:745)Causedby:kafka.common.FailedToSendMessageException:Failedtosendmessagesafter3tries.atkafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:93)atkafka.producer.Producer.send(Producer.scala:77)atkafka.javaapi.producer.Producer.send(Producer.scala:42)atorg.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:135)...3more07Jan201915:52:50,151INFO[SinkRunner−PollingRunner−DefaultSinkProcessor](kafka.utils.Loggingclass.info:68) - Connected to bigdata:9092 for producing
07 Jan 2019 15:52:50,160 WARN [SinkRunner-PollingRunner-DefaultSinkProcessor] (kafka.utils.Logging
c
l
a
s
s
.
w
a
r
n
:
89
)
−
F
a
i
l
e
d
t
o
s
e
n
d
p
r
o
d
u
c
e
r
r
e
q
u
e
s
t
w
i
t
h
c
o
r
r
e
l
a
t
i
o
n
i
d
243
t
o
b
r
o
k
e
r
9092
w
i
t
h
d
a
t
a
f
o
r
p
a
r
t
i
t
i
o
n
s
[
b
i
g
d
a
t
a
T
e
s
t
,
2
]
j
a
v
a
.
n
i
o
.
B
u
f
f
e
r
U
n
d
e
r
f
l
o
w
E
x
c
e
p
t
i
o
n
a
t
j
a
v
a
.
n
i
o
.
B
u
f
f
e
r
.
n
e
x
t
G
e
t
I
n
d
e
x
(
B
u
f
f
e
r
.
j
a
v
a
:
506
)
a
t
j
a
v
a
.
n
i
o
.
H
e
a
p
B
y
t
e
B
u
f
f
e
r
.
g
e
t
I
n
t
(
H
e
a
p
B
y
t
e
B
u
f
f
e
r
.
j
a
v
a
:
361
)
a
t
k
a
f
k
a
.
a
p
i
.
P
r
o
d
u
c
e
r
R
e
s
p
o
n
s
e
class.warn:89) - Failed to send producer request with correlation id 243 to broker 9092 with data for partitions [bigdataTest,2] java.nio.BufferUnderflowException at java.nio.Buffer.nextGetIndex(Buffer.java:506) at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) at kafka.api.ProducerResponse
class.warn:89)−Failedtosendproducerrequestwithcorrelationid243tobroker9092withdataforpartitions[bigdataTest,2]java.nio.BufferUnderflowExceptionatjava.nio.Buffer.nextGetIndex(Buffer.java:506)atjava.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361)atkafka.api.ProducerResponse.readFrom(ProducerResponse.scala:43)
at kafka.producer.SyncProducer.send(SyncProducer.scala:110)
at kafka.producer.async.DefaultEventHandler.kafka
p
r
o
d
u
c
e
r
producer
producerasync
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
DefaultEventHandler
DefaultEventHandler
s
e
n
d
(
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
s
c
a
l
a
:
259
)
a
t
k
a
f
k
a
.
p
r
o
d
u
c
e
r
.
a
s
y
n
c
.
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
send(DefaultEventHandler.scala:259) at kafka.producer.async.DefaultEventHandler
send(DefaultEventHandler.scala:259)atkafka.producer.async.DefaultEventHandler
a
n
o
n
f
u
n
anonfun
anonfundispatchSerializedData
2.
a
p
p
l
y
(
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
s
c
a
l
a
:
110
)
a
t
k
a
f
k
a
.
p
r
o
d
u
c
e
r
.
a
s
y
n
c
.
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
2.apply(DefaultEventHandler.scala:110) at kafka.producer.async.DefaultEventHandler
2.apply(DefaultEventHandler.scala:110)atkafka.producer.async.DefaultEventHandler
a
n
o
n
f
u
n
anonfun
anonfundispatchSerializedData
2.
a
p
p
l
y
(
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
s
c
a
l
a
:
102
)
a
t
s
c
a
l
a
.
c
o
l
l
e
c
t
i
o
n
.
T
r
a
v
e
r
s
a
b
l
e
L
i
k
e
2.apply(DefaultEventHandler.scala:102) at scala.collection.TraversableLike
2.apply(DefaultEventHandler.scala:102)atscala.collection.TraversableLikeWithFilterKaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲foreach$1.apply…anonfun$foreach
1.
a
p
p
l
y
(
H
a
s
h
M
a
p
.
s
c
a
l
a
:
98
)
a
t
s
c
a
l
a
.
c
o
l
l
e
c
t
i
o
n
.
m
u
t
a
b
l
e
.
H
a
s
h
M
a
p
1.apply(HashMap.scala:98) at scala.collection.mutable.HashMap
1.apply(HashMap.scala:98)atscala.collection.mutable.HashMap
a
n
o
n
f
u
n
anonfun
anonfunforeach
1.
a
p
p
l
y
(
H
a
s
h
M
a
p
.
s
c
a
l
a
:
98
)
a
t
s
c
a
l
a
.
c
o
l
l
e
c
t
i
o
n
.
m
u
t
a
b
l
e
.
H
a
s
h
T
a
b
l
e
1.apply(HashMap.scala:98) at scala.collection.mutable.HashTable
1.apply(HashMap.scala:98)atscala.collection.mutable.HashTableclass.foreachEntry(HashTable.scala:226)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
at scala.collection.TraversableLike
W
i
t
h
F
i
l
t
e
r
.
f
o
r
e
a
c
h
(
T
r
a
v
e
r
s
a
b
l
e
L
i
k
e
.
s
c
a
l
a
:
771
)
a
t
k
a
f
k
a
.
p
r
o
d
u
c
e
r
.
a
s
y
n
c
.
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
d
i
s
p
a
t
c
h
S
e
r
i
a
l
i
z
e
d
D
a
t
a
(
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
s
c
a
l
a
:
102
)
a
t
k
a
f
k
a
.
p
r
o
d
u
c
e
r
.
a
s
y
n
c
.
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
h
a
n
d
l
e
(
D
e
f
a
u
l
t
E
v
e
n
t
H
a
n
d
l
e
r
.
s
c
a
l
a
:
75
)
a
t
k
a
f
k
a
.
p
r
o
d
u
c
e
r
.
P
r
o
d
u
c
e
r
.
s
e
n
d
(
P
r
o
d
u
c
e
r
.
s
c
a
l
a
:
77
)
a
t
k
a
f
k
a
.
j
a
v
a
a
p
i
.
p
r
o
d
u
c
e
r
.
P
r
o
d
u
c
e
r
.
s
e
n
d
(
P
r
o
d
u
c
e
r
.
s
c
a
l
a
:
42
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
u
m
e
.
s
i
n
k
.
k
a
f
k
a
.
K
a
f
k
a
S
i
n
k
.
p
r
o
c
e
s
s
(
K
a
f
k
a
S
i
n
k
.
j
a
v
a
:
135
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
u
m
e
.
s
i
n
k
.
D
e
f
a
u
l
t
S
i
n
k
P
r
o
c
e
s
s
o
r
.
p
r
o
c
e
s
s
(
D
e
f
a
u
l
t
S
i
n
k
P
r
o
c
e
s
s
o
r
.
j
a
v
a
:
68
)
a
t
o
r
g
.
a
p
a
c
h
e
.
f
l
u
m
e
.
S
i
n
k
R
u
n
n
e
r
WithFilter.foreach(TraversableLike.scala:771) at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:102) at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:75) at kafka.producer.Producer.send(Producer.scala:77) at kafka.javaapi.producer.Producer.send(Producer.scala:42) at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:135) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner
WithFilter.foreach(TraversableLike.scala:771)atkafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:102)atkafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:75)atkafka.producer.Producer.send(Producer.scala:77)atkafka.javaapi.producer.Producer.send(Producer.scala:42)atorg.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:135)atorg.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)atorg.apache.flume.SinkRunnerPollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
查了网上的各种解决方案，kafka的properties文件的kafka.connection=主机名：端口、kafka，还有flume的job文件a2.sinks.k2A.brokerList = 主机号:端口号，都没问题。
这个问题困扰了我两天，仍然报错如下，重复消费数据，可以说是flume不断的发送给消费者
解决方案：
我安装的zookeeper版本是zookeeper-3.4.5-cdh5.7.6,
flume的版本是： flume-1.6.0-cdh5.7.6
kafka的版本是：kafka_2.11-0.8.2.1
将kafka的版本换成：kafka_2.11-0.9.0.1，不报错了，还是兼容性问题
扫一扫
3958
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
