[WARN] Network error when fetching messages:storm-kafka-0.8导致kafka读取数据丢失

最新推荐文章于 2022-04-29 18:07:21 发布

原创最新推荐文章于 2022-04-29 18:07:21 发布 · 2.8k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#storm #kafka

storm 同时被 2 个专栏收录

11 篇文章

订阅专栏

kafka

6 篇文章

订阅专栏

本文深入探讨了Kafka网络错误时数据丢失的问题，通过源码分析，解释了数据丢失的原因，并提供了可能的解决方案。重点在于网络异常处理及数据一致性维护。

2015-01-09T16:17:10.090+0800 s.k.KafkaUtils [WARN] Network error when fetching messages:
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_51]
        at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_51]
        at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_51]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_51]
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer.reconnect(SimpleConsumer.scala:57) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:79) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:71) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:109) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:109) ~[stormjar.jar:na]
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:108) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108) ~[stormjar.jar:na]
        at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:108) ~[stormjar.jar:na]
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) ~[stormjar.jar:na]
                                                                                 


 2015-01-09T16:17:10.091+0800 s.k.KafkaSpout [WARN] Fetch failed
storm.kafka.FailedFetchException: java.net.ConnectException: Connection refused
        at storm.kafka.KafkaUtils.fetchMessages(KafkaUtils.java:176) ~[stormjar.jar:na]
        at storm.kafka.PartitionManager.fill(PartitionManager.java:159) ~[stormjar.jar:na]
        at storm.kafka.PartitionManager.next(PartitionManager.java:123) ~[stormjar.jar:na]
        at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:141) ~[stormjar.jar:na]
        at backtype.storm.daemon.executor$fn__3373$fn__3388$fn__3417.invoke(executor.clj:565) [storm-core-0.9.3.jar:0.9.3]
        at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) [storm-core-0.9.3.jar:0.9.3]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]

1，由于网络异常，导致数据丢失：分析源码原因：

  //returns false if it's reached the end of current batch
    public EmitState next(SpoutOutputCollector collector) {
        if (_waitingToEmit.isEmpty()) {

          //fill是fetch message的方法
            fill();
        }
        while (true) {
            MessageAndRealOffset toEmit = _waitingToEmit.pollFirst();
            if (toEmit == null) {
                return EmitState.NO_EMITTED;
            }
            Iterable<List<Object>> tups = KafkaUtils.generateTuples(_spoutConfig, toEmit.msg);
            if (tups != null) {
                for (List<Object> tup : tups) {
                    collector.emit(tup, new KafkaMessageId(_partition, toEmit.offset));
                }
                break;
            } else {
                ack(toEmit.offset);
            }
        }
        if (!_waitingToEmit.isEmpty()) {
            return EmitState.EMITTED_MORE_LEFT;
        } else {
            return EmitState.EMITTED_END;
        }
    }

2，进入fill方法：

 private void fill() {
        long start = System.nanoTime();
        long offset;
        final boolean had_failed = !failed.isEmpty();

        // Are there failed tuples? If so, fetch those first.
        if (had_failed) {
            offset = failed.first();
        } else {
            offset = _emittedToOffset;
        }

        ByteBufferMessageSet msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, offset);

从上面的代码中看出，由于偏移量

emittedToOffset已经复制给offset，所以<pre name="code" class="java">fetchMessages异常的时候，下次读数据就从<pre name="code" class="java">emittedToOffset开始读取，所以导致数据丢失