kafka Drictor维护偏移量

最新推荐文章于 2025-03-16 07:01:03 发布

原创最新推荐文章于 2025-03-16 07:01:03 发布 · 332 阅读

0 ·

CC 4.0 BY-SA版权

Spark 同时被 2 个专栏收录

23 篇文章

订阅专栏

工具代码

16 篇文章

订阅专栏

Apache Spark 2.1.x以及spark-streaming-kafka-0-10使用新的的消费者API即异步提交API。你可以在你确保你处理后的数据已经妥善保存之后使用commitAsync API（异步提交 API）来向Kafka提交offsets。新的消费者API会以消费者组id作为唯一标识来提交offsets，将offsets提交到Kafka中。目前这还是实验性特性。

stream.foreachRDD { rdd =>

val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges

// some time later, after outputs have completed

stream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
s
}

在初始化 kafka stream 的时候，查看 zookeeper 中是否保存有 offset，有就从该 offset 进行读取，没有就从最新/旧进行读取。在消费 kafka 数据的同时，将每个 partition 的 offset 保存到 zookeeper 中进行备份

 val sparkConf = new SparkConf().setMaster("local[*]").setAppName("spark-streaming")
    val ssc = new StreamingContext(sparkConf, Seconds(10))
    val topic: String = "test"
    val kafkaParams = Map[String, Object](
      "bootstrap.servers" -> "master:9092",
      "key.deserializer" -> classOf[StringDeserializer],
      "value.deserializer" -> classOf[StringDeserializer],
      "group.id" -> "spark-streaming-group01",
      "auto.offset.reset" -> "earliest",
      "enable.auto.commit" -> (false: java.lang.Boolean)
    )
    var kafkaStream: InputDStream[ConsumerRecord[String, String]] = null
    val zkClient = new ZkClient("master")
    var fromOffsets: Map[TopicPartition, Long] = Map()
    val children = zkClient.countChildren("offsetDir")
    if (children > 0) {
      for (i <- 0 until children) {
        val partitionOffset = zkClient.readData[String]("offsetDir" + "/" + i)
        val tp = new TopicPartition(topic, i)
        fromOffsets += (tp -> partitionOffset.toLong)
        kafkaStream = KafkaUtils.createDirectStream[String, String](
          ssc, PreferConsistent, Subscribe[String, String](Set(topic), kafkaParams, fromOffsets)
        )
      }
    } else {
      kafkaStream = KafkaUtils.createDirectStream[String, String](
        ssc, PreferConsistent, Subscribe[String, String](Set(topic), kafkaParams)
      )
    }

纠错请联系 Q2320534398