Spark Streaming官方文档翻译基本概念之累加器、广播变量和检查点

本文链接：https://blog.youkuaiyun.com/qq_39315954/article/details/103810335

本文详细介绍了SparkStreaming中累加器和广播变量的使用方法，包括如何在驱动程序失败后重新实例化这些变量以确保应用程序的正确运行。通过实例展示了如何使用累加器进行计数和如何使用广播变量过滤数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

基本概念

累加器、广播变量和检查点(Accumulators, Broadcast Variables, and Checkpoints)

无法从Spark Streaming中的检查点恢复累加器和广播变量。如果启用了检查点并同时使用累加器或广播变量，则必须为累加器和广播变量创建延迟实例化的单例实例，以便在驱动程序失败重新启动后重新实例化它们。如下面的例子所示。

object WordBlacklist {

  @volatile private var instance: Broadcast[Seq[String]] = null

  def getInstance(sc: SparkContext): Broadcast[Seq[String]] = {
    if (instance == null) {
      synchronized {
        if (instance == null) {
          val wordBlacklist = Seq("a", "b", "c")
          instance = sc.broadcast(wordBlacklist)
        }
      }
    }
    instance
  }
}

object DroppedWordsCounter {

  @volatile private var instance: LongAccumulator = null

  def getInstance(sc: SparkContext): LongAccumulator = {
    if (instance == null) {
      synchronized {
        if (instance == null) {
          instance = sc.longAccumulator("WordsInBlacklistCounter")
        }
      }
    }
    instance
  }
}

wordCounts.foreachRDD { (rdd: RDD[(String, Int)], time: Time) =>
  // Get or register the blacklist Broadcast
  val blacklist = WordBlacklist.getInstance(rdd.sparkContext)
  // Get or register the droppedWordsCounter Accumulator
  val droppedWordsCounter = DroppedWordsCounter.getInstance(rdd.sparkContext)
  // Use blacklist to drop words and use droppedWordsCounter to count them
  val counts = rdd.filter { case (word, count) =>
    if (blacklist.value.contains(word)) {
      droppedWordsCounter.add(count)
      false
    } else {
      true
    }
  }.collect().mkString("[", ", ", "]")
  val output = "Counts at time " + time + " " + counts
})

查看完整的源代码。