SparkStreaming updateStateByKey 保存记录信息

最新推荐文章于 2025-09-10 10:06:31 发布

转载最新推荐文章于 2025-09-10 10:06:31 发布 · 96 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/zDanica/p/5471611.html

文章标签：

#大数据 #scala #开发工具

本文介绍如何使用Apache Spark Streaming进行状态更新处理，通过示例代码展示了如何配置Spark环境并利用updateStateByKey更新状态。同时，文章还提到了在集群环境下运行时可能遇到的分区数不匹配的问题及解决方案。

object SparkStreaming_StateFul {

def main(args: Array[String]): Unit = {
    Logger.getLogger("org.apache.spark").setLevel(Level.WARN)
    Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.OFF)

val conf = new SparkConf().setMaster("local[2]")
      .setAppName(this.getClass.getSimpleName)
      .set("spark.executor.memory", "2g")
      .set("spark.cores.max", "8")
      .setJars(Array("E:\\ScalaSpace\\Spark_Streaming\\out\\artifacts\\Spark_Streaming.jar"))
val context = new SparkContext(conf)

val updateFunc = (values : Seq[Int],state : Option[Int]) => {
val currentCount  = values.foldLeft(0)(_+_)
val previousCount = state.getOrElse(0) Some//step1 create streaming context
    val new ,Seconds10"."//step2 create a networkInputStream on get ip:port and count the words in input stream of \n delimited text
    val "218.193.154.79",12345val " "val ,1//使用updateStateByKey 来更新状态
    val Int

ssc.checkPoint 如果在集群上运行会报出如下的错误：

org.apache.spark.SparkException: Checkpoint RDD ReliableCheckpointRDD[9] at print at SparkStreaming_StateFul.scala:43(0) has different number of partitions from original RDD MapPartitionsRDD[8] at updateStateByKey at SparkStreaming_StateFul.scala:41(2)
	at org.apache.spark.rdd.ReliableRDDCheckpointData.doCheckpoint(ReliableRDDCheckpointData.scala:73)
	at org.apache.spark.rdd.RDDCheckpointData.checkpoint(RDDCheckpointData.scala:74)

这是因为没有将文件保存到hdfs环境中导致的

From WizNote

转载于:https://www.cnblogs.com/zDanica/p/5471611.html