Apache Spark Structured Streaming and watermarks

最新推荐文章于 2025-07-19 04:40:50 发布
最新推荐文章于 2025-07-19 04:40:50 发布
阅读量843 收藏
点赞数
分类专栏： SPARK 文章标签： spark structured streaming
本文深入探讨了Apache Spark中水印(watermark)的概念及其在不同场景下的应用，包括如何处理迟到的数据、水印在聚合操作中的作用等。通过具体示例展示了水印如何帮助过滤掉过期的数据记录，并确保流处理的准确性。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

"watermark" should "discard late data and accept 1 late but within watermark with window aggregation" in {
  val testKey = "watermark-window-test"
  val inputStream = new MemoryStream[(Timestamp, String)](1, sparkSession.sqlContext)
  val now = 5000L
  val aggregatedStream = inputStream.toDS().toDF("created", "name")
    .withWatermark("created", "2 seconds")
    .groupBy(window($"created", "2 seconds")).count()
 
  val query = aggregatedStream.writeStream.outputMode("update")
    .foreach(new ForeachWriter[Row]() {
      override def open(partitionId: Long, version: Long): Boolean = true
      override def process(processedRow: Row): Unit = {
        val window = processedRow.get(0)
        val rowRepresentation = s"${window.toString} -> ${processedRow.getAs[Long]("count")}"
        InMemoryKeyedStore.addValue(testKey, rowRepresentation)
      }
      override def close(errorOrNull: Throwable): Unit = {}
    }).start()
 
  new Thread(new Runnable() {
    override def run(): Unit = {
      inputStream.addData((new Timestamp(now), "a1"), (new Timestamp(now), "a2"),
        (new Timestamp(now-4000L), "b1"))
      while (!query.isActive) {
        // wait the query to activate
      }
      // The watermark is now computed as MAX(event_time) - watermark, i.e.:
      // 5000 - 2000 = 3000
      // Thus among the values sent above only "a6" should be accepted because it's within the watermark
      Thread.sleep(7000)
      val timeOutOfWatermark = 1000L
      inputStream.addData((new Timestamp(timeOutOfWatermark), "b2"), (new Timestamp(timeOutOfWatermark), "b3"),
        (new Timestamp(timeOutOfWatermark), "b4"), (new Timestamp(now), "a3"))
    }
  }).start()
 
  query.awaitTermination(25000)
 
  val readValues = InMemoryKeyedStore.getValues(testKey)
  println(s"All data=${readValues}")
  // As you can notice, the count for the window 0-2 wasn't updated with 3 fields (b2, b3 and b4) because they fall
  // before the watermark
  // Please see how this behavior changes in the next test where the watermark is defined to 10 seconds
  readValues should have size 3
  readValues should contain allOf("[1970-01-01 01:00:00.0,1970-01-01 01:00:02.0] -> 1",
    "[1970-01-01 01:00:04.0,1970-01-01 01:00:06.0] -> 2",
    "[1970-01-01 01:00:04.0,1970-01-01 01:00:06.0] -> 3")
}
 
"late data but within watermark" should "be aggregated in correct windows" in {
  val testKey = "watermark-window-test-accepted-data"
  val inputStream = new MemoryStream[(Timestamp, String)](1, sparkSession.sqlContext)
  val now = 5000L
  val aggregatedStream = inputStream.toDS().toDF("created", "name")
    .withWatermark("created", "10 seconds")
    .groupBy(window($"created", "2 seconds")).count()
 
  val query = aggregatedStream.writeStream.outputMode("update")
    .foreach(new ForeachWriter[Row]() {
      override def open(partitionId: Long, version: Long): Boolean = true
      override def process(processedRow: Row): Unit = {
        val window = processedRow.get(0)
        val rowRepresentation = s"${window.toString} -> ${processedRow.getAs[Long]("count")}"
        InMemoryKeyedStore.addValue(testKey, rowRepresentation)
      }
      override def close(errorOrNull: Throwable): Unit = {}
    }).start()
 
  new Thread(new Runnable() {
    override def run(): Unit = {
      inputStream.addData((new Timestamp(now), "a1"), (new Timestamp(now), "a2"),
        (new Timestamp(now-4000L), "b1"))
      while (!query.isActive) {
        // wait the query to activate
      }
      // The watermark is now computed as MAX(event_time) - watermark, i.e.:
      // 5000 - 2000 = 3000
      // Thus among the values sent above only "a6" should be accepted because it's within the watermark
      Thread.sleep(7000)
      val timeOutOfWatermark = 1000L
      inputStream.addData((new Timestamp(timeOutOfWatermark), "b2"), (new Timestamp(timeOutOfWatermark), "b3"),
        (new Timestamp(timeOutOfWatermark), "b4"), (new Timestamp(now), "a3"))
    }
  }).start()
 
  query.awaitTermination(25000)
 
  val readValues = InMemoryKeyedStore.getValues(testKey)
  // As you can notice, the count for the window 0-2 wasn't updated with 3 fields (b2, b3 and b4) because they fall
  // before the watermark
  // Please see how this behavior changes in the next test where the watermark is defined to 10 seconds
  readValues should have size 4
  readValues should contain allOf("[1970-01-01 01:00:00.0,1970-01-01 01:00:02.0] -> 1",
    "[1970-01-01 01:00:00.0,1970-01-01 01:00:02.0] -> 4",
    "[1970-01-01 01:00:04.0,1970-01-01 01:00:06.0] -> 2",
    "[1970-01-01 01:00:04.0,1970-01-01 01:00:06.0] -> 3")
}
 
"watermark after aggregation" should "not be allowed in append mode" in {
  val inputStream = new MemoryStream[(Timestamp, String)](1, sparkSession.sqlContext)
  inputStream.addData((new Timestamp(System.currentTimeMillis()), "a"),
    (new Timestamp(System.currentTimeMillis()), "b"))
  val aggregatedStream = inputStream.toDS().toDF("created", "name")
    .groupBy("created").count()
    .withWatermark("created", "10 seconds")
 
  val exception = intercept[AnalysisException] {
    val query = aggregatedStream.writeStream.outputMode("append")
      .foreach(new NoopForeachWriter[Row]()).start()
    query.awaitTermination(3000)
  }
 
  exception.message should include ("Append output mode not supported when there are streaming aggregations on streaming " +
    "DataFrames/DataSets without watermark")
}
 
"watermark applied on different field than the aggregation in append mode" should "make the processing fail" in {
  val inputStream = new MemoryStream[(Timestamp, String)](1, sparkSession.sqlContext)
  inputStream.addData((new Timestamp(System.currentTimeMillis()), "a"),
    (new Timestamp(System.currentTimeMillis()), "b"))
  val aggregatedStream = inputStream.toDS().toDF("created", "name")
    .withWatermark("created", "10 seconds")
    .agg(count("name"))
 
  val exception = intercept[AnalysisException] {
    // It fails only for append mode. If the watermark is not applied on grouped column it means no more
    // no less that this group is never finished and then that it would never output anything
    val query = aggregatedStream.writeStream.outputMode("append")
      .foreach(new NoopForeachWriter[Row]()).start()
    query.awaitTermination(3000)
  }
 
  exception.message should include ("Append output mode not supported when there are streaming aggregations on streaming " +
    "DataFrames/DataSets without watermark")
}
 
"watermark in append mode" should "emit the results only after watermark expiration" in {
  // Accordingly to the Spark's documentation
  // (https://spark.apache.org/docs/2.2.1/structured-streaming-programming-guide.html#handling-late-data-and-watermarking)
  // in the append mode the partial results are not updated. Only the final result is computed
  // and emitted to the result table:
  // ```
  // Similar to the Update Mode earlier, the engine maintains intermediate counts for each window.
  // However, the partial counts are not updated to the Result Table and not written to sink.
  // The engine waits for “10 mins” for late date to be counted, then drops intermediate state of a
  // window < watermark, and appends the final counts to the Result Table/sink.
  // ```
  // But please notice that in the append output mode the results
  // are emitted once the watermark passed. Here the results for 00:00:10 will be
  // emitted only when the watermark will pass to 00:00:19 (00:00:10 is now before the watermark), i.e. after updating
  // the watermark for the record (24000L, 5).
  val testKey = "watermark-append-mode"
  val inputStream = new MemoryStream[(Timestamp, Int)](1, sparkSession.sqlContext)
  val now = 5000L
  val aggregatedStream = inputStream.toDS().toDF("created", "name")
    .withWatermark("created", "5 second")
    .groupBy("created")
    .count()
 
  val query = aggregatedStream.writeStream.trigger(Trigger.ProcessingTime("2 seconds")).outputMode("append")
    .foreach(new ForeachWriter[Row]() {
      override def open(partitionId: Long, version: Long): Boolean = true
      override def process(processedRow: Row): Unit = {
        val rowRepresentation = s"${processedRow.getAs[Timestamp]("created")} -> ${processedRow.getAs[Long]("count")}"
        InMemoryKeyedStore.addValue(testKey, rowRepresentation)
        println(s"Processing ${rowRepresentation} at ${new Date(System.currentTimeMillis())}")
      }
      override def close(errorOrNull: Throwable): Unit = {}
    }).start()
 
  new Thread(new Runnable() {
    override def run(): Unit = {
      // send the first batch - max time = 10 seconds, so the watermark will be 5 seconds
      inputStream.addData((new Timestamp(now+5000), 1), (new Timestamp(now+5000), 2), (new Timestamp(now+5000), 3),
        (new Timestamp(now+5000), 4))
      while (!query.isActive) {
        // wait the query to activate
      }
      Thread.sleep(4000)
      inputStream.addData((new Timestamp(now), 6), (new Timestamp(6000), 7))
      Thread.sleep(4000)
      inputStream.addData((new Timestamp(24000L), 8))  // Only to update the watermark
      Thread.sleep(4000)
      inputStream.addData((new Timestamp(4000L), 9))
      Thread.sleep(4000)
      inputStream.addData((new Timestamp(now), 10))
    }
  }).start()
 
  query.awaitTermination(45000)
 
  val readValues = InMemoryKeyedStore.getValues(testKey)
  readValues should have size 2
  readValues should contain allOf("1970-01-01 01:00:10.0 -> 4", "1970-01-01 01:00:06.0 -> 1")
}
 
 
"the watermark" should "be used in aggregations others than windowing" in {
  val testKey = "watermark-count-aggregation"
  val inputStream = new MemoryStream[(Timestamp, Int)](1, sparkSession.sqlContext)
  val now = 5000L
  val aggregatedStream = inputStream.toDS().toDF("created", "number")
    .withWatermark("created", "1 second")
    .groupBy("created")
    .count()
 
  val query = aggregatedStream.writeStream./*trigger(Trigger.ProcessingTime("2 seconds")).*/outputMode("update")
    .foreach(new ForeachWriter[Row]() {
      override def open(partitionId: Long, version: Long): Boolean = true
      override def process(processedRow: Row): Unit = {
        val rowRepresentation = s"${processedRow.getAs[Timestamp]("created")} -> ${processedRow.getAs[Long]("count")}"
        InMemoryKeyedStore.addValue(testKey, rowRepresentation)
      }
      override def close(errorOrNull: Throwable): Unit = {}
    }).start()
 
  new Thread(new Runnable() {
    override def run(): Unit = {
      // send the first batch - max time = 10 seconds, so the watermark will be 9 seconds
      inputStream.addData((new Timestamp(now+5000), 1), (new Timestamp(now+5000), 2), (new Timestamp(now+5000), 3),
        (new Timestamp(now+5000), 4))
      while (!query.isActive) {
        // wait the query to activate
      }
      // Here the rows#5 and #6 are ignored because of the watermark
      // In the other side, the row#7 will be taken into account
      // To see what happens when the watermark doesn't exist, you can uncomment the line .withWatermark...
      // Without the watermark, the counter for *now* will be updated
      Thread.sleep(10000)
      inputStream.addData((new Timestamp(4000L), 5))
      inputStream.addData((new Timestamp(now), 6), (new Timestamp(11000), 7))
    }
  }).start()
 
  query.awaitTermination(45000)
 
  val readValues = InMemoryKeyedStore.getValues(testKey)
  println(s"All data=${readValues}")
  // As you can notice, the count for the window 0-2 wasn't updated with 3 fields (b2, b3 and b4) because they fall
  // before the watermark
  // Please see how this behavior changes in the next test where the watermark is defined to 10 seconds
  readValues should have size 2
  readValues should contain allOf("1970-01-01 01:00:10.0 -> 4", "1970-01-01 01:00:11.0 -> 1")
}
 
"the data arriving after the watermark and the state older than the watermark" should "not be discarded correctly" in {
  val testKey = "watermark-deduplicate"
  val testKeyLastProgress = "watermark-deduplicate-last-progress"
  val inputStream = new MemoryStream[(Timestamp, Int)](1, sparkSession.sqlContext)
  val now = 5000L
  val aggregatedStream = inputStream.toDS().toDF("created", "number")
    .withWatermark("created", "6 seconds")
    .dropDuplicates("number", "created")
 
  val query = aggregatedStream.writeStream.trigger(Trigger.ProcessingTime("2 seconds")).outputMode("update")
    .foreach(new ForeachWriter[Row]() {
      override def open(partitionId: Long, version: Long): Boolean = true
      override def process(processedRow: Row): Unit = {
        val rowRepresentation =
          s"${processedRow.getAs[Timestamp]("created").toString} -> ${processedRow.getAs[Int]("number")}"
        InMemoryKeyedStore.addValue(testKey, rowRepresentation)
        println(s"processing ${rowRepresentation}")
      }
      override def close(errorOrNull: Throwable): Unit = {}
    }).start()
 
  new Thread(new Runnable() {
    override def run(): Unit = {
      // Events sent before - they should be correctly deduplicated, i.e. (5000, 1), (5000, 2) and (10000, 2)
      // should be taken
      // As you can observe, the deduplication occurs with the pair (event time, value)
      inputStream.addData((new Timestamp(now), 1), (new Timestamp(now), 2),
        (new Timestamp(now), 1), (new Timestamp(now+5000), 2))
      while (!query.isActive) {
        // wait the query to activate
      }
      logInfo(s"Query was activated, sleep for 9 seconds before sending new data. Current timestamp " +
        s"is ${System.currentTimeMillis()}")
      Thread.sleep(11000)
      logInfo(s"Awaken at ${System.currentTimeMillis()} where the query status is ${query.lastProgress.json}")
      // In the logs we can observe the following entry:
      // ```
      // Filtering state store on: (created#5-T6000ms <= 4000000)
      // (org.apache.spark.sql.execution.streaming.StreamingDeduplicateExec:54)
      // ```
      // As you can correctly deduce, among the entries below:
      // - 1, 2 and 3 will be filtered
      // - 4 will be accepted
      // Moreover, the 4 will be used to compute the new watermark. Later in the logs we can observe the following:
      // ```
      // Filtering state store on: (created#5-T6000ms <= 6000000)
      // (org.apache.spark.sql.execution.streaming.StreamingDeduplicateExec:54)
      // ```
      inputStream.addData((new Timestamp(now), 1), (new Timestamp(now-1000), 2),
        (new Timestamp(now-3000), 3), (new Timestamp(now+7000), 4))
      Thread.sleep(9000)
      // Here the value 1 is after the watermark so automatically discarded
      InMemoryKeyedStore.addValue(testKeyLastProgress, query.lastProgress.json)
      inputStream.addData((new Timestamp(now), 1))
      Thread.sleep(7000)
      inputStream.addData((new Timestamp(now), 1))
      InMemoryKeyedStore.addValue(testKeyLastProgress, query.lastProgress.json)
    }
  }).start()
 
  query.awaitTermination(55000)
 
  val accumulatedValues = InMemoryKeyedStore.getValues(testKey)
  accumulatedValues should have size 4
  accumulatedValues should contain allOf("1970-01-01 01:00:10.0 -> 2", "1970-01-01 01:00:05.0 -> 1",
    "1970-01-01 01:00:05.0 -> 2", "1970-01-01 01:00:12.0 -> 4")
  val seenProgresses = InMemoryKeyedStore.getValues(testKeyLastProgress)
  // This progress represents the moment where all 4 rows are considered as within the watermark
  val initialProgress = seenProgresses(0)
  initialProgress should include("\"stateOperators\":[{\"numRowsTotal\":4,\"numRowsUpdated\":0}]")
  initialProgress should include("\"eventTime\":{\"watermark\":\"1970-01-01T00:00:04.000Z\"}")
  // This progress represents the moment where 2 rows (1970-01-01 01:00:05.0 -> 1 and 1970-01-01 01:00:05.0 -> 2)
  // were removed from the state store because of the watermark expiration
  val progressWithRemovedStates = seenProgresses(1)
  progressWithRemovedStates should include("\"stateOperators\":[{\"numRowsTotal\":2,\"numRowsUpdated\":0}]")
  progressWithRemovedStates should include("\"eventTime\":{\"watermark\":\"1970-01-01T00:00:06.000Z\"}")

}

ref:http://www.waitingforcode.com/apache-spark-structured-streaming/apache-spark-structured-streaming-watermarks/read

深入浅出Spark实时处理（二） --- Structured Streaming
杨鑫newlife的专栏
01-02 616
对比Spark Streaming可以参看文章深入浅出Spark实时处理（一） --- Spark Streaming Structured Streaming 是一个基于Spark-SQL的可伸缩且容错的流处理引擎。使用者可以像对待静态数据一样来处理流式计算。当流数据到达时，Spark-SQL 引擎将负责逐步递增的运行并更新结果。可以在Scala、Java、Python或者R语言中使用...
Spark Streaming Join with Watermark
tanzhangwen的专栏
06-26 1212
我们常常会有这样的应用场景，就是一个复杂系统可能是由多个模块组成，然后每个模块都会有自己的logging系统。最后我们在处理日志数据的时候，会希望能够把不同模块的日志数据join在一起。一个简单的例子就是，广告有显示的日志，然后当用户点击了广告后又会有点击的日志，那么如果我想把广告信息和点击信息组合在一起就需要从这两个日志流源得到的数据进行join。这在API文档里面有相应的介绍Stream-St...
「Spark-2.2.0」Structured Streaming - Watermarking操作详解
图特摩斯科技-博客
07-13 6820
很高兴spark-2.2.0于昨天发布，结构化流式处理在该版本中可用于生产环境。 Spark Streaming 中 Exactly Once 指的是： 每条数据从输入源传递到 Spark 应用程序 Exactly Once每条数据只会分到 Exactly Once batch 处理输出端文件系统保证幂等关系 streaming DataFrames/Datase
MacBook 下载 磁力（magnet） 链接的视频的app
最新发布
qq_45642410的博客
07-19 892
qBittorrent是一款免费开源的跨平台BT下载工具，支持磁力链接且无广告。Mac用户可通过官网下载.dmg安装包，安装时需在系统设置中手动允许打开未认证应用。安装完成后，用户可添加磁力链接开始下载，并支持匿名模式等可选设置。该软件界面简洁，是替代传统BT客户端的优秀选择。
Spark Streaming中withWatermark的简单尝试
统木木的博客
12-21 1195
我们在处理流数据的时候，往往会有实时性要求。可是如果我们直接按照程序所在服务器的当前时间计算又不行，比如当上游日志数据延迟了，则所有的这部分数据都会被抛弃掉。所以一般我们在记录日志的时候，加上日志的时间戳。这样我们在进行流处理的时候，就可以把日志记录的时间拿出来，根据这个时间来决定流处理是不是要往下进行。而往往我们会以最早到达的日志作为时间参考点，如果下一个日志比这个时间点晚的太多，就可以抛弃掉。...
Spark Structured Streaming EventTime下Watermark生成与Window触发
王佩的优快云博客
03-08 1932
本文旨在弄清楚Spark Structured Streaming EventTime下Watermark生成与Window触发相关问题。 窗口起止时间。 水印的生成。 对迟到数据的处理。 窗口销毁的时机。 Watermark与Update/Complete输出模式之间的关系。 测试数据 // 造的测试数据，如下: // eventTime: 北京时间 {"eventTi...
【spark床头书系列】Spark Structured Streaming 编程权威指南
wang2leee的博客
12-01 1427
Spark Structured Streaming 编程权威指南,看一篇就够了
Spark 面试题（十四）
闲依农圃邻，偶似山林客。
06-17 1461
1. 简述Spark Streaming的工作原理 ？ 2. 简述Spark输出文件的个数，如何合并小文件 ？ 3. 简述Spark的driver是怎么驱动作业流程的 ？ 4. 简述Spark SQL的劣势 ？ 5. 简述Spark Streaming和Structed Streaming ？ 6. 简述Spark为什么比Hadoop速度快 ？ 7. 简述Spark Streaming的双流join的过程，怎么做的 ？
sparkstructured
01-09
### Apache Spark Structured Streaming 使用指南 #### 创建和配置环境 为了使用Apache Spark Structured Streaming，需先设置好Spark环境并导入必要的依赖库。对于Maven项目，在`pom.xml`文件中加入如下依赖项[^2]...
StructuredStreaming: A Declarative API for Real-Time
weixin_38440581的博客
05-10 1779
AbstractWith the ubiquity of real-time data,organizations need streaming systems that are scalable, easy to use, and easyto integrate into business applications. Structured Streaming is a newhigh-leve...
Spark视频第4课_深度解析Spark Structured Streaming
12-25
机器学习(Machine Learning, ML)是一门多领域交叉学科，涉及概率论、统计学、逼近论、凸分析、算法复杂度理论等多门学科。专门研究计算机怎样模拟或实现人类的学习行为，以获取新的知识或技能，重新组织已有的知识结构使之不断改善自身的性能。
Spark Structured Steaming 聚合、watermark 以及 window操作，结合输出模式的研究总结
weixin_33672400的博客
05-08 1454
2019独角兽企业重金招聘Python工程师标准>>> ...
Spark结构化流之水印
踏雪无痕
06-06 2461
一.处理后期数据和加水印 现在考虑如果存在事件迟到了应用程序会发生什么。例如，应用程序在12:11接收在12:04（即事件时间）生成的单词。应用程序应使用12:04而不是12:11来更新窗口的旧计数12:00 - 12:10。这在基于窗口的分组中很自然地发生，结构化流可以长时间保持部分聚合的中间状态，以便后期数据可以正确更新旧窗口的聚合，如下所示。 但是，要连续几天运行此查询，系统必须限制其累积的中间内存状态量。这意味着系统需要知道何时可以从内存中删除旧状态聚合，因为应用程序将不再接收该聚合的最新数据。为
Spark Streaming的Event Time和WaterMark理解
coderlaw's study
05-19 5757
EventTime即事件真正生成的时间。例如一个用户在10：06点击 了一个按钮。记录在系统中为10：06。这条数据发送到Kafka，又到了Spark Streaming中处理，已经是10：08了。这个处理的时间就是process Time。Water Mark即 这个时间点之前的数据都到达了。例如现在是10:20 ,但是water Mark是 10：10 分，那么认为，10：10之前的数据都已经...
Spark2.3（三十四）：Spark Structured Streaming之withWaterMark和windows窗口是否可以实现最近一小时统计...
weixin_33877885的博客
12-02 653
WaterMark除了可以限定来迟数据范围，是否可以实现最近一小时统计？ WaterMark目的用来限定参数计算数据的范围：比如当前计算数据内max timestamp是12::00，waterMark限定数据分为是60 minutes，那么如果此时输入11:00之前的数据就会被舍弃不参与统计，视为来迟范围超出了60minutes限定范围。 那么，是否可以借助它实现最近一小时的数据统计呢？ ...
Spark基于事件时间的“状态”流的深层分析 - withWatermark与mapGroupsWithState的关系
Laurence的技术博客
06-27 4939
不管是基于watermark的窗口计算还是自维护的状态流，它们都是有状态的，watermark只是规定了数据进入“状态”（有资格参与状态计算）的条件，并没有（也不适合）声明状态的“退出”机制。对于watermark的窗口计算来说，它们的“退出”机制是：如果最近某个还处于active状态的窗口它的EndTime比当前批次中最新的一个事件时间减去watermark规定的阈值还要“早”，说明这个窗口所有...
19:Structured Streaming：Windows操作和WaterMarker-流处理引擎提供了哪些优秀机制
someInNeed的博客
01-28 2062
今天这一讲，咱们从功能的视角出发，继续来聊一聊 Structured Streaming 流处理引擎都为开发者都提供了哪些特性与能力，让你更灵活地设计并实现流处理应用。 Structured Streaming 怎样坐享其成？ 学习过计算模型之后，我们知道，不管是 Batch mode 的多个 Micro-batch、多个作业的执行方式，还是 Continuous mode 下的一个 Long running job，这些作业的执行计划，最终都会交付给 Spark SQL ...
Apache Spark Structured Streaming技术详解
Structured Streaming是Apache Spark中的一个流处理模型，它以微批处理的方式运行流数据。Structured Streaming将数据流抽象为一个无限的表格，支持从多种数据源进行实时数据处理，并能够与Spark SQL无缝集成。在这...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值