Spark Streaming更改数据结构

最新推荐文章于 2022-05-07 19:50:16 发布

小财迷，嘻嘻

最新推荐文章于 2022-05-07 19:50:16 发布

阅读量195

点赞数

分类专栏： spark 文章标签： Spark Streaming

本文链接：https://blog.youkuaiyun.com/weixin_48185778/article/details/111642981

版权

spark 专栏收录该内容

30 篇文章

订阅专栏

本文介绍如何在Spark Streaming中，通过transform函数将接收到的流式数据结构（如'java javajava'）转换为带有指定时间格式（如'2020122417:30:10'）的形式，以满足业务需求。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

我们在使用Spark Streaming处理流式数据时，业务需求需要更改数据结构，可以使用transform完成转化工作。

需求：输入：java scala java java，要求加上指定时间格式，
输出：
((java,20201224 17:30:10),3)
((scala,20201224 17:30:10),1)

import java.text.SimpleDateFormat

import org.apache.kafka.clients.consumer.{ConsumerConfig, ConsumerRecord}
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD
import org.apache.spark.streaming.dstream.{DStream, InputDStream}
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}
import org.apache.spark.streaming.{Seconds, StreamingContext}

object SparkTransform {
  def main(args: Array[String]): Unit = {
    val conf: SparkConf = new SparkConf().setAppName("SparkWindowDemo").setMaster("local[*]")

    val streamingContext = new StreamingContext(conf,Seconds(2))   //批处理时间设置为2秒,也就是采集时间

    val kafkaParams: Map[String, String] = Map(
      (ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG -> "192.168.136.20:9092"),
      (ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG -> "org.apache.kafka.common.serialization.StringDeserializer"),
      (ConsumerConfig.GROUP_ID_CONFIG -> "kafkaGroup2")
    )

    val kafkaStream: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream(
      streamingContext,
      LocationStrategies.PreferConsistent,
      ConsumerStrategies.Subscribe(Set("SparkKafkaDemo"), kafkaParams)
    )

    //业务需求需要更改数据结构是可以使用transform完成转化工作
    val numStream: DStream[((String, String), Int)] = kafkaStream.transform((rdd, timestamp) => {
      val format = new SimpleDateFormat("yyyyMMdd HH:mm:ss")
      val time: String = format.format(timestamp.milliseconds)
      val value: RDD[((String, String), Int)] = rdd.flatMap(x => x.value().split("\\s+"))
        .map(x => ((x, time), 1))
          .reduceByKey((x,y)=>x+y)
          .sortBy(x=>x._2,ascending = false)
      value
    })

    numStream.print()

    streamingContext.start()
    streamingContext.awaitTermination()
  }
}

创建生产信息进行测试

kafka-console-producer.sh --topic SparkKafkaDemo --broker-list 192.168.136.20:9092

# 输入：
java scala java java

# 输出：
((java,20201224 17:30:10),3)
((scala,20201224 17:30:10),1)

此项目的pom依赖如下：

    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.11</artifactId>
      <version>2.0.0</version>
    </dependency>

    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-streams</artifactId>
      <version>2.0.0</version>
    </dependency>

    <dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.4.5</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>2.4.5</version>
    </dependency>

    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-databind</artifactId>
      <version>2.6.6</version>
    </dependency>