Spark消费kafka数据 json中包含数组的数据类型

最新推荐文章于 2025-09-11 12:49:07 发布

原创

最新推荐文章于 2025-09-11 12:49:07 发布 · 812 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#spark #hdfs #kafka

JSON数据格式

{
    "header": {
        "traceId": "06ad872d5d5bfa0d", 
        "appName": "zeus-merchant", 
        "deviceType": null, 
        "version": null, 
        "userAgent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36", 
        "ip": "119.162.98.224", 
        "sysTime": "2021-06-11 13:36:27"
    }, 
    "body": [
        {
            "txnType": 1, 
            "txnCode": "10100720012021061111522994861596", 
            "txnSubCode": "18399", 
            "refCode": "10200720012021061113362702629760", 
            "txnValue": null, 
            "event": 1, 
            "subEvent": null, 
            "userId": "307426", 
            "extend": null
        }
    ]
}

代码：

import com.alibaba.fastjson.{JSON, JSONArray, JSONObject}
import org.apache.kafka.common.serialization.StringDeserializer
import org.apache.spark.SparkConf
import org.apache.spark.sql.types.{StringType, StructField, StructType}
import org.apache.spark.sql.{Row, SparkSession, functions}
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.streaming.kafka010.{ConsumerStrategies, KafkaUtils, LocationStrategies}

import scala.collection.mutable.ArrayBuffer

object cainiao {
  def main(args: Array[St