spark处理复杂json数据案例

阿拉吴悠

于 2020-06-29 14:51:29 发布

阅读量1.2k

点赞数 7

文章标签：大数据 spark

本文链接：https://blog.youkuaiyun.com/weixin_47994250/article/details/107018840

版权

本文介绍了如何使用Spark处理复杂的JSON数据，从导入数据到拆分数据，详细讲解了如何规划schema，拆分键值对，特别是对'cm'、'ap'、'et'字段的处理，以及如何通过explode进行行转列操作，最终将所有键值对解析并准备存储或写入Hive表。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

spark处理json数据

一、导入数据

源数据格式是这样的

1593136318832|
{
   
    "cm":{
   
        "ln":"-57.4",
        "sv":"V2.7.1",
        "os":"8.2.9",
        "g":"N92TN1PB@gmail.com",
        "mid":"1",
        "nw":"WIFI",
        "l":"pt",
        "vc":"12",
        "hw":"1080*1920",
        "ar":"MX",
        "uid":"1",
        "t":"1593104002139",
        "la":"-45.9",
        "md":"sumsung-4",
        "vn":"1.0.9",
        "ba":"Sumsung",
        "sr":"J"
    },
    "ap":"app",
    "et":[
        {
   
            "ett":"1593047699935",
            "en":"display",
            "kv":{
   
                "goodsid":"0",
                "action":"2",
                "extend1":"2",
                "place":"0",
                "category":"16"
            }
        },
        {
   
            "ett":"1593105286705",
            "en":"newsdetail",
            "kv":{
   
                "entry":"3",
                "goodsid":"1",
                "news_staytime":"36",
                "loading_time":"36",
                "action":"3",
                "showtype":"1",
                "category":"53",
                "type1":"201"
            }
        },
        {
   
            "ett":"1593044284730",
            "en":"ad",
            "kv":{
   
                "activityId":"1",
                "displayMills":"30370",
                "entry":"1",
                "action":"2",
                "contentType":"0"
            }
        },
        {
   
            "ett":"1593127793318",
            "en":"notification",
            "kv":{
   
                "ap_time":"1593085393178",
                "action":"1",
                "type":"4",
                "content":""
            }
        },
        {
   
            "ett":"1593051576646",
            "en":"active_background",
            "kv":{
   
                "active_source":"1"
            }
        },
        {

最低0.47元/天解锁文章