sparkSQL清洗埋点数据（java版）

最新推荐文章于 2023-04-24 16:02:01 发布

掉进悬崖的狼

最新推荐文章于 2023-04-24 16:02:01 发布

阅读量1.4k

点赞数

CC 4.0 BY-SA版权

分类专栏： Spark 大数据文章标签： sparkSQL 解析埋点数据 json

本文链接：https://blog.youkuaiyun.com/MrZhangBaby/article/details/88840464

本文介绍了如何使用SparkSQL进行埋点数据的清洗工作，特别是针对JSON格式的数据。通过Java编写程序，提取关键指标并将其存储到Mysql数据库中。尽管作者久未使用SparkSQL，但发现其功能依然强大且全面。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

数据格式:

{"actionTimes":"2018-11-25","actions":"搜索","bb":"v1.0","fromType":"Chrome/73.0.3683.75","fromURL":"https://www.nyist.com/s?wd=ip%E5%9C%B0%E5%9D%80&rsv_spt=1","ip":"120.50.10.233","requestMethod":"GET","sessionId":"0a1c7b51-4434-4ea0-ada9-d45ba788541c","title":"关键340洞察力","user_id":"6252"}
{"actionTimes":"2018-5-20","actions":"下单","bb":"v1.0","fromType":"Mozilla/5.0 (Windows NT 6.1; Win64; x64)","fromURL":"https://tv.qq.com/channel/child?listpage=1&channel=children&itype=3","ip":"181.94.33.139","requestMethod":"POST","sessionId":"79e33d3f-77e3-4120-ae75-c650e94e22f3","title":"关键273洞察力","user_id":"6106"}
{"actionTimes":"2018-7-21","actions":"下单","bb":"v1.0","fromType":"Mozilla/5.0 (Windows NT 6.1; Win64; x64)","fromURL":"https://www.mail.com/int?wd=ip%E5%9C%B0%E5%9D%80&rsv_spt=1","ip":"181.94.33.139","requestMethod":"GET","sessionId":"6bb2b015-36e7-4b7e-98c9-bc83365ea2cc","title":"关键504洞察力","user_id":"6918"}
{"actionTimes":"2018-9-27","actions":"登录","bb":"v1.0","fromType":"Chrome/73.0.3683.75","fromURL":"https://www.phone.com/int?wd=ip%E5%9C%B0%E5%9D%80&rsv_spt=1","ip":"115.44.31.64","requestMethod":"POST","sessionId":"8fff502c-fa29-4ce3-811b-e4eb944ef62e","title":"关键473洞察力","user_id":"3190"}
{"actionTimes":"2018-7-26","actions":"搜索","bb":"v1.0","fromType":"Chrome/73.0.3683.75","fromURL":"https://www.baidu.com/s?wd=ip%E5%9C%B0%E5%9D%80&rsv_spt=1","ip":"207.12.85.193","requestMethod":"POST","sessionId":"0a28e4aa-8d4a-4335-abad-9ff51f08f7fe","title":"关键371洞察力","user_id":"4569"}
{"actionTimes":"2018-5-5","actions":"浏览评论","bb":"v1.0","fromType":"IE/537.36 (KHTML, like Gecko) ","fromURL":"https://www.nyist.com/s?wd=ip%E5%9C%B0%E5%9D%80&rsv_spt=1","ip":"10.7.87.67","requestMethod":"POST","sessionId":"65af3572-d48b-4e49-aaf4-97f322d6ac10","title":"关键117洞察力","user_id":"212"}
{"actionTimes":"2018-0-19","actions":"搜索","bb":"v1.0","fromType":"Mozilla/5.0 (Windows NT 6.1; Win64; x64)","fromURL":"https://www.phone.com/int?wd=ip%E5%9C%B0%E5%9D%80&rsv_spt=1","ip":"181.94.33.139","requestMethod":"POST","sessionId":"d8c6e3fc-d557-4c26-855b-3053ead690d6","title":"关键34洞察力","user_id":"2367"}
{"actionTimes":"2018-5-27","actions":"下单","bb":"v1.0","fromType":"Chrome/73.0.3683.75","fromURL":"https://www.mail.com/int?wd=ip%E5%9C%B0%E5%9D%80&rsv_spt=1","ip":"241.21.27.237","requestMethod":"GET","sessionId":"8214cc26-d22f-4670-9d63-95ce87814c9a","title":"关键266洞察力","user_id":"5698"}
{"actionTimes":"2018-11-0",