<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[weixin_45687351的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/weixin_45687351</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; weixin_45687351]]></copyright><item><title><![CDATA[saprk的udtf函数一列转多列应用]]></title><link>https://blog.youkuaiyun.com/weixin_45687351/article/details/115555768</link><guid>https://blog.youkuaiyun.com/weixin_45687351/article/details/115555768</guid><author>weixin_45687351</author><pubDate>Fri, 09 Apr 2021 17:36:31 +0800</pubDate><description><![CDATA[saprk的udtf函数一列转多列应用
需求：查分outresult字段生成多个列
代码：
1.udtf部分
import com.alibaba.fastjson.JSONException;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.exec.UDFArgumentLeng]]></description><category></category></item><item><title><![CDATA[用户画像-算法实例:朴素贝叶斯算法]]></title><link>https://blog.youkuaiyun.com/weixin_45687351/article/details/104431030</link><guid>https://blog.youkuaiyun.com/weixin_45687351/article/details/104431030</guid><author>weixin_45687351</author><pubDate>Fri, 21 Feb 2020 17:12:24 +0800</pubDate><description><![CDATA[##利用朴素贝叶斯算法来预测明星出轨概率
package cn.doitedu.ml.bayes
import org.apache.spark.ml.linalg.Vectors
import cn.doitedu.commons.util.SparkUtil
import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql....]]></description><category></category></item><item><title><![CDATA[用户画像-算法实例KNN]]></title><link>https://blog.youkuaiyun.com/weixin_45687351/article/details/104400363</link><guid>https://blog.youkuaiyun.com/weixin_45687351/article/details/104400363</guid><author>weixin_45687351</author><pubDate>Wed, 19 Feb 2020 20:56:19 +0800</pubDate><description><![CDATA[import cn.doitedu.commons.util.SparkUtil
import org.apache.spark.ml.linalg
import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}

object Kn...]]></description><category></category></item><item><title><![CDATA[hive中的窗口函数]]></title><link>https://blog.youkuaiyun.com/weixin_45687351/article/details/103881792</link><guid>https://blog.youkuaiyun.com/weixin_45687351/article/details/103881792</guid><author>weixin_45687351</author><pubDate>Wed, 08 Jan 2020 16:38:47 +0800</pubDate><description><![CDATA[函数 + over()
over() :开窗口
1.Row_Number,Rank，Dense_Rank   这三个窗口函数的使用场景非常多
1.1 row_number()
对分区内的数据排序,多出来一列数据,从1到分区结尾
多用来求 连续天数,会用到date_sub函数
例子:
– 求有连续4天销售记录的店铺
1)
select
*,
row_number() over(partition b...]]></description><category></category></item><item><title><![CDATA[flume的核心组件及概念]]></title><link>https://blog.youkuaiyun.com/weixin_45687351/article/details/103825853</link><guid>https://blog.youkuaiyun.com/weixin_45687351/article/details/103825853</guid><author>weixin_45687351</author><pubDate>Fri, 03 Jan 2020 20:33:00 +0800</pubDate><description><![CDATA[1. agent: 核心的角色
2. Event：flume内部数据传输的封装形式
3. Transaction：事务控制机制
4. 拦截器
5. 选择器]]></description><category></category></item><item><title><![CDATA[DataFrame数据运算操作]]></title><link>https://blog.youkuaiyun.com/weixin_45687351/article/details/103812152</link><guid>https://blog.youkuaiyun.com/weixin_45687351/article/details/103812152</guid><author>weixin_45687351</author><pubDate>Thu, 02 Jan 2020 22:33:38 +0800</pubDate><description><![CDATA[一. SQL风格操作
核心要义：将DataFrame 注册为一个临时视图view，然后就可以针对view直接执行各种sql临时视图有两种：session级别视图，global级别视图；
session级别视图是Session范围内有效的，Session退出后，表就失效了；
全局视图则在application级别有效；
注意使用全局表时需要全路径访问：global_temp.people
// ap...]]></description><category></category></item><item><title><![CDATA[Spark SQL中创建 DataFrame 方法]]></title><link>https://blog.youkuaiyun.com/weixin_45687351/article/details/103811097</link><guid>https://blog.youkuaiyun.com/weixin_45687351/article/details/103811097</guid><author>weixin_45687351</author><pubDate>Thu, 02 Jan 2020 21:53:34 +0800</pubDate><description><![CDATA[在Spark SQL中SparkSession是创建DataFrames和执行SQL的入口
创建DataFrames有三种方式：
(1) 从一个已存在的RDD进行转换
(2) 从JSON/Parquet/CSV/ORC/JDBC等结构化数据源直接创建
(3) 从Hive Table进行查询返回
核心:
创建DataFrame，需要创建 “RDD  +  元信息schema定义”
rdd来自于数据
...]]></description><category></category></item></channel></rss>