<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[MAYIBO—BG的专栏]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/CPP_MAYIBO</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; CPP_MAYIBO]]></copyright><item><title><![CDATA[Hive源码分析——Alter Table操作执行流程分析]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/116278872</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/116278872</guid><author>CPP_MAYIBO</author><pubDate>Thu, 29 Apr 2021 23:40:19 +0800</pubDate><description><![CDATA[目录概述过程分析参数介绍初始化类型兼容性检查修改表名操作其他修改操作异常处理总结
概述
Alter Table是非常常见的一种DDL操作，在Hive中，使用Alter Table可以修改表名，修改表属性，添加字段，修改字段数据类型等等。需要注意的是，ALTER TABLE修改的只是MySQL中的相关元数据，不会对表数据产生影响，用户需要手动确保实际数据符合元数据的定义，否则查询不会返回预期的结果。但在分区数比较多的情况下，修改表名这类逻辑上非常简单的操作时间开销却很大。因此对Alter Table语句的具体]]></description><category></category></item><item><title><![CDATA[Hive优化—skew join优化原理详解]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/111350138</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/111350138</guid><author>CPP_MAYIBO</author><pubDate>Thu, 17 Dec 2020 23:50:58 +0800</pubDate><description><![CDATA[目录优化原理适用范围测试验证
优化原理
JOIN中倾斜键的处理思路最早是在HIVE-964中提出的，整体思路是使用独立的作业和mapjoin来处理倾斜的键。
用以处理倾斜键的MR作业数是表的数量减一(we can stream the last table, so big keys in the last table will not be a problem)
在执行JOIN的过程中，会将一个表中的大key（也就是倾斜的那部分数据，判断是否倾斜由配置项hive.skewjoin.key指定，默认是1000]]></description><category></category></item><item><title><![CDATA[YARN 作业 Staging目录的用途及配置]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/111349157</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/111349157</guid><author>CPP_MAYIBO</author><pubDate>Thu, 17 Dec 2020 23:10:44 +0800</pubDate><description><![CDATA[目录staging目录的用途MapReduce作业staging的配置示例Spark 作业Staging目录的配置示例
staging目录的用途
关于staging目录可能很多人都不太会关注，毕竟日常运行作业也用不到这些配置。不过了解它对于我们理解作业的执行流程也是有所帮助的，比如我们都会使用hadoop jar 或 spark-submit等命令来提交一个MR或Spark作业，然后我们就会看到在集群的某些计算节点上启动executor(MapRedece对应的是mapper和reducer)来执行任务。这]]></description><category></category></item><item><title><![CDATA[HiveQL DQL9—窗口函数]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104202719</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104202719</guid><author>CPP_MAYIBO</author><pubDate>Thu, 06 Feb 2020 22:28:52 +0800</pubDate><description><![CDATA[目录概述窗口聚合函数窗口排序函数窗口分析函数窗口表达式Row Typerange type参考
概述
Hive从0.11.0开始支持窗口函数，这是一组特殊的函数，可以扫描多个输入行来计算每个输出值。窗口函数通常和OVER, PARTITION BY, ORDER BY 以及 windowing specification一起使用。不同于和GROUP BY一起使用的常规聚合函数（每组只有一个结果值）...]]></description><category></category></item><item><title><![CDATA[HiveQL DQL8—数据采样]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104176277</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104176277</guid><author>CPP_MAYIBO</author><pubDate>Tue, 04 Feb 2020 22:35:36 +0800</pubDate><description><![CDATA[目录概述随机采样桶表采样示例：based on whole row示例：based on bucket column块采样示例：按行采样示例：按数据大小的百分比采样示例：按数据大小采样参考
概述
当数据量特别大时，我们可能需要获取一些数据的子集来加速数据分析。这就是采样，一种用于识别和分析数据子集以发现整个数据集中的模式和趋势的技术。在HQL中，采样数据有三种方式：随机采样、桶表采样和块采样。
随...]]></description><category></category></item><item><title><![CDATA[HiveQL DQL7—高级聚合]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104175674</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104175674</guid><author>CPP_MAYIBO</author><pubDate>Tue, 04 Feb 2020 22:27:58 +0800</pubDate><description><![CDATA[目录Grouping sets示例：使用一个字段对元素的GROUPING SETS示例：使用2个元素的GROUPING SETS示例：使用2个元素的GROUPING SETS，一个字段对，一个是单独字段示例：使用4个元素的GROUPING SETSRollup 和 Cube聚合条件过滤 HAVING参考
通过使用 GROUPING SETS, CUBE, 和 ROLLUP，Hive提供了高级聚合功...]]></description><category></category></item><item><title><![CDATA[HiveQL DQL6—基本聚合(GROUP BY)]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104084901</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104084901</guid><author>CPP_MAYIBO</author><pubDate>Sun, 26 Jan 2020 23:18:19 +0800</pubDate><description><![CDATA[目录概述使用示例示例：不使用GROUP BY的聚合示例：使用GROUP BY的聚合示例：同一个SELECT中使用多个聚合函数示例：与CASE WHEN THEN ELSE END,coalesce(...),  if(...)一起使用聚合函数示例：与coalesce(...),  if(...)一起使用聚合函数示例：GROUP BY子句中使用表达式示例：不允许聚合函数的嵌套示例：NULL值的处理示...]]></description><category></category></item><item><title><![CDATA[HiveQL DQL3—JOIN]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104079552</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104079552</guid><author>CPP_MAYIBO</author><pubDate>Fri, 24 Jan 2020 17:27:11 +0800</pubDate><description><![CDATA[目录概述隐式join表示法（Implicit join notation）非限定列引用（Unqualified column reference）ON子句支持复杂表达式（Complex expressions in ON clause）INNER JOINOUTER JOIN特殊的 joinMapJoinBucket MapJoinLEFT SEMI JOIN参考
概述
JOIN用于将两个或多个表...]]></description><category></category></item><item><title><![CDATA[HiveQL DQL4—UNION]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104078463</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104078463</guid><author>CPP_MAYIBO</author><pubDate>Thu, 23 Jan 2020 23:01:46 +0800</pubDate><description><![CDATA[目录概述示例UNION ALLUNIONOrder with UNION使用 join 实现 intercept使用 left join 实现 minus参考
概述
  当我们想将同一schema的数据组合在一起时，会经常使用set 操作。关系数据库中的常规set操作是INTERSECT、MINUS和UNION/UNION ALL。HQL只支持UNION 和 UNION ALL，二者的区别是UNI...]]></description><category></category></item><item><title><![CDATA[HiveQL DQL1—SELECT]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104078252</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104078252</guid><author>CPP_MAYIBO</author><pubDate>Thu, 23 Jan 2020 22:30:13 +0800</pubDate><description><![CDATA[目录语法介绍ALL 和 DISTINCT子句HAVING子句使用 IF 或 CASE WHEN 函数基于分区的查询基于正则表达式的列描述条件查询LIMIT 子句WHERE子句IN/NOT IN语句EXISTS/NOT EXISTS语句参考
语法介绍
[WITH CommonTableExpression (, CommonTableExpression)*]    (Note: Only avai...]]></description><category></category></item><item><title><![CDATA[HiveQL函数10—表生成函数（UDTF）]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104066064</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104066064</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 21:47:06 +0800</pubDate><description><![CDATA[目录1.explode(ARRAY&lt;T&gt; a)2.explode(MAP&lt;Tkey,Tvalue&gt; m)3.posexplode(ARRAY&lt;T&gt; a)4.inline(ARRAY&lt;STRUCT&lt;f1:T1,...,fn:Tn&gt;&gt; a)5.stack(int r,T1 V1,...,Tn/r Vn)6.json_tuple(string ...]]></description><category></category></item><item><title><![CDATA[HiveQL函数9—聚合函数（UDAF）]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065989</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065989</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 21:43:12 +0800</pubDate><description><![CDATA[目录1.count(*), count(expr), count(DISTINCT expr[, expr...])2.sum(col), sum(DISTINCT col)3.avg(col), avg(DISTINCT col)4.min(col)5.max(col)6.variance(col), var_pop(col)7.var_samp(col)8.stddev_pop(col)9.s...]]></description><category></category></item><item><title><![CDATA[HiveQL函数8—其他杂项函数]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065897</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065897</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 21:30:48 +0800</pubDate><description><![CDATA[目录1.java_method(class, method[, arg1[, arg2..]]) , reflect(class, method[, arg1[, arg2..]])2.hash(a1[, a2...])3.current_user()4.logged_in_user()5.current_database()6.md5(string/binary)7.sha1(string/bi...]]></description><category></category></item><item><title><![CDATA[HiveQL函数7—数据脱敏函数（Data Masking）]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065839</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065839</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 21:18:10 +0800</pubDate><description><![CDATA[目录1.mask(string str[, string upper[, string lower[, string number]]])2.mask_first_n(string str[, int n])3.mask_last_n(string str[, int n])4.mask_show_first_n(string str[, int n])5.mask_show_last_n(str...]]></description><category></category></item><item><title><![CDATA[HiveQL函数6—字符串函数]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065701</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065701</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 21:07:54 +0800</pubDate><description><![CDATA[目录1. ascii(string str)2.base64(binary bin)3.character_length(string str)4.chr(bigint|double A)5.concat(string|binary A, string|binary B...)6.context_ngrams(array&lt;array&lt;string&gt;&gt;, array&lt;s...]]></description><category></category></item><item><title><![CDATA[HiveQL函数5—条件函数]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065448</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065448</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 20:50:15 +0800</pubDate><description><![CDATA[目录1.if(boolean testCondition, T valueTrue, T valueFalseOrNull)2.isnull( a )3.isnotnull ( a )4.nvl(T value, T default_value)5.COALESCE(T v1, T v2, ...)6.CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] E...]]></description><category></category></item><item><title><![CDATA[HiveQL函数4—日期函数]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065247</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065247</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 20:19:45 +0800</pubDate><description><![CDATA[目录1.from_unixtime(bigint unixtime[, string format])2.unix_timestamp()3.unix_timestamp(string date)4.unix_timestamp(string date, string pattern)5.to_date(string timestamp)6.year(string date)7.quarter(d...]]></description><category></category></item><item><title><![CDATA[HiveQL函数3—集合、类型转换函数]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065151</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065151</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 19:52:52 +0800</pubDate><description><![CDATA[目录集合函数示例1示例2类型转换函数1. binary(string|binary)2. cast(expr as \)参考
集合函数
HiveQL支持的集合函数如下




函数名
返回类型
描述




size(Map&lt;K.V&gt;)
int
计算map的元素个数


size(Array)
int
计算数组的元素个数


map_keys(Map&lt;K.V&gt;)
array...]]></description><category></category></item><item><title><![CDATA[HiveQL函数2—数学函数]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065057</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104065057</guid><author>CPP_MAYIBO</author><pubDate>Tue, 21 Jan 2020 19:39:37 +0800</pubDate><description><![CDATA[目录1.round(DOUBLE a)2.round(DOUBLE a, INT d)3.bround(DOUBLE a)4.bround(DOUBLE a, INT d)5.floor(DOUBLE a)6.ceil(DOUBLE a), ceiling(DOUBLE a)7. rand(), rand(INT seed)8.exp(DOUBLE a), exp(DECIMAL a)9.ln(D...]]></description><category></category></item><item><title><![CDATA[HiveQL函数1—函数简介]]></title><link>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104057302</link><guid>https://blog.youkuaiyun.com/CPP_MAYIBO/article/details/104057302</guid><author>CPP_MAYIBO</author><pubDate>Mon, 20 Jan 2020 23:19:09 +0800</pubDate><description><![CDATA[目录简介参考
简介
HiveQL 内置了很多的函数。详细的列表可以参考 https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF ，本文介绍HiveQL的函数分类及如何查看函数的详细信息。
Hive中的函数分为如下几类：

数学函数
集合函数
类型转换函数
日期函数
条件函数
字符函数
数据掩码函数
Misc 函数
聚合...]]></description><category></category></item></channel></rss>