<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[macfei的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/macfei</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; macfei]]></copyright><item><title><![CDATA[hive函数]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/122471828</link><guid>https://blog.youkuaiyun.com/macfei/article/details/122471828</guid><author>macfei</author><pubDate>Thu, 13 Jan 2022 13:27:55 +0800</pubDate><description><![CDATA[关系运算符
以下运算符比较传递的操作数，并根据操作数之间的比较是否成立来生成TRUE或FALSE值。
操作员
操作数类型
描述
A = B
所有原始类型
如果表达式A等于表达式B，则为TRUE，否则为FALSE。
A == B	所有原始类型	=运算符的同义词。
A &lt;=&gt; B
所有原始类型
对于非空操作数，使用EQUAL（=）运算符返回相同的结果，但如果两个均为NULL，则返回TRUE，如果其中之一为NULL，则返回FALSE。（从0.9.0版开始。）
A &lt;&gt; B
所有原始类型
如]]></description><category></category></item><item><title><![CDATA[hive建表读取TXT文本]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/121698351</link><guid>https://blog.youkuaiyun.com/macfei/article/details/121698351</guid><author>macfei</author><pubDate>Fri, 03 Dec 2021 14:45:26 +0800</pubDate><description><![CDATA[1.建表语句
create table temp.mg_order(
father_id string,
order_id string,
user_id string,
account string,
created_time timestamp,
pay_status string,
pay_time timestamp,
money double,
pay_type string,
mall string,
order_type string,
order_status string
)ROW for]]></description><category></category></item><item><title><![CDATA[看hdfs指定文件夹大小]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/121540718</link><guid>https://blog.youkuaiyun.com/macfei/article/details/121540718</guid><author>macfei</author><pubDate>Thu, 25 Nov 2021 16:32:25 +0800</pubDate><description><![CDATA[看hdfs指定文件夹大小
hdfs dfs -du -s -h path

第一列是实际文件夹大小,第二列是备份大小


]]></description><category></category></item><item><title><![CDATA[hive中上个月第一天,最后一天]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/119803412</link><guid>https://blog.youkuaiyun.com/macfei/article/details/119803412</guid><author>macfei</author><pubDate>Thu, 19 Aug 2021 15:40:48 +0800</pubDate><description><![CDATA[每个月的第一天
trunc()

查看该函数的详细用法
desc function extended trunc;

上个月第一天
select trunc(add_months(current_date(),-1),'MM'); 

结果:
2021-07-01

每年第一天
select trunc(current_date(),'YY');

结果:
2021-01-01

每个月的最后一天
last_day()

上个月的最后一天
select  date_format(last_day(add_m]]></description><category></category></item><item><title><![CDATA[hive常用命令]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/118405634</link><guid>https://blog.youkuaiyun.com/macfei/article/details/118405634</guid><author>macfei</author><pubDate>Fri, 02 Jul 2021 10:48:24 +0800</pubDate><description><![CDATA[1.内部表转外部表
alter table temp.app_user_stage SET TBLPROPERTIES('EXTERNAL'='true');

2.外部表转内部表
alter table temp.app_user_stage SET TBLPROPERTIES('EXTERNAL'='false');

3.增加表分区
alter table temp.app_user_stage add partition(pt=20210701);

alter table dtwarehouse.]]></description><category></category></item><item><title><![CDATA[select * 有数据,select count(*) 没有数据]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/118405067</link><guid>https://blog.youkuaiyun.com/macfei/article/details/118405067</guid><author>macfei</author><pubDate>Fri, 02 Jul 2021 09:21:35 +0800</pubDate><description><![CDATA[analyze table dtwarehouse.app_user_stage_temp partition(pt=20210701) compute statistics;


]]></description><category></category></item><item><title><![CDATA[hue添加udf函数方法]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/115396571</link><guid>https://blog.youkuaiyun.com/macfei/article/details/115396571</guid><author>macfei</author><pubDate>Fri, 02 Apr 2021 13:46:09 +0800</pubDate><description><![CDATA[1.hue界面有一个设置的图标,
2.点开,会出现三个加号
3.files :上传jar包在hdfs上面的路径
functions:d第一个空格是自定义函数的名称,第二个空格是jar的全路径
4.使用自定义的函数

]]></description><category></category></item><item><title><![CDATA[sftp拉取数据到hive中]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/114885589</link><guid>https://blog.youkuaiyun.com/macfei/article/details/114885589</guid><author>macfei</author><pubDate>Tue, 16 Mar 2021 16:04:13 +0800</pubDate><description><![CDATA[sftp拉取数据到hive中
#sftp服务器地址
host=10.129.11.50
#端口
port=8888
sftp_user=user
#密码
password=passw0rd


#下载到本地的目录
localDir=/home/hive/nps/nps_dealer/${pt}/
[ ! -d $localDir ] &amp;&amp; mkdir -p $localDir
#sftp中待下载文件目录
remoteDir=/home/appuser/report/R

#要下载的文件为fi]]></description><category></category></item><item><title><![CDATA[hive时间格式的转换]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/114412339</link><guid>https://blog.youkuaiyun.com/macfei/article/details/114412339</guid><author>macfei</author><pubDate>Fri, 05 Mar 2021 21:21:47 +0800</pubDate><description><![CDATA[把2.02102261356020(double类型)转为2021-02-26 13:56:02
1.先截取前14位
select substr(2.02102261356020,0,15)


2.去除小数点
select regexp_replace(substr(2.02102261356020,0,15),'\\.','')


3.
select unix_timestamp( regexp_replace(substr(2.02102261356020,0,15),'\\.',''),'yyyy]]></description><category></category></item><item><title><![CDATA[HDFS安全模式]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/114020906</link><guid>https://blog.youkuaiyun.com/macfei/article/details/114020906</guid><author>macfei</author><pubDate>Wed, 24 Feb 2021 14:08:57 +0800</pubDate><description><![CDATA[主要就是4个命令语句
hdfs dfsadmin

hdfs dfsadmin -safemode

hdfs dfsadmin -safemode forceexit

hdfs dfsadmin -safemode leave


]]></description><category></category></item><item><title><![CDATA[spark脚本因为特殊符号报错]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/111882889</link><guid>https://blog.youkuaiyun.com/macfei/article/details/111882889</guid><author>macfei</author><pubDate>Tue, 29 Dec 2020 11:13:18 +0800</pubDate><description><![CDATA[这段SQL在hue里面测试成功,但是在spark脚本里面执行报错,
最终定位的问题是出现了一个中文的全角空格,把全角空格全部替换成空格就没有问题了.


![在这里插入图片描述](https://img-blog.csdnimg.cn/20201229104729405.png

...]]></description><category></category></item><item><title><![CDATA[Linux常用命令]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/111590053</link><guid>https://blog.youkuaiyun.com/macfei/article/details/111590053</guid><author>macfei</author><pubDate>Wed, 23 Dec 2020 14:57:52 +0800</pubDate><description><![CDATA[保存退出
shift+zz  /   :wq!

不保存退出
:q!

剪切
dd

粘贴
p

撤销
u

查看实时日志
tail -f  daily-mcf-2020-12-23.log

/string
搜索字符串 （字符串输入完之后，按enter键，然后按n查看下一个字符串的位置）
到最后一行
shift+g  /  G

第一行
gg

显示行号/隐藏行号
:set nu /  :set nonu

跳转到88行(先把行号显示出来使用较好)
88gg / 88G


...]]></description><category></category></item><item><title><![CDATA[cast ,case when的用法]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/110916626</link><guid>https://blog.youkuaiyun.com/macfei/article/details/110916626</guid><author>macfei</author><pubDate>Wed, 09 Dec 2020 11:17:27 +0800</pubDate><description><![CDATA[1.case when的用法

需求:将brand_type对应的数字翻译成品牌
select
     case
         when brand_type=1 then '荣威'
         when brand_type=2 then '名爵'
         when brand_type=4 then 'R标'
         else brand_type 
      end            as       attr1  
 from tb

2.手机号加密

//第一]]></description><category></category></item><item><title><![CDATA[hive登录设置]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/109802988</link><guid>https://blog.youkuaiyun.com/macfei/article/details/109802988</guid><author>macfei</author><pubDate>Thu, 19 Nov 2020 10:19:23 +0800</pubDate><description><![CDATA[hiveserver2

--启动hive的元数据服务
hive --service metastore
beeline -u jdbc:hive2://localhost:10000 -n root


--hive本地运行
set hive.exec.mode.local.auto=true;
--或者
set mapreduce.framework.name=local;

--查看建表语句
show create table 表名


...]]></description><category></category></item><item><title><![CDATA[coalesce的用法（将join后为null的值设置一个默认值）]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/109802817</link><guid>https://blog.youkuaiyun.com/macfei/article/details/109802817</guid><author>macfei</author><pubDate>Thu, 19 Nov 2020 10:14:55 +0800</pubDate><description><![CDATA[需求:C表的uid和B表的id进行关联,如果关联上则取值为’是’,否则为’否’
c.txt
1,北京
2,上海
3,合肥

b.txt
2,嘉定
3,六安
4,昌平

--建表语句
create table test.tb_c (
uid int ,
addr string 
)
row format delimited fields terminated by ',' ;
load data local inpath '/doit17/c.txt' into table tb_c ;


create t]]></description><category></category></item><item><title><![CDATA[在Linux中修改MySQL的密码]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/108968807</link><guid>https://blog.youkuaiyun.com/macfei/article/details/108968807</guid><author>macfei</author><pubDate>Thu, 08 Oct 2020 21:26:39 +0800</pubDate><description><![CDATA[1
先使用自己的密码账号登录MySQL
2
一些修改密码的命令会因为设置的密码简单而报错，输入这两个命令则不会出现这个问题。
mysql&gt; set global validate_password_policy=0;
mysql&gt; set global validate_password_length=1; 
  这个两个设置以后 密码很简单不会报错

3
设置自己的新密码
ALTER USER 'root'@'localhost' IDENTIFIED BY 'root';

4
quit退出]]></description><category></category></item><item><title><![CDATA[Spark广播变量]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/108813043</link><guid>https://blog.youkuaiyun.com/macfei/article/details/108813043</guid><author>macfei</author><pubDate>Sat, 26 Sep 2020 16:52:41 +0800</pubDate><description><![CDATA[广播的目的
广播变量是为了实现mapside join ,可以将Driver端的数据广播到属于该application的Executor，然后通过Driver广播变量返回的引用，获取事先广播到Executor的数据。
mapside join
当连接的两个表是一个比较大的表和一个比较小的时候，我们把比较小的table直接放到内存中去，然后再去对比较大的表格进行map操作。join就发生在map操作的时候，每当扫描一个大的table中的数据，就要去查看小表的数据，哪条与之相符，继而进行连接，这里的join并不]]></description><category></category></item><item><title><![CDATA[IDEA注释的设置]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/108292200</link><guid>https://blog.youkuaiyun.com/macfei/article/details/108292200</guid><author>macfei</author><pubDate>Sat, 29 Aug 2020 11:28:23 +0800</pubDate><description><![CDATA[File—&gt;Settings—&gt;Editor—&gt;Code Style—&gt;File and Code Templates




]]></description><category></category></item><item><title><![CDATA[Master is initializing 问题]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/108267999</link><guid>https://blog.youkuaiyun.com/macfei/article/details/108267999</guid><author>macfei</author><pubDate>Thu, 27 Aug 2020 20:27:21 +0800</pubDate><description><![CDATA[Master  is  initializing  问题
进入Hbase界面的时候,可以list看罗列出来的表,
但是create_namespace的时候,会出现卡在那里不动,
直到最后跳出连接超时或者

然后打算关闭Hbase  也会一直加载

最后的解决办法是:
**
1.先把hdfs上根目录下的hbase文件夹给删了**
先给hbase文件夹权限
hdfs dfs -chmod -R 777 /hbase

删除文件夹
hdfs dfs -rm -R /hbase

2.把zookeeper里面的h]]></description><category></category></item><item><title><![CDATA[去除Hive里面的info信息]]></title><link>https://blog.youkuaiyun.com/macfei/article/details/108241712</link><guid>https://blog.youkuaiyun.com/macfei/article/details/108241712</guid><author>macfei</author><pubDate>Wed, 26 Aug 2020 16:05:54 +0800</pubDate><description><![CDATA[关闭Hive里面info信息的两种方法
1.登录beeline的时候,输入如下语句
beeline--hiveconf hive.server2.logging.operation.level=NONE

2.在hive-site.xml 文件中加上如下语句
路径
/opt/apps/hive-3.1.2/conf


&lt;property&gt;
        &lt;name&gt;hive.server2.logging.operation.enabled&lt;/name&gt;
      ]]></description><category></category></item></channel></rss>