<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[qq_42422698的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/qq_42422698</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; qq_42422698]]></copyright><item><title><![CDATA[一个基于用户行为日志的资源推荐系统落地]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/145003631</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/145003631</guid><author>qq_42422698</author><pubDate>Wed, 08 Jan 2025 14:04:17 +0800</pubDate><description><![CDATA[一个基于用户行为日志的资源推荐系统]]></description><category></category></item><item><title><![CDATA[关于flinkOnYarn自动获取检查点并启动的探索]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/144399458</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/144399458</guid><author>qq_42422698</author><pubDate>Wed, 11 Dec 2024 14:52:37 +0800</pubDate><description><![CDATA[实现流任务自动执行]]></description><category></category></item><item><title><![CDATA[FlinkOnYarn-log4j打印问题]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/144271062</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/144271062</guid><author>qq_42422698</author><pubDate>Thu, 05 Dec 2024 17:12:19 +0800</pubDate><description><![CDATA[于是我把maven-pom里的log4j依赖都注释。]]></description><category></category></item><item><title><![CDATA[FlinkSql的Join]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/139526137</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/139526137</guid><author>qq_42422698</author><pubDate>Fri, 07 Jun 2024 14:46:01 +0800</pubDate><description><![CDATA[Temporal Join ]]></description><category></category></item><item><title><![CDATA[flinkCDC和doris实现即时查询]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/139477108</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/139477108</guid><author>qq_42422698</author><pubDate>Wed, 05 Jun 2024 17:22:00 +0800</pubDate><description><![CDATA[doris]]></description><category></category></item><item><title><![CDATA[flink数据仓库之考试模型]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/139476213</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/139476213</guid><author>qq_42422698</author><pubDate>Wed, 05 Jun 2024 16:42:09 +0800</pubDate><description><![CDATA[【代码】flink数据仓库之考试模型。]]></description><category></category></item><item><title><![CDATA[hive数据仓库之考试模型]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/139475343</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/139475343</guid><author>qq_42422698</author><pubDate>Wed, 05 Jun 2024 16:24:01 +0800</pubDate><description><![CDATA[hive数据仓库之考试模型。]]></description><category></category></item><item><title><![CDATA[DBever读取Elasticsearch]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/134649192</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/134649192</guid><author>qq_42422698</author><pubDate>Mon, 27 Nov 2023 16:56:58 +0800</pubDate><description><![CDATA[ES DBever]]></description><category></category></item><item><title><![CDATA[flink-OnYarn部署中关于zookeeper的匹配问题]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/132625544</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/132625544</guid><author>qq_42422698</author><pubDate>Fri, 01 Sep 2023 17:07:32 +0800</pubDate><description><![CDATA[flink-shaded-zookeeper-3.4.14.jar、flink-shaded-zookeeper-3.5.9.jar和flink-shaded-zookeeper-3.6.3.jar在flink1.14.4以及flink15+ onYarn 的适配问题]]></description><category></category></item><item><title><![CDATA[flinkSQL解析kafka数据源格式]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/130886998</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/130886998</guid><author>qq_42422698</author><pubDate>Fri, 26 May 2023 15:17:08 +0800</pubDate><description><![CDATA[flinksql   kafka   json]]></description><category></category></item><item><title><![CDATA[Doris 1.2.1-1升级1.2.4-1]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/130511802</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/130511802</guid><author>qq_42422698</author><pubDate>Fri, 05 May 2023 17:38:20 +0800</pubDate><description><![CDATA[doris升级]]></description><category></category></item><item><title><![CDATA[Doris服务正常-但是执行查询出现time-out]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/130511316</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/130511316</guid><author>qq_42422698</author><pubDate>Fri, 05 May 2023 17:29:48 +0800</pubDate><description><![CDATA[doris 1.2.1 报错]]></description><category></category></item><item><title><![CDATA[Doris+Flink搭建数据平台]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/129425595</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/129425595</guid><author>qq_42422698</author><pubDate>Fri, 10 Mar 2023 11:14:15 +0800</pubDate><description><![CDATA[Doris\flink\shell\python\canal\sql]]></description><category></category></item><item><title><![CDATA[Kafka 常用命令2]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/121339646</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/121339646</guid><author>qq_42422698</author><pubDate>Mon, 15 Nov 2021 18:01:57 +0800</pubDate><description><![CDATA[1、查看当前的集群Topic列表
./bin/kafka-topics.sh --list --zookeeper cdh-worker-1:2181/kafka

2、查看所有的Topic的详细信息
./bin/kafka-topics.sh --describe --zookeeper cdh-worker-1:2181/kafka
#如果要查看单个 topic 信息：可在上述命令后面添加 --topic &lt;topicName&gt;

3、创建Topic
./bin/kafka-topics.s]]></description><category></category></item><item><title><![CDATA[azkaban-任务执行成功，但是发送预警报错邮件]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/121203813</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/121203813</guid><author>qq_42422698</author><pubDate>Mon, 08 Nov 2021 11:07:08 +0800</pubDate><description><![CDATA[azkaban-任务执行成功，但是发送预警报错邮件
症状:邮件发送 status:PREPARING

任务执行情况：status=succees
解决办法：
azkaban节点 时钟同步

脚本内容：
[root@node116 ~]# cat ntp6_aliyun.sh 
#!/bin/bash
ntpdate ntp6.aliyun.com


...]]></description><category></category></item><item><title><![CDATA[pyspark到mysql]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/120530629</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/120530629</guid><author>qq_42422698</author><pubDate>Tue, 28 Sep 2021 15:58:48 +0800</pubDate><description><![CDATA[1 先上整体代码
import sys
sys.path.append('/data/gtair/tools/DWH')
import pandas as pd
from sqlalchemy import create_engine
engine = create_engine('mysql+pymysql://edu_ws:QbvkrBHKXO@chb2gc-pd-edu-dws.mysql.rds.aliyuncs.com:3306/edu_platform')

df_spark =ss.sql("]]></description><category></category></item><item><title><![CDATA[ambari+hdp下mysql元数据用户ambari、hive、root、azkaban用户密码修改]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/120162191</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/120162191</guid><author>qq_42422698</author><pubDate>Tue, 07 Sep 2021 17:14:49 +0800</pubDate><description><![CDATA[写在前面
ambari+hdp 安装集群的时候密码修改太简单，出于安全考虑，做各用户的密码修改。
1 mysql 用户管理表字段
CREATE TABLE `user` (
  `Host` char(60) COLLATE utf8_bin NOT NULL DEFAULT '',
  `User` char(32) COLLATE utf8_bin NOT NULL DEFAULT '',
  `Select_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL]]></description><category></category></item><item><title><![CDATA[hive函数之collect_set,collect_list,concat_ws,array_contains 联合使用]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/119999939</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/119999939</guid><author>qq_42422698</author><pubDate>Mon, 30 Aug 2021 16:37:33 +0800</pubDate><description><![CDATA[hive函数之collect_set/collect_list,concat_ws,array_contains 联合使用
1 表数据
select id,act_status from ods.ods_ta_task limit 6;

+----------------------+-------------+
|          id          | act_status  |
+----------------------+-------------+
| 34999455991526037]]></description><category></category></item><item><title><![CDATA[OLAP场景下大数据领域技术发展趋势]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/119931081</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/119931081</guid><author>qq_42422698</author><pubDate>Thu, 26 Aug 2021 14:48:30 +0800</pubDate><description><![CDATA[OLAP场景下大数据领域技术发展趋势
1 OLAP场景，共同特点：

绝大多数是读请求
数据批次更新，而不是单行更新
已添加到数据库的数据不能修改
对于读取，从数据库中提取相当多的行，但只提取列的一小部分，列存储
宽表，即每个表包含着大量的列
查询相对较少(通常每台服务器每秒查询数百次或更少)
事务不是必须的，对数据一致性要求低
查询结果明显小于源数据。数据经过聚合运算后可放入单个服务器的RAM中

2  发展趋势
step1
hive  hbase hdfs yarn hadoop zookeeper  ]]></description><category></category></item><item><title><![CDATA[hive合并小文件]]></title><link>https://blog.youkuaiyun.com/qq_42422698/article/details/119930694</link><guid>https://blog.youkuaiyun.com/qq_42422698/article/details/119930694</guid><author>qq_42422698</author><pubDate>Thu, 26 Aug 2021 14:30:52 +0800</pubDate><description><![CDATA[hive合并小文件

创建备用表

 create table new_table like old_table;


设置合并参数，再将原表数据插入到备用表

SET hive.merge.mapfiles = true;
SET hive.merge.mapredfiles = true;
SET hive.merge.size.per.task = 256000000;
SET hive.merge.smallfiles.avgsize = 134217728;
SET hive.exec.compr]]></description><category></category></item></channel></rss>