<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[github_28583061的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/github_28583061</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; github_28583061]]></copyright><item><title><![CDATA[数据中台实战课笔记]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/113698166</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/113698166</guid><author>github_28583061</author><pubDate>Sat, 23 Jan 2021 05:04:50 +0800</pubDate><description><![CDATA[数据中台元数据
元数据包括

数据字典 -- 描述数据的结构信息

表名，注释信息，表的产出任务，有哪些字段， 字段含义，字段类型


数据血缘 -- 表的加工关系，由哪些表加工而来

影响分析和故障溯源


数据特征 -- 数据的属性信息

存储空间大小，访问热度，主题域，分层，表关联的指标



业界元数据产品

开源的有 Netflix 的 Metacat、Apache Atlas；

me...]]></description><category></category></item><item><title><![CDATA[IDEA]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/106384859</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/106384859</guid><author>github_28583061</author><pubDate>Sat, 10 Oct 2020 00:25:27 +0800</pubDate><description><![CDATA[插件


translation


高亮提示 HighlightBracketPair


alibaba  java coding guidelines


lombok


maven helper


mybaitsX ， MyBatis Log Plugin ，


Bytecode Viewer


IdeaJad


其他

https://mp.weixin.qq.com/s?__...]]></description><category></category></item><item><title><![CDATA[structured-streaming-in-apache-spark]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295999</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295999</guid><author>github_28583061</author><pubDate>Tue, 28 Jul 2020 00:33:29 +0800</pubDate><description><![CDATA[Spark Structured Streaming
A new high-level API for streaming
https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html
]]></description><category></category></item><item><title><![CDATA[The_Internals_of_Apache_spark_3.0.0]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295983</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295983</guid><author>github_28583061</author><pubDate>Tue, 28 Jul 2020 00:33:29 +0800</pubDate><description><![CDATA[The Internals Of Apache Spark 3.0.0 笔记 --TODO
原文（https://books.japila.pl/apache-spark-internals/apache-spark-internals/index.html）
Resilient  Distributed Dataset

弹性分布式数据集，spark core模块对于数据集的核心抽象
RDD 就...]]></description><category></category></item><item><title><![CDATA[3_how_to_delete_update_merge_work]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295938</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295938</guid><author>github_28583061</author><pubDate>Tue, 28 Jul 2020 00:33:29 +0800</pubDate><description><![CDATA[Tech Talk | Diving into Delta Lake Part 3: How do DELETE, UPDATE, and MERGE work

https://www.youtube.com/watch?v=7ewmcdrylsA

update 更新操作


SQL :   UPDATE  table SET x=y  WHERE  predicate



底层操作进行了2...]]></description><category></category></item><item><title><![CDATA[Faster_SQL_Queries_on_Delta_Lake_with_Dynamic_File_Pruning]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295499</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295499</guid><author>github_28583061</author><pubDate>Tue, 28 Jul 2020 00:33:29 +0800</pubDate><description><![CDATA[通过动态文件剪裁实现在delta late上的快速查询


原文


两种传统的数据查询加速的方案：

以更快的速度处理数据
通过跳过无关的数据，减少数据量

Dynamic File Pruning (DFP) ： 动态文件裁剪是一种新的数据跳过（data-skipping, 直接跳过一些不相关的数据分区文件）技术

对于在delta lake 表上选择 非 分区列join的时候可以显著提高查询...]]></description><category></category></item><item><title><![CDATA[Apache_Spark_生产使用最佳实践]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295488</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295488</guid><author>github_28583061</author><pubDate>Tue, 28 Jul 2020 00:33:29 +0800</pubDate><description><![CDATA[Apache Spark 生产使用最佳实践

视频地址
git地址：https://github.com/montevideolabs/attractions-recommender

]]></description><category></category></item><item><title><![CDATA[SparkSQL自适应执行优化引擎]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/106385527</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/106385527</guid><author>github_28583061</author><pubDate>Tue, 28 Jul 2020 00:33:29 +0800</pubDate><description><![CDATA[Spark SQL 自适应执行优化引擎
背景

Adaptive Execution 将可以根据执行过程中的中间数据优化后续执行，从而提高整体执行效率。核心在于两点

执行计划可动态调整
调整的依据是中间结果的精确统计信息


spark 2.3 开始试验功能
spark 3.0 正式发布 自适应查询执行（Adaptive Query Execution）

现状与挑战

如何设置合适的shuff...]]></description><category></category></item><item><title><![CDATA[redis]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/106385240</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/106385240</guid><author>github_28583061</author><pubDate>Tue, 28 Jul 2020 00:33:29 +0800</pubDate><description><![CDATA[redis 和 memcached 有啥区别？

redis 支持复杂的数据结构

redis 相比 memcached 来说，拥有更多的数据结构，能支持更丰富的数据操作。如果需要缓存能够支持更复杂的结构和操作， redis 会是不错的选择。


redis 原生支持集群模式

memcached 没有原生的集群模式，需要依靠客户端来实现往集群中分片写入数据。


性能对比

redis 只使用单...]]></description><category></category></item><item><title><![CDATA[1_data_reliablility_for_data_lakes]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295948</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295948</guid><author>github_28583061</author><pubDate>Wed, 08 Jul 2020 06:58:59 +0800</pubDate><description><![CDATA[Delta Lake: Reliability and Data Quality for Data Lakes and Apache Spark by Michael Armbrust

视频连接

https://www.youtube.com/watch?v=GGkRwVHq-Zc
https://www.youtube.com/watch?v=7y0AAQ6qX5w&amp;list=PLT...]]></description><category></category></item><item><title><![CDATA[2_Enforcing_and_Evolving_the_Schema]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295944</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295944</guid><author>github_28583061</author><pubDate>Wed, 08 Jul 2020 06:58:59 +0800</pubDate><description><![CDATA[Tech Talk | Diving into Delta Lake Part 2: Enforcing and Evolving the Schema

https://www.youtube.com/watch?v=tjb10n5wVs8&amp;list=PLTPXxbhUt-YVPwG3OWNQ-1bJI_s_YRvqP&amp;index=16&amp;t=0s

主题内容

数据，和我...]]></description><category></category></item><item><title><![CDATA[Transaction_Log]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295461</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295461</guid><author>github_28583061</author><pubDate>Wed, 08 Jul 2020 06:58:59 +0800</pubDate><description><![CDATA[参考阅读

https://www.iteblog.com/archives/2584.html
https://www.iteblog.com/archives/2591.html
https://www.youtube.com/watch?v=F91G4RoA8is&amp;list=PLTPXxbhUt-YVPwG3OWNQ-1bJI_s_YRvqP&amp;index=15&amp;t=2...]]></description><category></category></item><item><title><![CDATA[5-sparksql-join]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/106385411</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/106385411</guid><author>github_28583061</author><pubDate>Wed, 08 Jul 2020 06:58:59 +0800</pubDate><description><![CDATA[join 操作
join 笛卡尔积
代码流程
spark.conf.set(&#34;spark.sql.autoBroadcastJoinThreshold&#34;, -1) // 关闭 broadcast hit
//1 加载数据成DataFrame
val df = spark.read.json(&#34;examples/src/main/resources/people.json&#...]]></description><category></category></item><item><title><![CDATA[一致性hash算法]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/106384335</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/106384335</guid><author>github_28583061</author><pubDate>Wed, 08 Jul 2020 06:58:59 +0800</pubDate><description><![CDATA[一致性hash算法
出现原因


分布式集群中，通过key 求 hash 取余节点数 决定 访问位置，实现数据分片


当节点数变化时（新增节点，节点出错），需要全部重新计算迁移


原理

一致性hash是对2^32取模， 假设有一个圆环：

对2 ^32取余是有依据的IPV4最大数量就是2^32，所以这样就能保证所有的ip的地址进行取余时不会重复—对应hash环上面的正数。



服务器位置确...]]></description><category></category></item><item><title><![CDATA[cdh_操作]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295487</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295487</guid><author>github_28583061</author><pubDate>Fri, 03 Jul 2020 09:37:22 +0800</pubDate><description><![CDATA[动态资源配置（CDH 6.1.0)
基础介绍

根据不同项目或不同用户，对yarn资源队列进行划分，达到资源管控，任务管控的目的
yarn资源池可以嵌套，可以在父资源池的基础上，更细粒度资源分配池
默认有一个名为root.default 的资源池，所有yarn applications运行在这个资源池。
root.users资源池，是一个父资源池,通过root.users.[username]规则...]]></description><category></category></item><item><title><![CDATA[docker]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/106384813</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/106384813</guid><author>github_28583061</author><pubDate>Fri, 03 Jul 2020 09:19:38 +0800</pubDate><description><![CDATA[docker  -- 精通Docker第三版 https://alanhou.org/portainer-gui-docker/
win10 安装docker

toolbox 安装
mklink /J &#34;C:\Users\Administrator.docker&#34; &#34;G:\vm\machines&#34;
C:\Users\Administrator.docker\mac...]]></description><category></category></item><item><title><![CDATA[ClusterInstall]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/106385655</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/106385655</guid><author>github_28583061</author><pubDate>Fri, 03 Jul 2020 09:17:22 +0800</pubDate><description><![CDATA[集群安装
cdh 安装

https://blog.youkuaiyun.com/u010003835/article/details/85007946
CDH6.1.0离线安装部署

https://blog.youkuaiyun.com/xsjzdrxsjzdr/article/details/86064094


https://www.cloudera.com/documentation/enterprise/5...]]></description><category></category></item><item><title><![CDATA[常用数据结构]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295973</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295973</guid><author>github_28583061</author><pubDate>Wed, 01 Jul 2020 09:38:48 +0800</pubDate><description><![CDATA[常用数据结构

https://mp.weixin.qq.com/s?__biz=MzI1MTIzMzI2MA==&amp;mid=2650565995&amp;idx=1&amp;sn=9a0325e8dfe9d366440f05e693a4d7e0&amp;chksm=f1fedbe8c68952fecfb467391a29f580082fb2c6522c47c37ae4802ae43a7ca...]]></description><category></category></item><item><title><![CDATA[kafka最佳实践]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295470</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295470</guid><author>github_28583061</author><pubDate>Wed, 01 Jul 2020 09:38:48 +0800</pubDate><description><![CDATA[Apache Kafka 最佳实践



参考

https://mp.weixin.qq.com/s?__biz=MzA5MTc0NTMwNQ==&amp;mid=2650721159&amp;idx=1&amp;sn=c2dcc4b183f59678c75582ca9738795e&amp;chksm=887dd6f1bf0a5fe79d1c8012cb57612064d348a92cf610...]]></description><category></category></item><item><title><![CDATA[spring_shrio_jwt]]></title><link>https://blog.youkuaiyun.com/github_28583061/article/details/109295447</link><guid>https://blog.youkuaiyun.com/github_28583061/article/details/109295447</guid><author>github_28583061</author><pubDate>Wed, 01 Jul 2020 09:38:48 +0800</pubDate><description><![CDATA[springboot + shrio + jwt
shrio

shiro中用户认证和用户授权是分开的。用户认证（可以理解为登陆）叫Authentication，用户授权叫Authorization，
Realm 是存储客户端安全数据如用户、角色、权限等的一个组件，可以理解为DAO。
shiro先用filter拦截请求，然后调用realm获取用户的认证、授权信息。

参考

https://blo...]]></description><category></category></item></channel></rss>