<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[jinjiating的专栏]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/jinjiating</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; jinjiating]]></copyright><item><title><![CDATA[Flink MySql CDC 源码剖析]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/155452889</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/155452889</guid><author>jinjiating</author><pubDate>Mon, 01 Dec 2025 15:14:23 +0800</pubDate><description><![CDATA[Flink CDC的MySqlSource组件通过MySqlSourceEnumerator和MySqlSourceReader实现MySQL数据的全量和增量同步。MySqlSourceEnumerator采用两种分片策略：MySqlBinlogSplitAssigner处理增量数据，MySqlHybridSplitAssigner先通过均分算法切分全量快照数据，再创建BinlogSplit读取增量变更。全量阶段多个Reader并发读取快照数据，完成后报告水位线；增量阶段由单个Reader读取BinlogS]]></description><category></category></item><item><title><![CDATA[Flink Source源码解析]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/154535483</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/154535483</guid><author>jinjiating</author><pubDate>Fri, 07 Nov 2025 16:27:21 +0800</pubDate><description><![CDATA[Source执行时由SourceOperatorCoordinator和SourceReader两部分组成，SourceOperatorCoordinator执行SourceSplitEnumerator逻辑，管理SourceSplit， 将SourceSplit分配给SourceReader；SourceReader是SourceOperator在启动时创建，SourceOperator和SourceCoordinatorProvider由SourceOperatorFactory创建。]]></description><category></category></item><item><title><![CDATA[Flink on Native K8S源码解析]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/151818952</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/151818952</guid><author>jinjiating</author><pubDate>Thu, 18 Sep 2025 10:00:33 +0800</pubDate><description><![CDATA[本文解析了Flink在Kubernetes上的实现原理，主要分析了Session模式的集群创建流程。启动入口为KubernetesSessionCli，通过KubernetesClusterDescriptor创建集群，最终返回RestClusterClient用于集群交互。重点剖析了JobManager Pod的初始化过程，包括各类Decorator如何定义资源配置和伴生对象。同时对比了Session和Application模式在Dispatcher等方面的差异，并介绍了TaskManager的创建流程，]]></description><category></category></item><item><title><![CDATA[Flink on Native K8S安装部署]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/150530661</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/150530661</guid><author>jinjiating</author><pubDate>Tue, 19 Aug 2025 17:26:03 +0800</pubDate><description><![CDATA[Flink on K8S部署指南 本文介绍了在Kubernetes上部署Flink会话集群的完整步骤： 准备K8S环境并下载Flink 1.18.1 创建专用namespace和registry secret 配置ServiceAccount并授权 修改flink-conf.yaml配置JVM参数、内存管理等 通过kubernetes-session.sh脚本启动集群，支持HA和OSS存储 使用Pod模板文件定义资源请求、挂载卷等配置 提供查看/删除K8S资源的命令 部署过程包含完整的鉴权、资源配置和持久化]]></description><category></category></item><item><title><![CDATA[Flink源码之Checkpoint执行流程]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132431655</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132431655</guid><author>jinjiating</author><pubDate>Tue, 22 Aug 2023 16:59:45 +0800</pubDate><description><![CDATA[以下对整个流程具体说明。]]></description><category></category></item><item><title><![CDATA[Flink源码之State创建流程]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132299910</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132299910</guid><author>jinjiating</author><pubDate>Tue, 15 Aug 2023 15:24:40 +0800</pubDate><description><![CDATA[Flink内置状态管理是相比其他分布式流式处理系统最大的优势之一，不用借助外部存储组件，就可实现高效可靠的分布式状态管理，极大降低了学习和使用成本。]]></description><category></category></item><item><title><![CDATA[Flink源码之StreamTask启动流程]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132275252</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132275252</guid><author>jinjiating</author><pubDate>Mon, 14 Aug 2023 14:25:39 +0800</pubDate><description><![CDATA[submitTask 方法核心就是构造org.apache.flink.runtime.taskmanager.Task实例，该实例继承自Runnable接口，有个Thread成员变量，构造完成后就启动线程执行Task逻辑。每个ExecutionVertex分配Slot后，JobMaster就会向Slot所在的TaskExecutor提交RPC请求执行Task，接口为TaskExecutorGateway::submitTask。]]></description><category></category></item><item><title><![CDATA[Flink源码之JobMaster启动流程]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132235155</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132235155</guid><author>jinjiating</author><pubDate>Fri, 11 Aug 2023 16:36:30 +0800</pubDate><description><![CDATA[在整个提交过程中，首先获取JobMasterService的Leader权限，然后对一个JobGraph生成一个JobMaster，JobMaster先将JobGraph转换为ExecutionGraph，转换核心逻辑在DefaultExecutionGraph::attachJobGraph方法中，最后为每个Execution申请Slot资源，对每个Execution向TM提交TaskDeploymentDescriptor调度执行。]]></description><category></category></item><item><title><![CDATA[Flink源码之TaskManager启动流程]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132206305</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132206305</guid><author>jinjiating</author><pubDate>Thu, 10 Aug 2023 11:49:35 +0800</pubDate><description><![CDATA[从启动命令flink-daemon.sh可以看出TaskManger入口类为org.apache.flink.runtime.taskexecutor.TaskManagerRunner这些服务在构造TaskExecutor时作为构造函数参数传入构造TaskExecutor前会先构造TaskManagerServices辅助TaskExecutor实现其核心功能TaskManager中管理Slot的实现类TaskSlotTableImpl，该实例记录了Slot的分配信息。]]></description><category></category></item><item><title><![CDATA[Flink源码之JobManager启动流程]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132193277</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132193277</guid><author>jinjiating</author><pubDate>Wed, 09 Aug 2023 17:35:48 +0800</pubDate><description><![CDATA[JobManager的启动过程就是创建三大组件RestServer/RM/Dispacher实例初始化的过程，RestSever通过Netty启动HTTP服务，RM/Dispacher被AkkaRpcService包装成AkkaActor提供本地或远程RPC服务，RestServer仅仅是接受请求解析消息后由具体Handler处理，JobGrap提交执行会转发给Dispatcher处理。]]></description><category></category></item><item><title><![CDATA[Flink源码之RPC]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132168993</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132168993</guid><author>jinjiating</author><pubDate>Tue, 08 Aug 2023 16:00:28 +0800</pubDate><description><![CDATA[因此实现一个具体的Component只要同时继承RpcGateway和RpcEndPoint就能实现将RpcGateway的业务接口暴露出去提供远程服务，在JobManager/TaskManger启动过程中，都是先创建一个RpcService实例，然后new出各个业务组件实例，这样各个组件完成初始化后就能各司其职，协调配合完成具体业务功能。组件间通信基于Actor, 负责RPC通信功能则被抽象成一个RPCEndPoint。这些接口的具体实现就是组件提供的核心功能。]]></description><category></category></item><item><title><![CDATA[Python登录QQ邮箱发送邮件]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/132161968</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/132161968</guid><author>jinjiating</author><pubDate>Tue, 08 Aug 2023 10:49:25 +0800</pubDate><description><![CDATA[Python发QQ邮件]]></description><category></category></item><item><title><![CDATA[Spark中Join实现原理]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/127973403</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/127973403</guid><author>jinjiating</author><pubDate>Mon, 21 Nov 2022 21:53:33 +0800</pubDate><description><![CDATA[spark中join实现方式]]></description><category></category></item><item><title><![CDATA[字节面试题求累计UV]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/127939888</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/127939888</guid><author>jinjiating</author><pubDate>Sat, 19 Nov 2022 18:07:52 +0800</pubDate><description><![CDATA[字节面试题求累计UV]]></description><category></category></item><item><title><![CDATA[Redis笔记]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/127938900</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/127938900</guid><author>jinjiating</author><pubDate>Sat, 19 Nov 2022 16:54:27 +0800</pubDate><description><![CDATA[Redids笔记]]></description><category></category></item><item><title><![CDATA[Roaring Bitmap原理]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/127930503</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/127930503</guid><author>jinjiating</author><pubDate>Fri, 18 Nov 2022 22:15:41 +0800</pubDate><description><![CDATA[最近面试字节，被问roaring bitmap原理，虽然说之前看过，时间久了细节忘了，再次mark下，这边文章讲的很透彻。]]></description><category></category></item><item><title><![CDATA[Java中5中发送HTTP请求的的方式]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/122135811</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/122135811</guid><author>jinjiating</author><pubDate>Fri, 24 Dec 2021 20:55:12 +0800</pubDate><description><![CDATA[发送HTTP请求时现代编程中的核心功能，也是我们学习一种编程语言时最先想做的事。在Java中，有多种方式实现此功能，比如JDK或第三方库，这篇文章将会介绍我所知的Java中HTTP客户端，如果你使用过其他的，那很好，请告诉我。这篇文章将会涉及：
JDK中:

HttpURLConnection
HttpClient

流行的第三方库:

ApacheHttpClient
OkHttp
Retrofit

我将会使用 NASA APIs 中每天一图Astronomy Picture of the Day做为示]]></description><category></category></item><item><title><![CDATA[ETL优化之道]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/122014042</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/122014042</guid><author>jinjiating</author><pubDate>Sat, 18 Dec 2021 16:43:39 +0800</pubDate><description><![CDATA[天下武功，唯快不破。
ETL Job优化的目标是执行快，耗费时间更短，总结起来有以下方式：


增加硬件资源，比如增大内存、CPU等，但优化往往是在资源不足的情况下


数据缓存，空间换时间


服务常驻，减少初始化化时间


减少输入数据量，常用以下技术

数据压缩，缓存中能存放更多数据
索引+BloomFilter，也是空间换时间，加快数据查找速度
列裁剪，原理也是提高了缓存利用率
谓词下推，减少扫描数据，一般要借助预分区或索引相关技术
数据分区分桶
避免重启读取相同数据，比如对同一张表多次查询UNIO]]></description><category></category></item><item><title><![CDATA[数据倾斜产生原因及应对之道]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/121998738</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/121998738</guid><author>jinjiating</author><pubDate>Fri, 17 Dec 2021 16:09:30 +0800</pubDate><description><![CDATA[产生数据倾斜的根本原因是某个task处理数据过多，执行时间较长，导致整个Job执行时间长，因此解决数据倾斜的根本之道就是避免过多的数据进入同一个task，总结起来主要有以下方式：
Group By产生数据倾斜


直接增加task并行度，简单粗暴，但并不一定能正在解决问题。倾斜是由多个key组合输入一个task导致数据过多, 这种方式有效，但如果存在单个hot key数据量大则无效。


预聚合，类似于Map Combine, 相当与把聚合运算提前分散到多个task, 减少最终聚合task的输入数据量
hi]]></description><category></category></item><item><title><![CDATA[理解LSM树:频繁写的数据库基石]]></title><link>https://blog.youkuaiyun.com/jinjiating/article/details/121960522</link><guid>https://blog.youkuaiyun.com/jinjiating/article/details/121960522</guid><author>jinjiating</author><pubDate>Wed, 15 Dec 2021 20:03:13 +0800</pubDate><description><![CDATA[log-structured merge-tree(LSM tree)是可支持大量写的一种数据结构，写操作被优化成顺序写，LSM树是许多数据库背后的核心数据结构，包括BigTable, Cassandra, Scylla, 和 RocksDB.
SSTables
LSM树在磁盘上以有序表(Sorted String Table)的形式存在。下图所示，SSTable存储了以key排序的key-value键值对，其中name为key。一个SSTable是由多个排序的文件组成，每个文件是一个Segment, 这些]]></description><category></category></item></channel></rss>