<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[cuichunchi的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/cuichunchi</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; cuichunchi]]></copyright><item><title><![CDATA[flink to starrocks 问题集锦....]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/131933092</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/131933092</guid><author>cuichunchi</author><pubDate>Wed, 26 Jul 2023 10:16:33 +0800</pubDate><description><![CDATA[报如图所示：提示所示：超出内存限制，生产上不知道BE设置了多少。]]></description><category></category></item><item><title><![CDATA[kali 安装无线网卡路程]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/131578787</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/131578787</guid><author>cuichunchi</author><pubDate>Thu, 06 Jul 2023 15:58:35 +0800</pubDate><description><![CDATA[输入上述命令之后，在执行 sudo apt-get update 和 sudo apt-get upgrade 就不会有问题了。apt upgrade && apt-get dist-upgrade 升级系统相关依赖。解决安装头文件问题，更新完内核，reboot重启，选择以及安装头文件的内核即可。apt upgrade 命令将尝试下载在apt服务器上具有更新的所有软件包。2、添加gpg，不然添加签名会报找不到有效的OpenPGP数据。安装aptitude来管理包，强烈推荐，不然很痛苦。]]></description><category></category></item><item><title><![CDATA[Better Explained 以通俗易懂的语言阐释数学]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/131113298</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/131113298</guid><author>cuichunchi</author><pubDate>Thu, 08 Jun 2023 18:05:58 +0800</pubDate><description><![CDATA[Finally understand math, don't memorize it.


]]></description><category></category></item><item><title><![CDATA[尝试 python flink jupyter]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/130883874</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/130883874</guid><author>cuichunchi</author><pubDate>Fri, 26 May 2023 11:50:14 +0800</pubDate><description><![CDATA[python flink]]></description><category></category></item><item><title><![CDATA[flink1.13.2 代码编译]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/130786686</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/130786686</guid><author>cuichunchi</author><pubDate>Sat, 20 May 2023 22:32:41 +0800</pubDate><description><![CDATA[执行命令：使用spotless来格式化代码。在编译或提交之前，请运行。执行：mvn clean install -DskipTests。]]></description><category></category></item><item><title><![CDATA[由于hadoop高可用状态切换，导致hive在指定数据库下建立以及读取表失败]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/129535988</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/129535988</guid><author>cuichunchi</author><pubDate>Tue, 14 Mar 2023 18:30:40 +0800</pubDate><description><![CDATA[问题描述：因为cdh集群，hadoop启用了HA高可用，之前node1节点为active，现在变成了node2节点为standby。而且之前hive建库的时候，LOCATION=hdfs://node1:8020/user/warehouse/dir.....定位：由于配置文件hive-site.xml、core-site.xml、hdfs-site.xml等有关配置文件都替换了最新，都是配置了hdfs://高可用地址/导致库里面的表默认都是获取DB的location来拼接的表的hdfs路径。]]></description><category></category></item><item><title><![CDATA[flink 实时维表关联方案]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/129321647</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/129321647</guid><author>cuichunchi</author><pubDate>Fri, 03 Mar 2023 15:40:12 +0800</pubDate><description><![CDATA[实时维表关联]]></description><category></category></item><item><title><![CDATA[Flink 解析kafka avro格式]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/127690696</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/127690696</guid><author>cuichunchi</author><pubDate>Fri, 04 Nov 2022 16:02:28 +0800</pubDate><description><![CDATA[flink 对接kafka avro]]></description><category></category></item><item><title><![CDATA[编译Iceberg源码以及检出分支并编译]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/126240417</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/126240417</guid><author>cuichunchi</author><pubDate>Tue, 09 Aug 2022 09:24:52 +0800</pubDate><description><![CDATA[构建iceberg源码]]></description><category></category></item><item><title><![CDATA[Flink中allowedLateness介绍与测试]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/123118655</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/123118655</guid><author>cuichunchi</author><pubDate>Thu, 24 Feb 2022 19:20:33 +0800</pubDate><description><![CDATA[默认情况下，当watermark通过end-of-window之后，再有之前的数据到达时，这些数据会被删除。

为了避免有些迟到的数据被删除，因此产生了allowedLateness的概念。

简单来讲，allowedLateness就是针对event time而言，对于watermark超过end-of-window之后，还允许有一段时间（也是以event time来衡量）来等待之前的数据到达，以便再次处理这些数据

默认情况下，如果不指定allowedLateness，其值是0，即对于watermark]]></description><category></category></item><item><title><![CDATA[Flink watermark 测试]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/123101871</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/123101871</guid><author>cuichunchi</author><pubDate>Wed, 23 Feb 2022 23:30:59 +0800</pubDate><description><![CDATA[代码如下：


package com.cuichunchi.watermark;

import org.apache.flink.api.common.eventtime.SerializableTimestampAssigner;
import org.apache.flink.api.common.eventtime.WatermarkStrategy;
import org.apache.flink.api.common.functions.MapFunction;
import org.apac]]></description><category></category></item><item><title><![CDATA[Flink Sort-shuffle简介]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/123091704</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/123091704</guid><author>cuichunchi</author><pubDate>Wed, 23 Feb 2022 15:40:53 +0800</pubDate><description><![CDATA[Flink Sort-Shuffle 实现简介sort-shuffle 使得 Flink 在应对大规模批数据处理任务时更加游刃有余。https://mp.weixin.qq.com/s?__biz=MzU3Mzg4OTMyNQ==&amp;mid=2247494455&amp;idx=1&amp;sn=7de90d250794ea095847d82d02b8dc30&amp;chksm=fd386575ca4fec634aae8b3d8567c60e3b9ab95d7c1d492c07a63bed0fc9e]]></description><category></category></item><item><title><![CDATA[Tez调优参数总结]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/122980931</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/122980931</guid><author>cuichunchi</author><pubDate>Thu, 17 Feb 2022 12:54:07 +0800</pubDate><description><![CDATA[hive.tez.auto.reducer.parallelism=false
hive.tez.bucket.pruning=false
hive.tez.bucket.pruning.compat=true
hive.tez.container.max.java.heap.fraction=0.8
hive.tez.container.size=768
hive.tez.cpu.vcores=-1
hive.tez.dynamic.partition.pruning=true
hive.tez.dyn.]]></description><category></category></item><item><title><![CDATA[Hive调优参数总结]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/122980923</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/122980923</guid><author>cuichunchi</author><pubDate>Thu, 17 Feb 2022 12:53:29 +0800</pubDate><description><![CDATA[MR程序 优化参数

1、客户端显示以及job任务名和优先级
##打印表头
set hive.cli.print.header=true;
set hive.cli.print.row.to.vertical=true;
set hive.cli.print.row.to.vertical.num=1;
##显示当前数据库
set hive.cli.print.current.db=true;
##job任务名
set mapreduce.job.name=p_${v_date};
##job优先级
se.]]></description><category></category></item><item><title><![CDATA[Flink CDC 问题以及测试记录（持续更新...）]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/122480296</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/122480296</guid><author>cuichunchi</author><pubDate>Thu, 13 Jan 2022 18:52:09 +0800</pubDate><description><![CDATA[问题及测试集锦
1、flink cdc 的 api兼容的ddl的结构变更吗？
2、cdc 比如增量消费 然后 kill任务 再恢复？
3、mysql cdc 设置了检查点，sink到kafka，重启作业，为啥全量的数据还是会重新发送到kafka
]]></description><category></category></item><item><title><![CDATA[Kafka 问题以及调优参数集锦]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/122333034</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/122333034</guid><author>cuichunchi</author><pubDate>Wed, 05 Jan 2022 22:03:46 +0800</pubDate><description><![CDATA[producer类参数：

1、request.timeout.ms:配置控制客户端等待请求响应的最长时间。如果在超时时间过去之前未收到响应，则客户端将在必要时重新发送请求，或者在重试次数用尽时使请求失败。默认值：30000ms。

注：该参数适合集群资源不高，带宽较低、且生产者频繁发送消息到topic。超时可能报以下错误信息，我在压测过程中发现的错误信息，并改为了60s，集群3台带宽4M（理想吞吐512KB/s）.

Expiring 7 record(s) for test_producer-0: 60]]></description><category></category></item><item><title><![CDATA[kafka压测多维度分析实战]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/122248067</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/122248067</guid><author>cuichunchi</author><pubDate>Fri, 31 Dec 2021 00:27:04 +0800</pubDate><description><![CDATA[设置虚拟机不同的带宽来进行模拟压测





---------kafka数据压测-------------------1、公司生产kafka集群硬盘：单台500G、共3台、日志保留7天。
  1.1 版本：1.1.0

-----2、压测kafka。
  2.1 使用kafka自带压测工具：bin/kafka-producer-perf-test.sh
   命令参数解释：
    --num-records ：总共发送多少条消息。
...]]></description><category></category></item><item><title><![CDATA[基于Flink 1.13.2问题集锦]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/121926090</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/121926090</guid><author>cuichunchi</author><pubDate>Tue, 14 Dec 2021 14:12:03 +0800</pubDate><description><![CDATA[问题：

1、使用flink mysql cdc 发现bigint unsigned类型的字段，capture以后转成了字符串类型，用的这个解析吧JsonDebeziumDeserializationSchema。

解决：在设置debeziumProperties方法里传入以下参数：

properties.setProperty("bigint.unsigned.handling.mode","long");

properties.setProperty("decimal.handling.mode"]]></description><category></category></item><item><title><![CDATA[Flink 1.13 一些配置参数记录]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/121888619</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/121888619</guid><author>cuichunchi</author><pubDate>Sun, 12 Dec 2021 17:35:20 +0800</pubDate><description><![CDATA[持续更新...

网络通信：


	
akka.ask.timeout：用于异步futures和阻塞调用Akka的超时，如果flink因为超时而失败，则可以尝试增加此值，超时可能是由于机器速度慢或网络拥挤造成的。超时值需要时间单位说明符（ms/s/min/h/d）。源码默认值：10s。所属源码类：AkkaOptions。
	
	
	
akka.tcp.timeout：用于和taskManager之间的通信，如果由于网络速度较慢而在连接TaskManager时遇到问题，则应增加此值。默认值：20s。源码类：A]]></description><category></category></item><item><title><![CDATA[hive 添加字段，字段值错位问题]]></title><link>https://blog.youkuaiyun.com/cuichunchi/article/details/121105043</link><guid>https://blog.youkuaiyun.com/cuichunchi/article/details/121105043</guid><author>cuichunchi</author><pubDate>Tue, 02 Nov 2021 18:03:09 +0800</pubDate><description><![CDATA[hive中在表结构中添加字段，并将字段指定到于业务库相对的位置：


alter table ods_test.ods_ppy_op_op_supplier_goods_quote_apply  add columns (storage_id bigint  comment '仓库id') cascade;   
alter table ods_test.ods_ppy_op_op_supplier_goods_quote_apply change storage_id storage_id bigi...]]></description><category></category></item></channel></rss>