<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[poplarandwillow的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/poplarandwillow</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; poplarandwillow]]></copyright><item><title><![CDATA[spring boot redis缓存实现]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/130291514</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/130291514</guid><author>poplarandwillow</author><pubDate>Fri, 21 Apr 2023 16:17:49 +0800</pubDate><description><![CDATA[另外，可以设置 allEntries 属性为 true，以移除指定缓存名称下的所有缓存项，例如在 clearCache 方法中。当调用 getFromCache 方法时，Spring 会自动将结果缓存到 Redis 中，并在下一次调用时从缓存中返回结果。在上述示例中，创建了一个 RedisCacheManager 实例，并设置了缓存项的过期时间为 30 分钟，使用了 GenericJackson2JsonRedisSerializer 序列化器序列化缓存值。也可以使用其他序列化器来序列化缓存值。]]></description><category></category></item><item><title><![CDATA[代码量统计- 主要针对java python 和vue]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/130109382</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/130109382</guid><author>poplarandwillow</author><pubDate>Wed, 12 Apr 2023 15:56:02 +0800</pubDate><description><![CDATA[统计文件夹下以[‘.py’, ‘.java’, ‘.scss’, ‘.css’, ‘.js’, ‘.vue’, ‘.ts’, ‘.json’, ‘.xml’, ‘yaml’, ‘.pom’] 结尾的代码行数。结果: 最终会形成一个excel文件文件中的内容如下。]]></description><category></category></item><item><title><![CDATA[python将py文件打包成exe可执行文件]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/130102582</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/130102582</guid><author>poplarandwillow</author><pubDate>Wed, 12 Apr 2023 11:14:04 +0800</pubDate><description><![CDATA[这将生成一个名为 main.exe 的可执行文件，并在当前目录下创建一个 dist 文件夹，其中包含 exe 文件及其所需的其他文件。创建一个 Python 脚本（例如 main.py），编写需要执行的代码。在命令行中运行 main.exe 文件即可启动应用程序。]]></description><category></category></item><item><title><![CDATA[【无标题】]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/130092039</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/130092039</guid><author>poplarandwillow</author><pubDate>Tue, 11 Apr 2023 19:57:56 +0800</pubDate><description><![CDATA[chatgpt 体验spring boot搭建## 标题。]]></description><category></category></item><item><title><![CDATA[Git clone error: unable to create file xxx: Filename too long]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/122960908</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/122960908</guid><author>poplarandwillow</author><pubDate>Wed, 16 Feb 2022 12:36:44 +0800</pubDate><description><![CDATA[在使用 git 时，提示
error: unable to create file xxx: Filename too long
error: unable to create file xxx: Filename too long
error: unable to create file xxx: Filename too long
error: unable to create file xxx: Filename too long
error: unable to create file xxxx: ]]></description><category></category></item><item><title><![CDATA[quartz 设计表结构]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/122318070</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/122318070</guid><author>poplarandwillow</author><pubDate>Wed, 05 Jan 2022 10:34:58 +0800</pubDate><description><![CDATA[quartz 设计表结构]]></description><category></category></item><item><title><![CDATA[海豚调度(DolphinSchedulerDolphinScheduler)2.0的下载及安装（单机版）]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/121762855</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/121762855</guid><author>poplarandwillow</author><pubDate>Tue, 07 Dec 2021 10:40:17 +0800</pubDate><description><![CDATA[海豚调度(DolphinSchedulerDolphinScheduler)2.0的下载及安装（单机版）]]></description><category></category></item><item><title><![CDATA[kafka面试]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/121161635</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/121161635</guid><author>poplarandwillow</author><pubDate>Fri, 05 Nov 2021 16:37:20 +0800</pubDate><description><![CDATA[1. Kafka 分布式的情况下，如何保证消息的顺序?
1.  kafka的单个分区partition中的数据是有序的

可以将相同的key发送到一个分区中

Kafka 分布式的单位是 Partition。如何保证消息有序，需要分几个情况讨论。
同一个 Partition 用一个 write ahead log 组织，所以可以保证 FIFO 的顺序。
不同 Partition 之间不能保证顺序。但是绝大多数用户都可以通过 message key 来定义，因为同一个 key 的 message 可以保证只发]]></description><category></category></item><item><title><![CDATA[spark 面试题]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/121162614</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/121162614</guid><author>poplarandwillow</author><pubDate>Fri, 05 Nov 2021 16:36:38 +0800</pubDate><description><![CDATA[1. spark 数据倾斜
2. spark shuffer过程
3. spark join的策略解析
1. spark 数据倾斜
Spark 数据倾斜的几种场景以及对应的解决方案，包括避免数据源倾斜，调整并行度，使用自定义 Partitioner，使用 Map 侧 Join 代替 Reduce 侧 Join（内存表合并），给倾斜 Key 加上随机前缀等。
什么是数据倾斜 对 Spark/Hadoop 这样的大数据系统来讲，数据量大并不可怕，可怕的是数据倾斜。数据倾斜指的是，并行处理的数据集中，某一部分（如]]></description><category></category></item><item><title><![CDATA[spark 函数]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/121120554</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/121120554</guid><author>poplarandwillow</author><pubDate>Wed, 03 Nov 2021 14:36:11 +0800</pubDate><description><![CDATA[https://spark.apache.org/docs/2.4.5/api/sql/index.html添加链接描述

]]></description><category></category></item><item><title><![CDATA[Flink面试总结`持续更新中........]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/121036397</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/121036397</guid><author>poplarandwillow</author><pubDate>Fri, 29 Oct 2021 15:54:21 +0800</pubDate><description><![CDATA[以前都是自己往小本本上总结, 后来小本本没了,  写到这里吧!  有问题请留言, 先谢过了
1. Flink 侧流输出
我们结合实际案例说明一下flink侧输出的用法，假设我们的需求是实时的从kafka接收生产数据，我们需要对迟到超过一定时长的数据或者不符合业务的数据另行处理：
第一步：定义OutputTag
// 定义 OutputTag 侧输出的数据格式可以不应和主流的数据格式一样
val delayOutputTag = OutputTag[String]("delay-side-output")

]]></description><category></category></item><item><title><![CDATA[jvm 堆内存使用和占用情况_查询]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/119572728</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/119572728</guid><author>poplarandwillow</author><pubDate>Tue, 10 Aug 2021 16:18:36 +0800</pubDate><description><![CDATA[1.查看进程id
jps
2. 查看堆内存的使用情况, 查看的是当前时间节点的
jmap -heap 进程id

jconsole可以查看图形界面式的堆内存 占用 cpu 等也可以查看


]]></description><category></category></item><item><title><![CDATA[jvm内存占用查询_问题]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/119571875</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/119571875</guid><author>poplarandwillow</author><pubDate>Tue, 10 Aug 2021 15:55:19 +0800</pubDate><description><![CDATA[使用 top 命令查询那个进程占用CPU占用过高


ps -H -eo pid,tid,%cpu | grep 进程id (用ps命令查看那个线程占用cpu过高)


jstack 进程id
可以查找到有问题的线程和代码  线程id是十进制的  需要换算为2进制



...]]></description><category></category></item><item><title><![CDATA[spark structured打通kafka与tablestore 的连接]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/119215452</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/119215452</guid><author>poplarandwillow</author><pubDate>Thu, 29 Jul 2021 16:56:04 +0800</pubDate><description><![CDATA[spark 连接kafka数据sink到tablestore中
直接上代码, 有不明白的可以留言
pom
&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven]]></description><category></category></item><item><title><![CDATA[spark streaming-sql tablestore统计数据]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/119115079</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/119115079</guid><author>poplarandwillow</author><pubDate>Mon, 26 Jul 2021 18:01:41 +0800</pubDate><description><![CDATA[spark数据源为ots 或者叫tablestore
DROP TABLE IF EXISTS source;
CREATE TABLE group_chat_source (
targetId STRING,targetType LONG, sendTime LONG COMMENT "group_chat_source"
)
USING tablestore
OPTIONS(
endpoint='${source_endpoint}',
access.key.id='${source_access_ke]]></description><category></category></item><item><title><![CDATA[spark tablestore数据读写]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/119114962</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/119114962</guid><author>poplarandwillow</author><pubDate>Mon, 26 Jul 2021 17:56:18 +0800</pubDate><description><![CDATA[直接上代码: 内容自己看


&lt;?xml version="1.0" encoding="UTF-8"?&gt;
&lt;project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apa]]></description><category></category></item><item><title><![CDATA[pyspark window安装和初始化]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/118765075</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/118765075</guid><author>poplarandwillow</author><pubDate>Thu, 15 Jul 2021 17:48:53 +0800</pubDate><description><![CDATA[今天开始搞pyspark

为了方便以后少走弯路直接下载 hadoop2.7.1 spark选择2.x即可

具体上一个博客! 注意可以先考率winutils.exe的版本在选择性下载



请点击这里


]]></description><category></category></item><item><title><![CDATA[pyspark报错 py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.isEncryptionEnabled does]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/118764997</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/118764997</guid><author>poplarandwillow</author><pubDate>Thu, 15 Jul 2021 17:44:56 +0800</pubDate><description><![CDATA[在执行一个程序的时候


conf = SparkConf().setAppName("miniProject").setMaster("local[1]")


报错内容


Traceback (most recent call last):
  File "D:/py_data/py_spark/demo_pyspark.py", line 9, in &lt;module&gt;
    sc = SparkContext.getOrCreate(conf)
  File "D:\py_data..]]></description><category></category></item><item><title><![CDATA[PG权限管理]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/115243582</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/115243582</guid><author>poplarandwillow</author><pubDate>Fri, 26 Mar 2021 14:39:29 +0800</pubDate><description><![CDATA[创建用户
CREATE USER app_user WITH PASSWORD ‘xxxxxx’;
创建schema  app模式
create schema app authorization app_user;
用户授权
grant all on schema app to app_user;
grant select, insert on all tables in schema app to app_user;
永久授权
–上边的方法赋值权限之后只能在现有的表上加上权限, 如果新增表之后还需要在执行]]></description><category></category></item><item><title><![CDATA[blink接入kafka数据  sink到postgre中]]></title><link>https://blog.youkuaiyun.com/poplarandwillow/article/details/114920936</link><guid>https://blog.youkuaiyun.com/poplarandwillow/article/details/114920936</guid><author>poplarandwillow</author><pubDate>Wed, 17 Mar 2021 13:53:48 +0800</pubDate><description><![CDATA[blink是阿里的开源组件
可以接入的数据源, 和目标源可以
https://help.aliyun.com/knowledge_list/62516.html?spm=a2c4g.11186623.6.750.5f7e7c842qR66N
废话不多说直接上代码
登录阿里子账号
右键-&gt;新建作业

要在pg数据库中创建要sink的数据表

```sql
-- 注意这里五个参数不能变
CREATE TABLE kafka_demo(
  messageKey VARBINARY,
  `message`]]></description><category></category></item></channel></rss>