<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[qq_42389487的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/qq_42389487</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; qq_42389487]]></copyright><item><title><![CDATA[Spark 行转列自定义UDAF]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/129817110</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/129817110</guid><author>qq_42389487</author><pubDate>Tue, 28 Mar 2023 15:30:07 +0800</pubDate><description><![CDATA[行转列自定义UDAF]]></description><category></category></item><item><title><![CDATA[记录sparksql解析json字符串]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/129689623</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/129689623</guid><author>qq_42389487</author><pubDate>Tue, 21 Mar 2023 15:26:44 +0800</pubDate><description><![CDATA[sparksql解析json字符串]]></description><category></category></item><item><title><![CDATA[scala插值字符串输出的坑]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/129689415</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/129689415</guid><author>qq_42389487</author><pubDate>Tue, 21 Mar 2023 15:23:03 +0800</pubDate><description><![CDATA[scala插值字符串坑]]></description><category></category></item><item><title><![CDATA[spark线上环境写入starRocks中文乱码]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/129672700</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/129672700</guid><author>qq_42389487</author><pubDate>Mon, 20 Mar 2023 18:50:59 +0800</pubDate><description><![CDATA[starRocks中文乱码]]></description><category></category></item><item><title><![CDATA[sparkSql写入starRocks]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/129673395</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/129673395</guid><author>qq_42389487</author><pubDate>Mon, 20 Mar 2023 18:44:06 +0800</pubDate><description><![CDATA[sparkSql写入starRocks]]></description><category></category></item><item><title><![CDATA[记录CDH-kafka配置外网]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/128468566</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/128468566</guid><author>qq_42389487</author><pubDate>Wed, 28 Dec 2022 14:57:52 +0800</pubDate><description><![CDATA[因为项目需要，需隔壁项目组协助拉取数据，但是公司用的阿里云，且两边的集群不在同一内网，故而需要开放kafka的端口给另外的项目组，以便于他们通过外网连接到kafka。记录以下配置，重启后生效。]]></description><category></category></item><item><title><![CDATA[记录impala中文会乱码的情况。]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/125636151</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/125636151</guid><author>qq_42389487</author><pubDate>Wed, 06 Jul 2022 11:23:48 +0800</pubDate><description><![CDATA[impala 视图中文乱码]]></description><category></category></item><item><title><![CDATA[200 port command successful. consider using pasv 425 failed to establish connection]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/125635672</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/125635672</guid><author>qq_42389487</author><pubDate>Wed, 06 Jul 2022 11:17:45 +0800</pubDate><description><![CDATA[windows->Linux：ftp报错200 port command successful. consider using pasv 425 failed to establish connection]]></description><category></category></item><item><title><![CDATA[spark java.net.BindException: Cannot assign requested address]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/123493488</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/123493488</guid><author>qq_42389487</author><pubDate>Tue, 15 Mar 2022 08:57:53 +0800</pubDate><description><![CDATA[网上找了下原因，大致上是由于客户端频繁的连服务器，由于每次连接都在很短的时间内结束，导致很多的TIME_WAIT，以至于用光了可用的端 口号，所以新的连接没办法绑定端口，即“Cannot assign requested address”。是客户端的问题不是服务器端的问题。通过netstat，的确看到很多TIME_WAIT状态的连接。



client端频繁建立连接，而端口释放较慢，导致建立新连接时无可用端口。



网上的解决方法：

1、先查看端口是否被占用。



2、执行命令修改如下2个内核参数 ]]></description><category></category></item><item><title><![CDATA[记录修改taildir source支持windows路径正则]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/123493292</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/123493292</guid><author>qq_42389487</author><pubDate>Tue, 15 Mar 2022 08:46:22 +0800</pubDate><description><![CDATA[TaildirMatcher(String fileGroup, String filePattern, boolean cachePatternMatching, int cachePatternMatchingDuration) {
 // store whatever came from configuration
 this.fileGroup = fileGroup;
 this.filePattern = filePattern;
 thi...]]></description><category></category></item><item><title><![CDATA[记录一次修改flume1.9 taildir source]]></title><link>https://blog.youkuaiyun.com/qq_42389487/article/details/122258188</link><guid>https://blog.youkuaiyun.com/qq_42389487/article/details/122258188</guid><author>qq_42389487</author><pubDate>Fri, 31 Dec 2021 16:03:03 +0800</pubDate><description><![CDATA[在测试代码中发现taildir不支持windows。且要上传很多小文件并记录文件名字。

使其支持windows很多大佬已经做到，我也是站在巨人的肩膀上，copy他们的。

这里记录修改taildir使其支持windows路径



在不断的了解flume机制的同时发现了根本问题，hdfs路径不支持冒号作为路径
因为我们获取到的fileHeaderKey 是绝对路径，所以是有冒号的

fileHeader 设置为 true ，可以将日志文件的绝对路径存储在事件的header中；
fileHeaderKey ]]></description><category></category></item></channel></rss>