<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[tianjun2012的专栏]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/tianjun2012</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; tianjun2012]]></copyright><item><title><![CDATA[构建基于docker的airflow并访问有kerberos认证的cdh集群]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/130276178</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/130276178</guid><author>tianjun2012</author><pubDate>Thu, 20 Apr 2023 22:22:32 +0800</pubDate><description><![CDATA[你没看错就是这个地方导致的，不能基于root安装，Ubuntu默认非root用户安教程一路起飞，centos一般都是root，一路各种掉坑，不然死活就是unhealthy，一查缺各种celery等各种依赖，但是用。，官网是入门的第一手好资料，虽说是官网，但大家环境各部相同，坑也是五花八门。进入pod查看，依赖都在，各种到头发，别问我是怎么知道的。可能需要用到的几个文件。]]></description><category></category></item><item><title><![CDATA[cdh集群元数据(mysql)kerberos账号及其权限分配查看方法]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/108768481</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/108768481</guid><author>tianjun2012</author><pubDate>Thu, 24 Sep 2020 10:39:22 +0800</pubDate><description><![CDATA[cdh集群通过sentry来管理kerberos账户权限

SELECT  sr.ROLE_NAME ,sdp.SERVER_NAME,sdp.DB_NAME ,sdp.TABLE_NAME,sdp.COLUMN_NAME,sdp.`ACTION` 
from sentry.SENTRY_DB_PRIVILEGE sdp
inner join sentry.SENTRY_ROLE_DB_PRIVILEGE_MAP srdp on sdp.DB_PRIVILEGE_ID=srdp.DB_PRIVILEGE_I]]></description><category></category></item><item><title><![CDATA[deepin qq 图片无法显示问题]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/107376984</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/107376984</guid><author>tianjun2012</author><pubDate>Thu, 16 Jul 2020 10:14:27 +0800</pubDate><description><![CDATA[禁止掉ipv6
在/etc/sysctl.conf中添加以下行：
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
后需可能需要用ipv6，此处先做个笔记，免得到时候，改了啥都不知道！

]]></description><category></category></item><item><title><![CDATA[structured streaming的checkpoint文件无限增长]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/106308378</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/106308378</guid><author>tianjun2012</author><pubDate>Sat, 23 May 2020 23:41:51 +0800</pubDate><description><![CDATA[目录structured streaming的checkpoint文件无限增长
structured streaming的checkpoint文件无限增长
原理和处理办法：
https://www.waitingforcode.com/apache-spark-structured-streaming/checkpoint-storage-structured-streaming/read#will_it_grow_indifinetely
Will it grow indefinitely?
No. Ap]]></description><category></category></item><item><title><![CDATA[大数据集群票据碰撞问题]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/105131822</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/105131822</guid><author>tianjun2012</author><pubDate>Fri, 27 Mar 2020 00:15:35 +0800</pubDate><description><![CDATA[kdc-kerberos 票据缓存在默认的/tmp/krb5cc_*这个文件，会导致有连个严重的问题：

linux系统会定期删除/tmp，这回导致偶发性的票据验证失败问题；
多用户并发执行任务的时候，票据会碰撞导致意外结果：

shell: 设置环境变量

export KRB5CCNAME=/home/keytab/krb5cc


python：改变缓存票据的存放路径



#!/usr/b...]]></description><category></category></item><item><title><![CDATA[Error in sasl_client_start (-4) SASL(-4): no mechanism available: No worthy mechs found]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/104480324</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/104480324</guid><author>tianjun2012</author><pubDate>Mon, 24 Feb 2020 17:10:45 +0800</pubDate><description><![CDATA[cdh或者hdp增加kerberos验证后，python链接大数据集群的调用都需要增加认证这一个环节：
如下报错的解决方案：
thriftpy.transport.TTransportException: TTransportException(type=1, message="Could not start SASL: b'Error in sasl_client_start (-4) SASL...]]></description><category></category></item><item><title><![CDATA[streamsets自定义插件部署方案]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/101206169</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/101206169</guid><author>tianjun2012</author><pubDate>Mon, 23 Sep 2019 15:10:01 +0800</pubDate><description><![CDATA[自定义JDBC Producer部署方案
目录1.背景简介2.部署步骤2.1 cm界面操作2.2上传驱动包2.3 上传插件2.4选择pipeline2.5 pipeline配置3.启动界面
1.背景简介
由于sdc缺少由kudu到mysql的upsert操作，前期通过spark-sql定制化开发了一套数据同步程序，但是由于spark-sql同步大量数据，效率十分低下，现基于sdc源码定制化了支持m...]]></description><category></category></item><item><title><![CDATA[spark-sql实现Kudu同步数据到mysql]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/100942403</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/100942403</guid><author>tianjun2012</author><pubDate>Tue, 17 Sep 2019 22:37:14 +0800</pubDate><description><![CDATA[Kudu同步数据到mysql实施方案

简介
目前kudu导出到mysql没有比较好的方案，临时借助spark-sql进行数据导出，处理逻辑是会把老的数据给删除再导入，已经完成了生产环境的上线。

需要传入的参数程序参数


			参数序号
			
			
			字段含义
			
			
			备注
			
		
			1
			
			
			同步的source表（含schema），必选
			...]]></description><category></category></item><item><title><![CDATA[日增百万级文件的实时采集的入坑出坑自强之路]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/94723680</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/94723680</guid><author>tianjun2012</author><pubDate>Fri, 05 Jul 2019 12:03:09 +0800</pubDate><description><![CDATA[写在前面的话
填坑不容易,入坑需谨慎!!!
流量数据数据是任何一家有志上市的科技公司不可能放弃的一个重点,但是对于好多公司而言,一本好好的经书,偏偏得歪着念,埋下一堆堪比定时炸弹的坑,后来入坑者后知后觉,从奔溃再到一点点的把坑填上,这就是一个程序员的自我修养吧! 怀着治病救人,普度众猿的慈悲之心,我把我的心酸泪贴出来,方便后来者,愿大家都活得更健康和谐.


...]]></description><category></category></item><item><title><![CDATA[snorkel调用spark的调试]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/86503743</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/86503743</guid><author>tianjun2012</author><pubDate>Wed, 16 Jan 2019 10:44:18 +0800</pubDate><description><![CDATA[前情提要
机器学习的库和工具多种多样诸如tensorflow这样的神器，最近在做体系解决方案，由于众所周知的原因，很大一部分公司，在实际的场景中是把数据都汇集到 了hadoop体系中（特别是文本类型数据，包括结构化的和非结构化的（音视频提取特征后也是文本数据）），如何在hadoop中高效的进行各种机器学习任务的开发（分布式开发，当然tensorflow也支持分布式，但是与hadoop我暂时还不知道...]]></description><category></category></item><item><title><![CDATA[python连接hive--impyla]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/84065186</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/84065186</guid><author>tianjun2012</author><pubDate>Wed, 14 Nov 2018 13:47:27 +0800</pubDate><description><![CDATA[impyla安装
连接hiveserv2的python的client端，可以提供impala、hive的使用功能。
python2.7
pip install thrift_sasl==0.2.1
pip install thrift
pip install six
pip install bit_array
pip install impyla

python3.3+
pip install t...]]></description><category></category></item><item><title><![CDATA[spark源码剖析（1）]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/80097877</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/80097877</guid><author>tianjun2012</author><pubDate>Thu, 26 Apr 2018 18:35:17 +0800</pubDate><description><![CDATA[WordCount例子流程简介

]]></description><category></category></item><item><title><![CDATA[windows10下关闭端口占用]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/79806760</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/79806760</guid><author>tianjun2012</author><pubDate>Tue, 03 Apr 2018 17:38:51 +0800</pubDate><description><![CDATA[windows10下关闭端口占用



windows10下关闭端口占用
占用查询端口的pid查询
关闭对应pid






很少用windows，但是本地开发一直都是windows系统，如果端口发生意外，没有被关闭，如下可解决。

占用查询端口的pid查询



C:\Users\helloworld&amp;gt;netstat -ano|findstr &quot;9097&quot;
  TCP    0.0.0....]]></description><category></category></item><item><title><![CDATA[springboot+mybatis+druid+postgreSql案例]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/79793116</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/79793116</guid><author>tianjun2012</author><pubDate>Mon, 02 Apr 2018 18:26:45 +0800</pubDate><description><![CDATA[springboot+mybatis+druid+postgreSql/mysql 示例



springboot+mybatis+druid+postgreSql/mysql 示例
简介
结构解析
代码剖析
entity层
Dao层
service层
协助层


启动







简介

最近加入一个新的团队，发现团队开发的底层调用时通过jdbctemplate拼凑sql，不能说这个不好，但...]]></description><category></category></item><item><title><![CDATA[通过反射来实现懒加载]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/79075344</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/79075344</guid><author>tianjun2012</author><pubDate>Tue, 16 Jan 2018 15:23:39 +0800</pubDate><description><![CDATA[代码写多了，得总结和借鉴，这是在kafka源码中拷贝下来的，一个小技巧： 
当时用的是jdk中的帮助类时，可以直接通过new关键字来实现， 
但是当依赖的时外部的类时，最好通过反射来实现，也就是说，只有在调用这个方法时才会去加载这个类以及相关的依赖



 // We should only have a runtime dependency on compression algorithms i]]></description><category></category></item><item><title><![CDATA[gradle构建工具构建web项目]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/79041054</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/79041054</guid><author>tianjun2012</author><pubDate>Fri, 12 Jan 2018 10:18:38 +0800</pubDate><description><![CDATA[越来越多的java项目通过gradle来构建，gradle比maven更加灵活，从github上拉下来的项目一堆都是通过gradle构建，不熟悉的化，会导致不小的麻烦，程序猿活到老学到老。 
初始搭建项目 
build.gradle如下：



group 'cmcc.cn.org'
version '1.0-SNAPSHOT'

apply plugin: 'java'
apply plugin]]></description><category></category></item><item><title><![CDATA[深入java重载和重写的内幕]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/78594757</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/78594757</guid><author>tianjun2012</author><pubDate>Tue, 21 Nov 2017 17:17:53 +0800</pubDate><description><![CDATA[重载Method Overload Resolution，也称Dispath，国内一般翻译为“静态分派”，但是严格来说，一般很少用再静态环境中。静态类方法中的演示package jdbc.test.es;/**
 * 方法静态分派演示
 * @author Administrator
 *
 */
public class MyStaticTest {    static abstract clas]]></description><category></category></item><item><title><![CDATA[elasticsearch选择器聚合，分组返回聚合结果]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/78582627</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/78582627</guid><author>tianjun2012</author><pubDate>Mon, 20 Nov 2017 15:21:50 +0800</pubDate><description><![CDATA[package tianjun.cmcc.es;import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;import org.apache.log4j.Logger;
import org]]></description><category></category></item><item><title><![CDATA[线程池技术简单实现]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/78443052</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/78443052</guid><author>tianjun2012</author><pubDate>Sat, 04 Nov 2017 13:35:25 +0800</pubDate><description><![CDATA[如上一篇数据库链接池看到，由于没有使用线程池技术，需要频繁的创建和销毁线程，这无疑十分消耗和浪费系统的资源，正式基于这个，我们可以探索，摸索线程池的原理，自己建立个简单的demo来解惑。]]></description><category></category></item><item><title><![CDATA[数据库链接池模型实现--多线程（非线程池）版]]></title><link>https://blog.youkuaiyun.com/tianjun2012/article/details/78443013</link><guid>https://blog.youkuaiyun.com/tianjun2012/article/details/78443013</guid><author>tianjun2012</author><pubDate>Sat, 04 Nov 2017 13:26:59 +0800</pubDate><description><![CDATA[链接池的实现##package chapter04.connection_pool.no_thread_pool;import java.sql.Connection;
import java.util.LinkedList;/**
 * Created by tianjun on 2017/11/4.
 */
public class ConnectionPool {
    private Li]]></description><category></category></item></channel></rss>