<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[chenzl的专栏]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/liangkiller</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; liangkiller]]></copyright><item><title><![CDATA[openssl 创建root证书]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/119677552</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/119677552</guid><author>liangkiller</author><pubDate>Fri, 13 Aug 2021 15:57:39 +0800</pubDate><description><![CDATA[作为CA,要有私钥(private keys)和公共证书(public certificates)这一密钥对;
最先要创建的是root密钥对;包括root私钥(ca.key.pem)和root证书(ca.cert.pem); 这些标识了CA
root CA不直接签发服务端或客户端证书,而是签发中间CA(intermediate CAs),代表root CA签发证书;这可以使root私钥离线保存,并尽可能少的使用; 私钥的泄漏是灾难性的;
准备目录
创建存储root密钥和证书的目录
mkdir /root/c]]></description><category></category></item><item><title><![CDATA[IntelliJ IDEA 开发zookeeper环境设置]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/115912885</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/115912885</guid><author>liangkiller</author><pubDate>Tue, 20 Apr 2021 17:07:53 +0800</pubDate><description><![CDATA[zookeeper版本是: apache-zookeeper-3.7.0
IntelliJ 版本是: IntelliJ IDEA 2021.1 (Community Edition)
1.使用IntelliJ 打开zookeeper源码目录:

2. 将conf文件夹下的zoo_sample.cfg复制为zoo.cfg

3. 创建数据存储目
右击apache-zookeeper-3.7.0[parent]， New-&gt;Directory，输入tmp/zookeeper

4. 编译zookeeper]]></description><category></category></item><item><title><![CDATA[pandas.Series.add参数fill_value]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/112993675</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/112993675</guid><author>liangkiller</author><pubDate>Fri, 22 Jan 2021 17:50:34 +0800</pubDate><description><![CDATA[fill_value: None或浮点数;填充NaN值;如果元素在2个series都是NaN,结果还是NaN;
pandas的一个重要功能,就是索引对齐,即在2个series里添加对应虚列,值为NaN
示例
a = pd.Series([1, 1, 1, np.nan], index=['a', 'b', 'c', 'd'])
b = pd.Series([1, np.nan, 1, np.nan], index=['a', 'b', 'd', 'e'])
a.add(b,fill_value=0)
---]]></description><category></category></item><item><title><![CDATA[kafka开启kerberos,报错server not found in kerberos database]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/107707159</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/107707159</guid><author>liangkiller</author><pubDate>Fri, 31 Jul 2020 10:20:17 +0800</pubDate><description><![CDATA[kafka开启了SASL(kerberos), server.properties配置为
sasl.enabled.mechanisms: GSSAPI
security.inter.broker.protocol: SASL_PLAINTEXT
ssl.mode.enable: false
allow.everyone.if.no.acl.found: true
sasl.port: 19092

服务端的jaas.conf内容为
KafkaServer {
com.sun.security.auth.m]]></description><category></category></item><item><title><![CDATA[Hadoop单机单实例安装]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/107386776</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/107386776</guid><author>liangkiller</author><pubDate>Thu, 16 Jul 2020 16:49:02 +0800</pubDate><description><![CDATA[国内镜像
//清华大学
https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/
//北京理工大学
http://mirror.bit.edu.cn/apache/hadoop/common/

JDK安装
yum remove java* -y
yum remove jdk* -y
wget  https://github.com/frekele/oracle-java/releases/download/8u212-b10/jdk-8u212-]]></description><category></category></item><item><title><![CDATA[hadoop HTTP 查询 JMX]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/107386581</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/107386581</guid><author>liangkiller</author><pubDate>Thu, 16 Jul 2020 16:43:02 +0800</pubDate><description><![CDATA[在Namenode information的web端口(默认9870)访问JMX
在页面的Utilites-&gt;metrics
http://192.168.19.160:9870/jmx

//通过qry参数来过滤MXBeanName(即json的name字段)
http://192.168.19.160:9870/jmx?qry=Hadoop:*
//查询具体的name
http://192.168.19.160:9870/jmx?qry=Hadoop:service=NameNode,name=J]]></description><category></category></item><item><title><![CDATA[prometheus metric_relabel_configs 详解加示例]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/105759782</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/105759782</guid><author>liangkiller</author><pubDate>Sat, 25 Apr 2020 23:49:37 +0800</pubDate><description><![CDATA[metric_relabel_configs
对拉取的指标样本数据进行标签处理;和relabel_config有相同的配置和动作;
无法应用于自动生成的时间序列指标,如up
一个用处是屏蔽太昂贵的时序数据;
详细解释参看: prometheus relabel_config 详解加示例
以prometheus的指标prometheus_engine_queries_concurrent_max为例...]]></description><category></category></item><item><title><![CDATA[prometheus relabel_config 详解加示例]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/105758857</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/105758857</guid><author>liangkiller</author><pubDate>Sat, 25 Apr 2020 23:03:46 +0800</pubDate><description><![CDATA[relabel_config配置
Relabeling(重定义标签),是在拉取(scraping)阶段前,修改target和它的labels;
在每个scrape_configs可以定义多个重定义标签的步骤;
默认的, target的job标签设置为配置文件里的job_name的值;
__address__设置为配置里的targets的值;
而instance标签的值,是重定义标签操作之后__ad...]]></description><category></category></item><item><title><![CDATA[SQL 练习]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103660554</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103660554</guid><author>liangkiller</author><pubDate>Mon, 23 Dec 2019 09:54:44 +0800</pubDate><description><![CDATA[1.  SQL 执行的顺序
FROM [ left_table ]
ON &lt; join_condition &gt;
&lt; join_type &gt; JOIN &lt; right_table &gt;
WHERE &lt; where_condition &gt;
GROUP BY &lt; group_by_list &gt;
WITH &lt; CUBE | RollUP &g...]]></description><category></category></item><item><title><![CDATA[将csv转为parquet]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103121837</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103121837</guid><author>liangkiller</author><pubDate>Mon, 18 Nov 2019 14:26:39 +0800</pubDate><description><![CDATA[csv内容
$ cat test.txt
1|2|3|test
2|4|6|wwww

使用pyspark
from pyspark import SparkContext,SparkConf
from pyspark.sql import SQLContext
from pyspark.sql.types import *

if __name__ == "__main__":
    sc =...]]></description><category></category></item><item><title><![CDATA[crontab PATH 问题]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103121259</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103121259</guid><author>liangkiller</author><pubDate>Mon, 18 Nov 2019 14:00:36 +0800</pubDate><description><![CDATA[背景
使用anaconda3的虚拟环境python2.7,crontab使用bash调用python,还是系统默认的2.6;
诊断
应用是环境PATH问题
测试脚本
$ cd /var/tmp
$ vi t.sh
#!/bin/bash
echo $PATH
pythonv -V

$ crontab -e
* * * * * cd /var/tmp &amp;&amp; sh t.sh &gt;...]]></description><category></category></item><item><title><![CDATA[使用Crontab调度计划任务]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103120238</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103120238</guid><author>liangkiller</author><pubDate>Mon, 18 Nov 2019 12:04:46 +0800</pubDate><description><![CDATA[Linux计划任务,可以按分钟,小时,天,月,星期几调度任务;
配置文件
有2种类型的配置文件

系统范围的计划任务
单用户的计划任务

Linux的分发版本不同,配置文件的路径会不同;
CentOS用户的计划任务文件在/var/spool/cron目录下;
Ubuntu在/cron/crontabs目录下;
系统范围的计划任务文件,有2个目录

/etc/crontab,系统范围的用户任务;
/...]]></description><category></category></item><item><title><![CDATA[从mysql导入hive脚本]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103005774</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103005774</guid><author>liangkiller</author><pubDate>Mon, 11 Nov 2019 10:08:09 +0800</pubDate><description><![CDATA[#!/bin/bash
#全量更新kn1_base_conf_group

#环境设置:u 不存在的变量报错;e 发生错误退出;pipefail 管道有错退出
set -euo pipefail

#########要更改变的变量#######
#mysql数据库信息
MYSQL_HOST="192.168.1.138"
MYSQL_PORT="63751"
MYSQL_USER='user'
M...]]></description><category></category></item><item><title><![CDATA[impala建表脚本]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103005738</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103005738</guid><author>liangkiller</author><pubDate>Mon, 11 Nov 2019 10:05:40 +0800</pubDate><description><![CDATA[IMPALA="sudo -u hdfs impala-shell -q"

db_name="chenzl"
table_name="kn1_base_conf_group"

$IMPALA&lt;&lt; EOF
USE ${db_name};
DROP TABLE ${table_name};
CREATE EXTERNAL TABLE ${table_name}
(
day_key   ...]]></description><category></category></item><item><title><![CDATA[hive建表脚本]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103005546</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103005546</guid><author>liangkiller</author><pubDate>Mon, 11 Nov 2019 09:55:57 +0800</pubDate><description><![CDATA[HIVE="sudo -u hdfs hive"

db_name="chenzl"
table_name="kn1_base_conf_group"

$HIVE&lt;&lt; EOF
USE ${db_name};
DROP TABLE ${table_name};
CREATE EXTERNAL TABLE ${table_name}
(
day_key               dat...]]></description><category></category></item><item><title><![CDATA[批量指定impalad的执行主机]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103005395</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103005395</guid><author>liangkiller</author><pubDate>Mon, 11 Nov 2019 09:47:23 +0800</pubDate><description><![CDATA[#!/bin/bash
#批量指定impalad的执行主机

DIR="/home/chenzl/data/BI"
LOG="/home/chenzl/tmp/log.log"


rm -f $LOG
`touch $LOG`

find $DIR -name "*.sh" | while read f; do
    # 有HIVE_DB字符串
    grep "HIVE_DB" $f &g...]]></description><category></category></item><item><title><![CDATA[在hive脚本末尾添加impala REFRESH]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/103005187</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/103005187</guid><author>liangkiller</author><pubDate>Mon, 11 Nov 2019 09:36:20 +0800</pubDate><description><![CDATA[在impala shell脚本的末尾,添加REFRESH db.tbname
#!/bin/bash
#有HIVE_DB和HIVE_TABLE变量的sh文件添加impala 命令

DIR="/home/chenzl/data/BI"
LOG="/home/chenzl/tmp/log.log"


rm -f $LOG
`touch $LOG`

find $DIR -name "*.sh" |...]]></description><category></category></item><item><title><![CDATA[azkaban 安装]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/102950007</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/102950007</guid><author>liangkiller</author><pubDate>Fri, 08 Nov 2019 10:15:24 +0800</pubDate><description><![CDATA[在hadoop领域，常见工作流调度系统有：Oozie,Azkaban,Cascading,Hamake
环境

centos 6.5
JDK 8
azkaban 3.79





hostname
ip




vps138
192.168.1.138


vps155
192.168.1.155


vps156
192.168.1.156


下载
$ cd /var/tmp
$ wget ...]]></description><category></category></item><item><title><![CDATA[MySQL 安装并修改密码策略]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/102948294</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/102948294</guid><author>liangkiller</author><pubDate>Thu, 07 Nov 2019 09:58:11 +0800</pubDate><description><![CDATA[mysql 安装,国内镜像,修改默认密码策略]]></description><category></category></item><item><title><![CDATA[CDH 生产环境安装]]></title><link>https://blog.youkuaiyun.com/liangkiller/article/details/102936916</link><guid>https://blog.youkuaiyun.com/liangkiller/article/details/102936916</guid><author>liangkiller</author><pubDate>Wed, 06 Nov 2019 16:00:08 +0800</pubDate><description><![CDATA[环境

centos 6.5
JDK 8
CHD 5.16
MySQL 5.7
python.2.6 (hue)

以三台vps为例: vps138(主),vps155(从),vps156(从)
Hostname设置
三台的hosts都要相同
$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localh...]]></description><category></category></item></channel></rss>