<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[DaiSnow的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/AlierSnow</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; AlierSnow]]></copyright><item><title><![CDATA[Prometheus配置WEB登录使用账密认证【basic-auth】]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/157551971</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/157551971</guid><author>AlierSnow</author><pubDate>Fri, 30 Jan 2026 17:42:04 +0800</pubDate><description><![CDATA[背景：默认的prometheus的web ui界面无需授权登录即可访问，为提高安全性，增加账密登录。prometheus官网文档：https://prometheus.io/docs/guides/basic-auth/]]></description><category></category></item><item><title><![CDATA[基于Python3的Selenium、BeautifulSoup实现网页数据爬取写入MySQL]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/157511807</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/157511807</guid><author>AlierSnow</author><pubDate>Fri, 30 Jan 2026 09:20:08 +0800</pubDate><description><![CDATA[背景：在数据采集阶段，有些场景需要定期从网站获取公开数据，故而进行网页爬虫开发。技术手段：selenium、beautifulsoup +ChromeDriver驱动包获取：https://googlechromelabs.github.io/chrome-for-testing/]]></description><category></category></item><item><title><![CDATA[DataX 新增Transformer 自定义转换器]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/157509726</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/157509726</guid><author>AlierSnow</author><pubDate>Thu, 29 Jan 2026 15:41:59 +0800</pubDate><description><![CDATA[在core/src/main/java/com/alibaba/datax/core/transport/transformer中新增自定义转换器类，（SM4加密）EncryptionSm4Transformer.java1、继承Transformer2、设置转换器名称3、重写evaluate方法。/***///SM4-加密//指明加密算法和秘钥@Overridetry {try {// 加密。]]></description><category></category></item><item><title><![CDATA[DataX更新hdfswriter、hdfsreader插件支持parquet]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/157508937</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/157508937</guid><author>AlierSnow</author><pubDate>Thu, 29 Jan 2026 14:37:36 +0800</pubDate><description><![CDATA[参考 https://github.com/wgzhao/Addax 魔改 DataX。github下载DataX源码，得到Datax-master项目源码。]]></description><category></category></item><item><title><![CDATA[DataX ElasticsearchWriter 插件改造]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/157506328</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/157506328</guid><author>AlierSnow</author><pubDate>Thu, 29 Jan 2026 11:47:36 +0800</pubDate><description><![CDATA[针对DataX自带的ElasticsearchWriter 插件，按照业务使用需要进行针对性改造。]]></description><category></category></item><item><title><![CDATA[数据治理DataHub安装部署]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/157503857</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/157503857</guid><author>AlierSnow</author><pubDate>Thu, 29 Jan 2026 10:55:53 +0800</pubDate><description><![CDATA[通过DataHub进行数据目录管理、数据治理、追踪数据血缘，分析数据集概况。github地址：https://github.com/datahub-project/datahub官网地址：https://datahubproject.io/docs/ datahub 模块介绍。]]></description><category></category></item><item><title><![CDATA[CDH集成Atlas]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/157503457</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/157503457</guid><author>AlierSnow</author><pubDate>Thu, 29 Jan 2026 09:52:27 +0800</pubDate><description><![CDATA[大数据平台进行数据治理需要，采用Apache Atlas进行数据治理。下载Atlas2.1.0版本源码包。前提：基于CDH（本文使用版本为6.3.2）集群搭建大数据集群，组件服务包含Hdfs、Hive、Hbase、Solr、Kafka、Sqoop、Zookeeper、Impala、Yarn、Spark、Oozie、Phoenix、Hue等。平台没有内置Atlas的Parcel需要额外扩展集成包。下载到windows。]]></description><category></category></item><item><title><![CDATA[IP地址解析为省、市、区县]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/152045930</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/152045930</guid><author>AlierSnow</author><pubDate>Wed, 24 Sep 2025 17:28:25 +0800</pubDate><description><![CDATA[IP地址解析]]></description><category></category></item><item><title><![CDATA[Flink CDC Oracle 用户权限不足 ORA-01031: insufficient privileges]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/127531172</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/127531172</guid><author>AlierSnow</author><pubDate>Wed, 26 Oct 2022 14:36:16 +0800</pubDate><description><![CDATA[Flink cdc 实时抽取Oracle数据表，提示：ORA-01031: insufficient privileges]]></description><category></category></item><item><title><![CDATA[字符集中文乱码转换为UTF8，如：å\u0085¬è¯\u0081ä¸\u009Aå\u008A¡ç±»å\u0088«æ\u009C\u0089è¯¯ï¼\u0081]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/126877981</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/126877981</guid><author>AlierSnow</author><pubDate>Thu, 15 Sep 2022 19:28:58 +0800</pubDate><description><![CDATA[字符集中文乱码转换为UTF8，如：å\u0085¬è¯\u0081ä¸\u009Aå\u008A¡ç±»å\u0088«æ\u009C\u0089è¯¯ï¼\u0081。]]></description><category></category></item><item><title><![CDATA[oracle 查找、替换 字符串中的 换行符]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/125210964</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/125210964</guid><author>AlierSnow</author><pubDate>Fri, 10 Jun 2022 10:00:39 +0800</pubDate><description><![CDATA[oracle 中实现字符串中查找、替换换行符 chr(10)、chr(13)等。]]></description><category></category></item><item><title><![CDATA[org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde Maven下载不下来]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/125105548</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/125105548</guid><author>AlierSnow</author><pubDate>Thu, 02 Jun 2022 18:51:43 +0800</pubDate><description><![CDATA[找了很多文章都提供以下两种方式
1）设置maven镜像仓库
2）pom文件增加
不过我试了都没有解决问题。
最终直接下载缺失的包，地址如下
https://public.nexus.pentaho.org/#browse/search=keyword=pentaho-aggdesigner-algorithm
将jar和pom放到maven对应路径下即可。
pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
pentaho-aggdesigner-algorithm-]]></description><category></category></item><item><title><![CDATA[Apache Http Server安全漏洞解决]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/124317535</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/124317535</guid><author>AlierSnow</author><pubDate>Thu, 21 Apr 2022 13:50:47 +0800</pubDate><description><![CDATA[场景：公司使用的大数据集群云服务器，安全扫描出严重漏洞，扫描漏洞如下：
安全漏洞扫描报告




端口
协议
服务
严重等级
漏洞




–
ICMP
–
弱
ICMP timestamp请求响应漏洞


–
UDP
–
弱
允许Traceroute探测


80
TCP
http
严重 严重 严重 严重 严重 弱
OpenSSL 安全漏洞(CVE-2022-0778) Apache HTTP Server 环境问题漏洞(CVE-2022-22720) Apache HTTP Server 输入验证错误漏]]></description><category></category></item><item><title><![CDATA[基于CM管理的CDH6.3.2集群集成Atlas2.1.0]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/123730617</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/123730617</guid><author>AlierSnow</author><pubDate>Fri, 25 Mar 2022 11:18:42 +0800</pubDate><description><![CDATA[基于CM管理的CDH6.3.2集群集成Atlas2.1.0
大数据平台进行数据治理需要，采用Apache Atlas进行数据治理。下载Atlas2.1.0版本源码包。下载https://www.apache.org/dyn/closer.cgi/atlas/2.1.0/apache-atlas-2.1.0-sources.tar.gz 到windows。
前提CDH集群已经搭建完成，组件服务包含Hdfs、Hive、Hbase、Solr、Kafka、Sqoop、Zookeeper、Impala、Yarn、Sp]]></description><category></category></item><item><title><![CDATA[基于CM管理的CDH集群集成Phoenix]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/123706560</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/123706560</guid><author>AlierSnow</author><pubDate>Thu, 24 Mar 2022 11:29:03 +0800</pubDate><description><![CDATA[基于CM管理的CDH集群集成Phoenix
背景介绍：原有的CDH集群，以Hive+Hbase+Impala进行数据处理查询，Hbase而言语法难懂，不同于寻常的SQL，Impala而言速度查询速度太慢且无法进行记录的修改。故而考虑用Phoenix来集成Hbase、Hive以解决上述问题。
前提：已经基于CM安装搭建CDH集群，本文采用parcels包的方式集成Phoenix到CDH集群。
下载Phoenix Parcel包和Jar包并上传到CM-Server节点
本文采用的phoenix版本包如下：
..]]></description><category></category></item><item><title><![CDATA[windows Server 2012 安装导入opencv-python 提示DLL找不到]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/122174486</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/122174486</guid><author>AlierSnow</author><pubDate>Mon, 27 Dec 2021 16:37:54 +0800</pubDate><description><![CDATA[项目背景：win server 2012 R2 + python3.6+opencv4.4.0+ffmpeg 实现视频图像处理
问题描述：import cv2 报错dll找不到
1.pip命令安装opencv：pip --default-timeout=100 install opencv-python==4.4.0.46 -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com
2.通过pip list 查看模块列表发现 numpy和]]></description><category></category></item><item><title><![CDATA[基于CDH6.2.0 搭建大数据集群]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/121809578</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/121809578</guid><author>AlierSnow</author><pubDate>Thu, 09 Dec 2021 11:56:50 +0800</pubDate><description><![CDATA[Cloudera Manager集群搭建
hostname按表格进行修改，/etc/hosts文件添加节点 Ip+hostname




hostname
ip
role




cdh-nn01
172.16.0.188
HadoopMaster，CM Server，Mysql，Hue，HS2


cdh-nn02
172.16.0.189
Hadoop Master，CMAgent，HS2


cdh-nn03
172.16.0.190
Hadoop Master，CMAgent，HS2,Impala]]></description><category></category></item><item><title><![CDATA[org.apache.parquet.schema.InvalidSchemaException: A group type can not be empty. Parquet does not su]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/118966472</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/118966472</guid><author>AlierSnow</author><pubDate>Wed, 21 Jul 2021 17:58:01 +0800</pubDate><description><![CDATA[1.读取cvs文件数据写入parquet文件
	val spark =SparkSession.builder().appName("test").master("local").getOrCreate()
	val file = "hdfs://clusters/test/demo.csv"
    val frame = spark.read.option("header","true").csv(file)
    val newfile = file.split("\\.")(0)+".parque]]></description><category></category></item><item><title><![CDATA[Java 基于poi 读写excel文件（ *.xlsx）]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/117075219</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/117075219</guid><author>AlierSnow</author><pubDate>Tue, 08 Jun 2021 18:25:12 +0800</pubDate><description><![CDATA[Java 通过poi读取excel文件需要使用XSSFWorkbook对象，需要如下依赖包
    &lt;dependency&gt;
      &lt;groupId&gt;org.apache.poi&lt;/groupId&gt;
      &lt;artifactId&gt;poi&lt;/artifactId&gt;
      &lt;version&gt;4.1.0&lt;/version&gt;
    &lt;/dependency&gt;

    &lt;dependency&g]]></description><category></category></item><item><title><![CDATA[org.apache.hadoop.security.AccessControlException]]></title><link>https://blog.youkuaiyun.com/AlierSnow/article/details/117514917</link><guid>https://blog.youkuaiyun.com/AlierSnow/article/details/117514917</guid><author>AlierSnow</author><pubDate>Thu, 03 Jun 2021 11:28:51 +0800</pubDate><description><![CDATA[通过spark程序HDFS进行文件写入操作时，程序报错org.apache.hadoop.security.AccessControlException: Permission denied: user=admin, access=WRITE, inode="/mydata":root:supergroup:drwxr-xr-x，根据提示可知是程序使用的是admin用户，它没有写权限。
报错信息如下：
Exception in thread "main" org.apache.hadoop.security]]></description><category></category></item></channel></rss>