<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[programmerDingl的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/programmerDingl</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; programmerDingl]]></copyright><item><title><![CDATA[Log 监控报警系统]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/135926355</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/135926355</guid><author>programmerDingl</author><pubDate>Tue, 30 Jan 2024 09:58:17 +0800</pubDate><description><![CDATA[实时监控并只接收错误信息。启动fliebeat。启动logstash。]]></description><category></category></item><item><title><![CDATA[AirFlow环境搭建]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/128252134</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/128252134</guid><author>programmerDingl</author><pubDate>Fri, 09 Dec 2022 14:17:39 +0800</pubDate><description><![CDATA[AirFlow 安装]]></description><category></category></item><item><title><![CDATA[Apache Doris]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/126008875</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/126008875</guid><author>programmerDingl</author><pubDate>Wed, 27 Jul 2022 10:13:07 +0800</pubDate><description><![CDATA[Apache Doris]]></description><category></category></item><item><title><![CDATA[朴素贝叶斯算法]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123556662</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123556662</guid><author>programmerDingl</author><pubDate>Thu, 17 Mar 2022 17:54:29 +0800</pubDate><description><![CDATA[浅谈朴素贝叶斯算法
一. 对朴素贝叶斯的理解

朴素贝叶斯算法是通过考虑特征概率来预测分类。之所以有朴素二字，是因为需要对于所研究的特征维度进行一个假设各个特征维度之间是相互独立的，则朴素贝叶斯成立。
对于假设独立性来说，很多情况下属性之间相关性较大时，分类效果不坏。
这一假设使得朴素贝叶斯法变得简单，但有时会牺牲一定的分类准确率。

二. 朴素贝叶斯的三个阶段
朴素贝叶斯分类分为三个阶段：
第一阶段——准备工作阶段，这个阶段的任务是为朴素贝叶斯分类做必要的准备，主要工作是根据具体情况确定特征属性，并对每个]]></description><category></category></item><item><title><![CDATA[普罗米修斯监控]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123260712</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123260712</guid><author>programmerDingl</author><pubDate>Thu, 03 Mar 2022 18:14:40 +0800</pubDate><description><![CDATA[普罗米修斯全方位监控安装部署


安装普罗米修斯
wget
https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz
tar -xvf prometheus-2.3.2.linux-amd64.tar.gz
启动
./prometheus
网页端口 ip:9090


安装kafka_exporter
wget&nbsp;https://github.com/da]]></description><category></category></item><item><title><![CDATA[java对于金额为汉字的处理转换]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123181555</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123181555</guid><author>programmerDingl</author><pubDate>Mon, 28 Feb 2022 14:19:29 +0800</pubDate><description><![CDATA[/**
     * 大写金额转数字
     *
     */
    public static String ChineseConvertToNumber(String chineseAmount) {
        if (chineseAmount == null || chineseAmount.length() &lt;= 0 || chineseAmount == "") {
            return null;
        }
        //移除计算干扰文...]]></description><category></category></item><item><title><![CDATA[java正则表达式剔除html标签]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123178079</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123178079</guid><author>programmerDingl</author><pubDate>Mon, 28 Feb 2022 11:06:16 +0800</pubDate><description><![CDATA[package com.dl.util;

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class DelHTMLUtil {
    private static final String regEx_script = "&lt;script[^&gt;]*?&gt;[\\s\\S]*?&lt;\\/script&gt;"; // 定义script的正则表达式
    private static fina]]></description><category></category></item><item><title><![CDATA[编辑距离算法比较两个字符串内容不同]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123177401</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123177401</guid><author>programmerDingl</author><pubDate>Mon, 28 Feb 2022 10:59:04 +0800</pubDate><description><![CDATA[package com.adt.util;

/**
 * 标记前后内容不同
 */
public class StringHighLightDifferentUtil {

    /**
         * 得到标记的字符串
         */
    public static String[] getMark(String a, String b) {
        String[] temp = getTempString(a, b);
        String[] result = ]]></description><category></category></item><item><title><![CDATA[flink整合ActiveMQ之ActiveMQSource]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123177266</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123177266</guid><author>programmerDingl</author><pubDate>Mon, 28 Feb 2022 10:42:51 +0800</pubDate><description><![CDATA[package com.dl.activeMQ

import java.util.Date

import javax.jms._
import org.apache.activemq.ActiveMQConnectionFactory
import org.apache.flink.configuration.Configuration
import org.apache.flink.streaming.api.functions.source.{RichSourceFunction, SourceFu]]></description><category></category></item><item><title><![CDATA[CDH运维]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123176759</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123176759</guid><author>programmerDingl</author><pubDate>Mon, 28 Feb 2022 10:30:17 +0800</pubDate><description><![CDATA[Cloudera Management Service
2020.04.13 10:22
发出警告，修改java内存


配置结束后重启服务，偶尔会出现红色警报
Cloudera Manager Server GC cpu使用率为总处理时间的10％或更多。
过了一会儿，红色警报自动消失。
HDFS
2020.04.13 10:47
警告：NameNode和SecondaryNameNode有不同的heapsizes



修改SecondaryNameNode Java堆栈大小与 NameNode 相同 4]]></description><category></category></item><item><title><![CDATA[CDH集群安装步骤]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/123176512</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/123176512</guid><author>programmerDingl</author><pubDate>Mon, 28 Feb 2022 10:25:49 +0800</pubDate><description><![CDATA[CDH集群安装步骤
准备工作：集群之前已经装好CentOS7，防火墙也已经关闭，对于mysql所配置的则是在另一台节点之上，已创建好cloudera-manageer的初始化数据库。
1.准备相关安装包，将安装包上传/data/

2.vi /etc/hosts 修改主机映射
3.配置ssh免密登录
4.安装ntp服务，配置各个节点时间同步，两个从节点与主节点同步，主节点与阿里云时间同步
yum -y install ntp


5.禁用透明页(所有节点，优化可选）
echo never &gt; /sys]]></description><category></category></item><item><title><![CDATA[匹配除中文括号外的其他标点符号]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/122194466</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/122194466</guid><author>programmerDingl</author><pubDate>Tue, 28 Dec 2021 15:28:10 +0800</pubDate><description><![CDATA[[\pP\p{Punct}&amp;&amp;[^（]&amp;&amp;[^）]]


]]></description><category></category></item><item><title><![CDATA[flink程序打包上传yarn]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/118490127</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/118490127</guid><author>programmerDingl</author><pubDate>Mon, 05 Jul 2021 14:56:56 +0800</pubDate><description><![CDATA[]]></description><category></category></item><item><title><![CDATA[logstash抽取数据库发送到kafka，以及使用logstash的something]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/118489554</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/118489554</guid><author>programmerDingl</author><pubDate>Mon, 05 Jul 2021 14:40:34 +0800</pubDate><description><![CDATA[input {
    jdbc {
        jdbc_connection_string =&gt; "jdbc:mysql://ip:PORT/database?serverTimezone=Asia/Shanghai&amp;useUnicode=true&amp;characterEncoding=utf8&amp;useSSL=false&amp;allowMultiQueries
=true&amp;zeroDateTimeBehavior=convertToNull"
        ]]></description><category></category></item><item><title><![CDATA[一些常用的大数据命令（日常更新）]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/118488643</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/118488643</guid><author>programmerDingl</author><pubDate>Mon, 05 Jul 2021 14:10:13 +0800</pubDate><description><![CDATA[一些常用的大数据命令（日常更新）

note：由于本人经常忘记各种命令（包括一些简单的），所以做一个总结

Linux常用命令

查找大文件	ls -lh  $(find /data -type f -size +100M)
查看目录文件占用最大      du -h --max-depth=1或者 du -sh *
查看磁盘使用	df -h
查找文件		find / -name ‘filename’
查找目录		find / -name ‘path’ -type d
查找内容		find . | xarg]]></description><category></category></item><item><title><![CDATA[大数据知识点（持续更新中。。。）]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/108980086</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/108980086</guid><author>programmerDingl</author><pubDate>Fri, 09 Oct 2020 15:49:07 +0800</pubDate><description><![CDATA[hdfs

hdfs组件



Namenode : 存储系统元数据、 namespace、管理datanode、接受datanode状态汇报
Datanode: 存储块数据，响应客户端的块的读写，接收namenode的块管理理指令
Block: HDFS存储数据的基本单位，默认值是128MB，实际块大小0~128MB
Rack: 机架，对datanode所在主机的物理标识，标识主机的位置，优化存储和计算



为什么HDFS不适合小文件存储？












情况
Namenode占用
Datan]]></description><category></category></item><item><title><![CDATA[Kettle导出mysql数据到hive]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/107979011</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/107979011</guid><author>programmerDingl</author><pubDate>Thu, 13 Aug 2020 14:43:00 +0800</pubDate><description><![CDATA[Kettle导出mysql数据到hive
通过Kettle导出mysql数据到hive有两种方案：
​	1. 将hive表直接作为表输出进行


先将数据上传到hdfs，然后执行SQL语句load到hive


具体步骤如下：
(1）将hive安装目录下，lib里面的所有hive开头的jar包转移至kettle的大数据jar包目录下
​		data-integration\plugins\pentaho-big-data-plugin\hadoop-configurations\hdp25\lib
(2) ]]></description><category></category></item><item><title><![CDATA[logstash 消费数据到kafka异常]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/107429972</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/107429972</guid><author>programmerDingl</author><pubDate>Sat, 18 Jul 2020 15:39:32 +0800</pubDate><description><![CDATA[报错 ：[logstash.outputs.kafka ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=&gt;1, :failures=&gt;1, :sleep=&gt;0.1}
原因： logstash 日志报错生产数据到 kafka 失败
解决办法：
     查看kafka配置，默认单条消息最大为1M，当单条消息长度超过1M时，就会出现发送到broker失败，从而导致消息在producer的队列中一直累]]></description><category></category></item><item><title><![CDATA[解决idea本地打包检测到多个版本的Scala库问题 Multiple versions of scala libraries detected]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/107188708</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/107188708</guid><author>programmerDingl</author><pubDate>Tue, 07 Jul 2020 19:44:01 +0800</pubDate><description><![CDATA[报错如下：

解决方法：
pom文件
note：声明scala.binary.version与scala.version版本一致即可
      &lt;properties&gt;
        &lt;scala.version&gt;2.**&lt;/scala.version&gt;
        &lt;scala.binary.version&gt;2.**&lt;/scala.binary.version&gt;
    &lt;/properties&gt;
  &lt;plugin&g]]></description><category></category></item><item><title><![CDATA[数据分析平台对接阿里云大数据MaxCompute]]></title><link>https://blog.youkuaiyun.com/programmerDingl/article/details/104939905</link><guid>https://blog.youkuaiyun.com/programmerDingl/article/details/104939905</guid><author>programmerDingl</author><pubDate>Wed, 18 Mar 2020 10:59:49 +0800</pubDate><description><![CDATA[数据分析平台对接MaxCompute
一．Mysql全量，增量实时同步到MaxCompute
方案一 从MySQL直接实时同步到MaxCompute
在MaxCompute的DataWorks控制台的数据集成模块可以实现实时同步，该模块的实时同步功能是根据Mysql Binlog日志进行数据的同步，但是该模块在测试中，并未开放，可以向阿里云服务的人员申请功能为自己的账号开放，即可配置使用。
该实时...]]></description><category></category></item></channel></rss>