<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[AaronLwx的博客]]></title><description><![CDATA[大数据]]></description><link>https://blog.youkuaiyun.com/xiaoxiongaa0</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; xiaoxiongaa0]]></copyright><item><title><![CDATA[基于Spring Boot搭建SOA架构级别的Web项目]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104276538</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104276538</guid><author>xiaoxiongaa0</author><pubDate>Wed, 12 Feb 2020 18:27:10 +0800</pubDate><description><![CDATA[Spring Boot版本选择为2.1.8

先在本地创建一个空的项目faces，然后导进Idea













创建faces-parent项目









在pom文件添加如下代码


    &lt;properties&gt;
        &lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourc...]]></description><category></category></item><item><title><![CDATA[Spring Boot整合MyBatis和Druid]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104265232</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104265232</guid><author>xiaoxiongaa0</author><pubDate>Tue, 11 Feb 2020 20:57:22 +0800</pubDate><description><![CDATA[快速创建Spring Boot应用





暂时先添加这么多依赖





项目生成之后打开pom文件看看



Spring Boot导入不进来，百度了一下，说要把*.lastUpdated文件删除。但我并没那么做，先去我的仓库看一看



发现Spring Boot的版本是2.1.8.RELEASE，估计是我之前已经安装过，所以我在新项目的文件把Spring Boot的版本改为2.1.8.R...]]></description><category></category></item><item><title><![CDATA[各种日志框架使用总结]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104128006</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104128006</guid><author>xiaoxiongaa0</author><pubDate>Fri, 31 Jan 2020 23:24:30 +0800</pubDate><description><![CDATA[目前市面上有如下日志框架：JUL、JCL、Jboss-logging、logback、log4j、log4j2、SLF4J等等


			日志的抽象层
			
			
			日志实现
			
		
			JCL(Jakarta Commons Logging) SLF4J(Simple Logging Facade for Java) jboss-logging
			
			
			log4j ...]]></description><category></category></item><item><title><![CDATA[使用Canal同步MySQL到ES]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104116866</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104116866</guid><author>xiaoxiongaa0</author><pubDate>Fri, 31 Jan 2020 10:40:36 +0800</pubDate><description><![CDATA[[hadoop@hadoop003 software]$ tar -xzvf elasticsearch-7.4.0-linux-x86_64.tar.gz -C ../app/

[hadoop@hadoop003 software]$ ln -s elasticsearch-7.4.0 elasticsearch

[hadoop@hadoop003 software]$ sudo v...]]></description><category></category></item><item><title><![CDATA[Canal部署]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104115674</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104115674</guid><author>xiaoxiongaa0</author><pubDate>Thu, 30 Jan 2020 16:50:23 +0800</pubDate><description><![CDATA[安装MySQL


[hadoop@hadoop003 canal-admin]$ sudo yum -y localinstall https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm

[hadoop@hadoop003 canal-admin]$ sudo yum -y install mysql-commu...]]></description><category></category></item><item><title><![CDATA[Wormhole部署&使用]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104105478</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104105478</guid><author>xiaoxiongaa0</author><pubDate>Wed, 29 Jan 2020 17:37:09 +0800</pubDate><description><![CDATA[[hadoop@hadoop003 software]$ tar -zxf wormhole-0.6.2.tar.gz -C ../app/

[hadoop@hadoop003 app]$ ln -s wormhole-0.6.2 wormhole

[hadoop@hadoop003 app]$ sudo vi /etc/profile

export WORMHOLE_HOME=/h...]]></description><category></category></item><item><title><![CDATA[Kylin初探]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104080439</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104080439</guid><author>xiaoxiongaa0</author><pubDate>Sat, 25 Jan 2020 22:23:46 +0800</pubDate><description><![CDATA[[root@hadoop001 ~]# $KYLIN_HOME/bin/sample.sh



一开始会报错，


ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache....]]></description><category></category></item><item><title><![CDATA[搭建DophinScheduler]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104081140</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104081140</guid><author>xiaoxiongaa0</author><pubDate>Fri, 24 Jan 2020 23:18:45 +0800</pubDate><description><![CDATA[各服务所在节点部署如下


			服务
			
			
			node01
			
			
			node02
			
			
			node03
			
		
			master
			
			
			√
			
			
			√
			
			
		
			worker/logServer
			
			
			
			√
			
			
			√
			
		
			alertServe...]]></description><category></category></item><item><title><![CDATA[搭建Kylin集群]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104078539</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104078539</guid><author>xiaoxiongaa0</author><pubDate>Fri, 24 Jan 2020 13:10:18 +0800</pubDate><description><![CDATA[[hadoop@hadoop001 software]$ tar -zxvf apache-kylin-2.6.4-bin.tar.gz -C ../app/

[hadoop@hadoop001 app]$ ln -s apache-kylin-2.6.4-bin kylin



准备Spark包

首先，在Ambari中安装Spark。


[hadoop@hadoop001 app...]]></description><category></category></item><item><title><![CDATA[深入使用DBus]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104073976</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104073976</guid><author>xiaoxiongaa0</author><pubDate>Thu, 23 Jan 2020 20:37:05 +0800</pubDate><description><![CDATA[中间的MySQL(hadoop002)即是主又是从，一定要及得加上log_slave_updates配置，否则在 MySQL主节点(hadoop001)上插入数据，MySQL从虽然可以同步数据，但是它不会级联生成binlog日志， canal就采集不到数据了。

在hadoop001节点上执行如下命令


[hadoop@hadoop001 ~]$ sudo vim /etc/my.cnf

se...]]></description><category></category></item><item><title><![CDATA[DBus集群部署]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104066335</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104066335</guid><author>xiaoxiongaa0</author><pubDate>Wed, 22 Jan 2020 01:25:07 +0800</pubDate><description><![CDATA[部署InfluxDB


[root@hadoop003 software]# wget  https://dl.influxdata.com/influxdb/releases/influxdb-1.1.0.x86_64.rpm

[root@hadoop003 software]# yum -y localinstall influxdb-1.1.0.x86_64.rpm




[root@...]]></description><category></category></item><item><title><![CDATA[搭建Ambari集群]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104061897</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/104061897</guid><author>xiaoxiongaa0</author><pubDate>Tue, 21 Jan 2020 21:48:28 +0800</pubDate><description><![CDATA[[root@hadoop001 ~]# visudo






[root@hadoop001 ssh]# useradd hadoop



SSH免密


[hadoop@hadoop001 ~]$ ssh-keygen

[hadoop@hadoop001 ~]$ cd .ssh

[hadoop@hadoop001 .ssh]$ pwd
/home/hadoop/.ssh

[h...]]></description><category></category></item><item><title><![CDATA[Ambari部署]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/103302634</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/103302634</guid><author>xiaoxiongaa0</author><pubDate>Sun, 01 Dec 2019 22:43:24 +0800</pubDate><description><![CDATA[生成ssh key


[root@hadoop001 .ssh]# ssh-keygen

[root@hadoop001 .ssh]# cat id_rsa.pub &gt;&gt; authorized_keys

[root@hadoop001 .ssh]# chmod 700 ~/.ssh

[root@hadoop001 .ssh]# chmod 600 ~/.ssh/author...]]></description><category></category></item><item><title><![CDATA[Hive 包版本引发的血案]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/100014831</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/100014831</guid><author>xiaoxiongaa0</author><pubDate>Thu, 22 Aug 2019 15:04:15 +0800</pubDate><description><![CDATA[刚才打开自己在阿里云的一台服务器，并准备在上面测试Hive的一些功能，原以为很顺利，没想到一上来一个show databases;就报错了，如图



奇怪了，以前是没有问题的啊，为啥会出现这个错误呢？

定睛一看，十有八九是jar包冲突了，问题就是出在guava这个包里头，于是跑去hive下的lib上看一看



会不会guava的版本太高或者太低了，先试一试。



先后把guava-13.0...]]></description><category></category></item><item><title><![CDATA[Flink清洗日志服务SLS的数据并求ACU&PCU]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/99648125</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/99648125</guid><author>xiaoxiongaa0</author><pubDate>Mon, 19 Aug 2019 12:26:42 +0800</pubDate><description><![CDATA[上文说到为什么使用Flink实时消费阿里云日志服务SLS的数据，并把阿里云上Flink消费SLS的代码粘贴到本地，做了相关修改之后成功把整个流程跑通了。但仅仅这样是不够的，从控制台上面输出的数据来看是个比较难看的字符串，可以说没多大用处。因此本文主要是继续使用Flink来对从日志服务SLS过来的数据做一系列的清洗，然后再计算几个工作中的指标。

相关ETL代码如下，就是把需要使用到的各个字段提取出...]]></description><category></category></item><item><title><![CDATA[Flink源码阅读之生成StreamGraph]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/99086610</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/99086610</guid><author>xiaoxiongaa0</author><pubDate>Sat, 10 Aug 2019 17:12:38 +0800</pubDate><description><![CDATA[首先，从一个流式计算的例子WordCount来入手。

我们知道这个程序只有env调用了execute方法才会正式的执行，那么execute方法里面一开始会先创建StreamGraph，然后再生成JobGraph，接着生成ExecutionGraph，最后生成物理执行计划。下面先介绍StreamGraph是如何生成的。



点击进入execute















通过调用这个...]]></description><category></category></item><item><title><![CDATA[Spark官网翻译--Quick Start]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/98940067</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/98940067</guid><author>xiaoxiongaa0</author><pubDate>Fri, 09 Aug 2019 11:20:27 +0800</pubDate><description><![CDATA[http://spark.apache.org/docs/latest/quick-start.html

Quick Start

Security
	Interactive Analysis with the Spark Shell
	Basics
		More on Dataset Operations
		Caching
	
	Self-Contained Applications
	Wh...]]></description><category></category></item><item><title><![CDATA[Flink实时消费阿里云日志服务SLS的数据]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/98856265</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/98856265</guid><author>xiaoxiongaa0</author><pubDate>Thu, 08 Aug 2019 14:29:19 +0800</pubDate><description><![CDATA[之前调研打算使用Spark Streaming来消费阿里云日志服务SLS的日志，具体的架构实现见之前的博客，大概流程就是使用Flume去收集阿里云日志服务SLS的数据，把采集到的数据sink到Kafka，最后Spark Streaming来消费。

咋一看上面整个链条太长，其实也是可以直接使用Spark Streaming来消费阿里云日志服务SLS的数据，但个人感觉自从阿里云收购了Flink之后，...]]></description><category></category></item><item><title><![CDATA[Offline离线项目效果演示]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/98508619</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/98508619</guid><author>xiaoxiongaa0</author><pubDate>Wed, 07 Aug 2019 17:47:23 +0800</pubDate><description><![CDATA[我们先从下面最简单的Demo入手，一步一步迭代


  // val time = sc.getConf.get("spark.time", "")
val time = "2019-06-08 10:14:09"
val parsedTime = DateUtils.parseToMinute(time)
val day = DateUtils.getDay(parsedTime) // "20...]]></description><category></category></item><item><title><![CDATA[Spark Streaming使用MySQL存储offset保证Exactly Once语义]]></title><link>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/97941306</link><guid>https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/97941306</guid><author>xiaoxiongaa0</author><pubDate>Fri, 02 Aug 2019 19:30:51 +0800</pubDate><description><![CDATA[1.准备

1.1 本次测试的运行环境如下

&lt;spark.version&gt;2.3.0&lt;/spark.version&gt;
	&lt;kafka.version&gt;0.10.0-kafka-2.1.1&lt;/kafka.version&gt;
	&lt;scala.version&gt;2.11.8&lt;/scala.version&gt;
	&lt;hadoop.ve...]]></description><category></category></item></channel></rss>