Movle

一.flume概览 1.概述:     Flume是一种分布式，可靠且可用的服务，用于有效地收集，聚合和移动大量日志数据。它具有基于流数据流的简单灵活的架构。它具有可靠的可靠性机制和许多故障 转移和恢复机制，具有强大的容错性。它使用简单的可扩展数据模型，允许在线分析应用程序。 2.大数据架构 数据采集(爬虫\日志数据\flume) 数据存储(hdfs/hive/hbase(nosql)) 数据计算(mapreduce/hive/sparkSQL/sparkStrea

1.下载压缩包 2.上传到linux:/opt/software 3.解压 cd /opt/software tar -zxvf apache-flume-1.6.0-bin.tar.gz -C /opt/moudule 4.重命名 cd /opt/module/flume/conf mv flume-env.sh.template flume-env.sh 5.修改配置 vi flume-env.sh 修改内容如下： export JAVA_HOME=/opt/module/jdk1.8.0_1

0.拦截器图解 1.常用拦截器 (1)时间戳拦截器 (2)主机名拦截器 (3)UUID拦截器&查询替换拦截器 (4)正则拦截器 2.自定义拦截器 (1)写自定义拦截器程序，即成flume的拦截器包 (2)打包 (3)上传到linux (4)修改flume.conf文件 (5)运行 ...

1.新建配置文件flumejob_telnet.conf #smple.conf: A single-node Flume configuration # Name the components on this agent 定义变量方便调用 加s可以有多个此角色 a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source 描述source角色 进行内容定制 # 此配置属于tcp source 必须是netc

1.新建配置文件flumejob_hdfs.conf,然后上传 # Name the components on this agent agent别名设置 a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source 设置数据源监听本地文件配置 # exec 执行一个命令的方式去查看文件 tail -F 实时查看 a1.sources.r1.type = exec # 要执行的脚本command tail -

1.新建配置文件flumejob_dir.conf # 定义别名 a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = spooldir # 监控的文件夹 a1.sources.r1.spoolDir = /opt/testdir # 上传成功后显示后缀名 a1.sources.r1.fileSuffix = .COMPLETED # 如论如何 加绝对路径的文

0.图解过程： 1.修改多配置文件,flumejob_1.conf,flumejob_2.conf,flumejob_3.conf (1)flumejob_1.conf a1.sources = r1 a1.sinks = k1 k2 a1.channels = c1 c2 # 将数据流复制给多个channel a1.sources.r1.selector.type = replicating # 2.source a1.sources.r1.type = exec a1.sources.r1.com

1.图示 2.新建配置文件： (1)flume-1.conf # 1 agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # 2 source a1.sources.r1.type = netcat a1.sources.r1.bind = hadoop2 a1.sources.r1.port = 55555 #3 sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = hadoop4 a1.sink

0.功能作用： 将时间戳放到event的header(Map<key,value>) 1.Timestamp.conf #1.定义agent名， source、channel、sink的名称 a4.sources = r1 a4.channels = c1 a4.sinks = k1 #2.具体定义source a4.sources.r1.type = spooldir a4.sources.r1.spoolDir = /opt/module/flume-1.8.0/upload #定义拦截

0.功能作用： 将时间戳放到event的header(Map<key,value>) 1.Host.conf #1.定义agent a1.sources= r1 a1.sinks = k1 a1.channels = c1 #2.定义source a1.sources.r1.type = exec a1.sources.r1.channels = c1 a1.sources.r1.command = tail -F /opt/plus #拦截器 a1.sources.r1.intercepto

1.uuid.conf a1.sources = r1 a1.sinks = k1 a1.channels = c1 a1.sources.r1.type = exec a1.sources.r1.channels = c1 a1.sources.r1.command = tail -F /opt/plus a1.sources.r1.interceptors = i1 #type的参数不能写成uuid，得写具体，否则找不到类 a1.sources.r1.interceptors.i1.type = or

1.search.conf #1 agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 #2 source a1.sources.r1.type = exec a1.sources.r1.channels = c1 a1.sources.r1.command = tail -F /opt/plus a1.sources.r1.interceptors = i1 a1.sources.r1.interceptors.i1.type = search_rep

1.filter.conf #1 agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 #2 source a1.sources.r1.type = exec a1.sources.r1.channels = c1 a1.sources.r1.command = tail -F /opt/plus a1.sources.r1.interceptors = i1 a1.sources.r1.interceptors.i1.type = regex_filt

1.extractor.conf #1 agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 #2 source a1.sources.r1.type = exec a1.sources.r1.channels = c1 a1.sources.r1.command = tail -F /opt/plus a1.sources.r1.interceptors = i1 a1.sources.r1.interceptors.i1.type = regex_e

1.写一个小写字母变成大写的拦截器； 2.编写代码： (1)pom.xml <dependencies> <!-- flume核心依赖 --> <dependency> <groupId>org.apache.flume</groupId> <artifactId>flume-ng-core</artifactId>