flume-ng-extends

最新推荐文章于 2021-10-04 14:35:26 发布

原创最新推荐文章于 2021-10-04 14:35:26 发布 · 3.1k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#flume-ng

flume-ng 专栏收录该内容

7 篇文章

订阅专栏

本文介绍如何使用Flume-ng-extends监听指定目录的文件写入事件，并触发数据收集流程。首先需要从GitHub获取源代码并编译打包。接着通过配置Agent来指定监听的目录路径，配置Sink为Logger类型以便于查看输出日志。

这个可以taildir文件，监听文件目录写入事件，只要往这个目录写入文件就能触发

首先需要flume-ng-extends：从获取源代码 https://github.com/jinoos/flume-ng-extends，maven编译打包

将flume-ng-extends-0.0.1-SNAPSHOT.jar,另外还要下载commons-vfs2-2.0.jar这个jar包（这个不然可能会报错）一起放到flume的lib下

2，agent配置

hadoop@stormspark:~/bigdata/flume-1.4.0-bin/dirTailTest$ cat ../conf/flume-ng-extend.properties 
a.sources = sources
a.sinks = sinks
a.channels = c

#configure sources
a.sources.sources.type = com.jinoos.flume.DirectoryTailSource
a.sources.sources.dirs = s0 s1
a.sources.sources.dirs.s0.path = /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s0in
a.sources.sources.dirs.s1.path = /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in
#congfigure sinks
a.sinks.sinks.type = logger
#configure channals
a.channels.c.type = memory
#bind channel
a.sources.sources.channels = c

启动这个agent

bin/flume-ng agent --conf conf  --conf-file conf/flume-ng-extend.properties --name a -Dflume.root.logger=INFO,console

然后往s0in,s1in目录丢进去一些文件

cp ../conf/*.properties  ./s1in/

可以看到已经触发了

2014-01-17 20:29:35,774 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:110)] Monitoried counter group for type: SOURCE, name: sources, registered successfully.
2014-01-17 20:29:35,774 (lifecycleSupervisor-1-4) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:94)] Component type: SOURCE, name: sources started
2014-01-17 20:29:53,314 (pool-3-thread-2) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileChanged(DirectoryTailSource.java:417)] /home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in/flume-ng-extend.propertiesis not in monitoring list.
2014-01-17 20:29:53,812 (pool-3-thread-5) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in
2014-01-17 20:29:53,812 (pool-3-thread-4) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in
2014-01-17 20:29:53,812 (pool-3-thread-6) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in
2014-01-17 20:29:53,812 (pool-3-thread-3) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in
2014-01-17 20:29:53,812 (pool-3-thread-7) [WARN - com.jinoos.flume.DirectoryTailSource$WorkerRunnable.fileCreated(DirectoryTailSource.java:341)] Occurred create event from un-indexed directory. ///home/hadoop/bigdata/flume-1.4.0-bin/dirTailTest/s1in