flume几种使用配置 avro spooling exec thrift

最新推荐文章于 2022-08-09 10:55:58 发布

JAVA日常笔记精选

最新推荐文章于 2022-08-09 10:55:58 发布

阅读量378

点赞数 1

分类专栏： flume

本文链接：https://blog.youkuaiyun.com/qq_29082603/article/details/103959952

版权

flume 专栏收录该内容

3 篇文章

订阅专栏

这篇博客详细介绍了Apache Flume的四种使用配置：Avro，Spooling，Exec和Thrift。通过这些配置，Flume能够高效地收集、聚合和移动大量日志数据。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

参考官网：

http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html

# avro source
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 4141
a1.sinks.k1.type = logger
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sinks.k1.channel = c1


# thrift source
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = thrift
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 4141
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sinks.k1.channel = c1
a1.sinks.k1.type = logger


# exec source
# 运行的时候可以指定命令  如tail -f 可以监视文件的变化
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /var/log/secure
a1.sources.r1.channels = c1
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sinks.k1.channel = c1
a1.sinks.k1.type = logger


# spooling source
# 监听文件夹变化，对新产生的文件进行读取，读取完毕之后，会对文件加.COMPLETED后缀，不会重复读取
# 一般都是用此种类型的源抽取日志文件
a1.channels = ch-1
a1.sources = src-1
a1.sinks = k1

a1.sources.src-1.type = spooldir
a1.sources.src-1.channels = ch-1
a1.sources.src-1.spoolDir = /var/log/apache/flumeSpool
a1.sources.src-1.fileHeader = true

a1.channels.ch-1.type = memory
a1.channels.ch-1.capacity = 1000
a1.channels.ch-1.transactionCapacity = 100

a1.sinks.k1.channel = c1
a1.sinks.k1.type = logger