Flume 收集Nginx日志 tail-to-avro avro-to-file_flume的avro收集nginx发送的数据-优快云博客

本文链接：https://blog.youkuaiyun.com/luyee2010/article/details/22171157

本文介绍了一种使用Apache Flume进行日志收集的具体配置方案，包括从文件中实时读取日志数据并转换为Avro格式，再将Avro数据持久化到滚动文件中的流程。配置中详细说明了source、sink及channel的设置。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

nginx，access.log日志约1.6w条/s，每1000w条数据约2000M,需要10min

tail-to-avro

agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

# Describe/configure spooldir source1
#agent1.sources.source1.type = spooldir
#agent1.sources.source1.spoolDir = /var/log/apache/flumeSpool1
#agent1.sources.source1.fileHeader = true

# Describe/configure tail -F source1
agent1.sources.source1.type = exec 
agent1.sources.source1.command = tail -F /tmp/log.log
agent1.sources.source1.channels = channel1

# Describe/configure nc source1
#agent1.sources.source1.type = netcat
#agent1.sources.source1.bind = localhost
#agent1.sources.source1.port = 44444

#configure host for source
agent1.sources.source1.interceptors = i1
agent1.sources.source1.interceptors.i1.type = host
agent1.sources.source1.interceptors.i1.hostHeader = hostname

# Describe logger  sink1
#agent1.sinks.sink1.type = logger

# Describe avro  sink1
agent1.sinks.sink1.type = avro
agent1.sinks.sink1.hostname = 172.16.10.175
agent1.sinks.sink1.port = 4545

# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.keep-alive = 120
agent1.channels.channel1.capacity = 500000
agent1.channels.channel1.transactionCapacity = 600

# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1

avro-to-rollfile
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1

# Describe/configure avro source

agent1.sources.source1.type = avro
agent1.sources.source1.bind = 0.0.0.0
agent1.sources.source1.port = 4545


# Describe logger sink1
#agent1.sinks.sink1.type = logger

# Describe file sink1
agent1.sinks.sink1.type = file_roll
agent1.sinks.sink1.sink.directory = /var/log/flume

# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.keep-alive = 120
agent1.channels.channel1.capacity = 500000
agent1.channels.channel1.transactionCapacity = 600

# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1