Flum_C:跨服务器实时采集日志

最新推荐文章于 2022-07-06 01:36:05 发布

原创最新推荐文章于 2022-07-06 01:36:05 发布 · 314 阅读

0 ·

CC 4.0 BY-SA版权

数仓专栏收录该内容

28 篇文章

订阅专栏

本文详细介绍了一种使用Flume进行跨服务器数据传输的方法。在A服务器上通过execsource从本地日志收集数据，经由memorychannel传递至avrosink，再发送到B服务器的avrosource接收，通过同一memorychannel将数据传给kafkasink，最终写入Kafka的hello_topic主题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

A服务器上：Agent: exec source + memory channel + avro sink
B服务器上 : Agent: avro source + memory channel + kafka sink

我们就在自己的服务器上传输不影响
A服务器上的配置文件:

exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel

exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /sda3/softs/flume-1.6.0-cdh5.14.0/data.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c

exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = hdp01
exec-memory-avro.sinks.avro-sink.port = 44444

exec-memory-avro.channels.memory-channel.type = memory

exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

B服务器上的配置文件:

avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel

avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = hdp01
avro-memory-kafka.sources.avro-source.port = 44444

avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = hdp01:9092
avro-memory-kafka.sinks.kafka-sink.topic = hello_topic
avro-memory-kafka.sinks.kafka-sink.batchSize = 5
avro-memory-kafka.sinks.kafka-sink.requiredAcks =1

avro-memory-kafka.channels.memory-channel.type = memory

avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel

在