整合Flume与Kafka完成实时数据采集

Tai_Park

于 2020-03-03 13:40:09 发布

阅读量591

点赞数 1

分类专栏： Spark和他的小伙伴们文章标签： flume kafka 数据采集

本文链接：https://blog.youkuaiyun.com/qq_36329973/article/details/104629923

版权

1.架构

2.Flume

第一个Agent配置 -> exec-memory-avro.conf：

exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel

exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /home/hadoop/data/data.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c

exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = hadoop000
exec-memory-avro.sinks.avro-sink.port = 44444

exec-memory-avro.channels.memory-channel.type = memory

exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel

第二个Agent配置 -> avro-memory-kafka.conf：