flume source netcatudp channels memory sinks kafka

最新推荐文章于 2024-11-22 09:12:30 发布

Gotham05

最新推荐文章于 2024-11-22 09:12:30 发布

阅读量209

点赞数

分类专栏：笔记

本文链接：https://blog.youkuaiyun.com/qq_37707251/article/details/104928833

版权

笔记专栏收录该内容

29 篇文章

订阅专栏

本文详细介绍如何使用Apache Flume的netcatudp源通过内存通道将数据发送到Kafka主题，配置包括源、通道和sink的类型，以及Kafka broker列表、主题名称和批量大小等关键参数。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

source:

netcatudp

channels:

memory

sinks:

org.apache.flume.sink.kafka.KafkaSink

tier1.sources  = source1
tier1.channels = channel1
tier1.sinks = sink1

tier1.sources.source1.type = netcatudp
tier1.sources.source1.bind = 0.0.0.0
tier1.sources.source1.port = 514
tier1.sources.source1.channels = channel1

tier1.channels.channel1.type = memory
tier1.channels.channel1.capacity = 100000
tier1.channels.channel1.transactionCapacity = 100000

tier1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink
tier1.sinks.sink1.topic = first_floor_test
tier1.sinks.sink1.brokerList = 172.168.1.17:9092,172.168.1.16:9092,172.168.1.14:9092
tier1.sinks.sink1.channel = channel1
tier1.sinks.sink1.batchSize = 100000
tier1.sinks.sink1.compression.type = snappy