flume的kafka到hdfs的配置

最新推荐文章于 2023-09-07 14:49:45 发布

原创最新推荐文章于 2023-09-07 14:49:45 发布 · 1.1k 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#flume #kafka2hdfs

flume 专栏收录该内容

3 篇文章

订阅专栏

本文介绍如何通过配置Flume实现与Kafka的数据集成，包括详细的Flume配置参数及解决依赖冲突的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.htrace-core-3.0.jar

2.commons-configuration-1.6.jar

3.hadoop-auth-2.6.0.jar

4.hadoop-common-2.6.0.jar

5.hadoop-hdfs-2.6.0.jar

6.hadoop-mapreduce-client-core-2.6.0.jar

遇到这些错误时，添加以上包就可以解决:

java.lang.NoClassDefFoundError: org/htrace/Trace

java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration

java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName

java.io.IOException: No FileSystem for scheme: hdfs

java.io.IOException: Incomplete HDFS URI, no host: hdfs://

flume的conf配置：

agent.sources=r1
agent.sinks=k1
agent.channels=c1


agent.sources.r1.type= org.apache.flume.source.kafka.KafkaSource
agent.sources.r1.kafka.bootstrap.servers=192.168.0.1:9092
agent.sources.r1.kafka.topics=TEST-TOPIC
#agent.sources.r1.serializer.class=kafka.serializer.StringEncoder
agent.sources.r1.kafka.consumer.group.id = flumetest
auto.commit.enable = true
agent.sources.r1.flumeBatchSize=1000
agent.sources.r1.useFlumeEventFormat=false
agent.sources.r1.restart=true
agent.sources.r1.batchSize=1000
agent.sources.r1.batchTimeout=3000
agent.sources.r1.channels=c1


agent.channels.c1.type=memory
agent.channels.c1.capacity=102400
agent.channels.c1.transactionCapacity=1000
agent.channels.c1.byteCapacity=134217728
agent.channels.c1.byteCapacityBufferPercentage=80

agent.sinks.k1.channel=c1
agent.sinks.k1.type=hdfs
agent.sinks.k1.hdfs.path=hdfs://master02:8020/tmp/flume/ds=%Y%m%d
agent.sinks.k1.hdfs.rollSize=0
agent.sinks.k1.hdfs.rollCount=0
agent.sinks.k1.hdfs.rollInterval=3600
agent.sinks.k1.hdfs.threadsPoolSize=30
agent.sinks.k1.hdfs.fileType=DataStream
agent.sinks.k1.hdfs.writeFormat=Text


## source 拦截器
agent.sources.r1.interceptors = i1
agent.sources.r1.interceptors.i1.type = org.com.flume.MyInterceptor$Builder