flume-ng总结/注意事项

本文介绍了在使用Flume-ng进行日志收集时遇到的一些常见问题及其解决方案,包括avroclient传输过程中数据丢失的问题、启动脚本中内存设置不当导致的错误以及server端memorychannel配置不当引发的问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

1. 在客户端,使用flume-ng 的avro client传输过程中会丢数据,如

$bin/flume-ng avro-client -H localhost -p 41414 -F /usr/logs/log.10


2. /bin目录下flume-ng启动脚本中的OPTS要设置的大一些,否则会报内存溢出的错误。默认是20m,如下:

JAVA_OPTS="-Xmx20m"

 

3. server端的memory channel的capacity和transactionCapacity一定要设置的比client的大,否则会报错,如下:

13 六月 2013 17:51:57,546 ERROR [pool-7-thread-1] (org.apache.flume.source.AvroSource.appendBatch:261)  - Avro source r1: Unable to process event batch. Exception follows.
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: ..}



 

[hadoop@master flume]$ bin/flume-ng agent -n a2 -c conf -f job/spooldir-flume-hdfs.conf -Dflume.root.logger=INFO,console Info: Sourcing environment configuration script /opt/module/flume/conf/flume-env.sh Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access Info: Including HBASE libraries found via (/usr/local/hbase/bin/hbase) for HBASE access Info: Including Hive libraries found via () for Hive access + exec /usr/local/jdk1.8/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/opt/module/flume/conf:/opt/module/flume/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hbase/conf:/usr/local/jdk1.8/lib/tools.jar:/usr/local/hbase:/usr/local/hbase/lib/shaded-clients/hbase-shaded-client-2.1.0.jar:/usr/local/hbase/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/usr/local/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar:/usr/local/hbase/lib/client-facing-thirdparty/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/usr/local/hbase/lib/client-facing-thirdparty/log4j-1.2.17.jar:/usr/local/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.25.jar:/usr/local/hbase/lib/client-facing-thirdparty/htrace-core-3.1.0-incubating.jar:/usr/local/hadoop/etc/hadoop:/usr/local/hbase/conf:/lib/*' -Djava.library.path= org.apache.flume.node.Application -n a2 -f job/spooldir-flume-hdfs.conf SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/module/flume/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found bin
03-29
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值