阿正的博客

若启动flume时提示上述错误，则修改配置文件

查看kafka官网配置 new comsumer.configsa1.sources.r1.kafka.consumer.max.partition.fetch.bytes=20485760flume配置文件.conf加上这条

项目需求是将线上服务器生成的日志信息实时导入kafka，采用agent和collector分层传输，app的数据通过thrift传给agent，agent通过avro sink将数据发给collector，collector将数据汇集后，发送给kafka，拓扑结构如下：现将调试过程中遇到的问题以及解决方法记录如下：1、 [ERROR - org.apache.thrift.ser...

1. 自定义拦截器实现说明1. 实现interceptor接口，并实现其方法，接口完全限定名为：org.apache.flume.interceptor.Interceptor;2. 自定义拦截器内部添加静态内部类，实现Builder接口，并实现其方法，接口完全限定名为：Interceptor.Builder以下是最简单的代码示例（每个方法的作用都有注释说明）：...

因为集群启用了kerberos认证，需要定期手动认证，不然启动的flume的application虽然启动着，但消费不到kafka的latest的数据。

nohup flume-ng agent -c /etc/flume-ng/conf -f /opt/flume-file/.conf -n a1 -Dflume.root.logger=INFO,console &

NoteBy default the File Channel uses paths for checkpoint and data directories that are within the user home as specified above. As a result if you have more than one File Channel instances activ...

https://blog.youkuaiyun.com/high2011/article/details/53282128https://blog.youkuaiyun.com/s243471087/article/details/84061536

运行Flume没多久就报下面的异常：2016-08-24 17:35:58,927 (Flume Thrift IPC Thread 8) [ERROR - org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:196)] Error while writing to required...

2016-04-21 05:40:51,393 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.java.la...

）当 sink 到 hdfs 时：) 需修改 flume-env.sh 配置，增添 hdfs 依赖库：　　FLUME_CLASSPATH="/root/TDH-Client/hadoop/hadoop/*:/root/TDHClient/hadoop/hadoop-hdfs/*:/root/TDH-Client/hadoop/hadoop/lib/*"实例：a1.source...