实现flume tailfsource解决丢数等问题

最新推荐文章于 2024-03-05 16:03:53 发布

wobenpingfan_y

最新推荐文章于 2024-03-05 16:03:53 发布

阅读量3.5k

点赞数

CC 4.0 BY-SA版权

文章标签： flume tail tailf tailsource

本文链接：https://blog.youkuaiyun.com/wobenpingfan_y/article/details/46911395

本文介绍了在使用Flume实时收集日志数据时，由于tail -F方法存在的丢数问题，转而采用自定义的TAILFSource进行优化。通过配置TAILFSource，如设置readingfilepath、historyfilepath、offsetrecordfile和batchSize等参数，以避免数据丢失并提高性能。同时提到了BufferedRandomAccessFile作为RandomAccessFile的优化方案。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

使用flume过程中，实时的收集数据时，一般使用exec的source，通过tail -F的形式收集日志内容，但是tail -F存在包括丢数在内一系列问题，实现tailfsource解决：

public class TailingFileSource extends AbstractSource implements
EventDrivenSource, Configurable {


private static final Logger logger = LoggerFactory
.getLogger(TailingFileSource.class);


private SourceCounter sourceCounter;
private long restartThrottle;
private boolean restart;
private Integer bufferCount;
private long batchTimeout;
private ExecutorService executor;
private Future<?> runnerFuture;
private String readingFilePath;
private String historyFilePath;
private String offsetRecordFile;
private ChannelProcessor channelProcessor;
private ReadingFile reader;


@Override
public void configure(Context context) {
// TODO Auto-generated method stub


restartThrottle = context
.getLong(
TailingFileSourceConfigurationConstants.CONFIG_RESTART_THROTTLE,
TailingFileSourceConfigurationConstants.DEFAULT_RESTART_THROTTLE);


restart = context.getBoolean(
TailingFileSourceConfigurationConstants.CONFIG_RESTART,
TailingFileSourceConfigurationConstants.DEFAULT_RESTART);


bufferCount = context.getInteger(
TailingFileSourceConfigurationConstants.CONFIG_BATCH_SIZE,
TailingFileSourceConfigurationConstants.DEFAULT_BATCH_SIZE);


batchTimeout = context.getLong(
TailingFileSourceConfigurationConstants.CONFIG_BATCH_TIME_OUT,
TailingFileSourceConfigurationConstants.DEFAULT_BATCH_TIME_OUT);


readingFilePath = context
.getString(TailingFileSourceConfigurationConstants.CONFIG_READING_FILE_PATH);


Preconditions.checkState(readingFilePath != null,
"The parameter readingFilePath must be specified");


historyFilePath = context
.getString(TailingFileSourceConfigurationConstants.CONFIG_HISTORY_FILE_PATH);


Preconditions.checkState(historyFilePath != null,
"The parameter historyFilePath must be specified");


offsetRecordFile = context
.getString(TailingFileSourceConfigurationConstants.CONFIG_OFFSET_RECORD_FILE);


Preconditions.checkState(offsetRecordFile != null,
"The parameter offsetRecordFile must be specified");


if (sourceCounter == null) {
sourceCounter = new SourceCounter(getName());
}
}


private static class ReadingFile implements Runnable {


public ReadingFile(String readingFilePath, int bufferCount,
long batchTimeout, SourceCounter sourceCounter,
String historyFilePath, String offsetRecordFile,
ChannelProcessor channelProcessor, boolean restart,
long restartThrottle) {
this.re