taildir Source

最新推荐文章于 2024-05-02 20:26:26 发布

原创最新推荐文章于 2024-05-02 20:26:26 发布 · 476 阅读

1 ·

CC 4.0 BY-SA版权

日志项目专栏收录该内容

10 篇文章

订阅专栏

博客指出使用exec source、tail -f进行日志监控存在不足，缺乏断点续传功能。同时提到Flume 1.6版本没有断点续传功能，而1.7版本具备该功能。

a1.sources = r1
a1.channels = c1
a1.sinks = k1


a1.sources.r1.type = TAILDIR
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = /home/atguigu/taildir/example.log
a1.sources.r1.filegroups.f2 =/home/atguigu/taildir/.*log.*
a1.sources.r1.fileHeader = true
a1.sources.ri.maxBatchCount = 1000

a1.channels.c1.type = memory

a1.sinks.k1.type = logger


a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

监控日志，如果使用exec source ，tail -f 这种监控，不好，没有断点续传的功能

Note This source is provided as a preview feature. It does not work on Windows.
Watch the specified files, and tail them in nearly real-time once detected new lines appended to the each files. If the new lines are being written, this source will retry reading them in wait for the completion of the write.

This source is reliable and will not miss data even when the tailing files rotate. It periodically writes the last read position of each files on the given position file in JSON format. If Flume is stopped or down for some reason, it can restart tailing from the position written on the existing position file.

但是flume 1.6是没有的，1.7版本有