FATAL: Spool Directory source source1: { spoolDir: /usr/local/logs }: Uncaught exception in SpoolDir

最新推荐文章于 2024-06-06 07:28:38 发布

原创最新推荐文章于 2024-06-06 07:28:38 发布 · 1.1w 阅读

1 ·

CC 4.0 BY-SA版权

大数据专栏收录该内容

6 篇文章

订阅专栏

本文介绍了一个关于Flume日志收集系统的配置案例，并详细记录了一次配置过程中遇到的异常及其解决方法。异常主要源于文件名重复使用导致的系统错误，通过清理特定目录下的文件和重启Flume服务得以解决。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

FATAL: Spool Directory source source1: { spoolDir: /usr/local/logs }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.

flume-conf.properties的配置如下：

#agent1表示代理名称
agent1.sources=source1
agent1.sinks=sink1
agent1.channels=channel1
#配置source1
agent1.sources.source1.type=spooldir
agent1.sources.source1.spoolDir=/usr/local/logs
agent1.sources.source1.channels=channel1
agent1.sources.source1.fileHeader = false
agent1.sources.source1.interceptors = i1
agent1.sources.source1.interceptors.i1.type = timestamp
#配置channel1
agent1.channels.channel1.type=file
agent1.channels.channel1.checkpointDir=/usr/local/logs_tmp_cp
agent1.channels.channel1.dataDirs=/usr/local/logs_tmp
#配置sink1
agent1.sinks.sink1.type=hdfs
agent1.sinks.sink1.hdfs.path=hdfs://spark01:9000/logs
agent1.sinks.sink1.hdfs.fileType=DataStream
agent1.sinks.sink1.hdfs.writeFormat=TEXT
agent1.sinks.sink1.hdfs.rollInterval=1
agent1.sinks.sink1.channel=channel1
agent1.sinks.sink1.hdfs.filePrefix=%Y-%m-%d

然后
本地文件夹：mkdir /usr/local/logs
HDFS文件夹：hdfs dfs -mkdir /logs

启动flume : flume-ng agent -n agent1 -c conf -f /usr/local/flume/conf/flume-conf.properties -Dflume.root.logger=DEBUG,console

在传文件的时候出现如下错误：
17/08/05 10:50:40 ERROR source.SpoolDirectorySource: FATAL: Spool Directory source source1: { spoolDir: /usr/local/logs }: Uncaught exception in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
java.lang.IllegalStateException: File name has been re-used with different files. Spooling assumptions violated for /usr/local/logs/install.log.COMPLETED
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.rollCurrentFile(ReliableSpoolingFileEventReader.java:376)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.retireCurrentFile(ReliableSpoolingFileEventReader.java:328)
at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:257)
at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:228)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

解决办法

删除/usr/local/logs/ 的文件
删除/usr/local/logs_tmp_cp 文件夹
删除/usr/local/logs_tmp 文件夹

然后重新启动flume就可以正常运行了