5.Flume实时监控读取日志数据，存储hdfs文件系统

最新推荐文章于 2024-04-10 16:44:49 发布

原创最新推荐文章于 2024-04-10 16:44:49 发布 · 3.2k 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#hive #hdfs #flume #hadoop #日志收集

flume 专栏收录该内容

5 篇文章

订阅专栏

本文介绍了一种通过Flume收集Hive运行日志的方法。首先通过tail命令实时读取Hive日志文件，然后使用Flume配置文件定义数据源、通道及接收器，将日志数据发送至HDFS中指定路径。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.收集Hive运行日志logs
命令

tail -f  /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/logs/hive.log

2.配置文件

# The configuration file needs to define the sources, 
# the channels and the sinks.

### define agent
a2.sources = r2
a2.channels = c2
a2.sinks = k2

### define sources
a2.sources.r2.type = exec
a2.sources.r2.command = tail -f /opt/cdh-5.3.6/hive-0.13.1-cdh5.3.6/logs/hive.log
a2.sources.r2.shell = /bin/bash -c

### define channels
a2.channels.c2.type = memory
a2.channels.c2.capacity = 1000
a2.channels.c2.transactionCapacity = 100

### define sink
a2.sinks.k2.type = hdfs
a2.sinks.k2.hdfs.path = hdfs://hadoop-CDH:8020/user/beifeng/flume/hive-logs/
#a2.sinks.k2.hdfs.path = hdfs://ns1/user/flume/hive-logs/ //当HDFS配置了高可用HA时
a2.sinks.k2.hdfs.fileType = DataStream 
a2.sinks.k2.hdfs.writeFormat = Text
a2.sinks.k2.hdfs.batchSize = 10

### bind the soures and  sink to the channel
a2.sources.r2.channels = c2
a2.sinks.k2.channel = c2

注意，当HDFS是配置了HA的时候，需要将hadoop的配置文件core-site.xml以及hdfs-site.xml等拷贝到Flume的配置文件夹conf下。
3.运行

bin/flume-ng agent \
-c conf \
-n a2 \
-f conf/flume-tail.conf \
-Dflume.root.logger=DEBUG,console

执行成功

2017-05-03 09:37:54,283 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:261)] Creating hdfs://hadoop-CDH:8020/user/beifeng/flume/hive-logs//FlumeData.1493818674204.tmp

tmp是默认格式，可以改变