Flume 配置方案

最新推荐文章于 2023-12-09 19:45:34 发布
最新推荐文章于 2023-12-09 19:45:34 发布
阅读量233 收藏 2
点赞数 4
CC 4.0 BY-SA版权
文章标签： Flume 配置方案
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/KamRoseLee/article/details/83753267

1：start case （single-node configuration）

#文件名：case1_example.conf
#配置内容：
# case1_example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
 
# Describe the sink
a1.sinks.k1.type = logger
 
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
 
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

#开始命令
flume-ng agent -c conf -f conf/case1_example.conf -n a1 -Dflume.root.logger=INFO,console

#启动参数说明
-c conf 指定配置目录为conf
-f conf/case1_example.conf 指定配置文件为conf/case1_example.conf
-n a1 指定agent名字为a1,需要与case1_example.conf中的一致
-Dflume.root.logger=INFO,console 指定DEBUF模式在console输出INFO信息

#在另一个终端进行测试
telnet 127.0.0.1 44444
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
hello world! 
OK

 
#在启动的终端查看console输出
2013-05-24 00:00:24,306 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D hello world!. }

2：Test Avro Source

#文件名：case2_avro.conf
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 4141
 
# Describe the sink
a1.sinks.k1.type = logger
 a1.sinks.k1.channel = c1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
 
#Start flume agent a1
flume-ng agent -c . -f case2_avro.conf -n a1 -Dflume.root.logger=INFO,console

#创建指定文件
echo "hello world" > /usr/logs/log.10
 
#使用avro-client发送文件
flume-ng avro-client -c . -H localhost -p 4141 -F /usr/logs/log.10
 
#在启动的终端查看console输出
2013-05-27 01:11:45,852 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 hello world }

3：Test Exec Source

#文件名：case3_exec.conf
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = cat /usr/logs/log.10
a1.sources.r1.channels = c1
 
# Describe the sink
a1.sinks.k1.type = logger
 a1.sinks.k1.channel = c1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
 
#启动flume agent a1
flume-ng agent -c . -f case3_exec.conf -n a1 -Dflume.root.logger=INFO,console
 
#追加内容到文件
echo "exec test" >> /usr/logs/log.10
 
#在启动的终端查看console输出
2018-10-27 01:50:12,825 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 hello world }
2018-10-27 01:50:12,826 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 65 78 65 63 20 74 65 73 74 exec test }
 
#如果要使用tail命令，必选使得file足够大才能看到输出内容
a1.sources.r1.command = tail -F /usr/logs/log.10
 
#生成足够多的内容在文件里
for i in {1..100};do echo "exec test$i" >> /usr/logs/log.10;echo $i;done
 
#可以在console看到output
2018-10-27 19:17:18,157 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.ExecSource.start(ExecSource.java:155)] Exec source starting with command:tail -n 5 -F /usr/logs/log.10
2018-10-27 19:19:50,334 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 65 78 65 63 20 74 65 73 74 37 exec test7 }

4：Test spool Source

#文件名：case4_spool.conf
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /usr/logs/flumeSpool
a1.sources.r1.fileHeader = true
a1.sources.r1.channels = c1
 
# Describe the sink
a1.sinks.k1.type = logger
 a1.sinks.k1.channel = c1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#启动flume agent a1
flume-ng agent -c . -f case4_spool.conf -n a1 -Dflume.root.logger=INFO,console
 
#追加内容到spool目录
echo "spool test1" > /usr/logs/flumeSpool/spool1.log
 
#在启动的终端查看console输出
2018-10-27 22:49:06,098 (pool-4-thread-1) [INFO - org.apache.flume.client.avro.SpoolingFileLineReader.retireCurrentFile(SpoolingFileLineReader.java:229)] Preparing to move file /usr/logs/flumeSpool/spool1.log to /usr/logs/flumeSpool/spool1.log.COMPLETED
2018-10-27 22:49:06,101 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{file=/usr/logs/flumeSpool/spool1.log} body: 73 70 6F 6F 6C 20 74 65 73 74 31 spool test1 }

5：Test Syslog tcp source

#文件名：case5_syslog.conf
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.port = 5140
a1.sources.r1.host = localhost
a1.sources.r1.channels = c1
 
# Describe the sink
a1.sinks.k1.type = logger
 a1.sinks.k1.channel = c1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#启动flume agent a1
flume-ng agent -c . -f case5_syslog.conf -n a1 -Dflume.root.logger=INFO,console
 
#测试产生syslog, <37>因为需要wire format数据，否则会报错” Failed to extract syslog wire entry”
echo "<37>hello via syslog" | nc localhost 5140
 
#在启动的终端查看console输出
2018-10-27 23:39:10,755 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 68 65 6C 6C 6F 20 76 69 61 20 73 79 73 6C 6F 67 hello via syslog }

6：Test Syslog udp source

#文件名：case6_syslogudp.conf
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = syslogudp
a1.sources.r1.port = 5140
a1.sources.r1.host = localhost
a1.sources.r1.channels = c1
 
# Describe the sink
a1.sinks.k1.type = logger
 a1.sinks.k1.channel = c1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#启动flume agent a1
flume-ng agent -c . -f case6_syslogudp.conf -n a1 -Dflume.root.logger=INFO,console
 
#测试产生syslog
echo "<37>hello via syslogudp" | nc -u localhost 5140
 
#在启动的终端查看console输出
2018-10-27 23:39:10,755 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 68 65 6C 6C 6F 20 76 69 61 20 73 79 73 6C 6F 67 hello via syslogudp }

7：Test HTTP source JSONHandler

#文件名：case7_httppost.conf
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1
 
# Describe the sink
a1.sinks.k1.type = logger
 a1.sinks.k1.channel = c1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#启动flume agent a1
flume-ng agent -c . -f case7_httppost.conf -n a1 -Dflume.root.logger=INFO,console
 
#生成JSON 格式的POST request
curl -X POST -d '[{ "headers" :{"namenode" : "namenode.example.com","datanode" : "random_datanode.example.com"},"body" : "really_random_body"}]' http://localhost:5140
 
#在启动的终端查看console输出
2018-10-27 01:17:47,186 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{namenode=namenode.example.com, datanode=random_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F 6D 5F 62 6F random_datanode_body

8：Test HDFS Sink

#文件名：case8_hdfs.conf
在/usr/local/apache-flume-1.3.1-bin/conf/flume-env.sh加入
export HADOOP_HOME=/usr/local/hadoop
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = syslogtcp
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1
 
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = hdfs://master:9000/user/hadoop/flume/collected/
a1.sinks.k1.hdfs.filePrefix = Syslog
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#启动flume agent a1
flume-ng agent -c . -f case8_hdfs.conf -n a1 -Dflume.root.logger=INFO,console
 
#测试产生syslog
echo "<37>hello via syslog to hdfs testing one" | nc -u localhost 5140
 
#在启动的终端查看console输出,文件生成成功
2018-10-27 00:53:58,078 (hdfs-k1-call-runner-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:208)] Creating hdfs://master:9000/user/hadoop/flume/collected//Syslog.1369814037714.tmp
2018-10-27 00:54:28,220 (hdfs-k1-roll-timer-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:427)] Renaming hdfs://master:9000/user/hadoop/flume/collected/Syslog.1369814037714.tmp to hdfs://master:9000/user/hadoop/flume/collected/Syslog.1369814037714
 
#在hadoop上查看文件
./hadoop dfs -cat hdfs://172.25.4.35:9000/user/hadoop/flume/collected/Syslog.1369814037714
SEQ!org.apache.hadoop.io.LongWritable"org.apache.hadoop.io.BytesWritable^;>Gv$hello via syslog to hdfs testing one

9：Test HDFS Sink

#文件名：case9_hdfs.conf
在/usr/local/apache-flume-1.3.1-bin/conf/flume-env.sh加入
export HADOOP_HOME=/usr/local/hadoop
#配置内容：
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1
 
# Describe the sink
a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path= = hdfs://master:9000/user/hadoop/flume/collected/%y-%m-%d/%H%M/%S
a1.sinks.k1.hdfs.filePrefix = Syslog.%{host}
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 10
a1.sinks.k1.hdfs.roundUnit = minute

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#启动flume agent a1
flume-ng agent -c . -f case9_hdfs.conf -n a1 -Dflume.root.logger=INFO,console

#生成JSON 格式的POST request, header的timestamp 参数如果格式不对则无法解析
需要生成13为的timestamp才能解析出正确的时间,包含MilliSec
#linux生成当前时间10位Unix timestamp
date +%s
#linux生成当前时间13位Unix timestamp
date +%s%N|awk '{print substr($0,1,13)}'
 
curl -X POST -d '[{ "headers":{"timestamp":"1369818213654","host":"cc-staging-loginmgr2"},"body": "hello via post"}]' http://localhost:5140
 
#在启动的终端查看console输出,文件生成成功
2018-10-27 02:03:38,646 (hdfs-k1-call-runner-4) [INFO - org.apache.flume.sink.hdfs.BucketWriter.doOpen(BucketWriter.java:208)] Creating hdfs://master:9000/user/hadoop/flume/collected/2018-10-27/0203/cc-staging-loginmgr2..1369818218614.tmp
2018-10-27 02:04:08,714 (hdfs-k1-roll-timer-0) [INFO - org.apache.flume.sink.hdfs.BucketWriter.renameBucket(BucketWriter.java:427)] Renaming hdfs://master:9000/user/hadoop/flume/collected/2018-10-27/0203/cc-staging-loginmgr2..1369818218614.tmp to hdfs://master:9000/user/hadoop/flume/collected/2018-10-27/0203/cc-staging-loginmgr2..1369818218614
 
#在hadoop上查看文件
./hadoop dfs -ls hdfs://172.25.4.35:9000/user/hadoop/flume/collected/2018-10-27/0203
Found 1 items
-rw-r--r-- 3 root supergroup 129 2018-10-27 02:04 /user/hadoop/flume/collected/2018-10-27/0203/cc-staging-loginmgr2..1369818218614

10：Test Avro Sink

#文件名：case10_avro.conf、case10_avro_sink.conf
#配置内容：
#case10_avro.conf
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
 
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 4545
 
# Describe the sink
a1.sinks.k1.type = logger
 a1.sinks.k1.channel = c1

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

#case10_avro_sink.conf
# Name the components on this agent
a2.sources = r1
a2.sinks = k1
a2.channels = c1
 
# Describe/configure the source
a2.sources.r1.type = syslogtcp
a2.sources.r1.port = 5140
a2.sources.r1.host = localhost
a2.sources.r1.channels = c1
 
# Describe the sink
a2.sinks.k1.type = avro
a2.sinks.k1.channel = c1
a2.sinks.k1.hostname = 172.25.4.23
a2.sinks.k1.port = 4545

# Use a channel which buffers events in memory
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100

#先启动Avro的Source,监听端口
flume-ng agent -c . -f case10_avro.conf -n a1 -Dflume.root.logger=INFO,console
 
#再启动Avro的Sink
flume-ng agent -c . -f case10_avro_sink.conf -n a2 -Dflume.root.logger=INFO,console
 
#可以看到已经建立连接
2018-10-27 19:23:00,237 (pool-5-thread-1) [INFO - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:171)] [id: 0x7a0e28bf, /172.25.4.32:14894 => /172.25.4.23:4545] CONNECTED: /172.25.4.32:14894
 
#在Avro Sink上生成测试log
echo "<37>hello via avro sink" | nc localhost 5140
 
#在Avro Source上可以看到log已经生成
2018-10-27 19:24:13,740 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{Severity=5, Facility=4} body: 68 65 6C 6C 6F 20 76 69 61 20 61 76 72 6F 20 73 hello via avro sink }

 

搬码日记
关注
flume系列之：flume配置写入zookeeper节点，根据zookeeper中flume配置启动flume agent
zhengzaifeidelushang的博客
03-18 1154
flume系列之：flume配置写入zookeeper节点，根据zookeeper中flume配置启动flume agent一、flume配置写入zookeeper方案二、读取flume配置并把flume配置写入zookeeper节点详细实现代码1.连接zookeeper创建节点2.读取flume配置并把配置写到zookeeper节点完整详细的代码三、安装部署flume 一、flume配置写入zookeeper方案 把flume配置写入到zookeeper节点 再根据zookeeper节点flume的配置
Flume数据采集配置(全网最详细)
互联网知识分享
05-10 265
监控目录面的不要修改内容 , 一旦监控过后就不会在监控了，所以最好使用在新创建一个文件在里面创建新内容, 而监控文件可以使用追加内容，而监控目录不支持追加内容。（2）被监控的目录，如果读取成功，就会给监控的目录下文件添加后缀名 以.COMPLETED结尾，表示读取成功。因为修改就会导致文件的所有内容都好比是新增，而source又会把文件重新读取一遍，所以最好使用追加的方式。特别注意,监控文件后 这个程序是退不出去的,记得把这个程序给杀死条。忽略所有以.tmp结尾的文件，\.表示进行转移符号， $表示结尾。
flume由于关闭方式错误启动报错
曦玥Moonight的博客
10-31 1067
flume使用命令启动 ../bin/flume-ng agent -c ./ -f ./flume.properties -n a1 -Dflume.root.logger=INFO,console 报错了： 2020-10-31 21:19:59,613 (lifecycleSupervisor-1-1) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.jav
flume和kafka合成报错ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleS
小蚯蚓的博客
09-23 4985
0.环境： flume1.6.0和kafka_2.10-0.8.2.1 1.问题描述： flume和kafka集成，flume输出到kafka： (1)flume配置文件： agent1.sources=avro-source agent1.channels=logger-channel agent1.sinks=kafka-sink #define source agent1...
hadoop报错report: Call From xxx to xxx failed on connect
似水流年
08-30 4707
flume异常日志： hdfs dfsadmin -report，报错如下： “report: Call From slave1.hadoop/192.168.1.106 to namenode:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more detai
flume 向 hdfs 上传文件时. java.net.ConnectException: 拒绝连接
weixin_46039745的博客
07-21 628
1.问题描述在利用 flume 向 hdfs 上传文件时出现了；连接不上 hdfs 的情况failed on connection exception: java.net.ConnectException: 拒绝连接2.原因flume配置flume-exec-hdfs.conf中的连接信息不对这个地址应该和hadoop配置中的 core-site.xml 中的端口号一致-- cd到 hadoop的目录下。
已解决：java.net.ConnectException: Call From XXXXX to XXXXX :8020 failed on connection Exception: 拒绝连接；
最新发布
qq_43408367的博客
12-09 1242
最近在忙着做一些日志监控类的项目，Flume做的，主要是针对几个业务数据库，监控一下运行日志，为啥？咱也不敢问，咱也不敢问、整个业务集群呢是在阿里云上搭建的，十几台服务器，在跑flume脚本的时候，发现了一个很奇怪的现象，正常的跑flume的测试脚本，采用logger sink呢，是OK的，但是使用HDFS sink呢，就挂壁了，我又让同事试了下 Kafka sink，OK的，搞郁闷了，一直折腾这玩意。
Flume配置双HA hdfsSink.docx
06-17
### Flume配置双HA HDFS Sink详解 #### 一、背景与需求分析 Apache Flume 是一款高可靠、高性能的服务，用于收集、聚合和移动大量日志数据。它具有简单的可扩展架构，易于定制和部署。然而，在某些情况下，用户...
flume 采集方案的编写
q2842360795的博客
10-31 548
1 avro+memory+logger 用avro-client进行测试 启动方案 flume-ng agent -c ./conf -f ./flumeconf/avro-mem-logger.properties -n a1 -Dflume.root.logger=INFO,console 因为用的是avro的source，那么必须使用avro-client进行测试 vim data/test.avro flume-ng avro-client -c /usr/local/flume/c
flume系列之：Error while trying to hflushOrSync，查看hdfs生成的文件发现文件损坏
zhengzaifeidelushang的博客
03-30 1946
一、flume完整报错如下所示 flume日志一直报如下错误： 13 Mar 2022 10:45:30,319 ERROR [hdfs-sink1-call-runner-400] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrS ync:269) - Error while trying to hflushOrSync! 264 25 Mar 2022 10:45:30,319 WARN [hdfs-sink1-roll-t
hadoop flume 监听hive日志文件的常见错误 flume链接HDFS失败之端口问题
qq_44641456的博客
05-21 940
21 五月 2022 17:08:53,228 INFO [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.BucketWriter.open:246) - Creating hdfs://hadoop102:9870/flume/20220521/17/logs-.1653124088090.tmp 21 五月 2022 17:08:53,249 WARN [hdfs-k2-call-runn...
启动flume出现ERROR lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.
Rashaun`s blog
01-08 7429
在启动flume的时候出现如下错误   ERROR lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@1e3aff56 counterGroup:{ name:null counters:{} } } - Exceptio...
Flume在Windows下启动报错[ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(Lifecy
weixin_45498137的博客
11-13 1792
Flume windows下启动报错[ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)] Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: position/test/test_log/position.json, skipToEnd: fal
启动Flume报错ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupe
qq_36588424的博客
11-06 2964
启动Flume报错： [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)] Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@65bf32fc counterGroup:{ name:null counters:{} } } - Except
Flume配置1——基础案例
weixin_51699336的博客
06-28 1601
Flume配置1——基础案例
Flume 的几种部署方案
Rashaun`s blog
12-04 577
1.单 Agent 采集数据     2.多 Agent 串联     3.多 Agent 合并串联     4.多路复用  
Flume自定义SyslogSink
阿达丶
11-09 1856
需求：收集客户端app应用日志，发送到其他代理agent上514syslog端口进行日志收集采样。 技术组件 ：flume-ng 版本：flume1.7 其他：由于flume本身默认没有syslog的sink，只有syslog的source，所以我们可以自己定义syslog sink 测试syslog的javaDemo，功能实现ip校验，发送多条syslog日志 ，直接java -jar运
Apache Flume安装与配置指南
在实际部署中，Flume配置文件通常会定义多个Source、Channel和Sink的组合，形成一个数据流处理链路。例如，一个Source可能从日志文件读取数据，然后将数据写入内存Channel，最后由Sink将数据持久化到HDFS。通过这种...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值