flume --收集日志到hbase中

最新推荐文章于 2022-02-22 14:25:24 发布

原创最新推荐文章于 2022-02-22 14:25:24 发布 · 371 阅读

0 ·

CC 4.0 BY-SA版权

flume 专栏收录该内容

11 篇文章

订阅专栏

本文详细解析了如何使用Apache Flume将数据从syslogtcp源收集，并通过SPILLABLEMEMORY通道存储到HBase数据库的具体配置过程。文章涵盖了syslogtcp源设置、SPILLABLEMEMORY通道参数配置及HBase接收器的详细说明。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

a1.sources = s1
a1.channels = c1
a1.sinks = k1

#tcp协议
a1.sources.s1.type = syslogtcp
a1.sources.s1.port = 5140
a1.sources.s1.host = wangfutai
a1.sources.s1.channels = c1

#channel
a1.channels.c1.type = SPILLABLEMEMORY
a1.channels.c1.memoryCapacity = 10000
a1.channels.c1.overflowCapacity = 1000000
a1.channels.c1.byteCapacity = 800000
a1.channels.c1.checkpointDir= /home/wangfutai/a/flume/checkpoint
a1.channels.c1.dataDirs = /home/wangfutai/a/flume/data

a1.sinks.k1.type = hbase
a1.sinks.k1.table = hadoop:flume_hbase
a1.sinks.k1.columnFamily = f1
a1.sinks.k1.serializer= org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.k1.channel = c1

a1.sinks.k1.serializer.regex=^([0-9]+),([a-z]+),([a-z]+)$
a1.sinks.k1.serializer.colNames=ROW_KEY,name,sex
a1.sinks.k1.serializer.rowKeyIndex=0