配置Flume的Hbase sink

最新推荐文章于 2022-11-11 14:50:38 发布

原创最新推荐文章于 2022-11-11 14:50:38 发布 · 1.1k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#flume

flume 专栏收录该内容

9 篇文章

订阅专栏

本文详细介绍了如何使用Apache Flume将数据收集并实时写入到HBase中。包括Zookeeper和HBase的启动步骤、Flume配置文件的设置、HBase表的创建过程、Flume的启动命令及消息发送测试等关键步骤。

一启动Zookeeper和Hbase

master节点

[root@master flumeSpool]# jps
4513 HRegionServer
4196 ResourceManager
4661 Jps
3752 QuorumPeerMain
3916 NameNode

slave1节点

[root@slave1 ~]# jps
2560 QuorumPeerMain
3009 HRegionServer
2643 DataNode
2826 NodeManager
3130 Jps
2703 SecondaryNameNode

slave2节点

[root@slave2 ~]# jps
2448 DataNode
2962 HRegionServer
3122 Jps
2563 NodeManager
2825 HMaster
2316 QuorumPeerMain

二准备配置文件k_hbse.conf

a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type = netcat
a1.sources.r1.bind=master
a1.sources.r1.port=8888

a1.channels.c1.type = memory

a1.sinks.k1.type = asynchbase
a1.sinks.k1.table = ttt
a1.sinks.k1.columnFamily = f1
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

三在Hbase中创建表

[root@slave2 ~]# hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017
hbase(main):001:0> create 'ttt','f1'
0 row(s) in 3.1440 seconds
=> Hbase::Table - ttt

四启动flume

flume-ng agent --conf-file /opt/apache-flume-1.7.0-bin/conf/k_hbse.conf --name a1

五发消息测试

[root@master flumeSpool]# nc master 8888
hello my dear
OK

六观察结果