Flume输出sink到HBase

最新推荐文章于 2022-11-11 15:56:54 发布

原创最新推荐文章于 2022-11-11 15:56:54 发布 · 1.4k 阅读

8 ·

CC 4.0 BY-SA版权

文章标签：

#Flume输出sink到HBase

HBase 同时被 2 个专栏收录

3 篇文章

订阅专栏

Flume

2 篇文章

订阅专栏

本文介绍如何使用Flume将本地文件数据实时导入HBase数据库，包括配置Flume以匹配特定的数据格式，以及在HBase中创建表并验证数据导入的成功。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、Hbase中建表

hbase(main):056:0>create 'movie','analyse'

2、Flume配置

a1.sources = r1   
a1.sinks = k1     
a1.channels = c1  

# Describe/configure the source
a1.sources.r1.type = spooldir  
#实时读取本地目录
a1.sources.r1.spoolDir = /root/flume_test
a1.sources.r1.deletePolicy=never 
a1.sources.r1.fileHeader = true  
a1.sources.r1.channels = c1   

# Use a channel which buffers events in memory
a1.channels.c1.type = memory   
a1.channels.c1.capacity = 1000   
a1.channels.c1.transactionCapacity = 100  

a1.sinks.k1.type = org.apache.flume.sink.hbase.HBaseSink 
#表名
a1.sinks.k1.table = movie
#列组
a1.sinks.k1.columnFamily = analyse
#通配符(数据格式为	1001::猫和老鼠::9 )
a1.sinks.k1.serializer.regex =(.*)::(.*)::(.*)
a1.sinks.k1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.k1.channel = c1  
#列
a1.sinks.k1.serializer.colNames = ROW_KEY,movie_name,rating
# 索引为0，即ROW_KEY(ROW_KEY是特殊字符)
a1.sinks.k1.serializer.rowKeyIndex = 0

３、启动flume

[root@quickstart]#flume-ng agent --conf conf /etc/flume-ng/conf --conf-file /etc/flume-ng/conf/flume.conf  --name a1 -Dflume.root.logger = INFO，console

4、导入数据到 /root/flume_test/,文件内容为

1001::tom::3
1002::jerry::5
1003::jack::4

5、查看HBase表

hbase(main):066:0> scan 'movie'
ROW                             COLUMN+CELL                                                                             
 1001                           column=analyse:movieid, timestamp=1545212471611, value=tom                              
 1001                           column=analyse:ratings, timestamp=1545212471611, value=3                                
 1002                           column=analyse:movieid, timestamp=1545212471611, value=jerry                            
 1002                           column=analyse:ratings, timestamp=1545212471611, value=5                                
 1003                           column=analyse:movieid, timestamp=1545212471611, value=jack                             
 1003                           column=analyse:ratings, timestamp=1545212471611, value=4                                
3 row(s) in 0.0440 seconds