Flume: hbase sink

本文详细介绍了Apache Flume配置文件及其使用场景,包括HTTP源、通道和HBasesink的配置。重点阐述了如何通过Flume将JSON格式的数据发送到HBase,并提供了实际的Curl命令示例进行验证。此外,还强调了在使用rowKey时的注意事项和如何在JSON数据中包含header信息的方法。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

flume.conf

a1.sinks.hbase-sink1.channel = ch1
a1.sinks.hbase-sink1.type = hbase
a1.sinks.hbase-sink1.table = users
a1.sinks.hbase-sink1.columnFamily= info
a1.sinks.hbase-sink1.serializer=org.apache.flume.sink.hbase.RegexHbaseEventSerializer
a1.sinks.hbase-sink1.serializer.regex=^(.+)\t(.+)\t(.+)$
a1.sinks.hbase-sink1.serializer.colNames=ROW_KEY,name,email
a1.sinks.hbase-sink1.serializer.rowKeyIndex=0
a1.sinks.hbase-sink1.serializer.depositHeaders=true

Note

A:In order to using rowKey, you should configure  rowKeyIndex=0   and  colNames=ROW_KEY.....  where in you post Josn data, rowkey must be the first filed. 

B: If you want to put the headers info of your json post, you must set depositHeaders=true

 

 

a1.sources.http-source1.channels = ch1
a1.sources.http-source1.type = http
a1.sources.http-source1.bind = 0.0.0.0
a1.sources.http-source1.port = 5140
a1.sources.http-source1.handler = org.apache.flume.source.http.JSONHandler

 

a1.channels = ch1
a1.sources = http-source1
a1.sinks = hbase-sink1

 

Hbase

#hbase shell

>create 'users' 'info'

 

Curl post json

curl -i -H 'content-type: application/json' -X POST 
-d '[{"headers":{"userId":"9","name":"ZhangZiYi","phoneNumber":"1522323222"},
"body":"9\tZhangZiYi\tzy@163.com"}]' http://192.168.10.204:5140

 

Hbase result

>scan 'users'



 

 

Note: the name column, which content comes from headers and boy of JSON, will just overwrite the same content. Acutally, you can specify different column names to save the same content in different cells.

 

 

References

http://flume.apache.org/FlumeUserGuide.html#hbasesinks

http://thunderheadxpler.blogspot.jp/2013/09/bigdata-apache-flume-hdfs-and-hbase.html

 

 

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值