python 访问hbase

通过thrift，我们可以使用python访问hbase。

关于thrift

thrift是一个跨语言服务的软件开发框架（Thrift is a software framework for scalable cross-language services development.）。

它的官方网站是：http://incubator.apache.org/thrift/

1 下载thrift

svn co http://svn.apache.org/repos/asf/incubator/thrift/trunk thrift

基本安装工具环境yum install automake libtool flex bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel./configure 配置 ，看到如下信息：

thrift 0.9.1
Building C++ Library ......... : no
Building C (GLib) Library .... : no
Building Java Library ........ : yes
Building C# Library .......... : no
Building Python Library ...... : yes
Building Ruby Library ........ : no
Building Haskell Library ..... : no
Building Perl Library ........ : no
Building PHP Library ......... : no
Building Erlang Library ...... : no
Building Go Library .......... : no
Building D Library ........... : no
thrift 0.9.1

Building C++ Library ......... : no

Building C (GLib) Library .... : no

Building Java Library ........ : yes

Building C# Library .......... : no

Building Python Library ...... : yes

Building Ruby Library ........ : no

Building Haskell Library ..... : no

Building Perl Library ........ : no

Building PHP Library ......... : no

Building Erlang Library ...... : no

Building Go Library .......... : no

Building D Library ........... : no

Java Library:

   Using javac ............... : javac

   Using java ................ : java

   Using ant ................. : /usr/bin/ant

Python Library:

   Using Python .............. : /usr/bin/python

make

make install 

查看thrift 版本  

thrift -version 

2. 安装python 的thrift 包

[root@YZSJHL19-81 py]# cd /opt/thrift-0.9.1/lib/py/

[root@YZSJHL19-81 py]# ls

build  compat  dist  Makefile  Makefile.am  Makefile.in  README  setup.cfg  setup.py  src  thrift.egg-info

[root@YZSJHL19-81 py]# python setup.py build

running build

running build_py

creating build

creating build/lib.linux-x86_64-2.6

....................
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.6 -c src/protocol/fastbinary.c -o build/temp.linux-x86_64-2.6/src/protocol/fastbinary.o
gcc -pthread -shared build/temp.linux-x86_64-2.6/src/protocol/fastbinary.o -L/usr/lib64 -lpython2.6 -o build/lib.linux-x86_64-2.6/thrift/protocol/fastbinary.so

[root@YZSJHL19-81 py]# python setup.py install
running install
running bdist_egg
running egg_info
......
Adding thrift 0.9.1 to easy-install.pth file
Installed /usr/lib64/python2.6/site-packages/thrift-0.9.1-py2.6-linux-x86_64.egg
Processing dependencies for thrift==0.9.1
Finished processing dependencies for thrift==0.9.1

3 生成python和hbase通讯的thrift包：

[root@YZSJHL19-81 thrift]# cd /opt/hbase/src/main/resources/org/apache/hadoop/hbase/thrift

[root@YZSJHL19-81 thrift]# ls

Hbase.thrift

[root@YZSJHL19-81 thrift]#  thrift --gen py:new_style Hbase.thrift

[root@YZSJHL19-81 thrift]# ls

gen-py Hbase.thrift 

[root@YZSJHL19-81 thrift]# ll -t 

总用量 28

drwxr-xr-x 3 root  root   4096 1月  14 21:11 gen-py.bak

-rwxr-xr-x 1 hbase hbase 23434 8月  28 2012 Hbase.thrift

[root@YZSJHL19-81 thrift]# mv gen-py /usr/lib/python2.6/site-packages/

4 启动hbase 的thrift服务 

hbase thrift -p 19090 -nonblocking start

5.编写测试程序

 1 #!/usr/bin/python

  2 #coding:utf8

  3 #author:guoqiang.ma

  4 #date:2014.1.15

  5 import sys

  6 #Hbase.thrift生成的py文件放在这里

  7 sys.path.append('/usr/lib/python2.6/site-packages/gen-py')

  8 from thrift import Thrift

  9 from thrift.transport import TSocket

 10 from thrift.transport import TTransport

 11 from thrift.protocol import TBinaryProtocol

 12

 13 from hbase import Hbase

 14 #如ColumnDescriptor 等在hbase.ttypes中定义

 15 from hbase.ttypes import *

 16

 17 # Make socket

 18 #此处可以修改地址和端口

 19 transport = TSocket.TSocket('localhost', 19090)

 20 # Buffering is critical. Raw sockets are very slow

 21 # 还可以用TFramedTransport,也是高效传输方式

 22 #transport = TTransport.TBufferedTransport(transport)

 23 transport = TTransport.TFramedTransport(transport)

 24 # Wrap in a protocol

 25 #传输协议和传输过程是分离的，可以支持多协议

 26 protocol = TBinaryProtocol.TBinaryProtocol(transport)

 27 #客户端代表一个用户

 28 client = Hbase.Client(protocol)

 29 #打开连接

 30 transport.open()

 31

 32 print client.getTableNames()

6.产看测试结果

python t.py 

[user_growth@YZSJHL19-81 guoqiang.ma]$ python t.py 

['atme', 'delete_friend', 'doing', 'doing2', 'fans_pages', 

.........................................

.........................................

.........................................]

参考资料：

http://www.cnblogs.com/hitandrew/archive/2013/01/21/2870419.html