HBase Compression

最新推荐文章于 2025-05-19 16:20:25 发布

macyang

最新推荐文章于 2025-05-19 16:20:25 发布

阅读量4.7k

点赞数

CC 4.0 BY-SA版权

分类专栏： database/nosql 文章标签： compression hbase algorithm command performance encoding

本文链接：https://blog.youkuaiyun.com/macyang/article/details/6618541

database/nosql 专栏收录该内容

102 篇文章

订阅专栏

本文介绍了HBase的压缩算法，包括GZIP, LZO和Snappy，并推荐使用Snappy，因其具备良好的编码/解码速度和压缩率。文章强调了在非压缩内容如JPEG图片之外的场景中启用压缩通常能提升整体性能，因为CPU的压缩和解压开销小于从磁盘读取更多数据的代价。同时，讲述了如何启用压缩以及可用的压缩算法特点。" 134620021,8658200,OpenGL在Qt中实现图元颜色设置,"['OpenGL', 'Qt', 'C++', '图形编程']

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Compression就是在用CPU换IO吞吐量/磁盘空间，如果没有什么特殊原因推荐针对Column Family设置compression，下面主要有三种算法: GZIP, LZO, Snappy，作者推荐使用Snappy，因为它有较好的Encoding/Decoding速度和可以接受的压缩率。

HBase comes with support for a number of compression algorithims that can be enabled at the column family level. Enabling compression is recommended unless you have a reason not to do so, for example, when using already compressed content, such as JPEG images. For every other use-case compression usually will yield an overall better performance, because the overhead of the CPU performing the compression and decompression is less than what is required to read more data from disk.

Available Codecs

You can choose from a fixed list of supported compression algorithms. They have different qualities when it comes to compression ratio, as well as CPU and installation requirements.

Table 11.1. Comparison between compression algorithms

Algorithm	% remaining	Encoding	Decoding
GZIP	13.4%	21 MB/s	118 MB/s
LZO	20.5%	135 MB/s	410 MB/s
Zippy/Snappy	22.2%	172 MB/s	409 MB/s

Note that some of the algorithms have a better compression ration while others are faster for the encoding, and a lot faster during decoding. Depending on your use-case you can choose one that suits you best.

Enabling Compression

Enabling compression requires the installation of the JNI and native compression libraries (unless you only want to use the Java code based GZIP compression), as described above, and specifying the chosen algorithm in the column family schema.

One way to accomplish this is during the creation of the table. The possible values are listed in the section called “Column Families”:

hbase(main):001:0> create 'testtable', { NAME => 'colfam1', COMPRESSION => 'GZ' }  
0 row(s) in 1.1920 seconds

hbase(main):012:0> describe 'testtable'                                            
DESCRIPTION                                                 ENABLED
{NAME => 'testtable', FAMILIES => [{NAME => 'colfam1',      true 
BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS 
=> '3', COMPRESSION => 'GZ', TTL => '2147483647', BLOCKSIZE
=> '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
1 row(s) in 0.0400 seconds

The describe shell command is used to read back the schema of the newly created table. You can see the compression is set to GZIP (using the shorter "GZ" value as required). Another option to enable - or change, or disable - the compression algorithm is using the alter command for existing tables:

hbase(main):013:0> create 'testtable2', 'colfam1'
0 row(s) in 1.1920 seconds

hbase(main):014:0> disable 'testtable2'
0 row(s) in 2.0650 seconds

hbase(main):016:0> alter 'testtable2', { NAME => 'colfam1', COMPRESSION => 'GZ' }
0 row(s) in 0.2190 seconds

hbase(main):017:0> enable 'testtable2'
0 row(s) in 2.0410 seconds

Note how the table was first disabled. This is necessary to perform the alteration of the column family definition. The final enable command brings the table back online.