hbase region与内存的关系

最新推荐文章于 2023-05-23 11:31:00 发布

转载最新推荐文章于 2023-05-23 11:31:00 发布 · 172 阅读

0 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/jixiangxiang/p/5520871.html

文章标签：

#大数据

本文介绍了在HBase中如何估算每个RegionServer可以承载的最大Region数量。通过公式计算，在默认设置下，拥有16GB RAM的RegionServer大约可以支持51个Region。文章还讨论了memstore内存使用对Region数量的影响。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

http://hbase.apache.org/book.html#ops.capacity.regions.count

In production scenarios, where you have a lot of data, you are normally concerned with the maximum number of regions you can have per server. too many regions has technical discussion on the subject. Basically, the maximum number of regions is mostly determined by memstore memory usage. Each region has its own memstores; these grow up to a configurable size; usually in 128-256 MB range, see hbase.hregion.memstore.flush.size. One memstore exists per column family (so there’s only one per region if there’s one CF in the table). The RS dedicates some fraction of total memory to its memstores (see hbase.regionserver.global.memstore.size). If this memory is exceeded (too much memstore usage), it can cause undesirable consequences such as unresponsive server or compaction storms. A good starting point for the number of regions per RS (assuming one table) is:

((RS memory) * (total memstore fraction)) / ((memstore size)*(# column families))

This formula is pseudo-code. Here are two formulas using the actual tunable parameters, first for HBase 0.98+ and second for HBase 0.94.x.

HBase 0.98.x

((RS Xmx) * hbase.regionserver.global.memstore.size) / (hbase.hregion.memstore.flush.size * (# column families))

HBase 0.94.x

((RS Xmx) * hbase.regionserver.global.memstore.upperLimit) / (hbase.hregion.memstore.flush.size * (# column families))+

If a given RegionServer has 16 GB of RAM, with default settings, the formula works out to 16384*0.4/128 ~ 51 regions per RS is a starting point. The formula can be extended to multiple tables; if they all have the same configuration, just use the total number of families.

转载于:https://www.cnblogs.com/jixiangxiang/p/5520871.html