hadoop balancer hbase balancer

Hadoop 均衡器

Hadoop在运行过程中，其datanode的块会越来越不平衡，不平衡的集群会导致部分datanode相对更繁忙。

Hadoop的均衡器是一个守护进程。它会重新分配块，将块从忙碌的datanode移到相对空闲的datanode。同时坚持复本策略，将复本分散到不同机架，以降低数据损坏率。

集群均衡标准：每个datanode的使用率和集群的使用率非常接近，差距不超过给定的阀值。

datanode使用率：该节点上已使用的空间与空间总量之间的比率；

集群的使用率：集群中已使用的空间与集群的总空间总量之间的比率。

为了降低集群负荷、避免干扰其他用户，均衡器后台运行，并限制不同节点之间复制数据的带宽，默认值为1MB/s。

Hadoop的均衡器默认是关闭的，启动需要执行%start-balancer.sh，集群均衡后会自动停止。

HBase 负载均衡

HBase的均衡器默认五分钟执行一次，由 hbase.balancer.period属性设置。均衡器运行后会尝试均匀分配region到所有region服务器。

先确定一个region分配计划，然后开始移动region。

通过均衡器的开关类控制开启和关闭状态。

HBase的均衡器默认是开启的，并周期性运行。

下面是官网的解释说明：

---

Hadoop Balancer

Over time, the distribution of blocks across datanodes can become unbalanced. An unbalanced cluster can affect locality for MapReduce, and it puts a greater strain on the highly utilized datanodes, so it’s best avoided.

The balancer program is a Hadoop daemon that redistributes blocks by moving them from overutilized datanodes to underutilized datanodes, while adhering to the block replica placement policy that makes data loss unlikely by placing block replicas on different racks (see Replica Placement). It moves blocks until the cluster is deemed to be balanced, which means that the utilization of every datanode (ratio of used space on the node to total capacity of the node) differs from the utilization of the cluster (ratio of used space on the cluster to total capacity of the cluster) by no more than a given threshold percentage. You can start the balancer with:

% start-balancer.sh

The -threshold argument specifies the threshold percentage that defines what it means for the cluster to be balanced. The flag is optional; if omitted, the threshold is 10%. At any one time, only one balancer may be running on the cluster.

The balancer runs until the cluster is balanced, it cannot move any more blocks, or it loses contact with the namenode. It produces a logfile in the standard log directory, where it writes a line for every iteration of redistribution that it carries out. Here is the output from a short run on a small cluster (slightly reformatted to fit the page):

Time Stamp      Iteration#  Bytes Already Moved  ...Left To Move  ...Being Moved
Mar 18, 2009 5:23:42 PM  0                 0 KB        219.21 MB       150.29 MB
Mar 18, 2009 5:27:14 PM  1            195.24 MB         22.45 MB       150.29 MB
The cluster is balanced. Exiting...
Balancing took 6.072933333333333 minutes

The balancer is designed to run in the background without unduly taxing the cluster or interfering with other clients using the cluster. It limits the bandwidth that it uses to copy a block from one node to another. The default is a modest 1 MB/s, but this can be changed by setting the dfs.datanode.balance.bandwidthPerSec property in hdfs-site.xml, specified in bytes.

---

HBase Load Balancing

The master has a built-in feature, called the balancer. By default, the balancer runs every five minutes, and it is configured by the hbase.balancer.period property. Once the balancer is started, it will attempt to equal out the number of assigned regions per region server so that they are within one region of the average number per server. The call first determines a new assignment plan, which describes which regions should be moved where. Then it starts the process of moving the regions by calling the unassign() method of the administrative API iteratively.

The balancer has an upper limit on how long it is allowed to run, which is configured using the hbase.balancer.max.balancing property anddefaults to half of the balancer period value, or two and a half minutes.

You can control the balancer by means of the balancer switch: either use the shell’s balance_switch command to toggle the balancer status between enabled and disabled, or use the balanceSwitch() API method to do the same. When you disable the balancer, it no longer runs as expected.

The balancer can be explicitly started using the shell’s balancer command, or using the balancer() API method. The time-controlled invocation mentioned previously calls this method implicitly. It will determine if there is any work to be done and return true if that is the case. The return value of false means that it was not able to run the balancer, because either it was switched off, there was no work to be done (all is balanced), or something else was prohibiting the process. One example for this is the region in transition list (see Main page): if there is a region currently in transition, the balancer will be skipped.

Instead of relying on the balancer to do its work properly, you can use the move command and API method to assign regions to other servers. This is useful when you want to control where the regions of a particular table are assigned. See Region Hotspotting for an example.