HDFS balancer实践

HDFS balancer

在hadoop的hdfs集群中新添加节点后，在新节点上运行./start-balancer.sh。

运行的日志信息如下，重复判断、迁移，速度相当慢。

为了降低集群负荷、避免干扰其他用户，均衡器被设计为在后台运行。在不同节点之间复制数据的带宽也是受限的，默认值是1MB/s。

2 over-utilized: [Source[192.168.70.94:50010, utilization=61.784966362172845], Source[192.168.70.92:50010, utilization=64.37189641437116]]

3 underutilized: [BalancerDatanode[192.168.60.153:50010, utilization=37.48304150996222], BalancerDatanode[192.168.60.151:50010, utilization=38.02279713694489], BalancerDatanode[192.168.60.154:50010, utilization=8.601460127855951]]

Need to move 1.71 TB to make the cluster balanced.

Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.153:50010

Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.151:50010

Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.154:50010

Will move 30 GB in this iteration

2015-10-13 09:21:29,585 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: namenodes = [hdfs://hmaster91/]2015-10-13 09:21:29,598 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: p         = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0]2015-10-13 09:21:30,350 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable2015-10-13 09:21:32,371 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.93:500102015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.153:500102015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.151:500102015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.94:500102015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.152:500102015-10-13 09:21:32,373 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.92:500102015-10-13 09:21:32,374 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.154:500102015-10-13 09:21:32,375 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 over-utilized: [Source[192.168.70.94:50010, utilization=61.784966362172845], Source[192.168.70.92:50010, utilization=64.37189641437116]]2015-10-13 09:21:32,375 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 underutilized: [BalancerDatanode[192.168.60.153:50010, utilization=37.48304150996222], BalancerDatanode[192.168.60.151:50010, utilization=38.02279713694489], BalancerDatanode[192.168.60.154:50010, utilization=8.601460127855951]]2015-10-13 09:21:32,380 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 1.71 TB to make the cluster balanced.2015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.153:500102015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.151:500102015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.154:500102015-10-13 09:21:32,381 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GB in this iteration2015-10-13 09:31:41,945 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515941_8775312 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:500102015-10-13 09:31:47,378 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610315_8869686 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.153:500102015-10-13 09:31:59,425 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082639980_8899357 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.94:500102015-10-13 09:32:02,929 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515919_8775290 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:500102015-10-13 09:32:03,938 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082510026_8769397 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:500102015-10-13 09:32:05,370 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1078753887_5013250 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.70.92:500102015-10-13 09:32:06,871 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610299_8869670 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.153:500102015-10-13 09:32:10,923 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610274_8869645 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.70.94:500102015-10-13 09:32:12,930 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082515963_8775334 with size=134217728 from 192.168.70.92:50010 to 192.168.60.151:50010 through 192.168.70.93:500102015-10-13 09:32:13,424 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082509872_8769243 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:500102015-10-13 09:32:13,429 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082610182_8869553 with size=134217728 from 192.168.70.93:50010 to 192.168.60.154:50010 through 192.168.60.151:500102015-10-13 09:32:17,423 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082509962_8769333 with size=134217728 from 192.168.70.94:50010 to 192.168.60.153:50010 through 192.168.70.94:500102015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.151:500102015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.94:500102015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.154:500102015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.93:500102015-10-13 09:32:43,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.70.92:500102015-10-13 09:32:43,570 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.153:500102015-10-13 09:32:43,570 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/192.168.60.152:500102015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 2 over-utilized: [Source[192.168.70.94:50010, utilization=61.598206824722595], Source[192.168.70.92:50010, utilization=64.2691664228416]]2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 3 underutilized: [BalancerDatanode[192.168.60.151:50010, utilization=38.248797525595634], BalancerDatanode[192.168.60.154:50010, utilization=8.900929142083783], BalancerDatanode[192.168.60.153:50010, utilization=37.771715581234325]]2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 1.68 TB to make the cluster balanced.2015-10-13 09:32:43,570 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.94:50010 to 192.168.60.151:500102015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.92:50010 to 192.168.60.154:500102015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 192.168.70.93:50010 to 192.168.60.153:500102015-10-13 09:32:43,571 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GB in this iteration2015-10-13 09:42:49,653 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082424981_8684352 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.94:500102015-10-13 09:42:51,028 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082269380_8528751 with size=125931484 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.93:500102015-10-13 09:43:12,136 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082558990_8818361 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.93:500102015-10-13 09:43:17,073 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082547829_8807200 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.92:500102015-10-13 09:43:17,135 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082558979_8818350 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.93:500102015-10-13 09:43:18,541 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082544853_8804224 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.93:500102015-10-13 09:43:20,635 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082685915_8945292 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.94:500102015-10-13 09:43:24,574 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082685889_8945266 with size=134217728 from 192.168.70.92:50010 to 192.168.60.154:50010 through 192.168.70.92:500102015-10-13 09:43:25,574 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082423441_8682812 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.92:500102015-10-13 09:43:27,540 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082675683_8935060 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.94:500102015-10-13 09:43:27,563 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082675703_8935080 with size=134217728 from 192.168.70.94:50010 to 192.168.60.151:50010 through 192.168.70.94:500102015-10-13 09:43:28,065 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Successfully moved blk_1082424970_8684341 with size=134217728 from 192.168.70.93:50010 to 192.168.60.153:50010 through 192.168.70.94:50010