Hadoop单节点磁盘均衡

Hadoop的balancer主要是针对于集群之间的数据均衡，但是有些时候节点内有新增磁盘或者数据出现在磁盘上不均衡时，需要做磁盘均衡，就是将其他已经写入数据的磁盘均衡到新增加的磁盘上去，大概分为以下三个步骤，（plan）计划，（execute）执行，（query）查询

如上图需要将Hadoop103节点/app的数据均衡到/app2中

1、生成plan计划（hdfs diskbalancer -plan node.example.com -bandwidth 100 --thresholdPercentage 10）

hdfs diskbalancer -plan node.example.com -bandwidth 100 --thresholdPercentage 10

1. hdfs diskbalancer -plan
    这是 HDFS Disk Balancer 的子命令，用于生成数据均衡计划。
    作用：分析指定节点的磁盘使用情况，生成一个均衡计划文件（JSON 格式），指导如何将数据从高使用率的磁盘迁移到低使用率的磁盘。

2. node.example.com
    这是目标 DataNode 的主机名或地址。
    作用：指定需要生成均衡计划的 DataNode 节点。

3. -bandwidth 100
    这是可选参数，用于限制数据迁移的带宽。
    作用：设置 Disk Balancer 在执行数据迁移时的最大带宽为 100 MB/s。
    单位：MB/s。
    默认值：如果不指定，HDFS 会使用 dfs.datanode.balance.bandwidthPerSec 配置的值（默认 10 MB/s）。
    意义：限制带宽可以避免 Disk Balancer 占用过多网络和磁盘资源，影响集群性能。

4. --thresholdPercentage 10
    这是可选参数，用于设置触发数据均衡的磁盘使用差异阈值。
    作用：当某个 DataNode 的磁盘使用率差异超过 10% 时，Disk Balancer 才会生成均衡计划。
    单位：百分比（%）。
    默认值：如果不指定，HDFS 会使用默认的阈值（通常为 10%）。
    意义：避免对磁盘使用差异较小的节点进行不必要的均衡操作。

 plan.json

[wwwroot@hadoop103 ~]$ hdfs dfs -cat /system/diskbalancer/2022-Nov-17-10-21-45/cdh192-22.plan.json
{"volumeSetPlans":[{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data10/","capacity":999716507648,"storageType":"DISK","used":518103326696,"reserved":0,"uuid":"DS-4ea78bac-85e4-4d7a-92f6-72d02502e30f","failed":false,"volumeDataDensity":-0.0534,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data7/","capacity":2399304445952,"storageType":"DISK","used":1115196706478,"reserved":0,"uuid":"DS-40172040-5a50-4f68-8201-208237f26d11","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":338470576696,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"},{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data11/","capacity":999716507648,"storageType":"DISK","used":464668232754,"reserved":0,"uuid":"DS-90f7fe25-1933-4167-8c14-2a9e2a143d0f","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data4/","capacity":2399304445952,"storageType":"DISK","used":1115196706478,"reserved":0,"uuid":"DS-b0cd02a7-2bf7-4457-abe9-14edd486d28e","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":259615596345,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"},{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data8/","capacity":999716507648,"storageType":"DISK","used":464668232754,"reserved":0,"uuid":"DS-176d5992-75bd-4767-9366-93df8a0fd159","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data3/","capacity":2399304445952,"storageType":"DISK","used":1115196706478,"reserved":0,"uuid":"DS-35e8ab71-a899-4ef7-be3f-d9c1f1c1561b","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":243296795330,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"},{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data9/","capacity":999716507648,"storageType":"DISK","used":464668232754,"reserved":0,"uuid":"DS-8b22a280-09ff-4f41-ae0a-55eaa62d300e","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data6/","capacity":2399304445952,"storageType":"DISK","used":1115196706478,"reserved":0,"uuid":"DS-5179da53-7b07-465e-b07e-b4a8795c85ce","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":190630379536,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"},{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data9/","capacity":999716507648,"storageType":"DISK","used":464668232754,"reserved":0,"uuid":"DS-8b22a280-09ff-4f41-ae0a-55eaa62d300e","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data2/","capacity":2399304445952,"storageType":"DISK","used":1107097449681,"reserved":0,"uuid":"DS-ad4ad5f7-4bc5-4cfd-80cb-cff8c6b837ca","failed":false,"volumeDataDensity":0.003400000000000014,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":181871783815,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"},{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data8/","capacity":999716507648,"storageType":"DISK","used":464668232754,"reserved":0,"uuid":"DS-176d5992-75bd-4767-9366-93df8a0fd159","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data5/","capacity":2399304445952,"storageType":"DISK","used":1115196706478,"reserved":0,"uuid":"DS-183b713c-f95d-45f5-8f1e-530ea8a15ec1","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":141152455299,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"},{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data11/","capacity":999716507648,"storageType":"DISK","used":464668232754,"reserved":0,"uuid":"DS-90f7fe25-1933-4167-8c14-2a9e2a143d0f","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data1/","capacity":2399304445952,"storageType":"DISK","used":1070758333399,"reserved":0,"uuid":"DS-ccc089fe-bb80-40d5-983e-fe678e7607b3","failed":false,"volumeDataDensity":0.018600000000000005,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":140740287938,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"},{"@class":"org.apache.hadoop.hdfs.server.diskbalancer.planner.MoveStep","sourceVolume":{"path":"/dfs/data10/","capacity":999716507648,"storageType":"DISK","used":518103326696,"reserved":0,"uuid":"DS-4ea78bac-85e4-4d7a-92f6-72d02502e30f","failed":false,"volumeDataDensity":-0.0534,"skip":false,"transient":false,"readOnly":false},"destinationVolume":{"path":"/dfs/data5/","capacity":2399304445952,"storageType":"DISK","used":1115196706478,"reserved":0,"uuid":"DS-183b713c-f95d-45f5-8f1e-530ea8a15ec1","failed":false,"volumeDataDensity":9.999999999998899E-5,"skip":false,"transient":false,"readOnly":false},"idealStorage":0.4648,"bytesToMove":47840136002,"volumeSetID":"9bbee851-9964-4d91-8a61-41ae86901e37"}],"nodeName":"cdh192-22","nodeUUID":"13584e04-9fdd-468b-96da-59e6be03259b","port":9867,"timeStamp":1668651705345}

上述plan.json主要字段解释

每个 MoveStep 对象包含以下关键字段：
    sourceVolume：数据来源磁盘的详细信息。
    destinationVolume：数据目标磁盘的详细信息。
    idealStorage：理想存储比例（即均衡后的目标使用率）。
    bytesToMove：需要迁移的数据量（单位：字节）。
    volumeSetID：磁盘组的唯一标识。
    bandwidth：数据迁移的带宽限制（单位：MB/s）。

文件末尾还包含了一些节点信息：
    nodeName：执行均衡的 DataNode 节点名称。
    nodeUUID：DataNode 的唯一标识。
    port：DataNode 的端口号。
    timeStamp：均衡计划生成的时间戳。

2、执行计划 （hdfs diskbalancer -execute /system/diskbalancer/2025-二月-17-09-02-01/hadoop103.plan.json）

3、 查看计划（hdfs diskbalancer -query hadoop103）

状态说明：
PLAN_UNDER_PROGRESS 计划进行中
PLAN_DONE 计划执行完成

4. 停止 Disk Balancer 操作（如果需要）

HDFS Disk Balancer 是一个后台任务，停止它的方法如下：

• 使用以下命令取消正在执行的 Disk Balancer 计划：

  hdfs diskbalancer -cancel <plan_file>

5、参考其他资料，未尝试

再后来百度大法，发现在还有个参数可以修改：
hdfs-site.xml
修改datanode写入策略，该配置默认是轮训，我们要改为以剩余空间考虑写入某块磁盘

<property>
    <name>dfs.datanode.fsdataset.volume.choosing.policy</name>
<value>org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy</value>
</property>

以下参数是配置各个磁盘的均衡阈值的，默认为10G。
在此节点的所有数据存储的目录中，找一个占用最大的，找一个占用最小的。
如果在两者之差在10G的范围内，那么块分配的方式是轮询。如下为英文原文。

This setting controls how much DN volumes are allowed to differ in terms of bytes of free disk space before they are considered imbalanced. If the free space of all the volumes are within this range of each other, the volumes will be considered balanced and block assignments will be done on a pure round robin basis.

<property>
    <name>dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold </name>
    <value>10737418240</value>
</property>

通过调整以上2个参数，应该就可以达到我们期望的效果了。
即当每个目录的剩余空间的最大值和最小值差距在10G以内时，使用轮询写入，否则优先写入空间比较大的目录。