vdbench设置多进程时，对maxdata的影响

最新推荐文章于 2024-08-19 16:07:30 发布

原创最新推荐文章于 2024-08-19 16:07:30 发布 · 1.4k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#linux #服务器 #运维

本文探讨了在vdbench中使用多进程时，maxdata参数如何影响程序执行。实验结果显示，vdbench会在所有JVMs完成总共10GB的数据读写后结束，而非每个JVM单独完成maxdata大小。通过调整jvms数量，可以更充分地测试硬盘性能，避免单进程限制。当单进程IOPS超过100,000时，建议调整jvm数量，但实际操作未出现警告。" 124911522,7056182,Win10引导区修复：固态硬盘系统无机械硬盘启动问题,"['Windows', '系统修复', '引导修复', '硬盘管理']

有时候为了测量硬盘IO的极限性能。我们会开启多进程的方式，如fio中的numjobs。vdbench中也有类似的参数，经查阅手册发现，设置vdbench的配置文件中jvms=nn即可。

这时候有了一个疑问，如果设运行时间elapsed非常大。maxdata=10G，那么整个程序是跑完40G才结束，还是一起跑完10G就结束了呢？在fio中是跑完40G才会结束。而在vdbench中，经过实验，发现其是JVMs一共跑完10G结束。即vdbench一共需要读写maxdata大的数据，而不是 jvms × maxdata的数据。

实验步骤

准备一个配置文件， 4k-read-16-jvms4.txt。文件内容如下：

hd=default,vdbench=/root/vdbench50406,user=root,jvms=4
sd=sd1,lun=/dev/sdb,openflags=o_direct,threads=16
wd=wd1,sd=sd*,rdpct=100,seekpct=100,xfersize=4k
rd=rd1,wd=wd1,iorate=max,elapsed=600,format=restart,maxdata=10G,interval=5,warmup=30

Note：配置文件中jvms=4前面，user=root后面，不能加空格哦，否则只能开启一个进程。
2. 调用cat /proc/diskstats，查看/dev/sdb读了多少次。

[root@node1 nvme_test]# cat /proc/diskstats
  11       0 sr0 0 0 0 0 0 0 0 0 0 0 0
   8      16 sdb 20297933 51 162388576 28350597 0 0 0 0 0 1805030 28338667
   8       0 sda 8759 48 632577 49030 19552 2755 305227 47172 0 28043 96051
   8       1 sda1 2032 0 52020 2398 34 0 4216 115 0 1023 2509
   8       2 sda2 6696 48 577213 46422 19518 2755 301011 47057 0 27585 93329
 253       0 dm-0 6851 0 568037 49260 22274 0 301011 57464 0 27542 106724
 253       1 dm-1 90 0 4920 421 0 0 0 0 0 338 421

可以看出sdb目前已经读了20297933次。

接下来检测jvms=4的情况，即运行4k-read-16-jvms4.txt这个配置文件。看看多线程时的情况。

[root@node1 nvme_test]# ../vdbench50406/vdbench -f 4k-read-16-jvms4.txt -o 4k-read-16-jvms4


Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Vdbench distribution: vdbench50406 Wed July 20 15:49:52 MDT 2016
For documentation, see 'vdbench.pdf'.

10:09:02.320 input argument scanned: '-f4k-read-16-jvms4.txt'
10:09:02.321 input argument scanned: '-o4k-read-16-jvms4'
10:09:02.505 Starting slave: /root/vdbench50406/vdbench SlaveJvm -m localhost -n localhost-10-220826-10.09.02.273 -l localhost-0 -p 5570
10:09:02.525 Starting slave: /root/vdbench50406/vdbench SlaveJvm -m localhost -n localhost-11-220826-10.09.02.273 -l localhost-1 -p 5570
10:09:02.546 Starting slave: /root/vdbench50406/vdbench SlaveJvm -m localhost -n localhost-12-220826-10.09.02.273 -l localhost-2 -p 5570
10:09:02.567 Starting slave: /root/vdbench50406/vdbench SlaveJvm -m localhost -n localhost-13-220826-10.09.02.273 -l localhost-3 -p 5570
10:09:03.094 All slaves are now connected
10:09:04.002 Starting RD=rd1; I/O rate: Uncontrolled MAX; elapsed=600 warmup=30; For loops: None

八月 26, 2022  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%
                             rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys
10:09:09.065         1    8320.20    32.50    4096 100.00    1.763    1.763    0.000   17.332    1.210  14.7   6.2   2.0
10:09:14.049         2    8631.40    33.72    4096 100.00    1.839    1.839    0.000   20.747    1.191  15.9   2.7   1.6
10:09:19.072         3    9101.80    35.55    4096 100.00    1.752    1.752    0.000   13.134    1.058  15.9   2.3   1.6
10:09:24.054         4    8642.20    33.76    4096 100.00    1.844    1.844    0.000   14.687    1.112  15.9   2.6   1.7
10:09:29.049         5    8663.40    33.84    4096 100.00    1.838    1.838    0.000   12.668    1.115  15.9   2.5   1.7
10:09:34.048         6    8352.80    32.63    4096 100.00    1.907    1.907    0.000   17.018    1.220  15.9   2.3   1.6
10:09:39.048         7    8999.40    35.15    4096 100.00    1.774    1.774    0.000   13.956    1.118  16.0   2.2   1.5
10:09:44.050         8    8768.60    34.25    4096 100.00    1.822    1.822    0.000   12.305    1.116  16.0   2.1   1.4
10:09:49.048         9    9354.20    36.54    4096 100.00    1.720    1.720    0.000   11.333    1.052  16.1   2.3   1.6
10:09:54.053        10    8937.60    34.91    4096 100.00    1.772    1.772    0.000   12.688    1.078  15.8   2.2   1.5
10:09:59.046        11    8737.60    34.13    4096 100.00    1.826    1.826    0.000   16.587    1.129  15.9   2.1   1.5
10:10:03.515
10:10:03.515 Message from slave localhost-0:
10:10:03.515 New messages found on /var/adm/messages. Do they belong to you?
10:10:03.516 /var/log/messages: Aug 26 10:10:01 localhost systemd: Started Session 537 of user root.
10:10:03.516
10:10:04.065        12    8556.40    33.42    4096 100.00    1.868    1.868    0.000   20.744    1.223  16.0   2.3   1.6
10:10:09.045        13    8765.60    34.24    4096 100.00    1.819    1.819    0.000   21.772    1.160  15.9   2.1   1.5
10:10:14.048        14    8795.60    34.36    4096 100.00    1.815    1.815    0.000   11.975    1.073  16.0   2.2   1.6
10:10:19.046        15    9128.80    35.66    4096 100.00    1.748    1.748    0.000   11.461    1.050  16.0   2.2   1.6
10:10:24.053        16    9485.20    37.05    4096 100.00    1.684    1.684    0.000   10.732    1.015  16.0   2.0   1.4
10:10:29.047        17    9250.40    36.13    4096 100.00    1.726    1.726    0.000   12.174    1.027  16.0   2.1   1.4
10:10:34.054        18    8730.60    34.10    4096 100.00    1.844    1.844    0.000   18.425    1.244  16.1   2.0   1.4
10:10:39.046        19    8414.80    32.87    4096 100.00    1.882    1.882    0.000   24.395    1.345  15.8   2.1   1.6
10:10:44.047        20    9082.80    35.48    4096 100.00    1.758    1.758    0.000   14.908    1.076  16.0   2.1   1.5
10:10:49.049        21    9132.40    35.67    4096 100.00    1.749    1.749    0.000   12.295    1.049  16.0   2.2   1.4
10:10:54.046        22    8805.20    34.40    4096 100.00    1.813    1.813    0.000   13.813    1.096  16.0   2.2   1.6
10:10:59.046        23    9039.40    35.31    4096 100.00    1.767    1.767    0.000   16.105    1.097  16.0   2.1   1.5
10:11:04.053        24    8641.20    33.75    4096 100.00    1.848    1.848    0.000   40.085    1.351  16.0   2.0   1.4
10:11:09.046        25    8710.00    34.02    4096 100.00    1.833    1.833    0.000   17.646    1.134  16.0   2.6   1.6
10:11:14.047        26    8983.00    35.09    4096 100.00    1.777    1.777    0.000   12.127    1.079  16.0   2.1   1.5
10:11:19.045        27    8897.80    34.76    4096 100.00    1.808    1.808    0.000   13.992    1.087  16.1   2.2   1.6
10:11:24.048        28    9045.60    35.33    4096 100.00    1.752    1.752    0.000   10.998    1.051  15.8   2.2   1.6
10:11:29.045        29    9390.60    36.68    4096 100.00    1.700    1.700    0.000   12.483    1.034  16.0   2.1   1.5
10:11:34.048        30    8974.40    35.06    4096 100.00    1.778    1.778    0.000   19.610    1.136  16.0   2.2   1.5

八月 26, 2022  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%
                             rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys
10:11:39.048        31    8946.20    34.95    4096 100.00    1.785    1.785    0.000   13.209    1.095  16.0   2.1   1.5
10:11:44.046        32    9059.60    35.39    4096 100.00    1.762    1.762    0.000   11.297    1.052  16.0   2.0   1.5
10:11:49.045        33    8844.80    34.55    4096 100.00    1.805    1.805    0.000   12.684    1.089  16.0   2.2   1.6
10:11:54.048        34    9323.80    36.42    4096 100.00    1.712    1.712    0.000   12.522    1.024  16.0   2.1   1.5
10:11:59.046        35    9553.20    37.32    4096 100.00    1.672    1.672    0.000   17.353    1.066  16.0   2.2   1.4
10:12:04.062        36    9195.20    35.92    4096 100.00    1.751    1.751    0.000   28.613    1.155  16.1   2.1   1.4
10:12:09.051        37    8826.20    34.48    4096 100.00    1.795    1.795    0.000   20.196    1.126  15.8   2.2   1.6
10:12:14.047        38   10107.20    39.48    4096 100.00    1.580    1.580    0.000   12.851    0.993  16.0   2.2   1.6
10:12:19.045        39    9167.60    35.81    4096 100.00    1.742    1.742    0.000   13.725    1.042  16.0   2.1   1.5
10:12:24.047        40    9659.40    37.73    4096 100.00    1.653    1.653    0.000   10.241    0.998  16.0   2.0   1.5
10:12:29.045        41    9519.40    37.19    4096 100.00    1.678    1.678    0.000   14.405    1.030  16.0   1.9   1.3
10:12:34.071        42    8868.20    34.64    4096 100.00    1.801    1.801    0.000   22.198    1.196  16.0   2.1   1.5
10:12:39.045        43    9038.20    35.31    4096 100.00    1.767    1.767    0.000   31.730    1.197  16.0   2.0   1.5
10:12:44.049        44    9175.00    35.84    4096 100.00    1.741    1.741    0.000   19.132    1.130  16.0   2.1   1.4
10:12:49.045        45    8924.80    34.86    4096 100.00    1.803    1.803    0.000   13.396    1.089  16.1   2.2   1.6
10:12:54.046        46    9167.80    35.81    4096 100.00    1.728    1.728    0.000   11.254    1.024  15.8   2.1   1.5
10:12:59.046        47    9217.20    36.00    4096 100.00    1.733    1.733    0.000   13.284    1.040  16.0   1.9   1.4
10:13:04.049        48    8697.60    33.98    4096 100.00    1.835    1.835    0.000   14.856    1.159  16.0   2.3   1.6
10:13:09.047        49    9019.20    35.23    4096 100.00    1.771    1.771    0.000   17.207    1.100  16.0   2.1   1.5
10:13:14.046        50   10196.80    39.83    4096 100.00    1.566    1.566    0.000   12.974    0.970  16.0   2.4   1.7
10:13:19.047        51   10506.20    41.04    4096 100.00    1.520    1.520    0.000   13.152    0.948  16.0   2.5   1.9
10:13:24.046        52   10751.60    42.00    4096 100.00    1.485    1.485    0.000   11.400    0.921  16.0   2.8   2.2
10:13:29.044        53   11139.00    43.51    4096 100.00    1.434    1.434    0.000    8.964    0.878  16.0   3.2   2.6
10:13:34.049        54   10251.40    40.04    4096 100.00    1.570    1.570    0.000   28.854    1.065  16.1   2.4   1.8
10:13:39.046        55   10083.00    39.39    4096 100.00    1.572    1.572    0.000   19.040    1.029  15.8   2.6   1.9
10:13:43.551
10:13:43.551 Message from slave localhost-0:
10:13:43.551 New messages found on /var/adm/messages. Do they belong to you?
10:13:43.551 /var/log/messages: Aug 26 10:13:42 localhost systemd: Started license manager.
10:13:43.551 /var/log/messages: Aug 26 10:13:42 localhost license_check[16722]: *** WARNING ***Your license is invalid.Please register to BigCloud Enterprise Linux(BC-Linux) to get the right license.You can use the command 'sys-id' to get the system UUID, then send to us to complete the registration.
10:13:43.551 /var/log/messages: Aug 26 10:13:42 localhost license-manager: *** WARNING ***Your license is invalid.Please register to BigCloud Enterprise Linux(BC-Linux) to get the right license.You can use the command 'sys-id' to get the system UUID, then send to us to complete the registration.
10:13:43.551
10:13:44.044        56    9883.00    38.61    4096 100.00    1.615    1.615    0.000   26.282    1.147  16.0   2.4   1.6
10:13:49.045        57    8880.60    34.69    4096 100.00    1.798    1.798    0.000   22.723    1.161  16.0   2.1   1.5
10:13:54.048        58    9358.00    36.55    4096 100.00    1.707    1.707    0.000   10.842    1.024  16.0   2.1   1.5
10:13:59.046        59    9643.60    37.67    4096 100.00    1.656    1.656    0.000   11.095    1.024  16.0   1.9   1.3
10:14:04.057        60   10086.20    39.40    4096 100.00    1.584    1.584    0.000   11.795    0.997  16.0   2.2   1.6

八月 26, 2022  interval        i/o   MB/sec   bytes   read     resp     read    write     resp     resp queue  cpu%  cpu%
                             rate  1024**2     i/o    pct     time     resp     resp      max   stddev depth sys+u   sys
10:14:09.051        61    9929.00    38.79    4096 100.00    1.608    1.608    0.000   23.773    1.045  16.0   2.3   1.6
10:14:14.047        62    9455.00    36.93    4096 100.00    1.687    1.687    0.000   10.631    1.032  16.0   2.2   1.6
10:14:14.052 Reached maxdata=10g. rd=rd1 shutting down after next interval.
10:14:15.060        63    1836.00     7.17    4096 100.00    1.742    1.742    0.000   11.255    1.065   3.2   3.5   1.7
10:14:15.075  avg_7-63    9125.69    35.65    4096 100.00    1.726    1.726    0.000   40.085    1.090  15.7   2.2   1.6
10:14:16.431 Vdbench execution completed successfully. Output directory: /root/nvme_test/4k-read-16-jvms4

调用cat /proc/diskstats，查看/dev/sdb读了多少次。

[root@node1 nvme_test]# cat /proc/diskstats
  11       0 sr0 0 0 0 0 0 0 0 0 0 0 0
   8      16 sdb 23162053 57 185301584 33091351 0 0 0 0 0 2116246 33079472
   8       0 sda 8759 48 632577 49030 19780 2825 309917 47885 0 28230 96764
   8       1 sda1 2032 0 52020 2398 34 0 4216 115 0 1023 2509
   8       2 sda2 6696 48 577213 46422 19746 2825 305701 47770 0 27772 94042
 253       0 dm-0 6851 0 568037 49260 22572 0 305701 58534 0 27729 107794
 253       1 dm-1 90 0 4920 421 0 0 0 0 0 338 421

可以看到。目前sb读了23162053次。根据每次4kB（vdbench中设置xfersize=4k）,程序运行期间大约读了10.9GB的数据。（还有30s是硬盘热身，warmup），所以应该说vdbench总体下来也读了10GB的数据（等于maxdata）。而不是maxdata × jvms。

23162053-20297933 = 2864120次读
2864120 * 4k / 1024 / 1024 = 10.9 GB

Note:

我们这里只用了10GB，对硬盘的真实性能测试并不是很准确。准确的maxdata应该等于硬盘的大小大一点点，如7.67GB的硬盘，maxdata应该设置为8GB比较合适。
有时候我们会发现单进程（single slave / single jvm）时，受单进程最大处理能力的限制，其测出来的IO有时候并非硬盘IO的最大性能，我们可以多设置几个JVM（如在配置文件中设置jvms=4），其更能穷尽硬盘的性能。根据vdbench提供的PDF中的说法，单个进程IOPS超过100,000就会警告，建议作者设置合理的jvm数量。然鹅我试过好像没有警告耶。。。