fio 测试zfs

# sudo arc_summary -s arc

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Jul 18 14:10:53 2024
Linux 6.8.8-2-pve                                             2.2.4-pve1
Machine: pve (x86_64)                                         2.2.4-pve1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    99.9 %    2.0 GiB
        Target size (adaptive):                       100.0 %    2.0 GiB
        Min size (hard limit):                         34.9 %  714.3 MiB
        Max size (high water):                            2:1    2.0 GiB
        Anonymous data size:                            0.0 %    0 Bytes
        Anonymous metadata size:                        0.0 %    0 Bytes
        MFU data target:                               37.5 %  765.4 MiB
        MFU data size:                                  2.5 %   51.1 MiB
        MFU ghost data size:                                     0 Bytes
        MFU metadata target:                           12.5 %  255.1 MiB
        MFU metadata size:                              0.4 %    7.6 MiB
        MFU ghost metadata size:                                 0 Bytes
        MRU data target:                               37.5 %  765.4 MiB
        MRU data size:                                 96.2 %    1.9 GiB
        MRU ghost data size:                                    38.2 MiB
        MRU metadata target:                           12.5 %  255.1 MiB
        MRU metadata size:                              0.9 %   17.8 MiB
        MRU ghost metadata size:                                 0 Bytes
        Uncached data size:                             0.0 %    0 Bytes
        Uncached metadata size:                         0.0 %    0 Bytes
        Bonus size:                                   < 0.1 %   31.6 KiB
        Dnode cache target:                            10.0 %  204.8 MiB
        Dnode cache size:                               0.1 %  252.7 KiB
        Dbuf size:                                    < 0.1 %  232.4 KiB
        Header size:                                    0.2 %    3.8 MiB
        L2 header size:                                 0.0 %    0 Bytes
        ABD chunk waste size:                         < 0.1 %   22.5 KiB

ARC hash breakdown:
        Elements max:                                              41.1k
        Elements current:                              40.6 %      16.7k
        Collisions:                                                 3.5k
        Chain max:                                                     2
        Chains:                                                       41

ARC misc:
        Deleted:                                                    1.1M
        Mutex misses:                                                392
        Eviction skips:                                           520.1k
        Eviction skips due to L2 writes:                               0
        L2 cached evictions:                                     0 Bytes
        L2 eligible evictions:                                 128.7 GiB
        L2 eligible MFU evictions:                    < 0.1 %   10.4 MiB
        L2 eligible MRU evictions:                    100.0 %  128.7 GiB
        L2 ineligible evictions:                                 1.4 GiB

cd /hdd-st400/

下面是顺序写测试

fio --name=seqwrite --rw=write --direct=1 --ioengine=psync --bs=128k --iodepth=32 --numjobs=10 --size=10G --group_reporting

4M

 fio --name=seqwrite --rw=write --direct=1 --ioengine=psync --bs=4m --iodepth=32 --numjobs=5 --size=10G --group_reporting

seqwrite: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=32
...
fio-3.33
Starting 5 processes
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
seqwrite: Laying out IO file (1 file / 10240MiB)
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
note: both iodepth >= 1 and synchronous I/O engine are selected, queue depth will be capped at 1
Jobs: 2 (f=2): [W(1),_(2),W(1),_(1)][97.4%][eta 00m:08s]
seqwrite: (groupid=0, jobs=5): err= 0: pid=23831: Thu Jul 18 13:58:52 2024
  write: IOPS=42, BW=170MiB/s (178MB/s)(50.0GiB/301831msec); 0 zone resets
    clat (usec): min=450, max=3485.8k, avg=116870.67, stdev=577125.36
     lat (usec): min=495, max=3485.8k, avg=117138.97, stdev=577112.00
    clat percentiles (usec):
     |  1.00th=[    652],  5.00th=[    947], 10.00th=[   1106],
     | 20.00th=[   1221], 30.00th=[   1303], 40.00th=[