本文通过fio工具对不同环境下RBD的读写性能进行了测试，包括k8s RBD volume、KVM实例直接通过内核访问RBD、KVM实例通过librbd访问RBD、主机访问RBD等场景，并给出了详细的测试结果。
使用fio对volume的性能进行了测试，对以下几种情况进行了测试和对比：
1. k8s with rbd volume(k8s running on kvm VM)
2. kvm VM accessing rbd through linux kernal
3. kvm VM accessing through librbd
4. host accessing RBD(linux kernal, rbd map)
5. k8s PVC
使用fio进行测试
fio -name=/root/tmpfile -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=16G -numjobs=1 -group_reporting=1
fio -name=/root/tmpfile -direct=1 -iodepth=32 -rw=randread-ioengine=libaio -bs=16k -size=16G -numjobs=1 -group_reporting=1
| item | kubernetes rbd volume | kvm VM by linux kernal | kvm VM by librbd | host accesing RBD | kubernates by Persistent Volumes(RBD) |
| 16k randread IOPS | 8566 | 9628 | 9685 | 相近 | 相近 |
| 16k randwrite IOPS | 2741 | 2754 | 3171 | 相近 | 相近 |
| 16k randread bw | 137058KB/s | 154052KB/s | 154975KB/s | 相近 | 相近 |
| 16k randwrite bw | 43857KB/s | 44078KB/s | 50744KB/s | 相近 | 相近 |
结论：
1. ceph rdb在guest VM和host上性能差不多
2. ceph rbd kernal和librbd性能相差不大，librbd性能相对好一些。
3. docker上的rdb volume和vm,还有host上的性能也差不多，docker上的rbd volume性能相当不错。
详细的测试结果：
================================
test1: kubernetes rbd volume:
================================
< -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=16G -numj>
/mnt/rbd/tmpfile: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [w] [100.0% done] [0KB/17952KB/0KB /s] [0/1122/0 iops] [eta 00m:00s]
/mnt/rbd/tmpfile: (groupid=0, jobs=1): err= 0: pid=1751: Tue May 9 05:06:27 2017
write: io=16384MB, bw=43857KB/s, iops=2741, runt=382544msec
slat (usec): min=4, max=1920.6K, avg=35.92, stdev=2042.96
clat (msec): min=2, max=4803, avg=11.63, stdev=54.37
lat (msec): min=2, max=4803, avg=11.67, stdev=54.41
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7],
| 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10],
| 70.00th=[ 11], 80.00th=[ 13], 90.00th=[ 17], 95.00th=[ 21],
| 99.00th=[ 39], 99.50th=[ 57], 99.90th=[ 229], 99.95th=[ 783],
| 99.99th=[ 2966]
bw (KB /s): min= 6, max=55648, per=100.00%, avg=46518.63, stdev=10622.23
lat (msec) : 4=0.49%, 10=63.86%, 20=30.01%, 50=5.04%, 100=0.40%
lat (msec) : 250=0.11%, 500=0.03%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec) : >=2000=0.03%
cpu : usr=1.94%, sys=7.92%, ctx=709934, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=16384MB, aggrb=43856KB/s, minb=43856KB/s, maxb=43856KB/s, mint=382544msec, maxt=382544msec
Disk stats (read/write):
rbd0: ios=0/1053645, merge=0/148156, ticks=0/12241232, in_queue=12241570, util=100.00%
<=randread -ioengine=libaio -bs=16k -size=16G -numjobs=1 -group_reporting=1
/mnt/rbd/tmpfile: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [160.2MB/0KB/0KB /s] [10.4K/0/0 iops] [eta 00m:00s]
/mnt/rbd/tmpfile: (groupid=0, jobs=1): err= 0: pid=1754: Tue May 9 05:09:41 2017
read : io=16384MB, bw=137058KB/s, iops=8566, runt=122410msec
slat (usec): min=2, max=6772, avg= 7.94, stdev=26.85
clat (usec): min=549, max=952927, avg=3725.35, stdev=9155.47
lat (usec): min=559, max=952936, avg=3733.60, stdev=9155.65
clat percentiles (usec):
| 1.00th=[ 1032], 5.00th=[ 1336], 10.00th=[ 1560], 20.00th=[ 1896],
| 30.00th=[ 2160], 40.00th=[ 2416], 50.00th=[ 2704], 60.00th=[ 2992],
| 70.00th=[ 3376], 80.00th=[ 3824], 90.00th=[ 4704], 95.00th=[ 5856],
| 99.00th=[25216], 99.50th=[44800], 99.90th=[126464], 99.95th=[166912],
| 99.99th=[374784]
bw (KB /s): min=30272, max=187360, per=100.00%, avg=137147.92, stdev=33606.47
lat (usec) : 750=0.01%, 1000=0.69%
lat (msec) : 2=23.18%, 4=58.73%, 10=14.74%, 20=1.33%, 50=0.87%
lat (msec) : 100=0.28%, 250=0.14%, 500=0.02%, 750=0.01%, 1000=0.01%
cpu : usr=2.70%, sys=8.61%, ctx=383200, majf=0, minf=153
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=16384MB, aggrb=137057KB/s, minb=137057KB/s, maxb=137057KB/s, mint=122410msec, maxt=122410msec
Disk stats (read/write):
rbd0: ios=1047827/4, merge=0/1, ticks=3754197/67, in_queue=3754965, util=100.00%
================================================================
test2: kvm instance:
================================================================
[root@localhost fio-2.1]# fio -name=/root/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=16G -numjobs=1 -group_reporting=1
/root/jzhao/tmpfile: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
/root/jzhao/tmpfile: Laying out IO file(s) (1 file(s) / 16384MB)
Jobs: 1 (f=1): [w] [100.0% done] [0KB/17968KB/0KB /s] [0/1123/0 iops] [eta 00m:00s]
/root/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=4062: Tue May 9 05:45:22 2017
write: io=16384MB, bw=44078KB/s, iops=2754, runt=380623msec
slat (usec): min=4, max=178921, avg=36.39, stdev=804.74
clat (msec): min=2, max=3347, avg=11.58, stdev=32.03
lat (msec): min=2, max=3347, avg=11.61, stdev=32.06
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7],
| 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 10],
| 70.00th=[ 12], 80.00th=[ 14], 90.00th=[ 18], 95.00th=[ 24],
| 99.00th=[ 45], 99.50th=[ 65], 99.90th=[ 161], 99.95th=[ 338],
| 99.99th=[ 1434]
bw (KB /s): min= 519, max=54464, per=100.00%, avg=45126.18, stdev=8457.41
lat (msec) : 4=0.47%, 10=61.20%, 20=30.95%, 50=6.61%, 100=0.51%
lat (msec) : 250=0.18%, 500=0.04%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec) : >=2000=0.01%
cpu : usr=2.01%, sys=8.31%, ctx=737590, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=1048576/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=16384MB, aggrb=44078KB/s, minb=44078KB/s, maxb=44078KB/s, mint=380623msec, maxt=380623msec
Disk stats (read/write):
rbd1: ios=0/1055672, merge=0/155683, ticks=0/12304677, in_queue=12304483, util=100.00%
[root@localhost fio-2.1]# fio -name=/root/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randread -ioengine=libaio -bs=16k -size=16G -numjobs=1 -group_reporting=1
/root/jzhao/tmpfile: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [165.1MB/0KB/0KB /s] [10.6K/0/0 iops] [eta 00m:00s]
/root/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=9642: Tue May 9 05:49:13 2017
read : io=16384MB, bw=154052KB/s, iops=9628, runt=108906msec
slat (usec): min=2, max=6237, avg= 7.41, stdev=21.18
clat (usec): min=564, max=738661, avg=3313.94, stdev=5667.71
lat (usec): min=569, max=738664, avg=3321.62, stdev=5667.82
clat percentiles (usec):
| 1.00th=[ 1080], 5.00th=[ 1384], 10.00th=[ 1608], 20.00th=[ 1928],
| 30.00th=[ 2192], 40.00th=[ 2480], 50.00th=[ 2736], 60.00th=[ 3056],
| 70.00th=[ 3408], 80.00th=[ 3888], 90.00th=[ 4640], 95.00th=[ 5536],
| 99.00th=[13632], 99.50th=[22400], 99.90th=[72192], 99.95th=[108032],
| 99.99th=[211968]
bw (KB /s): min=44960, max=185984, per=100.00%, avg=154205.67, stdev=22074.34
lat (usec) : 750=0.01%, 1000=0.47%
lat (msec) : 2=21.98%, 4=59.65%, 10=16.48%, 20=0.81%, 50=0.42%
lat (msec) : 100=0.11%, 250=0.05%, 500=0.01%, 750=0.01%
cpu : usr=3.01%, sys=9.16%, ctx=391490, majf=0, minf=152
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=1048576/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=16384MB, aggrb=154052KB/s, minb=154052KB/s, maxb=154052KB/s, mint=108906msec, maxt=108906msec
Disk stats (read/write):
rbd1: ios=1047621/4, merge=0/1, ticks=3322911/31, in_queue=3323436, util=100.00%
==========================================================
test3: kvm vm with librbd:
===========================================================
[root@localhost ~]# fio -name=/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=16k -size=10G -numjobs=1 -group_reporting=1
/jzhao/tmpfile: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
/jzhao/tmpfile: Laying out IO file(s) (1 file(s) / 10240MB)
Jobs: 1 (f=1): [w] [100.0% done] [0KB/80303KB/0KB /s] [0/5018/0 iops] [eta 00m:00s]
/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=30341: Wed May 10 05:11:08 2017
write: io=10240MB, bw=50744KB/s, iops=3171, runt=206640msec
slat (usec): min=5, max=2780.6K, avg=30.84, stdev=3443.02
clat (usec): min=118, max=2861.7K, avg=10055.79, stdev=30333.27
lat (usec): min=149, max=2861.7K, avg=10087.06, stdev=30526.98
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 908], 10.00th=[ 1496], 20.00th=[ 4448],
| 30.00th=[ 6432], 40.00th=[ 7776], 50.00th=[ 8896], 60.00th=[10048],
| 70.00th=[11328], 80.00th=[12992], 90.00th=[15808], 95.00th=[19328],
| 99.00th=[34560], 99.50th=[46848], 99.90th=[232448], 99.95th=[325632],
| 99.99th=[477184]
bw (KB /s): min= 3501, max=131206, per=100.00%, avg=52221.12, stdev=14813.34
lat (usec) : 250=0.01%, 500=0.02%, 750=2.63%, 1000=3.53%
lat (msec) : 2=5.57%, 4=6.50%, 10=41.52%, 20=35.77%, 50=4.01%
lat (msec) : 100=0.24%, 250=0.12%, 500=0.08%, >=2000=0.01%
cpu : usr=2.01%, sys=8.14%, ctx=241327, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=655360/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
WRITE: io=10240MB, aggrb=50744KB/s, minb=50744KB/s, maxb=50744KB/s, mint=206640msec, maxt=206640msec
Disk stats (read/write):
vda: ios=0/655668, merge=0/19280, ticks=0/6453652, in_queue=6468591, util=100.00%
[root@localhost ~]# fio -name=/jzhao/tmpfile -direct=1 -iodepth=32 -rw=randread -ioengine=libaio -bs=16k -size=10G -numjobs=1 -group_reporting=1
/jzhao/tmpfile: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=libaio, iodepth=32
fio-2.1
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [200.4MB/0KB/0KB /s] [12.9K/0/0 iops] [eta 00m:00s]
/jzhao/tmpfile: (groupid=0, jobs=1): err= 0: pid=666: Wed May 10 05:13:06 2017
read : io=10240MB, bw=154975KB/s, iops=9685, runt= 67661msec
slat (usec): min=2, max=3715, avg=12.52, stdev=14.56
clat (usec): min=137, max=601820, avg=3288.13, stdev=12099.75
lat (usec): min=437, max=601824, avg=3301.01, stdev=12099.73
clat percentiles (usec):
| 1.00th=[ 636], 5.00th=[ 884], 10.00th=[ 1080], 20.00th=[ 1368],
| 30.00th=[ 1608], 40.00th=[ 1832], 50.00th=[ 2040], 60.00th=[ 2256],
| 70.00th=[ 2512], 80.00th=[ 2864], 90.00th=[ 3568], 95.00th=[ 4704],
| 99.00th=[31872], 99.50th=[61696], 99.90th=[191488], 99.95th=[268288],
| 99.99th=[407552]
bw (KB /s): min=16192, max=219680, per=100.00%, avg=155079.54, stdev=49649.27
lat (usec) : 250=0.01%, 500=0.01%, 750=2.55%, 1000=5.21%
lat (msec) : 2=40.27%, 4=44.68%, 10=4.94%, 20=0.87%, 50=0.83%
lat (msec) : 100=0.34%, 250=0.23%, 500=0.06%, 750=0.01%
cpu : usr=5.63%, sys=17.22%, ctx=477444, majf=0, minf=152
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
issued : total=r=655360/w=0/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=10240MB, aggrb=154974KB/s, minb=154974KB/s, maxb=154974KB/s, mint=67661msec, maxt=67661msec
Disk stats (read/write):
vda: ios=653810/742, merge=0/136, ticks=2136381/17532, in_queue=2154169, util=99.96%
===========================================
扫一扫
1542
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
