MINIO 压测导致 内核态cpu 异常飙高记录

MINIO 压测导致 内核态cpu 异常飙高记录

集群规模

40c 500g 机器4台

2个zone，每个zone 2个set，每个set 4个driver

压测规模

cosBench 8 个 drivers

cosBench 压测条件：512个请求并发，写128k ，持续5分钟

压测结果

使用 iostat 命令查看 cpu 消耗比例

吞吐量降低至 0 ， 无法维持心跳机制，%system cpu使用率 逼近100%

具体消耗cpu代码

1. top + P （先键入top，后键入 P） ----> 找进程
2. pstack ----> 找代码

得出内容如下

Thread 1 (process 37893):

0 runtime.futex () at /usr/local/Cellar/go/1.14/libexec/src/runtime/sys_linux_amd64.s:568

1 0x0000000000430f46 in runtime.futexsleep (addr=0x3a72748 <runtime.m0+328>, val=0, ns=-1) at /usr/local/Cellar/go/1.14/libexec/src/runtime/os_linux.go:44

2 0x000000000040afcf in runtime.notesleep (n=0x3a72748 <runtime.m0+328>) at /usr/local/Cellar/go/1.14/libexec/src/runtime/lock_futex.go:151

3 0x000000000043b238 in runtime.stoplockedm () at /usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:1971

4 0x000000000043cea6 in runtime.schedule () at /usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2454

5 0x000000000043d27d in runtime.park_m (gp=0xc000783380) at /usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2690

6 0x000000000046551b in runtime.mcall () at /usr/local/Cellar/go/1.14/libexec/src/runtime/asm_amd64.s:318

7 0x0000000000465434 in runtime.rt0_go () at /usr/local/Cellar/go/1.14/libexec/src/runtime/asm_amd64.s:220

8 0x0000000000000000 in ?? ()

结果猜测

大量请求并发，导致cpu频繁消耗在上下文切换