Bootchart解析 - proc_diskstats.log

主要针对Linux系统的Bootchart文件进行分析，关于怎么采集Bootchart可以查看官方文档。

---

前言

本文主要针对Bootchart生成的proc_diskstats.log文件进行分析
Bootchart参考文章：elinux-Bootchart

---

一、proc_diskstats.log文件含义

顾名思义，proc_diskstats主要代表磁盘状态，当开启采集bootchart时，会在/data/bootchart/目录下生成proc_diskstats.log文件。这个文件用于显示磁盘、分区和统计信息，比起常规的cat /proc/diskstats 会增加一行采集信息的时间。

z3q:/ # cat /proc/diskstats
   1       0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       1 ram1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       2 ram2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       3 ram3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       4 ram4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       5 ram5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       6 ram6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       7 ram7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       8 ram8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       9 ram9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      10 ram10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      11 ram11 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      12 ram12 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      13 ram13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      14 ram14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      15 ram15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       0 loop0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       1 loop1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       2 loop2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       3 loop3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       4 loop4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       5 loop5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       6 loop6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       7 loop7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8       0 sda 9267 171 938604 3934 13 0 56 6 0 1850 4220 0 0 0 0
   8       1 sda1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8       2 sda2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8       3 sda3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8       4 sda4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8       5 sda5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8       6 sda6 9264 171 938580 3933 7 0 56 5 0 1850 4220 0 0 0 0
   8      16 sdb 7610 16 159552 2050 300 534 29696 550 0 880 2780 0 0 0 0
   8      17 sdb1 111 0 900 53 3 0 24 2 0 20 70 0 0 0 0
   8      18 sdb2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8      19 sdb3 7497 16 158636 1996 285 534 29672 545 0 860 2710 0 0 0 0

proc_diskstats.log文件样式如下，其中507、528、549即为采集时间：

507
   1       0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1       1 ram1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  ···
   1      14 ram14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   1      15 ram15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   7       0 loop0 35 1 3080 16 0 0 0 0 0 12 20 0 0 0 0
  ···
   7     120 loop15 10 0 80 4 0 0 0 0 0 4 4 0 0 0 0
 254       0 zram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8       0 sda 5190 4845 255144 5611 16 0 80 8 0 984 5556 0 0 0 0
   8       1 sda1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  ···
 259      18 sda34 1979 2543 89608 3023 3 0 24 0 0 260 3388 0 0 0 0
   8      16 sdb 5 0 40 22 0 0 0 0 0 24 24 0 0 0 0
   8      17 sdb1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   8      32 sdc 5 0 40 6 0 0 0 0 0 8 8 0 0 0 0
   8      48 sdd 5 0 40 3 0 0 0 0 0 4 4 0 0 0 0
   8      64 sde 66 1 1080 42 0 0 0 0 0 28 32 0 0 0 0
  ···
   8      80 sdf 5 0 40 4 0 0 0 0 0 0 0 0 0 0 0
  ···
   8      90 sdf10 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  31       0 mtdblock0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 252       0 dm-0 3311 0 105200 4568 0 0 0 0 0 380 4568 0 0 0 0
  ···
   7     128 loop16 12 0 96 5 0 0 0 0 0 8 8 0 0 0 0
 252      15 dm-15 6 0 48 4 0 0 0 0 0 4 4 0 0 0 0
   7     136 loop17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 252      16 dm-16 8 0 64 4 0 0 0 0 0 4 4 0 0 0 0

528
   1       0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
   ···
   8       0 sda 5236 4845 255512 5632 16 0 80 8 0 1012 5596 0 0 0 0
   ···
 252      20 dm-20 13 0 104 12 0 0 0 0 0 4 12 0 0 0 0

549
   1       0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

二、proc_diskstats.log文件解析

1. 参数解释

cat /proc/diskstats具体参数可以参考[Linux 运维 – 存储] /proc/diskstats详解和www.kernel.org

以sda举例

# sda
  8       0 sda 9267 171 938604 3934 13 0 56 6 0 1850 4220 0 0 0 0
# 第1列为 为设备号 - 8
# 第2列为 次设备号 - 0
# 第3列为 设备名称 - sda
# 第4列为 成功完成读的总次数 - 9267
# 第5列为 合并读次数 - 171 (为了效率可能会合并相邻的读和写，从而两次4K的读在它最终被处理到磁盘上之前可能会变成一次8K的读，才被计数（和排队），因此只有一次I/O操作。)
# 第6列为 读扇区的次数 - 938604 （读扇区大小，成功读过的扇区总次数）
# 第7列为 读花的时间（ms） - 3934

# 第8~11分别是写， 与4~7读对应， 依次是 13 0 56 0

# 第12列为 I/O的当前进度 - 0 (只有这个域应该是0，如果这个值为0，同时write_complete read_complete io_processing 一直不变可能就就是IO hang了)
# 第13列为 输入输入花的时间（ms） - 1850 (花在I/O操作上的毫秒数，这个域会增长只要field 就不为0)
# 第14列为 输入/输出操作花费的加权毫秒数 - 4220

# 第15列为 丢弃完成的成功数 - 0
# 第16列为 丢弃合并数 - 0
# 第17列为 丢弃扇区数 - 0
# 第18列为 丢弃花费时间(ms) - 0

# quote https://www.kernel.org/doc/Documentation/ABI/testing/procfs-diskstats
What:		/proc/diskstats
Date:		February 2008
Contact:	Jerome Marchand <jmarchan@redhat.com>
Description:
		The /proc/diskstats file displays the I/O statistics
		of block devices. Each line contains the following 14
		fields:

		==  ===================================
		 1  major number
		 2  minor mumber
		 3  device name
		 4  reads completed successfully
		 5  reads merged
		 6  sectors read
		 7  time spent reading (ms)
		 8  writes completed
		 9  writes merged
		10  sectors written
		11  time spent writing (ms)
		12  I/Os currently in progress
		13  time spent doing I/Os (ms)
		14  weighted time spent doing I/Os (ms)
		==  ===================================

		Kernel 4.18+ appends four more fields for discard
		tracking putting the total at 18:

		==  ===================================
		15  discards completed successfully
		16  discards merged
		17  sectors discarded
		18  time spent discarding
		==  ===================================

		Kernel 5.5+ appends two more fields for flush requests:

		==  =====================================
		19  flush requests completed successfully
		20  time spent flushing
		==  =====================================

		For more details refer to Documentation/admin-guide/iostats.rst

2. 文件解析

查看pybootchartgui源码，可以看到关于diskstats的解析，下面给出关键代码

def _parse_proc_disk_stat_log(file, numCpu):
    """
    Parse file for disk stats, but only look at the whole device, eg. sda,
    not sda1, sda2 etc. The format of relevant lines should be:
    {major minor name rio rmerge rsect ruse wio wmerge wsect wuse running use aveq}
    """
    disk_regex_re = re.compile ('^([hsv]d.|mtdblock\d|mmcblk\d|cciss/c\d+d\d+.*)$')

    # this gets called an awful lot.
    def is_relevant_line(linetokens):
        if len(linetokens) < 14:
            return False
        disk = linetokens[2]
        return disk_regex_re.match(disk)

    disk_stat_samples = []

    for time, lines in _parse_timed_blocks(file):
        sample = DiskStatSample(time)
        relevant_tokens = [linetokens for linetokens in map (lambda x: x.split(),lines) if is_relevant_line(linetokens)]

        for tokens in relevant_tokens:
            disk, rsect, wsect, use = tokens[2], int(tokens[5]), int(tokens[9]), int(tokens[12])
            sample.add_diskdata([rsect, wsect, use])

        disk_stat_samples.append(sample)

    disk_stats = []
    for sample1, sample2 in zip(disk_stat_samples[:-1], disk_stat_samples[1:]):
        interval = sample1.time - sample2.time
        if interval == 0:
            interval = 1
        sums = [ a - b for a, b in zip(sample1.diskdata, sample2.diskdata) ]
        readTput = sums[0] / 2.0 * 100.0 / interval
        writeTput = sums[1] / 2.0 * 100.0 / interval
        util = float( sums[2] ) / 10 / interval / numCpu
        util = max(0.0, min(1.0, util))
        disk_stats.append(DiskSample(sample2.time, readTput, writeTput, util))

    return disk_stats

• is_relevant_line 函数是找出相关的sda、sdb、hda、vda、mtdblock0、mmcblk0等，忽略ram、sda1等

disk_regex_re = re.compile ('^([hsv]d.|mtdblock\d|mmcblk\d|cciss/c\d+d\d+.*)$')
disk = 'sda'
print(disk_regex_re.match(disk))
<re.Match object; span=(0, 3), match='sda'>
disk = 'sda1'
print(disk_regex_re.match(disk))
None
disk = 'mmcblk0'
print(disk_regex_re.match(disk))
<re.Match object; span=(0, 7), match='mmcblk0'>

• _parse_timed_blocks 这个函数是将proc_diskstats.log进行分割，_parse_timed_blocks(file)返回一个列表，元素为元组形式，第一个元素为时间，第二个元素为按行读取的列表。

# 仅作示意
[(507, ['1       0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0','1       1 ram1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'],
 [528, ['1       0 ram0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0'])]

• 接下来解析出所有磁盘（sda\sdb\sdc所有的）本次采样时间的采样时间time、设备名disk、删除读取次数rsect、扇区写入次数wsect、 输入输出花的时间use
• 两次采样差值为变化量

interval = sample1.time - sample2.time  # 采样时间差，ms
readTput = sums[0] / 2.0 * 100.0 / interval  # sums[0] 为两次采样读扇区次数差
writeTput = sums[1] / 2.0 * 100.0 / interval  # sums[1] 为两次采样写扇区次数差
util = float( sums[2] ) / 10 / interval / numCpu  # sums[2] 为两次采样读写时间差
util = max(0.0, min(1.0, util))
disk_stats.append(DiskSample(sample2.time, readTput, writeTput, util))

扇区大小为512字节