1. kdump简介
coredump为用户态进程段错误等触发的crash现场转储机制, 可利用gdb进行corefile的调试.
kdump为kernel crash/panic等而触发的转储机制, 一般使用crash工具进行调试.
2. kdump原理
预留部分内存(如192MB) 供转储内核使用, 当panic/crash产生时 利用kexec系统调用直接拉起转储内核. 利用该内核进行crash现场的储存dump.
3. kdump使用
ubuntu配置的内核已使能kdump的支持, 故仅需要安装linux-crashdump包即可,
sudo apt-get install linux-crashdump
该包包含crash, kdump-tools, grub等相关依赖.
安装过程中会修改kernel cmdline已预留kdump转储内核内存空间. 以及使能kdump.
reboot重启系统后,
可用以下方式验证:
1). kdump-config show
river@river-C-Z370M-DH:~$ kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.0.21
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.0.21
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/vmlinuz-5.0.21 root=UUID=e4c42b5a-44ca-485a-9346-5628d0f46c97 ro quiet splash vt.handoff=1 nr_cpus=1 systemd.unit=kdump-tools-dump.service irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
2). echo c > /proc/sysrq-trigger
类似alt + sysrq + c键会触发一个空指针的kernel panic.
可验证kdump机制是否运行.
=============================================================================================
4. crash使用
crash system.map vmlinux kdumpfile
常用命令:
bt
crash> bt
PID: 7944 TASK: ffffa0c8a1564680 CPU: 2 COMMAND: "insmod"
#0 [ffffbb6c021eb820] machine_kexec at ffffffffa186afa3
#1 [ffffbb6c021eb880] __crash_kexec at ffffffffa19425c2
#2 [ffffbb6c021eb950] crash_kexec at ffffffffa1943421
#3 [ffffbb6c021eb970] oops_end at ffffffffa1832ff7
#4 [ffffbb6c021eb998] no_context at ffffffffa187b68e
#5 [ffffbb6c021eba40] __bad_area_nosemaphore at ffffffffa187b8a0
#6 [ffffbb6c021eba88] bad_area_nosemaphore at ffffffffa187ba26
#7 [ffffbb6c021eba98] __do_page_fault at ffffffffa187c497
#8 [ffffbb6c021ebb00] do_page_fault at ffffffffa187c6ae
#9 [ffffbb6c021ebb30] page_fault at ffffffffa24010ae
[exception RIP: strcmp+16]
RIP: ffffffffa220dc90 RSP: ffffbb6c021ebbe8 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffa0c8a53c6840 RCX: 0000000000000000
RDX: 0000000000000072 RSI: 0000000000000000 RDI: ffffffffa2b4dbb1
RBP: ffffbb6c021ebc10 R8: 00000000000271a0 R9: ffffffffa220709c
R10: fffff240c93aa600 R11: 0000000000000000 R12: ffffa0c8a525bf00
R13: 0000000000000000 R14: ffffa0c8a525bf10 R15: ffffffffc05d4400
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#10 [ffffbb6c021ebbe8] kset_find_obj at ffffffffa2205c1a
#11 [ffffbb6c021ebc18] driver_find at ffffffffa1ea3853
#12 [ffffbb6c021ebc30] driver_register at ffffffffa1ea38cf
#13 [ffffbb6c021ebc50] __platform_driver_register at ffffffffa1ea4cf6
#14 [ffffbb6c021ebc60] _MODULE_INIT_START_globalmem at ffffffffc05d703d [globalmem]
#15 [ffffbb6c021ebc78] do_one_initcall at ffffffffa180279a
#16 [ffffbb6c021ebcf0] do_init_module at ffffffffa193d000
#17 [ffffbb6c021ebd18] load_module at ffffffffa193f3f9
#18 [ffffbb6c021ebe58] __do_sys_finit_module at ffffffffa193f8dd
#19 [ffffbb6c021ebf20] __x64_sys_finit_module at ffffffffa193f95a
#20 [ffffbb6c021ebf30] do_syscall_64 at ffffffffa180429a
#21 [ffffbb6c021ebf50] entry_SYSCALL_64_after_hwframe at ffffffffa240008c
RIP: 00007f15d269e2e9 RSP: 00007ffeaa9a66b8 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 00005649ae149770 RCX: 00007f15d269e2e9
RDX: 0000000000000000 RSI: 00005649acedd338 RDI: 0000000000000003
RBP: 00005649acedd338 R8: 0000000000000000 R9: 00007f15d276ece0
R10: 0000000000000003 R11: 0000000000000246 R12: 0000000000000000
R13: 00005649ae14bf60 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 0000000000000139 CS: 0033 SS: 002b
dis -l
crash> dis -l ffffffffa1ea3853
/home/river/linux_workspace/kernel/linux/linux-5.0.21/drivers/base/driver.c: 215
0xffffffffa1ea3853 <driver_find+35>: mov %rax,%rbx
etc...