当linux系统内核发生崩溃的时候，可以通过KEXEC+KDUMP等方式收集内核崩溃之前的内存，生成一个转储文件vmcore。内核开发者通过分析该vmcore文件就可以诊断出内核崩溃的原因，从而进行操作系统的代码改进。那么Crash就是一个被广泛使用的内核崩溃转储文件分析工具。
对调试来讲，gdb是非常适合的，但gdb始终是调试native的工具，不支持kernel信息显示，比如task信息之类的。crash补足了这个短板，由Dave Anderson开发和维护的一个内存转储分析工具，是基于GDB开发的 (GDB适用于用户进程的coredump，而Crash扩展了GDB，使其适用于linux kernel coredump)，目前它的最新版本是8.0.1(with GDB 10.2)。在没有统一标准的内存转储文件的格式的情况下，Crash工具支持众多的内存转储文件格式，包括：
- Live linux系统
- kdump产生的正常的和压缩的内存转储文件
- 由makedumpfile命令生成的压缩的内存转储文件
- 由Netdump生成的内存转储文件
- 由Diskdump生成的正常的或压缩的内存转储文件
- 由Kdump生成的xen host内存转储文件
- 由原生xendump生成的xen guest内存转储文件
- 由ELF格式的xendump生成的xen guest
- 由Kdump生成的xen hypervisor内存转储文件
- 由virsh dump生成的KVM guest内存转储文件
- IBM的390/390x的内存转储文件
- LKCD生成的内存转储文件
- Mcore生成的内存转储文件
而MTK在KE时会抓取full dump文件：SYS_COREDUMP，则可以用crash来调试。
crash使用gdb作为它的内部引擎，crash中的很多命令和语法都与gdb相同。如果曾经使用过gdb，就会发现crash并不是很陌生。如果想获得crash更多的命令和相关命令的详细说明，可以使用crash的内部命令help来获取：
| 命令 | 说明 | 例子 |
| * | 指针的快捷方式，用于代替struct/union | *page 0xc02943c0：显示0xc02943c0地址的page结构体 |
| files | 显示已打开的所有文件的信息 | files 462：显示进程462的已打开文件信息 |
| mach | 显示与机器相关的参数信息 | mach：显示CPU型号，核数，内存大小等 |
| sys | 显示特殊系统的数据 | sys config：显示CONFIG_xxx配置宏状态 |
| timer | 无参数。按时间的先后顺序显示定时器队列的数据 | timer：显示详细信息 |
| mod | 显示已加载module的详细信息 | mod：列出所有已加载module信息 |
| runq | 显示runqueue信息 | runq：显示所有runqueue里的task |
| tree | 显示基数树/红黑树结构 | tree -t rbtree -o vmap_area.rb_node vmap_area_root：显示所有红黑树vmap_area.rb_node节点地址 |
| fuser | 显示哪些task使用了指定的文件/socket | fuser /usr/lib/libkfm.so.2.0.0：显示使用了该文件的所有进程 |
| mount | 显示已挂载的文件系统信息 | mount：当前已挂载的文件系统信息 |
| ipcs | 显示System V IPC信息 | ipcs：显示系统中System V IPC信息 |
| ps | 显示进程状态 | ps：类似ps命令 |
| struct | 显示结构体的具体内容 | struct vm_area_struct c1e44f10：显示c1e44f10结构 |
| union | 显示联合体的具体内容，用法与struct一致 | union bdflush_param：显示bdflush_param结构 |
| waitq | 列出在等待队列中的所有task。参数可以指定队列的名称、内存地址等 | waitq buffer_wait：显示buffer_wait等待队列信息 |
| irq | 显示中断编号的所有信息 | irq 18：显示中断18的信息 |
| list | 显示链表的内容 | list task_struct.p_pptr c169a000：显示c169a000地址所指task里p_pptr链表 |
| log | 显示内核的日志，以时间的先后顺序排列 | log -m：显示kernel log |
| dev | 显示数据关联着的块设备分配，包括端口使用、内存使用及PCI设备数据 | dev：显示字符/块设备相关信息 |
| sig | 显示一个或者多个task的signal-handling数据 | sig 8970：显示进程8970的信号处理相关信息 |
| task | 显示指定内容或者进程的task_struct的内容 | task -x：显示当前进程task_struct等内容 |
| swap | 无参数。显示已配置好的交换设备信息 | swap：交换设备信息 |
| search | 在给定范围的用户、内核虚拟内存或者物理内存搜索值 | search -u deadbeef：在用户内存搜索0xdeadbeef |
| bt | 显示调用栈信息 | bt：显示当前调用栈 |
| net | 显示各种网络相关的数据 | net：显示网络设备列表 |
| vm | 显示task的基本虚拟内存信息 | vm：类似于/proc/self/maps |
| btop | 把一个16进制地址转换成它的分页号 | N/A |
| ptob | 该命令与btop相反，是把一个分页号转换成地址 | N/A |
| vtop | 显示用户或内核虚拟内存所对应的物理内存 | N/A |
| ptov | 该命令与vtop相反。把物理内存转换成虚拟内存 | N/A |
| pte | 16进制页表项转换为物理页地址和页的位设置 | N/A |
| alias | 显示或建立一个命令的别名 | alias kp kmem -p：以后用kp命令相当于kmem -p |
| foreach | 用指定的命令枚举 | foreach bt：显示所有进程的调用栈 |
| repeat | 循环执行指定命令 | repeat -1 p jiffies：每个1s执行p jiffies |
| ascii | 把16进制表示的字符串转化成ascii表示的字符串 | ascii 62696c2f7273752f：结果为/usr/lib |
| set | 设置要显示的内容，内容一般以进程为单位，也可以设置当前crash的内部变量 | set -p：切换到崩溃进程的上下文环境 |
| p | print的缩写，打印表达式的值。表达式可以为变量，也可以为结构体 | N/A |
| dis | disassemble的缩写。把一个命令或者函数分解成汇编代码 | dis sys_signal：反汇编sys_signal函数 |
| whatis | 搜索数据或者类型的信息 | whatis linux_binfmt：显示linux_binfmt结构体 |
| eval | 计算表达式的值，及把计算结果或者值显示为16、10、8和2进制 | N/A |
| kmem | 显示当前kernel使用内存状况 | kmem -i：显示kernel使用内存状况 |
| sym | 显示符号所在的虚拟地址，或虚拟地址对应的符号 | sym jiffies：显示jiffies地址 |
| rd | 显示指定内存的内容。缺少的输出格式是十六进制输出 | rd -a linux_banner：显示linux_banner内容 |
| wr | 根据参数指定的写内存。在定位系统出错的地方时，一般不使用该命令 | wr my_debug_flag 1：修改my_debug_flag值为1 |
| gdb | 执行GDB原生命令 | gdb help：执行gdb的help命令 |
| extend | 动态装载或卸载crash额外的动态链接库 | N/A |
| q | 退出 | N/A |
| exit | 同q，退出 | N/A |
| help | 帮助命令 | N/A |
White Paper: Crash Utility
by David Anderson
Copyright © 2003, 2008, 2022 by Red Hat, Inc.
Copyright © 2022 by NEC Corporation
Permission is granted to copy, distribute and/or modify this document under the terms of
the GNU Free Documentation License, Version 1.2 or any later version published by the
Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU Free Documentation License".
Content
- Why Crash?
- Prerequisites
- Build Procedure
- Invocation
- Command Input
- Command Output
- Crash Context
- Builtin Help
- The Command Set
- Crash Usage: A Case Study
- Command Extensions
- GNU Free Documentation License
Abstract
The crash analysis utility is loosely based on the SVR4 UNIX crash command, but has been significantly enhanced by completely merging it with the GNU gdb debugger. The marriage of the two effectively combines the kernel-specific nature of the traditional UNIX crash utility with the source code level debugging capabilities of gdb. The utility can be used to investigate:
- Live Linux systems
- Linux kernel core dumps created by the Kdump facility
- Compressed Linux kernel core dumps created by the makedumpfile command (from kdump dumpfiles)
- Linux kernel core dumps created from by the Red Hat Netdump facility
- Linux kernel core dumps created from by the Red Hat Diskdump facility
- Compressed Linux kernel core dumps created by the Red Hat Diskdump facility
- Xen host Linux kernel core dumps created by the Kdump facility
- Xen guest Linux kernel core dumps created by the original xendump facility
- Xen guest Linux kernel core dumps created by the ELF-format xendump facility
- Xen hypervisor core dumps created by the Kdump facility
- KVM guest Linux kernel core dumps created by the virsh dump facility
- s390 Linux kernel core dumps created by the IBM standalone core dump facility.
- s390x Linux kernel core dumps created by the IBM standalone core dump facility.
- Linux kernel core dumps created by the LKCD (Linux Kernel Crash Dumps) Sourceforge project
- Linux kernel core dumps created by the Mcore patch offered by Mission Critical Linux
The current set of commands consist of common kernel core analysis tools such as kernel stack back traces of all processes, source code disassembly, formatted kernel structure and variable displays, virtual memory data, dumps of linked-lists, etc., along with several commands that delve deeper into specific kernel subsystems. Relevant gdb commands may also be entered, which in turn are passed on to the gdb module for execution.
The crash utility is designed to be independent of Linux version dependencies. When new kernel source code impacts the correct functionality of crash and its command set, the utility will be updated to recognize new kernel code changes while maintaining backwards compatibility with earlier releases. The most current version of the crash utility may be found here: https://crash-utility.github.io/
Why Crash?
The Linux operating system originally lacked a built-in, traditional UNIX-like kernel crash dump facility. This was initially addressed by the Mission Critical Linux Mcore kernel patch and the LKCD (Linux Kernel Crash Dump) kernel patch from SGI in 1999, and later by the Red Hat Netdump facility in 2002, and the Red hat Diskdump facility in 2004. The upstream Linux community finally settled upon the adoption of the Kdump crash dump facility in 2006.
However, the creation of a kernel crash dump file is only half of the picture; a utility is required to be able to recognize the dumpfile format in order to read it, and to offer a useful set of commands to make sense of it.
Furthermore, to examine the contents of a live system's kernel internals from user space, the only readily available option has been to use gdb on /proc/kcore. While gdb is an incredibly powerful tool, it is designed to debug user programs, and is not at all "kernel-aware". Consequently, using gdb alone has limited usefulness when looking at kernel memory, essentially constrained to the printing of kernel data structures if the vmlinux file was built with the -g C flag, the disassembly of kernel text, and raw data dumps. Furthermore, distributions such as Red Hat Enterprise Linux have limited the access to /proc/kcore, making it unusable as a kernel memory source.
As far as kernel crash dump files are concerned, the Red Hat Netdump and uncompressed Diskdump facilities, and the Kdump facility create dump files that are readable by gdb, but aside from giving it the capability of displaying the panicking task's stack trace, it has the same constraints as when reading /proc/kcore. However, gdb cannot read LKCD, Mcore, Xen or s390/s390x dump files.
That being the state of things, the crash utility was developed as a convenient means to cover all bases, i.e., all listed dumpfile formats as well as live systems. Moreover, it is also designed to be easily enhanced to suit the specific needs of the kernel developers or analysts using it; the builtin command set can easily be extended or enhanced, and external command modules may be written and dynamically attached.
Prerequisites
The crash utility has the following prerequisites:
kernel object file:
A vmlinux kernel object file, often referred to as the namelist in this document, which must have been built with the -g C flag so that it will contain the debug data required for symbolic debugging.In RHEL3 installations, the vmlinux file associated with the running kernel is split into two files, a stripped version found in the /boot directory; which has have the operating system release string appended to it, for example, vmlinux-2.4.21-4.ELsmp. The stripped file in /boot contains a link to its associated debuginfo file, which is located in the /usr/lib/debug/boot directory.
In RHEL4, RHEL5 and RHEL6 installations, the vmlinux file is part of the kernel debuginfo package, and is found in the relevant /usr/lib/debug/lib/modules/<release> directory.
Ideally the kernel object file is the same kernel object file that is associated with the memory image. However, in circumstances where the vmlinux file associated with the crash dump or live system was not built with the -g flag, there are work-arounds discussed later in the Invocation section.
memory image:
This may consist of a kernel crash dump file generated from any of the supported dump facilties, or live system memory accessed via /dev/mem or its replacement in RHEL4/RHEL5/RHEL6, the /dev/crash driver. If no dump file argument is issued on the crash command line, live system memory will be used by default. When examining a live system, root privileges are required.
platform processor types:
The crash utility is actively developed and tested on the x86, x86_64, ia64, ppc64, arm, s390 and s390x processors. Legacy support for the Alpha and 32-bit PowerPC platforms exists, but no longer actively maintained.
Linux kernel versions:
The crash utility is backwards-compatible to at least Red Hat 6.0 (Linux version 2.2.5-15), up to Red Hat Enterprise Linux 5 (Linux version 2.6.18+). Due to the constantly shifting sands of the upstream kernel internals, immediate support for the latest kernel versions cannot be guaranteed. However, modifications are constantly being implemented to support changes in upstream kernel versions. The intent has always been to make the utility independent of Linux version dependencies, building in recognition of major kernel code changes so as to adapt to new kernel versions, while maintaining backwards compatibility.
Build Procedure
Starting with the RHEL3 release, the crash utility is automatically installed during system installation if the Development Tools package set is selected. However, for all other kernel versions, or if it was not selected during system installation, the binary RPM can be installed, or if desired, the sources re-built and installed.
If the crash utility is not pre-installed, and if all dependencies are met on the target system, install the binary RPM like so:
# rpm -ivh crash-4.0-8.11.i386.rpm Preparing... ########################################### [100%] 1:crash ########################################### [100%] # |
The crash executable will be installed in the /usr/bin directory.
Alternatively, the crash source code can be rebuilt. The crash utility's source files come packaged in two typical formats, a compressed tar image. or a source RPM file, So, for example, crash version 4.0-8.11 can be built from either crash-4.0-8.11.tar.gz or crash-4.0-8.11.src.rpm.
The latest "upstream" version of the crash utility, available in both file formats, can be found here: https://crash-utility.github.io/
In either case, the source file layout consists of a top-level directory containing a set of crash-specific files, a compressed tar image containing the full, unmodified, gdb source tree, and a small number of modified gdb files required to merge the two entities. The build procedure does the following:
- the unmodified gdb sources are extracted into a subdirectory of the top-level crash source directory, and overlayed by the small set of modified gdb files.
- the files in the gdb source tree are built first, creating the libbfd.a, libreadline.a, libopcodes.a, libiberty.a and libgdb.a libraries.
- the crash sources files in the top-level directory are then compiled into a crashlib.a library.
- the objects are all linked into the crash executable, located in the top-level directory.
Depending upon the speed of the host system, the complete build may take several minutes, primarily due to the time consumed by the build of the gdb portion.
Building from the tar image
To build from the compressed tar image, simply uncompress/extract the source files, cd into the resultant source directory, and enter make:
# tar xvzmf crash-4.0-8.11.tar.gz
crash-4.0-8.11/
crash-4.0-8.11/main.c
crash-4.0-8.11/tools.c
crash-4.0-8.11/global_data.c
crash-4.0-8.11/memory.c
crash-4.0-8.11/filesys.c
crash-4.0-8.11/help.c
crash-4.0-8.11/task.c
crash-4.0-8.11/kernel.c
crash-4.0-8.11/test.c
crash-4.0-8.11/gdb_interface.c
crash-4.0-8.11/configure.c
crash-4.0-8.11/net.c
crash-4.0-8.11/dev.c
crash-4.0-8.11/alpha.c
crash-4.0-8.11/x86.c
crash-4.0-8.11/ppc.c
crash-4.0-8.11/ia64.c
crash-4.0-8.11/s390.c
crash-4.0-8.11/s390x.c
crash-4.0-8.11/s390dbf.c
crash-4.0-8.11/ppc64.c
crash-4.0-8.11/x86_64.c
crash-4.0-8.11/extensions.c
crash-4.0-8.11/remote.c
crash-4.0-8.11/va_server.c
crash-4.0-8.11/va_server_v1.c
crash-4.0-8.11/symbols.c
crash-4.0-8.11/cmdline.c
crash-4.0-8.11/lkcd_common.c
crash-4.0-8.11/lkcd_v1.c
crash-4.0-8.11/lkcd_v2_v3.c
crash-4.0-8.11/lkcd_v5.c
crash-4.0-8.11/lkcd_v7.c
crash-4.0-8.11/lkcd_v8.c
crash-4.0-8.11/lkcd_fix_mem.c
crash-4.0-8.11/s390_dump.c
crash-4.0-8.11/lkcd_x86_trace.c
crash-4.0-8.11/netdump.c
crash-4.0-8.11/diskdump.c
crash-4.0-8.11/xendump.c
crash-4.0-8.11/unwind.c
crash-4.0-8.11/unwind_decoder.c
crash-4.0-8.11/unwind_x86_32_64.c
crash-4.0-8.11/xen_hyper.c
crash-4.0-8.11/xen_hyper_command.c
crash-4.0-8.11/xen_hyper_global_data.c
crash-4.0-8.11/xen_hyper_dump_tables.c
crash-4.0-8.11/defs.h
crash-4.0-8.11/xen_hyper_defs.h
crash-4.0-8.11/va_server.h
crash-4.0-8.11/vas_crash.h
crash-4.0-8.11/netdump.h
crash-4.0-8.11/diskdump.h
crash-4.0-8.11/xendump.h
crash-4.0-8.11/unwind.h
crash-4.0-8.11/unwind_i.h
crash-4.0-8.11/rse.h
crash-4.0-8.11/unwind_x86.h
crash-4.0-8.11/unwind_x86_64.h
crash-4.0-8.11/lkcd_vmdump_v1.h
crash-4.0-8.11/lkcd_vmdump_v2_v3.h
crash-4.0-8.11/lkcd_dump_v5.h
crash-4.0-8.11/lkcd_dump_v7.h
crash-4.0-8.11/lkcd_dump_v8.h
crash-4.0-8.11/lkcd_x86_trace.h
crash-4.0-8.11/lkcd_fix_mem.h
crash-4.0-8.11/ibm_common.h
crash-4.0-8.11/Makefile
crash-4.0-8.11/gdb-6.1/
crash-4.0-8.11/gdb-6.1/gdb/
crash-4.0-8.11/gdb-6.1/gdb/Makefile.in
crash-4.0-8.11/gdb-6.1/gdb/main.c
crash-4.0-8.11/gdb-6.1/gdb/symtab.c
crash-4.0-8.11/gdb-6.1/gdb/target.c
crash-4.0-8.11/gdb-6.1/gdb/symfile.c
crash-4.0-8.11/gdb-6.1/gdb/elfread.c
crash-4.0-8.11/gdb-6.1/gdb/ui-file.c
crash-4.0-8.11/gdb-6.1/gdb/utils.c
crash-4.0-8.11/gdb-6.1/gdb/dwarf2read.c
crash-4.0-8.11/gdb-6.1/gdb/ppc-linux-tdep.c
crash-4.0-8.11/gdb-6.1/Makefile.in
crash-4.0-8.11/gdb-6.1/include/
crash-4.0-8.11/gdb-6.1/include/obstack.h
crash-4.0-8.11/gdb-6.1.patch
crash-4.0-8.11/COPYING
crash-4.0-8.11/.rh_rpm_package
crash-4.0-8.11/crash.8
crash-4.0-8.11/extensions/
crash-4.0-8.11/extensions/Makefile
crash-4.0-8.11/extensions/echo.c
crash-4.0-8.11/extensions/dminfo.c
crash-4.0-8.11/extensions/libsial/
crash-4.0-8.11/extensions/libsial/Makefile
crash-4.0-8.11/extensions/libsial/mkbaseop.c
crash-4.0-8.11/extensions/libsial/README
crash-4.0-8.11/extensions/libsial/README.sial
crash-4.0-8.11/extensions/libsial/sial_alloc.c
crash-4.0-8.11/extensions/libsial/sial_api.c
crash-4.0-8.11/extensions/libsial/sial_api.h
crash-4.0-8.11/extensions/libsial/sial_builtin.c
crash-4.0-8.11/extensions/libsial/sial_case.c
crash-4.0-8.11/extensions/libsial/sial_define.c
crash-4.0-8.11/extensions/libsial/sial_func.c
crash-4.0-8.11/extensions/libsial/sial.h
crash-4.0-8.11/extensions/libsial/sial_input.c
crash-4.0-8.11/extensions/libsial/sial.l
crash-4.0-8.11/extensions/libsial/sial-lsed
crash-4.0-8.11/extensions/libsial/sial_member.c
crash-4.0-8.11/extensions/libsial/sial_node.c
crash-4.0-8.11/extensions/libsial/sial_num.c
crash-4.0-8.11/extensions/libsial/sial_op.c
crash-4.0-8.11/extensions/libsial/sialpp.l
crash-4.0-8.11/extensions/libsial/sialpp-lsed
crash-4.0-8.11/extensions/libsial/sialpp.y
crash-4.0-8.11/extensions/libsial/sial_print.c
crash-4.0-8.11/extensions/libsial/sial_stat.c
crash-4.0-8.11/extensions/libsial/sial_str.c
crash-4.0-8.11/extensions/libsial/sial_type.c
crash-4.0-8.11/extensions/libsial/sial_util.c
crash-4.0-8.11/extensions/libsial/sial_var.c
crash-4.0-8.11/extensions/libsial/sial.y
crash-4.0-8.11/extensions/sial.c
crash-4.0-8.11/extensions/sial.mk
crash-4.0-8.11/gdb-6.1.tar.gz
crash-4.0-8.11/README
# cd crash-4.0-8.11
# make
TARGET: X86
CRASH: 4.0-8.11
GDB: 6.1
gdb-6.1/gdb/CONTRIBUTE
gdb-6.1/gdb/COPYING
gdb-6.1/gdb/ChangeLog
gdb-6.1/gdb/ChangeLog-1990
gdb-6.1/gdb/ChangeLog-1991
gdb-6.1/gdb/ChangeLog-1992
gdb-6.1/gdb/ChangeLog-1993
gdb-6.1/gdb/ChangeLog-1994
gdb-6.1/gdb/ChangeLog-1995
gdb-6.1/gdb/ChangeLog-1996
gdb-6.1/gdb/ChangeLog-1997
gdb-6.1/gdb/ChangeLog-1998
gdb-6.1/gdb/ChangeLog-1999
gdb-6.1/gdb/ChangeLog-2000
gdb-6.1/gdb/ChangeLog-2001
gdb-6.1/gdb/ChangeLog-2002
gdb-6.1/gdb/ChangeLog-2003
gdb-6.1/gdb/ChangeLog-3.x
gdb-6.1/gdb/MAINTAINERS
gdb-6.1/gdb/NEWS
gdb-6.1/gdb/PROBLEMS
gdb-6.1/gdb/README
gdb-6.1/gdb/TODO
gdb-6.1/gdb/abug-rom.c
gdb-6.1/gdb/acconfig.h
gdb-6.1/gdb/acinclude.m4
gdb-6.1/gdb/aclocal.m4
gdb-6.1/gdb/ada-exp.y
gdb-6.1/gdb/ada-lang.c
gdb-6.1/gdb/ada-lang.h
(complete output not shown)
ar -rs crashlib.a main.o tools.o global_data.o memory.o filesys.o help.o task.o
build_data.o kernel.o test.o gdb_interface.o net.o dev.o alpha.o x86.o ppc.o ia6
4.o s390.o s390x.o s390dbf.o ppc64.o x86_64.o extensions.o remote.o va_server.o
va_server_v1.o symbols.o cmdline.o lkcd_common.o lkcd_v1.o lkcd_v2_v3.o lkcd_v5.
o lkcd_v7.o lkcd_v8.o lkcd_fix_mem.o s390_dump.o netdump.o diskdump.o xendump.o
lkcd_x86_trace.o unwind_v1.o unwind_v2.o unwind_v3.o unwind_x86_32_64.o xen_hype
r.o xen_hyper_command.o xen_hyper_global_data.o xen_hyper_dump_tables.o
ar: creating crashlib.a
gcc -g -O2 \
-o `cat mergeobj` libgdb.a \
../bfd/libbfd.a ../readline/libreadline.a ../opcodes/libopcod
es.a ../libiberty/libiberty.a -lm -lncurses ../libiberty/libiberty.a -ldl
-rdynamic `cat mergelibs`
#
|
The resultant crash executable will be located in the top-level source directory. Install it in /usr/bin by entering:
# make install /usr/bin/install crash /usr/bin # |
Building from the source RPM
To build from the source RPM, install the crash-4.0-8.11.src.rpm, cd to the appropriate SPECS directory, and build the package:
# rpm -Uvh crash-4.0-8.11.src.rpm 1:crash ########################################### [100%] # cd /usr/src/redhat/SPECS # rpmbuild -ba crash.spec Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.19005 + umask 022 + cd /usr/src/redhat/BUILD + LANG=C + export LANG + unset DISPLAY + cd /usr/src/redhat/BUILD + rm -rf crash-4.0-8.11 + /bin/gzip -dc /usr/src/redhat/SOURCES/crash-4.0-8.11.tar.gz + tar -xvvf - drwxr-xr-x root/root 0 2002-09-12 16:51:40 crash-4.0-8.11/ -rwxrwxr-x root/root 31916 2002-09-12 16:51:40 crash-4.0-8.11/main.c -rwxrwxr-x root/root 103454 2002-09-12 16:51:40 crash-4.0-8.11/tools.c -rwxrwxr-x root/root 5802 2002-09-12 16:51:40 crash-4.0-8.11/global_data.c -rwxrwxr-x root/root 225343 2002-09-12 16:51:40 crash-4.0-8.11/memory.c -rwxrwxr-x root/root 75492 2002-09-12 16:51:40 crash-4.0-8.11/filesys.c -rwxrwxr-x root/root 211519 2002-09-12 16:51:40 crash-4.0-8.11/help.c -rwxrwxr-x root/root 110604 2002-09-12 16:51:40 crash-4.0-8.11/task.c -rwxrwxr-x root/root 101805 2002-09-12 16:51:40 crash-4.0-8.11/kernel.c -rwxrwxr-x root/root 2198 2002-09-12 16:51:40 crash-4.0-8.11/test.c -rwxrwxr-x root/root 18949 2002-09-12 16:51:40 crash-4.0-8.11/gdb_interface.c -rwxrwxr-x root/root 20239 2002-09-12 16:51:40 crash-4.0-8.11/configure.c -rwxrwxr-x root/root 29931 2002-09-12 16:51:40 crash-4.0-8.11/net.c -rwxrwxr-x root/root 99654 2002-09-12 16:51:40 crash-4.0-8.11/dev.c -rwxrwxr-x root/root 76146 2002-09-12 16:51:40 crash-4.0-8.11/alpha.c -rwxrwxr-x root/root 74638 2002-09-12 16:51:40 crash-4.0-8.11/x86.c -rwxrwxr-x root/root 42109 2002-09-12 16:51:40 crash-4.0-8.11/ppc.c -rwxrwxr-x root/root 76357 2002-09-12 16:51:40 crash-4.0-8.11/ia64.c (complete output not shown) Requires: libc.so.6 libc.so.6(GLIBC_2.0) libc.so.6(GLIBC_2.1) libc.so.6(GLIBC_2. 2) libc.so.6(GLIBC_2.2.3) libc.so.6(GLIBC_2.3) libdl.so.2 libdl.so.2(GLIBC_2.0) libdl.so.2(GLIBC_2.1) libm.so.6 libm.so.6(GLIBC_2.0) libncurses.so.5 libz.so.1 Processing files: crash-debuginfo-4.0-8.11 Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHave Prefix) <= 4.0-1 Checking for unpackaged file(s): /usr/lib/rpm/check-files /var/tmp/crash-root Wrote: /usr/src/redhat/SRPMS/crash-4.0-8.11.src.rpm Wrote: /usr/src/redhat/RPMS/i386/crash-4.0-8.11.i386.rpm Wrote: /usr/src/redhat/RPMS/i386/crash-debuginfo-4.0-8.11.i386.rpm # |
Lastly, install the binary RPM, which copies the crash executable to the /usr/bin directory:
# rpm -ivh /usr/src/redhat/RPMS/i386/crash-4.0-8.11.i386.rpm Preparing... ########################################### [100%] 1:crash # |
Invocation
When crash is run on a dumpfile, at least two arguments are always required:
- The kernel object filename, often referred to as the kernel namelist. When initially built from the kernel sources, its name is vmlinux. In RHEL3 installations, it is copied to the /boot directory, where the operating system release number is appended, as in vmlinux-2.4.21-4.ELsmp. In RHEL4, RHEL5 and RHEL6 installations, the vmlinux file is part of the kernel debuginfo package, and is located in the relevant /usr/lib/debug/lib/modules/<release> directory.
- The dumpfile name, typically named vmcore.
For example, if both files are located in the current directory:
# crash vmlinux vmcore |
When crash is run on a live system, /dev/mem is used as the memory image. In RHEL4, RHEL5 and RHEL6, the /dev/mem driver is restricted on x86 and x86_64 systems, and is replaced by the /dev/crash driver. In any case, only the kernel object filename is required:
# crash vmlinux |
Furthermore, when crash is run on a live system, the vmlinux argument is not required when the kernel object file is located in any of the following locations:
- /boot
- /usr/lib/debug/lib/modules/<release>
- / (root directory)
- any subdirectory of /usr/src
- /usr/src/redhat/BUILD/kernel-x.x.x/linux-<release>
When the vmlinux file is not entered on the command line, a search will be made in all of the directories above until a kernel object file is found that contains a version string that matches the running kernel, as indicated by /proc/version. If a matching kernel is found, then crash may be invoked on a live system simply by entering:
# crash |
In the examples above, it is presumed that the vmlinux kernel has been built with the -g C flag, which traditionally has not been done by default. To address this requirement, starting with Red Hat Enterprise Linux 3 (RHEL3), all RHEL kernels are now built with -g C flag. The manner of accessing the debug data for RHEL3, RHEL4, RHEL5 and RHEL6 kernels is described in the following sections. Unfortunately, since RHEL2.1 kernels are not built with -g, the kernel must be rebuilt; directions for rebuilding RHEL2.1 kernels can be found here.
RHEL3 Kernels
In RHEL3, the vmlinux kernel debug information is stripped and stored in a separate debuginfo file. The stripped vmlinux file in /boot has an embedded link to its associated debuginfo file in /usr/lib/debug/boot, so that the crash utility (and the built-in gdb module) knows where to find it:
| RHEL3 kernel namelist in /boot | RHEL3 kernel debuginfo file in /usr/lib/debug/boot |
| vmlinux-<release>.EL vmlinux-<release>.ELsmp vmlinux-<release>.ELhugemem | vmlinux-<release>.EL.debug vmlinux-<release>.ELsmp.debug vmlinux-<release>.ELhugemem.debug |
The debuginfo files for a specific kernel <release> come from a separate RPM that must be installed for the crash utility to work. For example, the i686 RPM for the examples above would be named kernel-debuginfo-<release>.i686.rpm, and would install the debuginfo file for all three of the kernel flavors.
For example, to run crash on a live system, the associated debuginfo package must be installed:
# uname -r 2.4.21-4.ELsmp # rpm -ivh kernel-debuginfo-2.4.21-4.EL.i686.rpm Preparing... ########################################### [100%] 1:kernel-debuginfo ########################################### [100%] # ls /usr/lib/debug/boot vmlinux-2.4.21-4.EL.debug vmlinux-2.4.21-4.ELhugemem.debug vmlinux-2.4.21-4.ELsmp.debug # |
Accordingly, if the running kernel's vmlinux file is in one the search locations above, and its associated debuginfo file is located in the /usr/lib/debug/boot directory or in the current directory from which crash is invoked, no arguments are required to run on a live system:
# crash |
However, if the linked debuginfo file is not in either of those locations, it can be added to the crash command line along with the vmlinux filename. So, for example, if the debuginfo file was located in /tmp:
# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug |
For analyzing dumpfiles however, the vmlinux file name must be on the command line along with the dumpfile name, as in the following examples:
# crash /boot/vmlinux-2.4.21-4.ELsmp vmcore |
or if the debuginfo file is not in the standard location:
# crash /boot/vmlinux-2.4.21-4.ELsmp /tmp/vmlinux-2.4.21-4.ELsmp.debug vmcore |
RHEL4 Kernels
The procedure has been made much simpler for RHEL4 kernels. The kernel is built with the -g flag, and the resultant vmlinux file is stored in the associated debuginfo package. After installing the debuginfo package, the vmlinux file for each kernel flavor of a given RHEL4 release will be installed in the directory named:
/usr/lib/debug/lib/modules/<release><flavor>/vmlinux
where for i686 kernels, <flavor> can be either hugemem, smp, or nothing (for uniprocessor kernels). For example:
# uname -r 2.6.9-6.39.ELsmp # rpm -ivh kernel-debuginfo-2.6.9-6.39.EL.i686.rpm Preparing... ########################################### [100%] 1:kernel-debuginfo ########################################### [100%] # # find /usr/lib/debug/lib/modules/2.6.9-6.39.EL* -name vmlinux /usr/lib/debug/lib/modules/2.6.9-6.39.ELhugemem/vmlinux /usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux /usr/lib/debug/lib/modules/2.6.9-6.39.EL/vmlinux # |
Once the debuginfo package is installed, crash can be invoked on the live system with no arguments, because the vmlinux file will be found automatically:
# crash |
To run crash on a dumpfile, however, the appropriate vmlinux file and the dumpfile name must both be on the command line, as in:
# crash /usr/lib/debug/lib/modules/2.6.9-6.39.ELsmp/vmlinux vmcore |
RHEL5 Kernels
RHEL5 kernels are also built with the -g flag, and the resultant vmlinux file is stored in an associated debuginfo package. Unlike RHEL4, the kernel debuginfo packages are split into one package per flavor plus a "common" package user by all kernel flavors. After installing the debuginfo package, the vmlinux file for each kernel flavor of a given RHEL5 release will be installed in the directory named:
/usr/lib/debug/lib/modules/<release><flavor>/vmlinux
where for i686 kernels, there are 4 possible <flavor>s, either PAE, xen, debug, or nothing (for non-PAE SMP kernels). For example:
# rpm -ivh kernel-debuginfo-2.6.18-66.el5.i686.rpm \
kernel-PAE-debuginfo-2.6.18-66.el5.i686.rpm \
kernel-xen-debuginfo-2.6.18-66.el5.i686.rpm \
kernel-debuginfo-common-2.6.18-66.el5.i686.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo-common########################################### [ 25%]
2:kernel-debuginfo ########################################### [ 50%]
3:kernel-PAE-debuginfo ########################################### [ 75%]
4:kernel-xen-debuginfo ########################################### [100%]
# find /usr/lib/debug/lib/modules/2.6.18-66.el5* -name vmlinux
/usr/lib/debug/lib/modules/2.6.18-66.el5/vmlinux
/usr/lib/debug/lib/modules/2.6.18-66.el5PAE/vmlinux
/usr/lib/debug/lib/modules/2.6.18-66.el5xen/vmlinux
#
|
Once the debuginfo package is installed, crash can be invoked on the live system with no arguments, because the vmlinux file will be found automatically:
# crash |
To run crash on a dumpfile, however, the appropriate vmlinux file and the dumpfile name must both be on the command line, as in:
# crash /usr/lib/debug/lib/modules/2.6.18-66.el5/vmlinux vmcore |
RHEL6 Kernels
RHEL6 kernels are also built with the -g flag, and the resultant vmlinux file is stored in an associated debuginfo package. Like RHEL5, the kernel debuginfo packages are split into one package per flavor plus a "common" package user by all kernel flavors. After installing the debuginfo package, the vmlinux file for each kernel flavor of a given RHEL6 release will be installed in the directory named:
/usr/lib/debug/lib/modules/<release><flavor>/vmlinux
where for x86_64 kernels, there are only 2 possible <flavor>s, either the standard kernel or the debug kernel. For example:
# rpm -ivh kernel-debuginfo-common-2.6.32-70.el6.x86_64.rpm \
kernel-debuginfo-2.6.32-70.el6.x86_64.rpm \
kernel-debug-debuginfo-2.6.32-70.el6.x86_64.rpm
Preparing... ########################################### [100%]
1:kernel-debuginfo-common########################################### [ 25%]
2:kernel-debuginfo ########################################### [ 50%]
3:kernel-debug-debuginfo ########################################### [100%]
# find /usr/lib/debug/lib/modules/2.6.32-70.el6* -name vmlinux
/usr/lib/debug/lib/modules/2.6.32-70.el6/vmlinux
/usr/lib/debug/lib/modules/2.6.32-70.el6debug/vmlinux
#
|
Once the debuginfo package is installed, crash can be invoked on the live system with no arguments, because the vmlinux file will be found automatically:
# crash |
To run crash on a dumpfile, however, the appropriate vmlinux file and the dumpfile name must both be on the command line, as in:
# crash /usr/lib/debug/lib/modules/2.6.32-70.el6/vmlinux vmcore |
RHEL2.1 Kernels (or kernels built without -g flag)
If the running kernel was not built with the -g C flag, then it is necessary to rebuild a kernel of the same configuration with the -g C flag. The essential change done by this kernel rebuild task is a modification of top-level Makefile of the kernel source tree, such that the CFLAGS definition contains the -g flag. For example, this is the line that must be changed:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common
|
by adding the -g flag:
CFLAGS := $(CPPFLAGS) -Wall -Wstrict-prototypes -Wno-trigraphs -O2 \
-fno-strict-aliasing -fno-common -g
|
For example, since RHEL2.1 kernels are not built with -g, a kernel rebuild is required. For a detailed example of how to rebuild a RHEL2.1 kernel with the -g flag, please refer to these directions.
Upon rebuilding the kernel, a new vmlinux file will be created that contains the debug data required by crash. However, the symbol values will not match those of the running or dumped kernel. To deal with this inequity, the actual symbol values can be gathered from either the original non-debug vmlinux file or from its associated System.map file. That being the case, two arguments must be supplied to crash to fully describe the running/dumped kernel, the newly-created vmlinux file compiled with -g, as well as a source of the real symbol values. So, for example, if the vmlinux file built with -g were renamed to vmlinux.dbg, the invocation line would look like this on a live system:
# crash vmlinux vmlinux.dbg (or) # crash /boot/System.map vmlinux.dbg (or) # crash -S vmlinux.debug |
The -S argument above is simply an alternative to entering the default /boot/System.map string.
Similarly, when looking at a dumpfile, two arguments are required to describe the dumped kernel, along with the vmcore image:
# crash vmlinux vmlinux.dbg vmcore (or) # crash /boot/System.map vmlinux.dbg vmcore (or) # crash -S vmlinux.dbg vmcore |
Again, for a detailed example of how to rebuild a RHEL2.1 kernel with the -g flag, refer to these directions.
Invocation output
The arguments may be entered in any order. If the file arguments are not in the current directory, absolute pathnames must be used. When in doubt, simply enter crash -h to get an explanation of the command line arguments:
# crash -h
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
[namelist]
The [namelist] argument is a pathname to an uncompressed kernel image
(a vmlinux file) that has been compiled with the "-g" switch, or
that has an accessible, associated, debuginfo file. If the [dumpfile]
argument is entered, then the [namelist] argument must be entered
If the [namelist] argument is not entered when running on a live
system, a search will be made in several typical directories for
for a kernel namelist file that matches the live system.
[dumpfile]
The [dumpfile] argument is a pathname to a kernel memory core dump
file. If the [dumpfile] argument is not entered, the session will be
invoked on the live system using /dev/mem, which usually requires root
privileges.
[mapfile]
If the live system kernel, or the kernel from which the [dumpfile]
was derived, was not compiled with the -g switch, then the additional
[mapfile] argument is required. The [mapfile] argument may consist
of either the associated System.map file, or the non-debug kernel
namelist. However, if the [mapfile] argument is used, then the
[namelist] argument must be a kernel namelist of a similar kernel
version that was built with the -g switch.
[-S]
Use "/boot/System.map" as the [mapfile].
Examples when running on a live system:
$ crash
$ crash /usr/tmp/vmlinux
$ crash /boot/System.map vmlinux.dbg
$ crash -S vmlinux.dbg
$ crash vmlinux vmlinux.dbg
Examples when running on a dumpfile:
$ crash vmlinux vmcore
$ crash /boot/System.map vmlinux.dbg vmcore
$ crash -S vmlinux.dbg vmcore
$ crash vmlinux vmlinux.dbg vmcore
[-h [opt]]
The -h option alone displays this message. If the [opt] argument is
a crash command name, the help page for that command is displayed. If
the string "input" is entered, a page describing the various crash
command line input options is displayed. If the string "output" is
entered, a page describing command line output options is displayed.
[-v]
Display the versions of crash and gdb making up this executable.
[-s]
Do not display any version, GPL, or crash initialization data; proceed
directly to the "crash>" prompt.
[-i file]
Execute the crash command(s) in [file] prior to accepting any user
input from the "crash>" prompt.
[-d num]
Set crash debug level [num]. The higher the number, the more debug data
will be printed during crash runtime.
|
Given that all invocation arguments are in order, here is an example of a successful invocation on a dumpfile, running a kernel that was built with -g, along with a vmcore dump file was created by the Red Hat Netdump facility:
# crash vmlinux-2.4.20-2.1.15.entsmp vmcore
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: vmlinux-2.4.20-2.1.15.entsmp
DUMPFILE: vmcore
CPUS: 1
DATE: Wed Mar 12 10:12:56 2003
UPTIME: 00:38:25
LOAD AVERAGE: 1.16, 0.74, 0.30
TASKS: 60
NODENAME: dhcp64-220.boston.redhat.com
RELEASE: 2.4.20-2.1.15.entsmp
VERSION: #1 SMP Tue Mar 11 16:12:22 EST 2003
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 0
COMMAND: "swapper"
TASK: c038e000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
|
This next example shows the output when the panicking kernel was not built with -g. In this case, a similar kernel type was built with -g, and the resultant kernel object file was renamed as vmlinux.dbg. Note that there will be a message concerning the patching of gdb data; this indicates that the non-matching symbol values from the vmlinux.dbg are being over-written by the correct symbol values found in the original vmlinux file:
# crash vmlinux vmlinux.dbg vmcore
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
please wait... (patching 16053 gdb minimal_symbol values)
DEBUG KERNEL: vmlinux.dbg
DUMPFILE: vmcore
CPUS: 1
DATE: Wed Mar 27 11:02:31 2002
UPTIME: 00:07:24
LOAD AVERAGE: 0.43, 0.42, 0.19
TASKS: 68
NODENAME: anderson.boston.redhat.com
RELEASE: 2.4.9-26beta.48enterprise
VERSION: #1 SMP Thu Mar 21 12:33:05 EST 2002
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
|
Invocation on a live system looks essentially the same, except that the DUMPFILE will be indicated as /dev/mem. In the following example, no arguments were entered, because the running RHEL3 kernel was found in the /boot directory, and its associated debuginfo file in the /usr/lib/debug/boot directory. The debuginfo file is listed next to the DEBUGINFO tag:
# crash
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /boot/vmlinux-2.4.21-4.ELhugemem
DEBUGINFO: /usr/lib/debug/boot/vmlinux-2.4.21-4.ELhugemem.debug
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Aug 21 11:24:38 2003
UPTIME: 1 days, 23:14:11
LOAD AVERAGE: 0.14, 0.10, 0.08
TASKS: 106
NODENAME: crash.boston.redhat.com
RELEASE: 2.4.21-4.ELhugemem
VERSION: #1 SMP Wed Aug 13 21:33:31 EDT 2003
MACHINE: i686 (1993 Mhz)
MEMORY: 511.5 MB
PID: 4757
COMMAND: "crash"
TASK: 19b82000
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash>
|
Invocation Errors
Invocation errors will cause the crash session to abort upon initialization. Typically they occur as the result of one of the following reasons:
- The vmlinux file contains no debug data (i.e., was built without the -g flag), and no additional debug kernel object file name was entered on the command line. The error message will be of the form:
crash: /boot/vmlinux-2.4.18-14: no debugging data available
- The vmlinux file does not match the dumpfile. The error message will be of the form:
crash: vmlinux and tmp/vmcore do not match!
- The vmlinux file could not be found on a live system. The error message will be of the form:
crash: cannot find booted kernel -- please enter namelist argument
- The associated debuginfo file cannot be found. The error message will be of the form:
crash: /boot/vmlinux-2.4.21-4.ELsmp: no debugging data available crash: vmlinux-2.4.21-4.ELsmp.debug: debuginfo file not found
- The crash utility binary does not match the vmlinux and/or vmcore arguments. The error message will be of the form:
WARNING: machine type mismatch: crash utility: X86 vmlinux: X86_64 crash: vmlinux: not a supported file format
Command Input
Upon a successful session invocation on a dump file or a live kernel, the crash> prompt will appear. Interactive crash commands are gathered using the GNU readline library, taking advantage of its command line history mechanism, and its vi or emacs command line editing modes. Commands may also be issued to crash from a file.
The command line history consists of a numbered list of previously-run commands. The full list of commands may be viewed by entering h at any time. For example:
crash> h [1] bt -a [2] ps [3] foreach bt [4] set [5] dis -rl c0221141 crash> |
Commands in the history list may be re-run in the following manners
- To re-run the last command executed, simply enter r or !! and then ENTER.
- Enter r followed by the appropriate history list number, and then ENTER.
- Enter r followed by a uniquely-identifying set of characters from the beginning of the previously-entered command string, and then ENTER.
- Recycle back through the command history list using the up-arrow and down-arrow keys until the desired command is re-displayed, and then ENTER.
- Recycle back through the command history list using the key-strokes appropriate for the command line editing mode being used (vi or emacs) until the desired command is re-displayed, and then ENTER.
The command line editing mode may be set to either vi (the default) or emacs. The mode may set in the following manners, listed in increasing order of precedence:
- Do none of the following, in which case the default vi editing mode will be used.
- Set the EDITOR environment variable to either vi or emacs.
- Create an entry in a .crashrc file in the user's $HOME directory. The entry must be a line of one of the following forms:
set vi set emacs
- Create an entry in a .crashrc file be located in the current directory, in the form shown in (3) above.
- Use the -e command line option, as in:
# crash -e [vi | emacs] ...
Given either editing mode, any previously entered command line can be brought back by entering the mode-specific key-stroke(s), the command line edited using the proper mode, and then run by hitting ENTER.
Command Line Input from a File
An input file consisting of a list of commands may be fed to crash in the following manners:
crash> < inputfile |
- Upon invocation, as in:
# crash -i inputfile
- Upon invocation, by entering the commands in a .crashrc file, which can be either in the user's $HOME directory or in the current directory.
- On the command line during a crash session, as in:
In all of the three cases above, after the list of commands in the file have completed, the crash> prompt will appear and commands may then be entered interactively (unless one of the file commands happens to be the exit command).
Numerical arguments are typically presumed to be decimal unless the argument contains an a, b, c, d, e or f. In those cases, the preceding 0x is not required. For hexadecimal numbers that do not contain one of those 6 characters, the preceding 0x is required. So, for example, a value of 1 gigabyte would have to be expressed as 0x40000000, whereas 3 gigabytes could be expressed as c0000000.
It should be noted that several commands will only accept hexadecimal numerical arguments. For example, the rd ("read") command only accepts hexadecimal addresses. Therefore a read from user address of 0x40017000 could be entered as:
crash> rd 40017000 40 40017000: 20000824 00000010 00000048 00000063 $.. ....H...c... 40017010: 00000082 000000ba 000000bb 000000cd ................ 40017020: 000000ce 000000cf 000000d7 000000db ................ 40017030: 000000dc 000000dd 000000de 000000e2 ................ 40017040: 000000ed 00000167 6c676e45 20687369 ....g...English 40017050: 61636f6c 6620656c 7420726f 55206568 locale for the U 40017060: 46004153 20656572 74666f53 65726177 SA.Free Software 40017070: 756f4620 7461646e 2c6e6f69 636e4920 Foundation, Inc 40017080: 3935002e 6d655420 20656c70 63616c50 ..59 Temple Plac 40017090: 202d2065 74697553 33332065 42202c30 e - Suite 330, B |
Command Output
crash commands can often be verbose, and it's helpful to control the output, as well as to be able to scroll backwards to view previous command output. So, by default, command output that would overflow the user's display screen is piped to /usr/bin/less, along with a prompt line that informs the user how to scroll forward, backward, or to quit the command. For example, here is an example of what a ps command might look like:
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 c030a000 RU 0.0 0 0 [swapper]
1 0 0 cff98000 IN 0.2 1412 468 init
2 1 0 c1446000 IN 0.0 0 0 [keventd]
3 1 0 cfffa000 IN 0.0 0 0 [kapm-idled]
4 0 0 cfff8000 IN 0.0 0 0 [ksoftirqd_CPU0]
5 0 0 cffee000 IN 0.0 0 0 [kswapd]
6 0 0 cffec000 IN 0.0 0 0 [kreclaimd]
7 0 0 c1826000 IN 0.0 0 0 [bdflush]
8 0 0 c1824000 IN 0.0 0 0 [kupdated]
9 1 0 cff90000 IN 0.0 0 0 [mdrecoveryd]
13 1 0 cf07a000 IN 0.0 0 0 [kjournald]
89 1 0 ce804000 IN 0.0 0 0 [khubd]
184 1 0 ce4d4000 IN 0.0 0 0 [kjournald]
572 1 0 cd938000 IN 0.0 440 48 dhcpcd
637 1 0 ce4a4000 IN 0.2 1476 612 syslogd
642 1 0 cd92c000 IN 0.2 2092 432 klogd
663 1 0 ce2bc000 IN 0.2 1564 612 portmap
691 1 0 cd84a000 IN 0.3 1652 668 rpc.statd
803 1 0 cd756000 IN 0.2 1400 452 apmd
828 1 0 cd6c2000 IN 0.3 18024 684 ypbind
830 828 0 cd76e000 IN 0.3 18024 684 ypbind
831 830 0 cd71c000 IN 0.3 18024 684 ypbind |
-- MORE -- forward: <SPACE>, <ENTER> or j backward: b or k quit: q |
This default output scrolling behavior can be turned off by entering the following line in a .crashrc file located in either the $HOME or current directories:
set scroll off |
During runtime, the following commands (or their respective builtin aliases) can be used to turn the scrolling behavior off, and back on, again:
crash> set scroll off scroll: off crash> set scroll on scroll: on crash> alias ORIGIN ALIAS COMMAND builtin man help builtin ? help builtin quit q builtin sf set scroll off builtin sn set scroll on builtin hex set radix 16 builtin dec set radix 10 builtin g gdb builtin px p -x builtin pd p -d builtin for foreach builtin size * builtin dmesg log builtin last ps -l crash> sf scroll: off crash> sn scroll: on crash> |
Alternatively, command output may be redirected to a pipe or to a file using standard shell redirection syntax. For examples:
crash> task | grep uid uid = 3369, euid = 3369, suid = 3369, fsuid = 3369, crash> foreach bt > bt.all crash> ps >> process.data crash> kmem -i | grep SLAB > slab.pages crash> |
When a command's output is redirected to a pipe or file, the default /usr/bin/less behavior is turned off for that particular command.
The default numerical output radix for non-pointer values is decimal, which is most often noticed when using the builtin gdb capability of printing formatted data structures. During runtime, the following commands (or their respective builtin aliases) can be used to toggle the output radix from decimal to hexadecimal, and back again:
crash> set radix 16 output radix: 16 (hex) crash> set scroll 10 output radix: 10 (decimal) crash> alias ORIGIN ALIAS COMMAND builtin man help builtin ? help builtin quit q builtin sf set scroll off builtin sn set scroll on builtin hex set radix 16 builtin dec set radix 10 builtin g gdb builtin px p -x builtin pd p -d builtin for foreach builtin size * builtin dmesg log crash> hex output radix: 16 (hex) crash> dec output radix: 10 (decimal) crash> |
Alternatively, the px or pd aliases coerce the "print" command p, to override the current output radix. For example, here the changing value of jiffies on a live system is printed using the current default radix, then in hexadecimal, and lastly in decimal:
crash> p jiffies jiffies = $4 = 69821055 crash> px jiffies jiffies = $5 = 0x42963aa crash> pd jiffies jiffies = $6 = 69821656 crash> |
Crash Context
Upon a successful invocation of a crash session, one of the existing Linux tasks is selected as the current context. It is important to be aware of the current context because several crash commands are "context-sensitive", meaning that the command is executed from the view-point of the current context. Therefore, the output of context-sensitive commands can vary depending upon which context is current.
Upon invocation of a crash session, the selection of the current context is based upon the following criteria:
On dumpfiles:
- The task that was running when die() was called.
- The task that was running when panic() was called.
- The task that was running when an ALT-SYSRQ-c keyboard interrupt was received.
- The task that was running when the character "c" was echoed to /proc/sysrq-trigger.
On a live system:
- the crash task itself.
The current context selection is shown in the session invocation data. For example, here is a session begun on a dumpfile that was created when an insmod task's attempt to install a module resulted in an "oops" violation:
# crash tmp/vm*
crash 4.0-8.11
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: tmp/vmlinux
DEBUG KERNEL: tmp/vmlinux.dbg
DUMPFILE: tmp/vmcore
CPUS: 1
DATE: Wed Mar 27 11:02:31 2002
UPTIME: 00:07:24
LOAD AVERAGE: 0.43, 0.42, 0.19
TASKS: 68
NODENAME: anderson.boston.redhat.com
RELEASE: 2.4.9-26beta.48enterprise
VERSION: #1 SMP Thu Mar 21 12:33:05 EST 2002
MACHINE: i686 (501 Mhz)
MEMORY: 128 MB
PANIC: "Oops: 0002" (check log for details)
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
|
During runtime, the current context can always be displayed by entering the set command with no arguments:
crash> set
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
|
The current context can be changed to a new task via the set command. Either of two "handles" may be used to identify a task, the PID number, or the kernel address of the task's task_struct. For example:
crash> set 1
PID: 1
COMMAND: "init"
TASK: c7f98000
CPU: 0
STATE: TASK_RUNNING
crash> set c0a52000
PID: 1503
COMMAND: "cat"
TASK: c0a52000
CPU: 0
STATE: TASK_INTERRUPTIBLE
crash>
|
Alternatively, the current context can be set to the task running on a given CPU number, or back to the panicking task. Using the same dumpfile session shown above, in which there is only one CPU, the original context may be restored using the -c CPU-number or the -p ("panic task") options:
crash> set -c 0
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash> set -p
PID: 1696
COMMAND: "insmod"
TASK: c74de000
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash>
|
It is important to be aware that several crash commands are context-sensitive. For example, the files command displays data about the open files of a task. If it is issued with no arguments, it displays the open files data of the current context. In this example, the current context happens to be PID 642, the klogd daemon:
crash> files PID: 642 TASK: cd92c000 CPU: 0 COMMAND: "klogd" ROOT: / CWD: / FD FILE DENTRY INODE TYPE PATH 0 ce06c800 ce29ec60 cd8df900 REG /proc/kmsg 1 ce06cf20 ce29ebe0 cd8df740 SOCK socket:/[858] 2 ce06c5c0 ce2423a0 ce462c80 REG /boot/System.map-2.4.9-e.3enterprise |
However, if the files command is issued with either of the two task handles as an argument, then it will display the open files data of the specified task. In this example, PID 12731 is specified:
crash> files 12731 PID: 12731 TASK: c8150000 CPU: 0 COMMAND: "vi" ROOT: / CWD: /tmp FD FILE DENTRY INODE TYPE PATH 0 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11 1 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11 2 c988cd80 ced919a0 c87fc3c0 CHR /dev/pts/11 4 c2927ae0 c6cad8a0 cd6d5040 REG /tmp/.crontab.12730.swp 5 c2927a80 c6cad9a0 c5764ac0 REG /tmp/crontab.12730 |
This type of context-sensitive behaviour is also exhibited by the vm, bt, sig, set, net and task commands. Unless a PID or task address is specified as an argument, the output will reflect data concerning the current context.
Other commands may simply default to the current context. For example, the rd command can read memory from an address that is specified as a user-space address. Since the rd command does not accept a PID or task address as an argument, it would be necessary to be aware that the user-space access will come from the address space of the current context.
Builtin Help
Readily available help information is built into the crash utility. During a session, entering the help command with no argument shows the following menu:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Each command has its own man-like help page, which can be viewed by clicking on the command name above. Each help page details the syntax of the command and its available options, a description of the command in general, a description of each option, and a set of examples. During a crash session, a command's help page can be displayed by entering help followed by the command name. So, for example, to get help on how to use the set command:
crash> help set
NAME
set - set a process context or internal crash variable
SYNOPSIS
set [pid | taskp | [-c cpu] | -p] | [crash_variable [setting]] | -v
DESCRIPTION
This command either sets a new context, or gets the current context for
display. The context can be set by the use of:
pid a process PID.
taskp a hexadecimal task_struct pointer.
-c cpu sets the context to the active task on a cpu (dumpfiles only).
-p sets the context to the panic task, or back to the crash task on
a live system.
-v display the current state of internal crash variables.
If no argument is entered, the current context is displayed. The context
consists of the PID, the task pointer, the CPU, and task state.
This command may also be used to set internal crash variables. If no value
argument is entered, the current value of the crash variable is shown. These
are the crash variables, acceptable arguments, and purpose:
scroll on | off controls output scrolling.
scroll less /usr/bin/less as the output scrolling program.
scroll more /bin/more as the output scrolling program.
scroll CRASHPAGER use CRASHPAGER environment variable as the
output scrolling program.
radix 10 | 16 sets output radix to 10 or 16.
refresh on | off controls internal task list refresh.
print_max number set maximum number of array elements to print.
console device-name sets debug console device.
debug number sets crash debug level.
core on | off if on, drops core when the next error message
is displayed.
hash on | off controls internal list verification.
silent on | off turns off initialization messages; turns off
crash prompt during input file execution.
(scrolling is turned off if silent is on)
edit vi | emacs set line editing mode (from .crashrc file only).
namelist filename name of kernel (from .crashrc file only).
dumpfile filename name of core dumpfile (from .crashrc file only).
zero_excluded on | off controls whether excluded pages from a dumpfile
should return zero-filled memory.
Internal variables may be set in four manners:
1. entering the set command in $HOME/.crashrc.
2. entering the set command in .crashrc in the current directory.
3. executing an input file containing the set command.
4. during runtime with this command.
During initialization, $HOME/.crashrc is read first, followed by the
.crashrc file in the current directory. Set commands in the .crashrc file
in the current directory override those in $HOME/.crashrc. Set commands
entered with this command or by runtime input file override those
defined in either .crashrc file. Multiple set command arguments or argument
pairs may be entered in one command line.
EXAMPLES
Set the current context to task c2fe8000:
crash> set c2fe8000
PID: 15917
COMMAND: "bash"
TASK: c2fe8000
CPU: 0
STATE: TASK_INTERRUPTIBLE
Set the context back to the panicking task:
crash> set -p
PID: 698
COMMAND: "gen12"
TASK: f9d78000
CPU: 2
STATE: TASK_RUNNING (PANIC)
Turn off output scrolling:
crash> set scroll off
scroll: off (/usr/bin/less)
Show the current state of crash internal variables:
crash> set -v
scroll: on (/usr/bin/less)
radix: 10 (decimal)
refresh: on
print_max: 256
console: /dev/pts/2
debug: 0
core: off
hash: on
silent: off
edit: vi
namelist: vmlinux
dumpfile: vmcore
zero_excluded: off
Show the current context:
crash> set
PID: 1525
COMMAND: "bash"
TASK: c1ede000
CPU: 0
STATE: TASK_INTERRUPTIBLE
|
If for some reason a crash session cannot be invoked, but help information for a particular crash command is desired, the same help page can be displayed from a shell command line using the -h option to crash:
# crash -h ascii
NAME
ascii - translate a hexadecimal string to ASCII
SYNOPSIS
ascii value ...
DESCRIPTION
Translates 32-bit or 64-bit hexadecimal values to ASCII. If no argument
is entered, an ASCII chart is displayed.
EXAMPLES
Translate the hexadecimal value of 0x62696c2f7273752f to ASCII:
crash> ascii 62696c2f7273752f
62696c2f7273752f: /usr/lib
Display an ASCII chart:
crash> ascii
0 1 2 3 4 5 6 7
+-------------------------------
0 | NUL DLE SP 0 @ P ' p
1 | SOH DC1 ! 1 A Q a q
2 | STX DC2 " 2 B R b r
3 | ETX DC3 # 3 C S c s
4 | EOT DC4 $ 4 D T d t
5 | ENQ NAK % 5 E U e u
6 | ACK SYN & 6 F V f v
7 | BEL ETB ` 7 G W g w
8 | BS CAN ( 8 H X h x
9 | HT EM ) 9 I Y i y
A | LF SUB * : J Z j z
B | VT ESC + ; K [ k {
C | FF FS , < L \ l |
D | CR GS _ = M ] m }
E | SO RS . > N ^ n ~
F | SI US / ? O - o DEL
#
|
Lastly, help concerning command input and output can be displayed by entering help input or help output during runtime, or crash -h input or crash -h output from a shell command line.
The Command Set
Each crash command generally falls into one of the following categories:
The remainder of this section breaks the command set into categories, and gives a short description of each command in that category. However, for complete details and examples, recall that the crash utility has a self-contained help page for each command; to view the full help page, click on the command name next to its description below.
Symbolic Display of Kernel Text or Data
The following commands typically take full advantage of the power of gdb to display kernel data structures symbolically.
| Command | | Description |
| struct | | Displays a formatted kernel data structure type located at a given address, or at an address referred to by a symbol; if no address is specified, the structure definition is displayed. The output can be narrowed down to a singular member of the structure, or to display the offset of every member from the beginning of the structure. A count may be appended to display an array of structures. Its usage is so common that two short-cuts exist such that the user need not enter the "struct" command name:
|
| union | | Same as struct command, but used for kernel data types defined as unions instead of structures.. |
| * | | "Pointer-to" command which can be used in lieu of entering struct or union; the gdb module first determines whether the argument is a structure or a union, and then calls the appropriate function. |
| p | | Displays the contents of a kernel variable; the arguments are passed on to gdb's print command for proper formatting. Two builtin aliases, px and pd, set the numerical output radix to hexadecimal or decimal for the print operation, temporarily overriding the current default. |
| whatis | | Displays all available symbol table information concerning a data type or a data symbol. |
| sym | | Translates a kernel symbol name to its kernel virtual address and section, or a kernel virtual address to its symbol name and section. It can also be used to dump the complete list of kernel symbols, or to query the symbol list for all symbols containing a given sub-string. |
| dis | | Disassembles the text of complete kernel function, or from a specified address for a given number of instructions, or from the beginning of a function up to a specified address. |
System State
The majority of crash commands come from the following set of "kernel-aware" commands, which delve into various kernel subsystems on a system-wide or per-task basis. The task-specific commands are context-sensitive, meaning that they act upon the current context unless a PID or task address is specified as an argument.
| Command | | Description |
| bt | | Arguably the most useful crash command, bt displays a task's kernel stack back-trace, including full exception frame dumps. It is context-sensitive, although the -a option will display the stack traces of the active task on each CPU. This command is often used within the foreach wrapper command in order to display the back traces of all tasks with one command. |
| dev | | Displays data concerning the character and block device assignments, I/O port usage, I/O memory usage, and PCI device data. |
| files | | This context-sensitive command displays the task's current root directory and working directories, and then for each open file descriptor, shows:
|
| fuser | | Displays a list of tasks that reference a specified filename or inode address as the current root or working directory, an open file descriptor, or which mmap the file. |
| irq | | Display data concerning interrupt request numbers and bottom-half handling. |
| kmem | | This command has numerous options that delve into the state of several kernel memory subsystems:
|
| log | | Dumps the kernel message buffer chronologically, accounting for any wrap-around. |
| mach | | Displays machine and/or processor specific data. |
| mod | | Displays the list of currently-loaded kernel modules. More importantly, it loads the debug data from the module object files if they are available, allowing symbolic debugging capability of kernel modules. |
| mount | | For each mounted filesystem, or for just a specified filesystem, displays:
|
| net | | Displays various network-related data:
|
| ps | | Useful process status command, in typical Linux ps command type output, containing:
|
| pte | | This command translates the contents of a PTE into its physical page address and page bit settings, or if it references a swap location, the swap device and offset. |
| runq | | Displays list of tasks on the run queue. |
| sig | | A context-sensitive command which displays a task's signal information, including:
|
| swap | | For each configured swap device, this command displays the same data that is shown by the Linux command swapon -s. |
| sys | | Re-displays the same system-related data that is seen during crash initialization:
|
| task | | This context-sensitive command displays a task's complete task_struct contents, or one or more members of the structure. This command is often used within the foreach wrapper command in order to display task_struct data for all tasks with one command. |
| timer | | Displays the timer queue entries in chronological order, listing the target function names, the current value of jiffies, and the expiration time of each entry. |
| vm | | This powerful, context-sensitive command displays a wealth of information concerning a task's virtual memory data, including:
|
| vtop | | This context-sensitive command translates a user or kernel virtual address to its physical address. Also displayed are:
|
| waitq | | Lists the tasks linked on a specified kernel wait queue. |
Utility Functions
The following commands are a set of useful helper commands serving various purposes, some simple, others quite powerful.
| Command | | Description |
| ascii | | Translates a numerical value into its ASCII components; with no arguments, displays an ASCII chart. |
| btop | | Translates a byte value (physical address) to its page number. |
| eval | | A simple calculator, evaluates an expression and displays the result in hexadecimal, decimal, octal and binary, and optionally showing the bit numbers set in the result. |
| list | | Dumps the entries of a linked list of structures. It can handle lists of structures that are singly-linked with simple "next" pointers, or those with embedded list_head structures. The output may be constrained to simply display the address of each structure in the list, or if directed, also dump each complete structure, or just one member of each structure. The gathered list entries are hashed, so a corrupted list that loops back upon itself will be recognized. |
| ptob | | translates a page frame number to its byte value (physical address). |
| ptov | | Translates a physical address into a kernel virtual address by adding the appropriate PAGE_OFFSET value. |
| search | | Searches a range of user or kernel memory space for given value, with an optional "don't care" bit-mask argument. |
| rd | | Displays a specified amount of user virtual, kernel virtual, or physical memory in several formats, such as 8, 16, 32 or 64 bit values, hexadecimal or decimal, symbolically, and with ASCII translations. When reading user virtual addresses, the command is context-sensitive. |
| wr | | Modifies the contents of memory on a live system. Write permission on /dev/mem is required; this command should obviously be used with great care. The write operation is constrained to one 8, 16, 32 or 64 bit location. |
Session Control Commands
The following commands typcally aid in the efficient running of a crash session.
| Command | | Description |
| alias | | Creates a single-word alias for a command string. Several aliases are built into crash; user-defined aliases may also be defined in a .crashrc file, or during a crash session by entering it on the command line or reading it from an input file. |
| exit | | Shuts down the crash session (same as q). |
| extend | | Extend the crash command set by dynamically loading a shared object library containing one or more user-written commands. |
| foreach | | Quite often it is helpful, or even necessary, to run the same crash context-sensitive command on a number of tasks by just entering one command. This wrapper command sets off the execution of a given crash command on each of a defined set of tasks, temporarily changing the current context to that of the targeted task before running the command. The set of tasks that are issued the given command can be defined by:
|
| gdb | | This command passes its arguments directly to gdb for processing. This is typically not necessary, but where ambiguities between crash and gdb command names exist, this will force the command to be executed by gdb. |
| repeat | | This wrapper command repeats a crash command indefinitely, optionally delaying a given number of seconds between each command execution. Obviously this command is only useful when running on a live system. |
| set | | This primary purpose for this command is to set the crash context to a new task, or to display the current context. It can also be used to view or change one of a set of internal crash variables that modify program behavior, such as the default output radix or scrolling behavior. It can be called from the foreach wrapper command for viewing the context data of each task. |
| q | | Shuts down the crash session (same as exit). |
Crash Usage: A Case Study
The steps taken to debug a kernel crash dump are not etched in stone, and the crash commands used to debug a kernel issue vary according to the problem exhibited. The section contains of a case study that shows how the capabilities of the crash utility were used to to debug a specific kernel problem. However, before doing so, it should be noted that the following commands are typically the most commonly-used:
bt Display the backtrace of the current context, or as specified with arguments. This command is typically the first command entered after starting a dumpfile session. Since the initial context is the panic context, it will show the function trace leading up to the kernel panic. bt -a will show the trace of the active task on each CPU, since there may be an interrelationship between the panicking task on one CPU and the running task(s) on the other CPU(s). When bt is given as the argument to foreach. displays the backtraces of all tasks. struct Print the contents of a data structure at a specified address. This command is so common that it is typically unnecessary to enter the struct command name on the command line; if the first command line argument is not a crash or gdb command, but it is the name of a known data structure, then all the command line arguments are passed to the struct command. So for example, the following two commands yield the same result:
set Set a new task context by PID, task address, or cpu. Since several crash commands are context-sensitive, it's helpful to be able to change the context to avoid having to pass the PID or task address to those context-sensitive commands in order to access the data of a task that is not the current context. p Prints the contents of a kernel variable; since it's a gateway to the print command of the mbedded gdb module, it can also be used to print complex C language expressions. rd Read memory, which may be either kernel virtual, user virtual, or physical, and display it several different formats and sizes. ps Lists basic task information for each process; it can also display parent and child hierarchies. log Dump the kernel log_buf, which often contains clues leading up to a subsequent kernel crash. foreach Execute a crash command on all tasks, or those specified, in the system; can be used with bt, vm, task, files, net, set, sig and vtop. files Dump the open file descriptor data of a task; most usefully, the file, dentry and inode structure addresses for each open file descriptor. vm Dump the virtual memory map of a task, including the vital information concerning each vm_area_struct making up a task's address space. It can also dump the physical address of each page in the address space, or if not mapped, its location in a file or on the swap device.
A Case Study: "kernel BUG at pipe.c:120!"
Upon bringing up a crash session, a great deal of information can be gained just by the invocation data. Here is what what displayed in this particular case:
...
KERNEL: vmlinux-2.4.9-e.10.13enterprise-g
DUMPFILE: vmcore-incomplete
CPUS: 2
DATE: Mon Feb 17 08:20:56 2003
UPTIME: 4 days, 20:04:41
LOAD AVERAGE: 0.95, 1.04, 1.25
TASKS: 110
NODENAME: testbox.redhat.com
RELEASE: 2.4.9-e.10.13enterprise
VERSION: #1 SMP Mon Feb 3 12:59:26 EST 2003
MACHINE: i686 (2788 Mhz)
MEMORY: 6 GB
PANIC: "kernel BUG at pipe.c:120!"
PID: 20571
COMMAND: "imp"
TASK: d1566000
CPU: 1
STATE: TASK_RUNNING (PANIC)
crash>
|
In this case the PANIC string "kernel BUG at pipe.c:120!" points to the exact kernel source code line at which the panic occurred.
Then, getting a backtrace of panicking task is typically the first order of the day:
crash> bt
PID: 20571 TASK: d1566000 CPU: 1 COMMAND: "imp"
#0 [d1567e44] die at c010785c
#1 [d1567e54] do_invalid_op at c0107b2c
#2 [d1567f0c] error_code (via invalid_op) at c01073de
EAX: 0000001d EBX: ed87b2e0 ECX: c02f6064 EDX: 00005fa1 EBP: 00001000
DS: 0018 ESI: f640e740 ES: 0018 EDI: 00001000
CS: 0010 EIP: c0150b6d ERR: ffffffff EFLAGS: 00010292
#3 [d1567f48] pipe_read at c0150b6d
#4 [d1567f6c] sys_read at c01468d4
#5 [d1567fc0] system_call at c01072dc
EAX: 00000003 EBX: 0000000a ECX: 40b4e05c EDX: 00002000
DS: 002b ESI: 00002000 ES: 002b EDI: 40b4e05c
SS: 002b ESP: bffe9e88 EBP: bffe9eb8
CS: 0023 EIP: 40aaa1d4 ERR: 00000003 EFLAGS: 00000286
|
The backtrace shows that the call to die() was generated by an invalid_op exception. The exception was caused by the BUG() call in the pipe_read() function:
if (count && PIPE_WAITING_WRITERS(*inode) &&
!(filp->f_flags & O_NONBLOCK)) {
/*
* We know that we are going to sleep: signal
* writers synchronously that there is more
* room.
*/
wake_up_interruptible_sync(PIPE_WAIT(*inode));
if (!PIPE_EMPTY(*inode))
BUG();
goto do_more_read;
}
|
In the code segment above, the pipe_read() code has previously down'd the semaphore of the inode associated with the pipe, giving it exclusive access. It had read all data in the pipe, but still needed more to satisfy the count requested. Finding that there was a writer with more data -- and who was waiting on the semaphore -- it woke up the writer. However, after doing the wakeup, it did a sanity-check on the pipe contents, and found that it was no longer empty -- which is theoretically impossible since it was still holding the semaphore. It appeared that the writer process wrote to the pipe while the reader process still had exclusive access -- somehow overriding the semaphore.
Since the semaphore mechanism was seemingly not working, it was first necessary to look at the actual semaphore structure associated with the pipe's inode. This first required looking at the first argument to the pipe_read() function; the whatis command shows that it is a struct file pointer:
crash> whatis pipe_read ssize_t pipe_read(struct file *, char *, size_t, loff_t *); crash> |
Using the bt -f option, each frame in the backtrace is expanded to show all stack data in the frame. Looking at the expansion of the sys_read() frame, we can see that the last thing pushed on the stack before calling pipe_read() was the file pointer address of edf3f740:
...
#3 [d1567f48] pipe_read at c0150b6d
[RA: c01468d6 SP: d1567f4c FP: d1567f6c SIZE: 36]
d1567f4c: c026701c 00000078 fffffff2 00001000
d1567f5c: 00000000 edf3f740 ffffffea 00002000
d1567f6c: c01468d6
#4 [d1567f6c] sys_read at c01468d4
[RA: c01072e3 SP: d1567f70 FP: d1567fc0 SIZE: 84]
d1567f70: edf3f740 40b4f05c 00002000 edf3f760
d1567f80: c03683d0 fffffffb 00000001 c0120d3b
d1567f90: 00000046 00000046 0000000b c0350960
d1567fa0: 0000000b f639eb00 c0108e0e 00000020
d1567fb0: d1566000 00002000 40b4e05c bffe9eb8
d1567fc0: c01072e3
... |
The task at hand is finding the inode containing the suspect semaphore from the file structure address. The file structure's f_dentry member points to its dentry structure, whose d_inode member in turn points to the pipe's inode. The struct command can be used to dump the complete contents of a data structure at a given address; by tagging the .member onto the structure name, we can print just the member desired. By following the structure chain, the inode address can be determined like so:
crash> struct file.f_dentry edf3f740
f_dentry = 0xdb0ec440,
crash> struct dentry.d_inode db0ec440
d_inode = 0xf640e740,
crash> struct inode.i_sem f640e740
i_sem = {
count = {
counter = 2
},
sleepers = 0,
wait = {
lock = {
lock = 1
},
task_list = {
next = 0xf640e7ac,
prev = 0xf640e7ac
}
}
},
crash>
|
The dump of the semaphore structure above showed the problem: the counter value of 2 is illegal. It should never be greater than 1; in this case a value of 2 allows two successful down operations, i.e., giving two tasks access to the pipe at the same time.
(As an aside, determining the inode address above could also be accomplished by using the context-sensitive files command, which dumps the associated file, dentry and inode structure addresses for each open file descriptor of a task. The dumped file descriptor list would contain one with a reference to the file structure at edf3f740, and would also show the associated inode address of f640e740.)
Before getting a dumpfile, this same panic had occurred several times. It was erroneously presumed that the problem was in the pipe-handling code, but it was eventually determined not to be the case. By instrumenting a kernel with debug code, the starting counter value of a pipe was found to be 3. Compounding that problem was the fact that the inode slab cache is one of a few special cases that presume that the freed inode's contents are left in a legitimate state so that they do not have to be completely reinitialized with each subsequent reallocation. So when the pipe's inode was created, it received an inode with a bogus counter value.
Confirming the existence of bogus inode structures in the slab cache was a multi-stepped procedure. Using the command kmem command to access the inode slab cache, we can get the addresses of all free and currently-allocated inodes. Since there are typically several thousand inodes, the output is extremely verbose, but here is the beginning of it:
crash> kmem -S inode_cache CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE c7666564 inode_cache 448 11563 12339 1371 4k SLAB MEMORY TOTAL ALLOCATED FREE d1d82000 d1d82040 9 9 0 FREE / [ALLOCATED] [d1d82040] [d1d82200] [d1d823c0] [d1d82580] [d1d82740] [d1d82900] [d1d82ac0] [d1d82c80] [d1d82e40] SLAB MEMORY TOTAL ALLOCATED FREE f4e52000 f4e52040 9 7 2 FREE / [ALLOCATED] f4e52040 (cpu 1 cache) f4e52200 (cpu 1 cache) [f4e523c0] [f4e52580] [f4e52740] [f4e52900] [f4e52ac0] [f4e52c80] [f4e52e40] ... |
In the truncated output above, all of the inode address in the slab cache are dumped; the ones currently in use are surrounded by brackets, the free ones are not. So, for example, the inodes at addresses f4e52040 and f4e52200 are free; the others are not. The full output was piped to a script that pulled out just the free inode addresses (i.e., output lines starting with three spaces), and redirected them into a file. The file was modified to be a crash input file by making each extracted inode address to be the arguments of the struct command, using its short-cut method that allows the dropping of the struct command name; therefore the input file contained hundreds of crash commands of the form:
inode.i_sem f4e52040 inode.i_sem f4e52200 inode.i_sem f5cdc040 inode.i_sem f5cdc200 inode.i_sem f5cdc3c0 inode.i_sem f5cdc580 ... |
Note that the struct command would be used by default above, as documented in its help page; if the first command line argument is not a crash or gdb command, but it is the name of a known data structure, it passes the arguments to the struct command.
Using the capability of feeding an input file, in this case consisting of hundreds of short-cut struct commands like those above, the output was again quite verbose, consisting of structure member dumps of the form:
crash> < input.file
crash> inode.i_sem f4e52040
i_sem = {
count = {
counter = 1
},
sleepers = 0,
wait = {
lock = {
lock = 1
},
task_list = {
next = 0xf4e520ac,
prev = 0xf4e520ac
}
}
},
crash> inode.i_sem f4e52200
i_sem = {
count = {
counter = 1
},
sleepers = 0,
wait = {
lock = {
lock = 1
},
task_list = {
next = 0xf4e5226c,
prev = 0xf4e5226c
}
}
},
...
|
However, it was a simple matter of piping the output to grep, and looking for counter values not equal to 1:
crash> < input.file | grep counter | grep -v "= 1"
counter = 3
counter = 3
counter = 3
counter = 3
crash>
|
This turned out to be the smoking gun. Another round of debugging with an instrumented kernel that trapped attempts to free an inode with a semaphore counter of 3 caught the perpetrator in the act.
Command Extensions
Frequently users wish to add an additional option to an existing crash command, or add a brand new command, in order to address a kernel issue they are debugging or developing. For those reasons, the crash utility was designed with extensibility in mind. There are two ways to add new functionality:
- adding new code and compiling it into the crash executable,
- creating a shared object library that can be dynamically loaded by using the extend command.
This section consists of a quick guide that describes how to get started using both methods.
Adding new code and compiling it into the crash executable
The current set of crash commands can be seen by entering the help command with no arguments:
crash> help * files mod runq union alias foreach mount search vm ascii fuser net set vtop bt gdb p sig waitq btop help ps struct whatis dev irq pte swap wr dis kmem ptob sym q eval list ptov sys exit log rd task extend mach repeat timer crash version: 4.0-8.11 gdb version: 6.1 For help on any command above, enter "help". For help on input options, enter "help input". For help on output options, enter "help output". crash> |
For each command in the menu above, there is an entry in a data structure in the file global_data.c, which is located in the top-level directory of the crash source code tree:
/*
* To add a new command, declare it in defs.h and enter it in this table.
*/
struct command_table_entry base_command_table[] = {
{"*", cmd_pointer, help_pointer, 0},
{"alias", cmd_alias, help_alias, 0},
{"ascii", cmd_ascii, help_ascii, 0},
{"bt", cmd_bt, help_bt, REFRESH_TASK_TABLE},
{"btop", cmd_btop, help_btop, 0},
{"dev", cmd_dev, help_dev, 0},
{"dis", cmd_dis, help_dis, 0},
{"eval", cmd_eval, help_eval, 0},
{"exit", cmd_quit, help_exit, 0},
{"extend", cmd_extend, help_extend, 0},
{"files", cmd_files, help_files, REFRESH_TASK_TABLE},
{"foreach", cmd_foreach, help_foreach, REFRESH_TASK_TABLE},
{"fuser", cmd_fuser, help_fuser, REFRESH_TASK_TABLE},
{"gdb", cmd_gdb, help_gdb, REFRESH_TASK_TABLE},
{"help", cmd_help, help_help, 0},
{"irq", cmd_irq, help_irq, 0},
{"kmem", cmd_kmem, help_kmem, 0},
{"list", cmd_list, help__list, REFRESH_TASK_TABLE},
{"log", cmd_log, help_log, 0},
{"mach", cmd_mach, help_mach, 0},
{"mod", cmd_mod, help_mod, 0},
{"mount", cmd_mount, help_mount, 0},
{"net", cmd_net, help_net, REFRESH_TASK_TABLE},
{"p", cmd_p, help_p, 0},
{"ps", cmd_ps, help_ps, REFRESH_TASK_TABLE},
{"pte", cmd_pte, help_pte, 0},
{"ptob", cmd_ptob, help_ptob, 0},
{"ptov", cmd_ptov, help_ptov, 0},
{"q", cmd_quit, help_quit, 0},
{"rd", cmd_rd, help_rd, 0},
{"repeat", cmd_repeat, help_repeat, 0},
{"runq", cmd_runq, help_runq, REFRESH_TASK_TABLE},
{"search", cmd_search, help_search, 0},
{"set", cmd_set, help_set, REFRESH_TASK_TABLE},
{"sig", cmd_sig, help_sig, REFRESH_TASK_TABLE},
{"struct", cmd_struct, help_struct, 0},
{"swap", cmd_swap, help_swap, 0},
{"sym", cmd_sym, help_sym, 0},
{"sys", cmd_sys, help_sys, REFRESH_TASK_TABLE},
{"task", cmd_task, help_task, REFRESH_TASK_TABLE},
{"test", cmd_test, NULL, HIDDEN_COMMAND},
{"timer", cmd_timer, help_timer, 0},
{"union", cmd_union, help_union, 0},
{"vm", cmd_vm, help_vm, REFRESH_TASK_TABLE},
{"vtop", cmd_vtop, help_vtop, REFRESH_TASK_TABLE},
{"waitq", cmd_waitq, help_waitq, REFRESH_TASK_TABLE},
{"whatis", cmd_whatis, help_whatis, 0},
{"wr", cmd_wr, help_wr, 0},
{(char *)NULL}
};
|
Each entry consists of the following simple data structure:
struct command_table_entry { /* one for each command in menu */
char *name;
cmd_func_t func;
char **help_data;
ulong flags;
};
|
The structure members consist of:
name The character string that appears in the help menu output. It must be unique among other crash commands, and preferably should not clash with any gdb command. func A pointer to the function that performs the command. To add an option to an existing command, find the file that contains this function, and modify it as desired. help_data A pointer to an array of character strings that make up the help data for that command. Although optional, it's certainly useful to create help data for any new commands or options. flags If REFRESH_TASK_TABLE, the set of running tasks on a live system will be updated just prior to executing the command.
If HIDDEN_COMMAND, the command will not be shown in the help menu. The only command with this flag is the test command, whose function cmd_test() exists solely as an builtin aid for quickly developing new or temporary commands.
For a newly-written command to appear in the help menu, it simply requires a reference to it in the structure above. To test a new command without adding it to the menu, use the hidden "test" command, found in test.c:
void
cmd_test(void)
{
int c;
while ((c = getopt(argcnt, args, "")) != EOF) {
switch(c)
{
default:
argerrs++;
break;
}
}
if (argerrs)
cmd_usage(pc->curcmd, SYNOPSIS);
while (args[optind]) {
;
optind++;
}
}
|
The test command contains the basic template used by crash commands to accept dash-arguments, which are fielded by the getopt() routine, while all other command line arguments are fielded in the while loop. To add an option to the test command (or any other existing command), simply fit it into the appropriate argument-gathering mechanism. Or, for that matter, if no arguments are required, put the functionality at the end of the command's function. Here is a trivial example of a change to the cmd_test() function, with the modifications highlighted:
void
cmd_test(void)
{
int c;
while ((c = getopt(argcnt, args, "xa:")) != EOF) {
switch(c)
{
case 'x':
fprintf(fp, "arg: -x\n");
break;
case 'a':
fprintf(fp, "arg: -a %s\n", optarg);
break;
default:
argerrs++;
break;
}
}
if (argerrs)
cmd_usage(pc->curcmd, SYNOPSIS);
while (args[optind]) {
fprintf(fp, "arg: %s\n", args[optind]);
;
optind++;
}
fprintf(fp, "do test work here...\n");
}
|
Then re-compile crash by entering make:
# make
TARGET: X86
CRASH: 4.0-8.11
GDB: 6.1
cc -c -g -DX86 -D_FILE_OFFSET_BITS=64 build_data.c
cc -c -g -DX86 -D_FILE_OFFSET_BITS=64 test.c
ar -rs crashlib.a main.o tools.o global_data.o memory.o filesys.o help.o task.o
build_data.o kernel.o test.o gdb_interface.o net.o dev.o alpha.o x86.o ppc.o ia6
4.o s390.o s390x.o s390dbf.o ppc64.o x86_64.o extensions.o remote.o va_server.o
va_server_v1.o symbols.o cmdline.o lkcd_common.o lkcd_v1.o lkcd_v2_v3.o lkcd_v5.
o lkcd_v7.o lkcd_v8.o lkcd_fix_mem.o s390_dump.o netdump.o diskdump.o xendump.o
lkcd_x86_trace.o unwind_v1.o unwind_v2.o unwind_v3.o unwind_x86_32_64.o xen_hype
r.o xen_hyper_command.o xen_hyper_global_data.o xen_hyper_dump_tables.o
gcc -g -O2 \
-o `cat mergeobj` libgdb.a \
../bfd/libbfd.a ../readline/libreadline.a ../opcodes/libopcod
es.a ../libiberty/libiberty.a -lm -lncurses ../libiberty/libiberty.a -ldl
-rdynamic `cat mergelibs`
#
|
Then run the test command with the tree possible argument types:
crash> test -x -a dasharg this that the other arg: -x arg: -a dasharg arg: this arg: that arg: the arg: other do test work here... crash> |
The easiest way to implement a new command's functionality is to use an existing command as a template. There is a broad range of utility routines that handle argument strings, read memory, access data structures and their members, display data, and so on, that obviate the need to reinvent the wheel to accomplish a task. Look at what other similar commands do, and copy their mechanisms.
Creating a shared object library and loading it with extend
While adding a new command and/or command option in the manner above is useful, it would require the same integration mechanism with each subsequent release of the crash utility. Since that could become tedious, another extension mechanism exists in which share objects containing one or more crash commands can be written, and then dynamically attached to a running crash session, using the extend command. Once loaded, the command(s) in the shared object library automatically appear in the help menu, as if they were compiled into the crash executable. As an quick aid in creating a shared object, the help page for the extend contains an example C program tagged onto the end, which adds a new echo command (which simply echoes back all arguments). The C program piece can be cut and pasted into a file, say echo.c for example, and then compiled like so:
# gcc -nostartfiles -shared -rdynamic -o echo.so echo.c -fPIC -D<machine-type> $(TARGET_CFLAGS) |
where <machine-type> is the appropriate architecture X86, X86_64, IA64, PPC64, S390 or S390X), and where $(TARGET_CFLAGS) is -D_FILE_OFFSET_BITS=64 on 32-bit architectures, and -m64 on ppc64. So for an x86 build, the compile line would be:
# gcc -nostartfiles -shared -rdynamic -o echo.so echo.c -fPIC -DX86 -D_FILE_OFFSET_BITS=64 |
The resultant echo.so file may be dynamically linked into crash during runtime using the extend command:
crash> extend echo.so ./echo.so: shared object loaded crash> |
Or, to automatically load the shared library during crash session initialization, put the following string into a .crashrc file located in the current directory, or in the user's $HOME directory:
extend echo.so |
Here is the help menu once the library is loaded; note the integration of the new echo command:
crash> help * extend mach repeat timer alias files mod runq union ascii foreach mount search vm bt fuser net set vtop btop gdb p sig waitq dev help ps struct whatis dis irq pte swap wr echo kmem ptob sym q eval list ptov sys exit log rd task crash version: 4.0-8.11 gdb version: 6.1 For help on any command above, enter "help". For help on input options, enter "help input". For help on output options, enter "help output". |
With this extension mechanism, the most that would be required to use the shared library with subsequent versions of crash would be a simple re-compile of the echo.c file.
扫一扫
1300
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
