debug vmcore on CentOS 7.2

# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
# uname -a

Linux dev-r630-04 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

 

安装下面两个rpm包

-rw-r--r--. 1 root root 277M Dec 22  2015 kernel-debuginfo-3.10.0-327.el7.x86_64.rpm

-rw-r--r--. 1 root root  45M Dec 22  2015 kernel-debuginfo-common-x86_64-3.10.0-327.el7.x86_64.rpm

 

centos 7.5
http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-common-x86_64-3.10.0-862.el7.x86_64.rpm
http://debuginfo.centos.org/7/x86_64/kernel-debuginfo-3.10.0-862.el7.x86_64.rpm

 

下载地址：

http://debuginfo.centos.org/7/x86_64/

安装crash

# yum install crash

用下面的命令trigger一个panic：

# echo 1 > /proc/sys/kernel/sysrq;

# echo c > /proc/sysrq-trigger

重启后，用下面的命令分析：

# cd /var/crash

# ln -s 127.0.0.1-2017-10-22-22\:52\:31/vmcore vmcore.0

# /bin/crash /var/crash/vmcore.0 /usr/lib/debug/lib/modules/3.10.0-327.el7.x86_64/vmlinux

crash 7.1.9-2.el7
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.


GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...


      KERNEL: /usr/lib/debug/lib/modules/3.10.0-327.el7.x86_64/vmlinux
    DUMPFILE: /var/crash/vmcore.0  [PARTIAL DUMP]
        CPUS: 8
        DATE: Wed Oct 25 01:37:54 2017
      UPTIME: 18:36:56
LOAD AVERAGE: 0.10, 0.07, 0.06
       TASKS: 678
    NODENAME: dev-r630-04
     RELEASE: 3.10.0-327.el7.x86_64
     VERSION: #1 SMP Thu Nov 19 22:10:57 UTC 2015
     MACHINE: x86_64  (2599 Mhz)
      MEMORY: 63.8 GB
       PANIC: "SysRq : Trigger a crash"
         PID: 3203
     COMMAND: "bash"
        TASK: ffff88084ec3dc00  [THREAD_INFO: ffff880079e18000]
         CPU: 4
       STATE: TASK_RUNNING (SYSRQ)


crash> net
   NET_DEVICE     NAME   IP ADDRESS(ES)
ffff880851a8c000  lo     127.0.0.1
ffff88084f7a4000  eno1   10.12.205.14
ffff880851b54000  eno2
ffff880850a9a000  eno3
ffff880850a98000  eno4
ffff88104fa3e000  virbr0 192.168.122.1
ffff881043448000  virbr0-nic

 

debug live kernel，直接运行crash就行

# crash

crash 7.1.9++
Copyright (C) 2002-2017  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.


GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...


      KERNEL: /usr/lib/debug/lib/modules/3.10.0-327.el7.x86_64/vmlinux
    DUMPFILE: /dev/crash
        CPUS: 8
        DATE: Wed Oct 25 01:50:52 2017
      UPTIME: 00:11:20
LOAD AVERAGE: 0.04, 0.04, 0.05
       TASKS: 658
    NODENAME: dev-r630-04
     RELEASE: 3.10.0-327.el7.x86_64
     VERSION: #1 SMP Thu Nov 19 22:10:57 UTC 2015
     MACHINE: x86_64  (2600 Mhz)
      MEMORY: 63.8 GB
         PID: 3308
     COMMAND: "crash"
        TASK: ffff88104fde9700  [THREAD_INFO: ffff88104f318000]
         CPU: 5
       STATE: TASK_RUNNING (ACTIVE)


crash>

 

如果要源码编译crash的话，加上lzo support

# make lzo