Linux Debugging 2 - System Profiling

1. The /proc filesystem

一个表示进程的虚拟文件系统，每个进程一个目录，名字是进程号，里面存有关于一个进程的各种信息，可以监管一个进程，依据map和fd信息做出诊断。同时/proc也可用于控制和诊断核心以及硬件。

1) kernel and hardware 诊断信息

refer : http://blog.chinaunix.net/u/31/showart_602243.html

例如： APM(advanced power management), execdomains(registered execution domains), fb(frame buffer information).

2) kernel tunable parameters /proc/sys 例如，

/home/a/j/nomad2:pwd /proc/sys/net/ipv4 /home/a/j/nomad2:ls conf neigh tcp_mem icmp_echo_ignore_all route tcp_moderate_rcvbuf icmp_echo_ignore_broadcasts tcp_abc tcp_mtu_probing icmp_errors_use_inbound_ifaddr tcp_abort_on_overflow tcp_no_metrics_save icmp_ignore_bogus_error_responses tcp_adv_win_scale tcp_orphan_retries icmp_ratelimit tcp_allowed_congestion_control tcp_reordering icmp_ratemask tcp_app_win tcp_retrans_collapse igmp_max_memberships tcp_available_congestion_control tcp_retries1 igmp_max_msf tcp_base_mss tcp_retries2 inet_peer_gc_maxtime tcp_congestion_control tcp_rfc1337 inet_peer_gc_mintime tcp_dma_copybreak tcp_rmem inet_peer_maxttl tcp_dsack tcp_sack inet_peer_minttl tcp_ecn tcp_slow_start_after_idle inet_peer_threshold tcp_fack tcp_stdurg ip_default_ttl tcp_fin_timeout tcp_syn_retries ip_dynaddr tcp_frto tcp_synack_retries ip_forward tcp_frto_response tcp_syncookies ip_local_port_range tcp_keepalive_intvl tcp_timestamps ip_no_pmtu_disc tcp_keepalive_probes tcp_tso_win_divisor ip_nonlocal_bind tcp_keepalive_time tcp_tw_recycle ipfrag_high_thresh tcp_low_latency tcp_tw_reuse ipfrag_low_thresh tcp_max_orphans tcp_window_scaling ipfrag_max_dist tcp_max_ssthresh tcp_wmem ipfrag_secret_interval tcp_max_syn_backlog tcp_workaround_signed_windows ipfrag_time tcp_max_tw_buckets /home/a/j/nomad2:echo 1 >ip_forward /home/a/j/nomad2:sysctl -w net.ipv4.ip_forward=1

3) 进程诊断信息

/home/a/j/nomad2:pwd /proc/22034 /home/a/j/nomad2:ls -F attr/ clear_refs cpuset exe@ io maps mountstats oom_score smaps status auxv cmdline cwd@ fd/ limits mem numa_maps root@ stat task/ cgroup coredump_filter environ fdinfo/ loginuid mounts oom_adj sched statm wchan

2. 中断

the following table shows the IRQ#, number of interrupts, responsible PIC, and registered handler.

/home/a/j/nomad2:pwd /proc /home/a/j/nomad2:cat interrupts CPU0 CPU1 0: 342 0 IO-APIC-edge timer 1: 2 0 IO-APIC-edge i8042 4: 2 0 IO-APIC-edge 8: 1 0 IO-APIC-edge rtc 9: 0 0 IO-APIC-fasteoi acpi 12: 4 0 IO-APIC-edge i8042 14: 459271 0 IO-APIC-edge libata 15: 0 0 IO-APIC-edge libata 16: 7635628 0 IO-APIC-fasteoi eth0 20: 338070 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb4, libata 21: 0 0 IO-APIC-fasteoi uhci_hcd:usb2 22: 0 0 IO-APIC-fasteoi uhci_hcd:usb3 NMI: 0 0 Non-maskable interrupts LOC: 8867982 8281590 Local timer interrupts RES: 182963 469095 Rescheduling interrupts CAL: 242 90 function call interrupts TLB: 7039 16037 TLB shootdowns TRM: 0 0 Thermal event interrupts THR: 0 0 Threshold APIC interrupts SPU: 0 0 Spurious interrupts ERR: 0

/proc/stat的内容如下，

/home/a/j/nomad2:cat /proc/stat cpu 3987145 0 1306724 6475556 8213 4534 21324 0 0 cpu0 1991828 0 660103 3214978 7115 4534 12830 0 0 cpu1 1995316 0 646621 3260578 1098 0 8494 0 0 intr 8452956 343 2 0 0 2 0 0 0 1 0 0 0 4 0 461229 0 7652206 0 0 0 339169 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ctxt 23576136 btime 1291929146 processes 231620 procs_running 3 procs_blocked 0

interrupts are per-process, and in SMP, multiple interrupts can be handled concurrently by the different CPUs.

3. /proc/kallsyms : all expoted module symbols from the kernel, and module name is in []

/home/a/j/nomad2:head /proc/kallsyms ffffffff80200000 A _text ffffffff80200000 T startup_64 ffffffff802000b7 t ident_complete ffffffff80200110 T secondary_startup_64 ffffffff802001a8 T initial_code ffffffff802001b0 T init_rsp ffffffff802001b8 t bad_address ffffffff802001c0 T early_idt_handler ffffffff8020020b t early_recursion_flag ffffffff8020020f t early_idt_msg

the symbols are entry points, which are used primarily by insmod when modules are inserted, and another use of ksyms is to map kernel structures when peeking into the kernel memory, by using /dev/kmem.

4. Dmesg (print or control the kernel ring buffer)

kernel messages may be rerouted to user mode via klogd.

/home/a/j/nomad2:dmesg |more [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Linux version 2.6.24-22-generic (buildd@crested) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #1 SMP Mon Nov 24 19:35:06 UTC 2008 (Ubuntu 2.6.24-22.45-generic) [ 0.000000] Command line: root=UUID=e021d2fd-f6b7-40fa-b6b3-3d086aa71a80 ro quiet splash [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: 0000000000000000 - 00000000000a0000 (usable) [ 0.000000] BIOS-e820: 0000000000100000 - 00000000dffc0000 (usable) [ 0.000000] BIOS-e820: 00000000dffc0000 - 00000000dffcfc00 (ACPI data) [ 0.000000] BIOS-e820: 00000000dffcfc00 - 00000000dffff000 (reserved) [ 0.000000] BIOS-e820: 00000000f0000000 - 00000000f4000000 (reserved) [ 0.000000] BIOS-e820: 00000000fec00000 - 00000000fed00400 (reserved) [ 0.000000] BIOS-e820: 00000000fed13000 - 00000000feda0000 (reserved) [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved) [ 0.000000] BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved) [ 0.000000] BIOS-e820: 0000000100000000 - 00000001ffffe000 (usable) [ 0.000000] BIOS-e820: 00000001ffffe000 - 0000000200000000 (reserved) [ 0.000000] BIOS-e820: 0000000200000000 - 0000000220000000 (usable) [ 0.000000] Entering add_active_range(0, 0, 160) 0 entries of 3200 used [ 0.000000] Entering add_active_range(0, 256, 917440) 1 entries of 3200 used [ 0.000000] Entering add_active_range(0, 1048576, 2097150) 2 entries of 3200 used [ 0.000000] Entering add_active_range(0, 2097152, 2228224) 3 entries of 3200 used

in /var/log,

/home/a/j/nomad2:pwd /var/log /home/a/j/nomad2:ls Bastille daemon.log.2.gz dpkg.log.12.gz kern.log.3.gz mysql.log.2.gz syslog.6.gz account daemon.log.3.gz dpkg.log.2.gz lastlog mysql.log.3.gz sysstat apache2 daemon.log.4.gz dpkg.log.3.gz lastlog.1 mysql.log.4.gz udev apparmor debug dpkg.log.4.gz lpr.log mysql.log.5.gz user.log apt debug.0 dpkg.log.5.gz mail.err mysql.log.6.gz user.log.0 auth.log debug.1.gz dpkg.log.6.gz mail.info mysql.log.7.gz user.log.1.gz auth.log.0 debug.2.gz dpkg.log.7.gz mail.log news user.log.2.gz auth.log.1.gz debug.3.gz dpkg.log.8.gz mail.warn pycentral.log user.log.3.gz auth.log.2.gz dist-upgrade dpkg.log.9.gz messages scrollkeeper.log user.log.4.gz auth.log.3.gz dmesg faillog messages.0 scrollkeeper.log.1 uucp.log auth.log.4.gz dmesg.0 fontconfig.log messages.1.gz scrollkeeper.log.2 uucp.log.0 auth.log.5.gz dmesg.1.gz fsck messages.2.gz sulog vsftpd.log auth.log.6.gz dmesg.2.gz gdm messages.3.gz syslog vsftpd.log.1 boot dmesg.3.gz installer messages.4.gz syslog.0 vsftpd.log.2 btmp dmesg.4.gz iptraf messages.5.gz syslog.1.gz vsftpd.log.3 btmp.1 dpkg.log kern.log mysql syslog.2.gz vsftpd.log.4 daemon.log dpkg.log.1 kern.log.0 mysql.err syslog.3.gz wtmp daemon.log.0 dpkg.log.10.gz kern.log.1.gz mysql.log syslog.4.gz wtmp.1 daemon.log.1.gz dpkg.log.11.gz kern.log.2.gz mysql.log.1.gz syslog.5.gz wtmp.report

5. Devices

/sys for devices/modules just like /proc for processes.

/home/a/j/nomad2:pwd /sys/module /home/a/j/nomad2:ls -F 8250/ battery/ fan/ jbd/ nfs_acl/ pty/ sr_mod/ ac/ bitblit/ fbcon/ keyboard/ parport/ rcupdate/ sunrpc/ acpi/ button/ font/ libata/ parport_pc/ rd/ tg3/ aerdriver/ capability/ fuse/ lockd/ pata_acpi/ scsi_mod/ thermal/ amd64_agp/ cdrom/ i8042/ lp/ pci_hotplug/ sd_mod/ tileblit/ apparmor/ dcdbas/ iTCO_vendor_support/ mbcache/ pcspkr/ serio_raw/ uhci_hcd/ ata_generic/ ehci_hcd/ iTCO_wdt/ mousedev/ printk/ sg/ usbcore/ ata_piix/ evdev/ ip_tables/ mtrr/ processor/ shpchp/ vt/ atkbd/ ext3/ iptable_filter/ nfs/ psmouse/ softcursor/ x_tables/

6. 2 kind processes

migration(runqueue -> idle) and watchdog(make sure cpu active)