How do I collect performance data with perf which is readable on another machine?

How do I collect performance data with perf which is readable on another machine?

 

环境

• Red Hat Enterprise Linux 8
• Red Hat Enterprise Linux 7
• Red Hat Enterprise Linux 6

问题

• How do I collect performance data with perf which is readable on another machine?
• System performance is slow, how can I identify where user processes and the kernel are spending CPU time?

决议

Install relevant packages

• Add the debuginfo channel to your system. (needed for installing the kernel-debuginfo package)

• Install perf and kernel debugging symbols for the version you are running:

  Raw

  yum install bzip2 perf kernel-debuginfo-`uname -r`
  

  Note: The command "perf archive" described later requires the bzip2 binary. In some versions of the perf package, the bzip2 binary is not a dependency and running perf archive will fail if it is not installed.

Collect data

1. (Optional) Create a custom directory to capture perf data:

   Raw

   mkdir -p /tmp/perf; cd $_
   

2. Capture performance data with perf record command:

   • To capture data of a specific command, use:

     Raw

     perf record -g -- <command>
     

     This will capture performance data for the whole execution of the specified command (including all potential child processes) or until interrupted.

   • To capture global data, use:

     Raw

     perf record -a -g -- sleep X
     

     Where X is number of seconds, after which the recording should stop. Recommended: 10-60 seconds.
     Global data collection can produce very large amount of data. Therefore it is not recommended to run global recording for long periods of time.

   The perf record command will create a perf.data file in the current active directory.

   Note: You can also run perf record without specifying any command. This way, however, the recording runs indefinitely and needs to be manually stopped, for example by interrupting it with Ctrl+C.

   For more information see man perf-record.

3. Generate archive containing debug symbols for the recorded data:

   Raw

   perf archive
   

   This will pack all relevant debug symbols for the latest recorded data. These debug symbols are required for the data to be interpretable on another (different) machine. Without them the data would only show address pointers (hexadecimal numbers) of the recorded functions.

   The perf archive command will create a perf.data.tar.gz archive file in the current active directory.

   Note: The perf archive command does not archive also the actual data, although the name might suggest such.

   For more information see man perf-archive.

Upload data for analysis

• At this point you should have 2 files generated in the current active directory:

  • perf.data - these are the actual recorded data;
  • perf.data.tar.gz - this archive contains debug symbols needed to interpret the data.

• Important: In order to analyse the recorded data on another (different) machine, both these files need to be uploaded!

根源

perf is a Performance analysis tool for Linux. The man page explains the tool:

Raw

Performance counters for Linux are a new kernel-based subsystem that
provide a framework for all things performance analysis. It covers 
hardware level (CPU/PMU), Performance Monitoring Unit) features and
software features (software counters, tracepoints) as well.