linux kernel ftrace 之wakeup tracer and wakeup_rt tracer

Linux内核调试利器:ftrace原理与应用详解 函数追踪是Linux内核调试的核心技术之一,通过在运行时动态修改代码实现低开销监控。ftrace作为内核原生追踪框架,采用nop指令替换和环形缓冲区等机制,既保证了性能又提供了丰富的调试信息。在系统性能优化、内核问题定位等场景中,ftrace能够精确追踪函数调用关系、执行耗时等关键指标。特别是结合动态过滤器和事件触发功能,开发者可以针对特定进程或代码路径进行深度分析。本文以调度延迟和内存分配问题为例,展示了如何利用function_graph追踪器和wakeup_rt追踪器解决实际问题,同时对比了ftrac 阅读详情

对于 ftrace 的function tracer/function graph tracer 用的比较多,本文介绍 wakeup tracer/wakeup_rt tracer。

开始介绍之前,先扯两句别的,在一开始想了解wakeup/wakeup_rt tracer 的时候,因为内核Documents 中有ftrace的文档介绍,
直接在工程中的ftrace.txt 看了一下wakeup/wakeup_rt tracer 的介绍,当场看糊涂了,写的完全驴唇不对马嘴,工程中的kernel 是3.4的,
当时就怀疑Documents 应该是没跟上代码更新,看了一下linus torvals tree, 果然更新了。建议直接看最新版本的介绍:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/trace/ftrace.txt


1.wakeup/wakeup_rt tracer 是什么?

下面有详细的英文介绍,简单来说,wakeup/wakeup_rt tracer 是ftrace 的一个tracing utilities,是两种tracer.
区别在于wakeup tracer 是Trace 所有的进程中(实时/非实时进程)的schedule latency, 而wakeup_rt 只Trace 实时进程的最大schedule latency.


  "wakeup"
Traces and records the max latency that it takes for
the highest priority task to get scheduled after
it has been woken up.
        Traces all tasks as an average developer would expect.


  "wakeup_rt"
        Traces and records the max latency that it takes for just
        RT tasks (as the current "wakeup" does). This is useful
        for those interested in wake up timings of RT tasks.


2. wakeup tracer How-to



$ adb shell cat /sys/kernel/debug/tracing/available_tracers
function_graph wakeup_rt wakeup preemptirqsoff preemptoff irqsoff function nop


$ adb shell "echo wakeup > /sys/kernel/debug/tracing/current_tracer"
$ adb shell "echo 0 > /sys/kernel/debug/tracing/tracing_max_latency"
$ adb shell "echo 1 > /sys/kernel/debug/tracing/tracing_on"
$ adb shell cat /sys/kernel/debug/tracing/trace




# tracer: wakeup
#
# wakeup latency trace v1.1.5 on 3.4.0-gd26777b-00428-gd66afe1-dirty
# --------------------------------------------------------------------
# latency: 21865 us, #11408/11408, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:2)//wakeup latency 是21865us
#    -----------------
#    | task: PowerManagerSer-943 (uid:1000 nice:-4 policy:0 rt_prio:0)//trace 的进程是PowerManagerSer-943
#    -----------------
#//下面是详细的trace,向你解释为什么PowerManagerSer-943 进程从wake up 到真正scheduled in 得以执行用了21865us.
 //第一列是执行wakeup PowerManagerSer-943 进程的时候正在运行的进程,后面是运行于哪个CPU,中断开关情况,need-resched flag 的情况,
 //中断服务程序中,还是软中断中,抢占深度
 //当你看到第一列中的进程优先级并不高时,PowerManagerSer-943 为啥得不到执行,原因无非就是中断被关了,无法调度,或是抢占深度count 置位,无法
 //抢占。
#                  _------=> CPU#            
#                 / _-----=> irqs-off        
#                | / _----=> need-resched    
#                || / _---=> hardirq/softirq 
#                ||| / _--=> preempt-depth   
#                |||| /     delay             
#  cmd     pid   ||||| time  |   caller      
#     \   /      |||||  \    |   /           
   <...>-1108    0d..5    6us+:   1108:120:R   + [000]   943:116:R PowerManagerSer
   <...>-1108    0d..5    9us+: 0
   <...>-1108    0d..4   11us : check_preempt_curr <-ttwu_do_wakeup
   <...>-1108    0d..4   12us : check_preempt_wakeup <-check_preempt_curr
   <...>-1108    0d..4   13us : update_curr <-check_preempt_wakeup
   <...>-1108    0d..4   15us : wakeup_preempt_entity <-check_preempt_wakeup
   <...>-1108    0d..4   16us : _raw_spin_unlock <-try_to_wake_up
   <...>-1108    0d..4   17us+: sub_preempt_count <-_raw_spin_unlock
   <...>-1108    0d..3   19us+: ttwu_stat <-try_to_wake_up
...


kworker/-31782   0d..2 21843us+: update_stats_wait_end <-set_next_entity
kworker/-31782   0d..2 21849us : _raw_spin_unlock_irq <-__schedule
kworker/-31782   0...2 21850us+: sub_preempt_count <-_raw_spin_unlock_irq
kworker/-31782   0...2 21852us : __schedule
kworker/-31782   0...2 21853us :  31782:120:S ==> [000]   943:116:R PowerManagerSer


没一列的详述:
  cmd: The name of the process in the trace.


  pid: The PID of that process.


  CPU#: The CPU which the process was running on.


  irqs-off: 'd' interrupts are disabled. '.' otherwise.
   Note: If the architecture does not support a way to
 read the irq flags variable, an 'X' will always
 be printed here.


  need-resched:
'N' both TIF_NEED_RESCHED and PREEMPT_NEED_RESCHED is set,
'n' only TIF_NEED_RESCHED is set,
'p' only PREEMPT_NEED_RESCHED is set,
'.' otherwise.


  hardirq/softirq:
'H' - hard irq occurred inside a softirq.
'h' - hard irq is running
's' - soft irq is running
'.' - normal context.


  preempt-depth: The level of preempt_disabled


The above is mostly meaningful for kernel developers.


  time: When the latency-format option is enabled, the trace file
output includes a timestamp relative to the start of the
trace. This differs from the output when latency-format
is disabled, which includes an absolute timestamp.


  delay: This is just to help catch your eye a bit better. And
needs to be fixed to be only relative to the same CPU.
The marks are determined by the difference between this
current trace and the next trace.
 '$' - greater than 1 second
 '#' - greater than 1000 microsecond
 '!' - greater than 100 microsecond
 '+' - greater than 10 microsecond
 ' ' - less than or equal to 10 microsecond.


  The rest is the same as the 'trace' file.


  Note, the latency tracers will usually end with a back trace
  to easily find where the latency occurred.


3.wakeup_rt tracer How-to



$ adb shell cat /sys/kernel/debug/tracing/available_tracers
function_graph wakeup_rt wakeup preemptirqsoff preemptoff irqsoff function nop


$ adb shell "echo wakeup_rt > /sys/kernel/debug/tracing/current_tracer"
$ adb shell "echo 0 > /sys/kernel/debug/tracing/tracing_max_latency"
$ adb shell "echo 1 > /sys/kernel/debug/tracing/tracing_on"
$ adb shell cat /sys/kernel/debug/tracing/trace


# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.4.0-gd26777b-00428-gd66afe1-dirty
# --------------------------------------------------------------------
# latency: 1973 us, #967/967, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:1)
#    -----------------
#    | task: mdss_fb0-319 (uid:0 nice:0 policy:1 rt_prio:16)
#    -----------------
#
#                  _------=> CPU#            
#                 / _-----=> irqs-off        
#                | / _----=> need-resched    
#                || / _---=> hardirq/softirq 
#                ||| / _--=> preempt-depth   
#                |||| /     delay             
#  cmd     pid   ||||| time  |   caller      
#     \   /      |||||  \    |   /           
surfacef-253     2d..4    2us+:    253:112:R   + [000]   319: 83:R mdss_fb0
surfacef-253     2d..4    8us+: 0
RenderTh-2982    0dN.3   16us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   19us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   21us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   23us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   25us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   27us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   29us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   31us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   33us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   34us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   36us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   38us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   40us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   42us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   44us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   45us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   47us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   49us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   51us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   53us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   55us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   57us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   59us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   60us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   63us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   64us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   66us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   68us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   70us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   71us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   73us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   76us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   77us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   79us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   81us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   83us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   84us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   86us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   88us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   90us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   92us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   95us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   96us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   98us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  100us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  102us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  103us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  106us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  108us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  110us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  112us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  114us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  116us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  117us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  119us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  121us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  123us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  124us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  127us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  129us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  131us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  133us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  135us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  136us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  138us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  140us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  141us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  143us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  145us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  147us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  149us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  150us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  152us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  154us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  156us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  158us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  160us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  162us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  164us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  165us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  167us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  169us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  171us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  173us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  176us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  178us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  180us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  182us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  183us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  185us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  187us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  189us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  192us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  193us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  196us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  197us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  199us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  201us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  203us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  205us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  208us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  209us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  211us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  213us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  215us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  217us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  218us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  220us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  224us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  226us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  228us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  229us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  231us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  233us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  235us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  237us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  239us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  241us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  243us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  244us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  246us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  248us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  250us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  252us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  253us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  258us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  259us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  261us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  263us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  265us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  267us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  269us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  271us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  273us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  275us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  276us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  278us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  280us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  282us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  284us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  286us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  289us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  290us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  292us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  294us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  296us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  298us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  299us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  301us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  305us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  307us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  308us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  310us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  312us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  314us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  316us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  317us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  321us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  323us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  325us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  327us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  329us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  331us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  332us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  335us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  337us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  339us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  341us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  343us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  344us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  346us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  348us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  350us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  353us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  355us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  357us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  359us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  360us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  362us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  364us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  366us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  370us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  372us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  374us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  375us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  377us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  380us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  381us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  384us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  386us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  388us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  393us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  395us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  397us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  403us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  405us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  407us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  408us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  410us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  412us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  414us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  423us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  425us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  427us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  429us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  431us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  432us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  434us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  436us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  438us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  440us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  442us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  443us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  445us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  447us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  449us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  450us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  452us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  454us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  456us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  458us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  460us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  461us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  463us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  466us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  467us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  469us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  471us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  473us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  475us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  476us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  478us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  481us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  482us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  484us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  486us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  488us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  490us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  492us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  493us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  495us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  497us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  499us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  501us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  503us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  504us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  506us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  508us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  510us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  511us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  515us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  516us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  518us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  520us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  522us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  523us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  525us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  527us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  531us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  532us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  534us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  536us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  538us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  540us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  542us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  543us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  546us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  548us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  550us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  551us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  553us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  555us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  557us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  558us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  562us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  564us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  566us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  567us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  569us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  571us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  573us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  574us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  576us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  578us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  580us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  582us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  583us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  585us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  587us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  589us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  591us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  592us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  594us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  596us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  598us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  601us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  603us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  604us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  607us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  608us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  611us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  613us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  615us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  617us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  618us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  620us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  622us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  624us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  628us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  629us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  631us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  633us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  635us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  637us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  639us+: __rcu_read_unlock <-thread_group_cputime
RenderTh-2982    0dN.1  640us+: nsecs_to_jiffies <-thread_group_times
RenderTh-2982    0dN.1  643us+: nsecs_to_jiffies64 <-nsecs_to_jiffies
RenderTh-2982    0dN.1  646us+: _raw_spin_unlock_irq <-do_sys_times
RenderTh-2982    0dN.1  648us+: gic_handle_irq <-__irq_svc
RenderTh-2982    0dN.1  651us+: handle_IPI <-gic_handle_irq
RenderTh-2982    0dN.1  653us+: scheduler_ipi <-handle_IPI
RenderTh-2982    0dN.1  656us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0dN.1  659us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0dN.1  661us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0dN.1  663us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0dN.1  665us+: irq_enter <-handle_IRQ
RenderTh-2982    0dN.1  667us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0dN.1  669us+: add_preempt_count <-irq_enter
RenderTh-2982    0dNh1  671us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0dNh1  673us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0dNh1  676us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0dNh1  678us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0dNh1  680us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2  682us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0dNh2  684us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0dNh2  685us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1  687us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0dNh1  691us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0dNh1  693us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0dNh1  695us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0dNh1  697us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0dNh1  699us+: adreno_read <-adreno_regread
RenderTh-2982    0dNh1  701us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1  705us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0dNh1  707us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0dNh1  709us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2  711us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2  713us+: __queue_work <-queue_work_on
RenderTh-2982    0dNh2  715us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0dNh2  716us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh3  718us+: get_cwq <-__queue_work
RenderTh-2982    0dNh3  720us+: insert_work <-__queue_work
RenderTh-2982    0dNh3  724us+: get_pool_nr_running <-insert_work
RenderTh-2982    0dNh3  725us+: wake_up_worker <-insert_work
RenderTh-2982    0dNh3  727us+: wake_up_process <-wake_up_worker
RenderTh-2982    0dNh3  729us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0dNh3  731us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0dNh3  733us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh4  735us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0dNh4  737us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0dNh4  739us+: __rcu_read_lock <-select_task_rq_fair
RenderTh-2982    0dNh4  741us+: source_load <-select_task_rq_fair
RenderTh-2982    0dNh4  743us+: target_load <-select_task_rq_fair
RenderTh-2982    0dNh4  745us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  747us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  749us+: cpu_avg_load_per_task <-select_task_rq_fair
RenderTh-2982    0dNh4  751us+: idle_cpu <-select_task_rq_fair
RenderTh-2982    0dNh4  753us+: __rcu_read_unlock <-select_task_rq_fair
RenderTh-2982    0dNh4  755us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0dNh4  757us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0dNh4  758us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh5  760us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0dNh5  762us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  764us+: activate_task <-ttwu_activate
RenderTh-2982    0dNh5  765us+: enqueue_task <-activate_task
RenderTh-2982    0dNh5  767us+: update_rq_clock <-enqueue_task
RenderTh-2982    0dNh5  770us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0dNh5  772us+: update_curr <-enqueue_task_fair
RenderTh-2982    0dNh5  774us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0dNh5  776us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0dNh5  778us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0dNh5  780us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0dNh5  782us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0dNh5  784us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0dNh5  787us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh6  789us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0dNh6  792us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0dNh6  793us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh5  796us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0dNh5  797us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0dNh5  799us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  802us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0dNh5  804us+: resched_task <-check_preempt_curr
RenderTh-2982    0dNh5  806us+: smp_send_reschedule <-resched_task
RenderTh-2982    0dNh5  808us+: gic_raise_softirq <-smp_send_reschedule
RenderTh-2982    0dNh5  812us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0dNh5  813us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh4  815us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0dNh4  817us+: __rcu_read_lock <-ttwu_stat
RenderTh-2982    0dNh4  820us+: __rcu_read_unlock <-ttwu_stat
RenderTh-2982    0dNh4  821us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0dNh4  823us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh3  825us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0dNh3  827us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh2  829us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1  831us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0dNh1  833us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0dNh1  835us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2  837us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2  839us+: __queue_work <-queue_work_on
RenderTh-2982    0dNh2  841us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0dNh2  842us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh3  844us+: get_cwq <-__queue_work
RenderTh-2982    0dNh3  846us+: insert_work <-__queue_work
RenderTh-2982    0dNh3  848us+: get_pool_nr_running <-insert_work
RenderTh-2982    0dNh3  849us+: wake_up_worker <-insert_work
RenderTh-2982    0dNh3  851us+: wake_up_process <-wake_up_worker
RenderTh-2982    0dNh3  853us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0dNh3  854us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0dNh3  856us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh4  858us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0dNh4  860us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0dNh4  862us+: __rcu_read_lock <-select_task_rq_fair
RenderTh-2982    0dNh4  864us+: source_load <-select_task_rq_fair
RenderTh-2982    0dNh4  866us+: target_load <-select_task_rq_fair
RenderTh-2982    0dNh4  868us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  869us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  871us+: cpu_avg_load_per_task <-select_task_rq_fair
RenderTh-2982    0dNh4  874us+: idle_cpu <-select_task_rq_fair
RenderTh-2982    0dNh4  876us+: __rcu_read_unlock <-select_task_rq_fair
RenderTh-2982    0dNh4  878us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0dNh4  880us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0dNh4  881us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh5  883us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0dNh5  885us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  887us+: activate_task <-ttwu_activate
RenderTh-2982    0dNh5  889us+: enqueue_task <-activate_task
RenderTh-2982    0dNh5  891us+: update_rq_clock <-enqueue_task
RenderTh-2982    0dNh5  893us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0dNh5  895us+: update_curr <-enqueue_task_fair
RenderTh-2982    0dNh5  897us+: update_min_vruntime <-update_curr
RenderTh-2982    0dNh5  900us+: cpuacct_charge <-update_curr
RenderTh-2982    0dNh5  902us+: __rcu_read_lock <-cpuacct_charge
RenderTh-2982    0dNh5  903us+: __rcu_read_unlock <-cpuacct_charge
RenderTh-2982    0dNh5  905us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0dNh5  907us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0dNh5  909us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0dNh5  911us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0dNh5  913us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0dNh5  914us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0dNh5  917us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh6  919us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0dNh6  921us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0dNh6  923us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh5  925us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0dNh5  927us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0dNh5  929us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  934us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0dNh5  936us+: check_preempt_wakeup <-check_preempt_curr
RenderTh-2982    0dNh5  938us+: update_curr <-check_preempt_wakeup
RenderTh-2982    0dNh5  940us+: wakeup_preempt_entity <-check_preempt_wakeup
RenderTh-2982    0dNh5  942us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0dNh5  944us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh4  946us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0dNh4  948us+: __rcu_read_lock <-ttwu_stat
RenderTh-2982    0dNh4  951us+: __rcu_read_unlock <-ttwu_stat
RenderTh-2982    0dNh4  953us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0dNh4  955us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh3  957us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0dNh3  959us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh2  961us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1  965us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0dNh1  967us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0dNh1  969us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1  972us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0dNh1  974us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0dNh1  976us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2  982us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0dNh2  984us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0dNh2  986us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1  987us+: irq_exit <-handle_IRQ
RenderTh-2982    0dNh1  989us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.2  991us+: idle_cpu <-irq_exit
RenderTh-2982    0dN.2  993us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0dN.2  995us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.1  997us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0dN.1  999us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0dN.1 1001us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0dN.1 1003us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0dN.1 1004us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0dN.1 1006us+: irq_enter <-handle_IRQ
RenderTh-2982    0dN.1 1008us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0dN.1 1010us+: add_preempt_count <-irq_enter
RenderTh-2982    0dNh1 1012us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0dNh1 1014us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0dNh1 1016us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0dNh1 1018us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0dNh1 1020us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1022us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1023us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0dNh2 1025us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1027us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0dNh1 1030us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1031us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0dNh1 1033us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0dNh1 1035us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0dNh1 1036us+: adreno_read <-adreno_regread
RenderTh-2982    0dNh1 1038us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1042us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0dNh1 1044us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0dNh1 1046us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1048us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1050us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1052us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0dNh1 1053us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0dNh1 1055us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1057us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1058us+: __queue_work <-queue_work_on
RenderTh-2982    0dNh2 1061us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0dNh2 1063us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh3 1064us+: get_cwq <-__queue_work
RenderTh-2982    0dNh3 1066us+: insert_work <-__queue_work
RenderTh-2982    0dNh3 1068us+: get_pool_nr_running <-insert_work
RenderTh-2982    0dNh3 1070us+: wake_up_worker <-insert_work
RenderTh-2982    0dNh3 1072us+: wake_up_process <-wake_up_worker
RenderTh-2982    0dNh3 1073us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0dNh3 1075us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0dNh3 1077us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh4 1079us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0dNh4 1081us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh3 1083us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0dNh3 1084us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh2 1087us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1088us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0dNh1 1090us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0dNh1 1092us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1095us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1097us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0dNh1 1098us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1100us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1102us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1104us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1106us+: irq_exit <-handle_IRQ
RenderTh-2982    0dNh1 1108us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.2 1110us+: idle_cpu <-irq_exit
RenderTh-2982    0dN.2 1112us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0dN.2 1113us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.1 1115us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0dN.1 1118us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0dN.1 1120us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0dN.1 1122us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0dN.1 1124us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0dN.1 1126us+: irq_enter <-handle_IRQ
RenderTh-2982    0dN.1 1127us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0dN.1 1129us+: add_preempt_count <-irq_enter
RenderTh-2982    0dNh1 1132us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0dNh1 1133us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0dNh1 1135us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0dNh1 1137us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0dNh1 1139us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1141us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1143us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0dNh2 1145us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1147us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0dNh1 1149us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1150us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0dNh1 1152us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0dNh1 1154us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0dNh1 1155us+: adreno_read <-adreno_regread
RenderTh-2982    0dNh1 1159us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1163us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0dNh1 1164us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0dNh1 1166us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1168us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1170us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1171us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0dNh1 1173us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0dNh1 1175us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1177us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1179us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1181us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0dNh1 1182us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0dNh1 1184us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1187us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1191us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0dNh1 1193us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1194us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1197us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1198us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1200us+: irq_exit <-handle_IRQ
RenderTh-2982    0dNh1 1202us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.2 1204us+: idle_cpu <-irq_exit
RenderTh-2982    0dN.2 1206us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0dN.2 1208us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.1 1210us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0.N.1 1212us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0.N.. 1214us+: __schedule <-preempt_schedule
RenderTh-2982    0.N.. 1216us+: add_preempt_count <-__schedule
RenderTh-2982    0.N.1 1218us+: rcu_note_context_switch <-__schedule
RenderTh-2982    0.N.1 1220us+: rcu_sched_qs <-rcu_note_context_switch
RenderTh-2982    0dN.1 1222us+: rcu_preempt_qs <-rcu_note_context_switch
RenderTh-2982    0.N.1 1224us+: _raw_spin_lock_irq <-__schedule
RenderTh-2982    0dN.1 1225us+: add_preempt_count <-_raw_spin_lock_irq
RenderTh-2982    0dN.2 1227us+: update_rq_clock <-__schedule
RenderTh-2982    0dN.2 1229us+: put_prev_task_fair <-__schedule
RenderTh-2982    0dN.2 1231us+: update_curr <-put_prev_task_fair
RenderTh-2982    0dN.2 1233us+: calc_delta_mine <-update_curr
RenderTh-2982    0dN.2 1235us+: update_min_vruntime <-update_curr
RenderTh-2982    0dN.2 1237us+: cpuacct_charge <-update_curr
RenderTh-2982    0dN.2 1240us+: __rcu_read_lock <-cpuacct_charge
RenderTh-2982    0dN.2 1242us+: __rcu_read_unlock <-cpuacct_charge
RenderTh-2982    0dN.2 1244us+: __enqueue_entity <-put_prev_task_fair
RenderTh-2982    0dN.2 1246us+: pick_next_task_stop <-__schedule
RenderTh-2982    0dN.2 1248us+: pick_next_task_rt <-__schedule
RenderTh-2982    0dN.2 1250us+: dequeue_pushable_task <-pick_next_task_rt
RenderTh-2982    0d..2 1253us+: _raw_spin_unlock_irq <-__schedule
RenderTh-2982    0d..2 1256us+: gic_handle_irq <-__irq_svc
RenderTh-2982    0d..2 1259us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0d..2 1260us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0d..2 1262us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0d..2 1264us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0d..2 1265us+: irq_enter <-handle_IRQ
RenderTh-2982    0d..2 1267us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0d..2 1269us+: add_preempt_count <-irq_enter
RenderTh-2982    0d.h2 1271us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0d.h2 1272us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0d.h2 1274us+: handle_percpu_devid_irq <-generic_handle_irq
RenderTh-2982    0d.h2 1276us+: arch_timer_handler_cp15 <-handle_percpu_devid_irq
RenderTh-2982    0d.h2 1278us+: hrtimer_interrupt <-arch_timer_handler_cp15
RenderTh-2982    0d.h2 1280us+: ktime_get <-hrtimer_interrupt
RenderTh-2982    0d.h2 1282us+: arch_counter_read <-ktime_get
RenderTh-2982    0d.h2 1283us+: arch_counter_get_cntpct <-arch_counter_read
RenderTh-2982    0d.h2 1285us+: _raw_spin_lock <-hrtimer_interrupt
RenderTh-2982    0d.h2 1287us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h3 1289us+: __run_hrtimer <-hrtimer_interrupt
RenderTh-2982    0d.h3 1292us+: __remove_hrtimer <-__run_hrtimer
RenderTh-2982    0d.h3 1295us+: _raw_spin_unlock <-__run_hrtimer
RenderTh-2982    0d.h3 1296us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h2 1298us+: tick_sched_timer <-__run_hrtimer
RenderTh-2982    0d.h2 1300us+: ktime_get <-tick_sched_timer
RenderTh-2982    0d.h2 1302us+: arch_counter_read <-ktime_get
RenderTh-2982    0d.h2 1303us+: arch_counter_get_cntpct <-arch_counter_read
RenderTh-2982    0d.h2 1305us+: update_process_times <-tick_sched_timer
RenderTh-2982    0d.h2 1307us+: account_process_tick <-update_process_times
RenderTh-2982    0d.h2 1309us+: account_system_time <-account_process_tick
RenderTh-2982    0d.h2 1311us+: __rcu_read_lock <-account_system_time
RenderTh-2982    0d.h2 1313us+: __rcu_read_unlock <-account_system_time
RenderTh-2982    0d.h2 1315us+: run_local_timers <-update_process_times
RenderTh-2982    0d.h2 1316us+: hrtimer_run_queues <-run_local_timers
RenderTh-2982    0d.h2 1318us+: raise_softirq <-run_local_timers
RenderTh-2982    0d.h2 1320us+: __raise_softirq_irqoff <-raise_softirq
RenderTh-2982    0d.h2 1322us+: rcu_check_callbacks <-update_process_times
RenderTh-2982    0d.h2 1323us+: rcu_is_cpu_rrupt_from_idle <-rcu_check_callbacks
RenderTh-2982    0d.h2 1325us+: rcu_bh_qs <-rcu_check_callbacks
RenderTh-2982    0d.h2 1327us+: rcu_preempt_qs <-rcu_check_callbacks
RenderTh-2982    0d.h2 1329us+: rcu_pending <-rcu_check_callbacks
RenderTh-2982    0d.h2 1331us+: __rcu_pending <-rcu_pending
RenderTh-2982    0d.h2 1332us+: cpu_needs_another_gp <-__rcu_pending
RenderTh-2982    0d.h2 1335us+: __rcu_pending <-rcu_pending
RenderTh-2982    0d.h2 1337us+: cpu_needs_another_gp <-__rcu_pending
RenderTh-2982    0d.h2 1338us+: __rcu_pending <-rcu_pending
RenderTh-2982    0d.h2 1340us+: cpu_needs_another_gp <-__rcu_pending
RenderTh-2982    0d.h2 1342us+: raise_softirq <-rcu_check_callbacks
RenderTh-2982    0d.h2 1343us+: __raise_softirq_irqoff <-raise_softirq
RenderTh-2982    0d.h2 1345us+: printk_tick <-update_process_times
RenderTh-2982    0d.h2 1347us+: scheduler_tick <-update_process_times
RenderTh-2982    0d.h2 1349us+: _raw_spin_lock <-scheduler_tick
RenderTh-2982    0d.h2 1351us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h3 1353us+: update_rq_clock <-scheduler_tick
RenderTh-2982    0d.h3 1355us+: update_cpu_load <-scheduler_tick
RenderTh-2982    0d.h3 1357us+: sched_avg_update <-update_cpu_load
RenderTh-2982    0d.h3 1359us+: task_tick_rt <-scheduler_tick
RenderTh-2982    0d.h3 1360us+: update_curr_rt <-task_tick_rt
RenderTh-2982    0d.h3 1362us+: cpuacct_charge <-update_curr_rt
RenderTh-2982    0d.h3 1364us+: __rcu_read_lock <-cpuacct_charge
RenderTh-2982    0d.h3 1366us+: __rcu_read_unlock <-cpuacct_charge
RenderTh-2982    0d.h3 1368us+: sched_avg_update <-update_curr_rt
RenderTh-2982    0d.h3 1369us+: _raw_spin_lock <-update_curr_rt
RenderTh-2982    0d.h3 1371us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h4 1373us+: balance_runtime <-update_curr_rt
RenderTh-2982    0d.h4 1376us+: _raw_spin_unlock <-update_curr_rt
RenderTh-2982    0d.h4 1377us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h3 1380us+: _raw_spin_unlock <-scheduler_tick
RenderTh-2982    0d.h3 1381us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h2 1383us+: idle_cpu <-scheduler_tick
RenderTh-2982    0d.h2 1385us+: trigger_load_balance <-scheduler_tick
RenderTh-2982    0d.h2 1386us+: raise_softirq <-trigger_load_balance
RenderTh-2982    0d.h2 1388us+: __raise_softirq_irqoff <-raise_softirq
RenderTh-2982    0d.h2 1390us+: idle_cpu <-trigger_load_balance
RenderTh-2982    0d.h2 1392us+: __rcu_read_lock <-trigger_load_balance
RenderTh-2982    0d.h2 1394us+: __rcu_read_unlock <-trigger_load_balance
RenderTh-2982    0d.h2 1396us+: run_posix_cpu_timers <-update_process_times
RenderTh-2982    0d.h2 1398us+: hrtimer_forward <-tick_sched_timer
RenderTh-2982    0d.h2 1401us+: ktime_add_safe <-hrtimer_forward
RenderTh-2982    0d.h2 1403us+: ktime_add_safe <-hrtimer_forward
RenderTh-2982    0d.h2 1404us+: _raw_spin_lock <-__run_hrtimer
RenderTh-2982    0d.h2 1406us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h3 1408us+: enqueue_hrtimer <-__run_hrtimer
RenderTh-2982    0d.h3 1411us+: _raw_spin_unlock <-hrtimer_interrupt
RenderTh-2982    0d.h3 1413us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h2 1414us+: tick_program_event <-hrtimer_interrupt
RenderTh-2982    0d.h2 1416us+: clockevents_program_event <-tick_program_event
RenderTh-2982    0d.h2 1418us+: ktime_get <-clockevents_program_event
RenderTh-2982    0d.h2 1420us+: arch_counter_read <-ktime_get
RenderTh-2982    0d.h2 1422us+: arch_counter_get_cntpct <-arch_counter_read
RenderTh-2982    0d.h2 1423us+: arch_timer_set_next_event_cp15 <-clockevents_program_event
RenderTh-2982    0d.h2 1425us+: gic_eoi_irq <-handle_percpu_devid_irq
RenderTh-2982    0d.h2 1428us+: irq_exit <-handle_IRQ
RenderTh-2982    0d.h2 1429us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d..3 1431us+: __do_softirq <-irq_exit
RenderTh-2982    0d..3 1433us+: __local_bh_disable <-__do_softirq
RenderTh-2982    0d.s3 1436us+: gic_handle_irq <-__irq_svc
RenderTh-2982    0d.s3 1438us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0d.s3 1440us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0d.s3 1442us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0d.s3 1443us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0d.s3 1445us+: irq_enter <-handle_IRQ
RenderTh-2982    0d.s3 1447us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0d.s3 1449us+: add_preempt_count <-irq_enter
RenderTh-2982    0d.H3 1451us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0d.H3 1453us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0d.H3 1455us+: handle_percpu_devid_irq <-generic_handle_irq
RenderTh-2982    0d.H3 1457us+: arch_timer_handler_cp15 <-handle_percpu_devid_irq
RenderTh-2982    0d.H3 1459us+: gic_eoi_irq <-handle_percpu_devid_irq
RenderTh-2982    0d.H3 1461us+: irq_exit <-handle_IRQ
RenderTh-2982    0d.H3 1462us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s4 1465us+: idle_cpu <-irq_exit
RenderTh-2982    0d.s4 1467us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0d.s4 1469us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s3 1474us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0d.s3 1480us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0d.s3 1485us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0d.s3 1488us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0d.s3 1490us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0d.s3 1498us+: irq_enter <-handle_IRQ
RenderTh-2982    0d.s3 1503us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0d.s3 1509us+: add_preempt_count <-irq_enter
RenderTh-2982    0d.H3 1519us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0d.H3 1525us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0d.H3 1528us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0d.H3 1533us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0d.H3 1535us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H4 1537us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0d.H4 1539us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0d.H4 1540us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H3 1542us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0d.H3 1545us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0d.H3 1547us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0d.H3 1548us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0d.H3 1550us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0d.H3 1552us+: adreno_read <-adreno_regread
RenderTh-2982    0d.H3 1553us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0d.H3 1557us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0d.H3 1558us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0d.H3 1560us+: add_preempt_count <-queue_work
RenderTh-2982    0d.H4 1562us+: queue_work_on <-queue_work
RenderTh-2982    0d.H4 1564us+: __queue_work <-queue_work_on
RenderTh-2982    0d.H4 1566us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0d.H4 1567us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H5 1569us+: get_cwq <-__queue_work
RenderTh-2982    0d.H5 1571us+: insert_work <-__queue_work
RenderTh-2982    0d.H5 1573us+: get_pool_nr_running <-insert_work
RenderTh-2982    0d.H5 1574us+: wake_up_worker <-insert_work
RenderTh-2982    0d.H5 1576us+: wake_up_process <-wake_up_worker
RenderTh-2982    0d.H5 1578us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0d.H5 1580us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0d.H5 1581us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H6 1583us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0d.H6 1585us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H7 1587us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0d.H7 1589us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H6 1590us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0d.H6 1593us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0d.H6 1595us+: __rcu_read_lock <-select_task_rq_fair
RenderTh-2982    0d.H6 1597us+: source_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1599us+: target_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1601us+: effective_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1603us+: effective_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1604us+: cpu_avg_load_per_task <-select_task_rq_fair
RenderTh-2982    0d.H6 1606us+: idle_cpu <-select_task_rq_fair
RenderTh-2982    0d.H6 1608us+: __rcu_read_unlock <-select_task_rq_fair
RenderTh-2982    0d.H6 1610us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0d.H6 1612us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0d.H6 1613us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H7 1615us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0d.H7 1617us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.H7 1618us+: activate_task <-ttwu_activate
RenderTh-2982    0d.H7 1620us+: enqueue_task <-activate_task
RenderTh-2982    0d.H7 1622us+: update_rq_clock <-enqueue_task
RenderTh-2982    0d.H7 1624us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0d.H7 1627us+: update_curr <-enqueue_task_fair
RenderTh-2982    0d.H7 1628us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0d.H7 1630us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0d.H7 1632us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0d.H7 1634us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0d.H7 1636us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0d.H7 1637us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0d.H7 1639us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H8 1642us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0d.H8 1644us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0d.H8 1646us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H7 1648us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0d.H7 1649us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0d.H7 1651us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.H7 1654us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0d.H7 1658us+: resched_task <-check_preempt_curr
RenderTh-2982    0d.H7 1660us+: smp_send_reschedule <-resched_task
RenderTh-2982    0d.H7 1662us+: gic_raise_softirq <-smp_send_reschedule
RenderTh-2982    0d.H7 1665us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0d.H7 1667us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H6 1668us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0d.H6 1670us+: __rcu_read_lock <-ttwu_stat
RenderTh-2982    0d.H6 1672us+: __rcu_read_unlock <-ttwu_stat
RenderTh-2982    0d.H6 1674us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0d.H6 1676us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H5 1678us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0d.H5 1680us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H4 1681us+: sub_preempt_count <-queue_work
RenderTh-2982    0d.H3 1683us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0d.H3 1685us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0d.H3 1686us+: add_preempt_count <-queue_work
RenderTh-2982    0d.H4 1688us+: queue_work_on <-queue_work
RenderTh-2982    0d.H4 1690us+: __queue_work <-queue_work_on
RenderTh-2982    0d.H4 1692us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0d.H4 1694us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H5 1695us+: get_cwq <-__queue_work
RenderTh-2982    0d.H5 1697us+: insert_work <-__queue_work
RenderTh-2982    0d.H5 1699us+: get_pool_nr_running <-insert_work
RenderTh-2982    0d.H5 1701us+: wake_up_worker <-insert_work
RenderTh-2982    0d.H5 1702us+: wake_up_process <-wake_up_worker
RenderTh-2982    0d.H5 1704us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0d.H5 1705us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0d.H5 1707us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H6 1709us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0d.H6 1711us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H5 1712us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0d.H5 1714us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H4 1716us+: sub_preempt_count <-queue_work
RenderTh-2982    0d.H3 1718us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0d.H3 1720us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0d.H3 1722us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0d.H3 1725us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0d.H3 1727us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0d.H3 1728us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H4 1730us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0d.H4 1732us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0d.H4 1734us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H3 1736us+: irq_exit <-handle_IRQ
RenderTh-2982    0d.H3 1738us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s4 1740us+: idle_cpu <-irq_exit
RenderTh-2982    0d.s4 1741us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0d.s4 1743us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s3 1745us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0..s3 1748us+: run_timer_softirq <-__do_softirq
RenderTh-2982    0..s3 1749us+: hrtimer_run_pending <-run_timer_softirq
RenderTh-2982    0..s3 1752us+: _raw_spin_lock_irq <-run_timer_softirq
RenderTh-2982    0d.s3 1754us+: add_preempt_count <-_raw_spin_lock_irq
RenderTh-2982    0d.s4 1757us+: _raw_spin_unlock_irq <-run_timer_softirq
RenderTh-2982    0..s4 1758us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0..s3 1761us+: delayed_work_timer_fn <-run_timer_softirq
RenderTh-2982    0..s3 1762us+: get_work_cwq <-delayed_work_timer_fn
RenderTh-2982    0..s3 1764us+: __queue_work <-delayed_work_timer_fn
RenderTh-2982    0..s3 1766us+: get_gcwq <-__queue_work
RenderTh-2982    0..s3 1768us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0d.s3 1769us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.s4 1771us+: get_cwq <-__queue_work
RenderTh-2982    0d.s4 1773us+: insert_work <-__queue_work
RenderTh-2982    0d.s4 1775us+: get_pool_nr_running <-insert_work
RenderTh-2982    0d.s4 1777us+: wake_up_worker <-insert_work
RenderTh-2982    0d.s4 1779us+: wake_up_process <-wake_up_worker
RenderTh-2982    0d.s4 1780us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0d.s4 1782us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0d.s4 1784us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.s5 1787us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0d.s5 1789us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0d.s5 1791us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0d.s5 1793us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0d.s5 1795us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.s6 1796us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0d.s6 1798us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.s6 1800us+: activate_task <-ttwu_activate
RenderTh-2982    0d.s6 1801us+: enqueue_task <-activate_task
RenderTh-2982    0d.s6 1804us+: update_rq_clock <-enqueue_task
RenderTh-2982    0d.s6 1806us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0d.s6 1807us+: update_curr <-enqueue_task_fair
RenderTh-2982    0d.s6 1809us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0d.s6 1811us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0d.s6 1813us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0d.s6 1815us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0d.s6 1817us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0d.s6 1820us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0d.s6 1821us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.s7 1824us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0d.s7 1825us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0d.s7 1827us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.s6 1829us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0d.s6 1831us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0d.s6 1833us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.s6 1837us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0d.s6 1839us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0d.s6 1840us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.s5 1842us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0d.s5 1845us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0d.s5 1847us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.s4 1849us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0..s4 1850us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0..s3 1852us+: _raw_spin_lock_irq <-run_timer_softirq
RenderTh-2982    0d.s3 1854us+: add_preempt_count <-_raw_spin_lock_irq
RenderTh-2982    0d.s4 1856us+: _raw_spin_unlock_irq <-run_timer_softirq
RenderTh-2982    0..s4 1857us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0..s3 1859us+: rcu_bh_qs <-__do_softirq
RenderTh-2982    0..s3 1861us+: run_rebalance_domains <-__do_softirq
RenderTh-2982    0..s3 1863us+: rebalance_domains <-run_rebalance_domains
RenderTh-2982    0..s3 1865us+: update_shares <-rebalance_domains
RenderTh-2982    0..s3 1868us+: __rcu_read_lock <-update_shares
RenderTh-2982    0..s3 1870us+: __rcu_read_unlock <-update_shares
RenderTh-2982    0..s3 1872us+: __rcu_read_lock <-rebalance_domains
RenderTh-2982    0..s3 1874us+: msecs_to_jiffies <-rebalance_domains
RenderTh-2982    0..s3 1876us+: load_balance <-rebalance_domains
RenderTh-2982    0..s3 1878us+: find_busiest_group <-load_balance
RenderTh-2982    0..s3 1881us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1883us+: target_load <-find_busiest_group
RenderTh-2982    0..s3 1885us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1888us+: update_group_power <-find_busiest_group
RenderTh-2982    0..s3 1889us+: msecs_to_jiffies <-update_group_power
RenderTh-2982    0..s3 1891us+: scale_rt_power <-update_group_power
RenderTh-2982    0..s3 1895us+: source_load <-find_busiest_group
RenderTh-2982    0..s3 1896us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1899us+: source_load <-find_busiest_group
RenderTh-2982    0..s3 1901us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1904us+: __rcu_read_unlock <-rebalance_domains
RenderTh-2982    0..s3 1906us+: rcu_bh_qs <-__do_softirq
RenderTh-2982    0..s3 1907us+: rcu_process_callbacks <-__do_softirq
RenderTh-2982    0..s3 1910us+: __rcu_process_callbacks <-rcu_process_callbacks
RenderTh-2982    0..s3 1911us+: force_quiescent_state <-__rcu_process_callbacks
RenderTh-2982    0..s3 1913us+: rcu_process_gp_end.isra.23 <-__rcu_process_callbacks
RenderTh-2982    0..s3 1917us+: check_for_new_grace_period <-__rcu_process_callbacks
RenderTh-2982    0..s3 1918us+: cpu_needs_another_gp <-__rcu_process_callbacks
RenderTh-2982    0..s3 1920us+: __rcu_process_callbacks <-rcu_process_callbacks
RenderTh-2982    0..s3 1922us+: force_quiescent_state <-__rcu_process_callbacks
RenderTh-2982    0..s3 1924us+: rcu_process_gp_end.isra.23 <-__rcu_process_callbacks
RenderTh-2982    0..s3 1926us+: check_for_new_grace_period <-__rcu_process_callbacks
RenderTh-2982    0..s3 1927us+: cpu_needs_another_gp <-__rcu_process_callbacks
RenderTh-2982    0..s3 1929us+: __rcu_process_callbacks <-rcu_process_callbacks
RenderTh-2982    0..s3 1932us+: rcu_process_gp_end.isra.23 <-__rcu_process_callbacks
RenderTh-2982    0d.s3 1934us+: _raw_spin_trylock <-rcu_process_gp_end.isra.23
RenderTh-2982    0d.s3 1936us+: add_preempt_count <-_raw_spin_trylock
RenderTh-2982    0d.s4 1938us+: _raw_spin_unlock_irqrestore <-rcu_process_gp_end.isra.23
RenderTh-2982    0..s4 1939us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0..s3 1941us+: check_for_new_grace_period <-__rcu_process_callbacks
RenderTh-2982    0d.s3 1943us+: _raw_spin_trylock <-check_for_new_grace_period
RenderTh-2982    0d.s3 1944us+: add_preempt_count <-_raw_spin_trylock
RenderTh-2982    0d.s4 1946us+: _raw_spin_unlock_irqrestore <-check_for_new_grace_period
RenderTh-2982    0d.s4 1948us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0..s3 1950us+: cpu_needs_another_gp <-__rcu_process_callbacks
RenderTh-2982    0..s3 1952us+: rcu_bh_qs <-__do_softirq
RenderTh-2982    0d.s3 1953us+: __local_bh_enable <-__do_softirq
RenderTh-2982    0d.s3 1955us+: sub_preempt_count <-__local_bh_enable
RenderTh-2982    0d..3 1957us+: idle_cpu <-irq_exit
RenderTh-2982    0d..3 1959us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0d..3 1961us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d..2 1962us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0...2 1965us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0...2 1968us : __schedule
RenderTh-2982    0...2 1970us :   2982:116:R ==> [000]   319: 83:R mdss_fb0
ftrace与trace_cmd自动化布署方式 trace_cmd ftrace使用 阅读详情

相关推荐

Ubuntu实时内核部署与深度调优:从延迟尖峰到稳定低延迟的实战指南

在追求极致性能的实时计算领域,Linux内核的调度策略是决定系统响应确定性的核心。通用Linux内核采用的完全公平调度器(CFS)虽然保证了整体吞吐量和公平性,但其非抢占式的设计内核段和不可预测的中断处理,容易导致关键任务被延迟,形成影响系统稳定性的延迟尖峰。为解决这一问题,Linux实时内核(PREEMPT_RT)通过将大部分内核代码变为可抢占式,并引入优先级继承等机制,从根本上提升了系统的实时响应能力。其技术价值在于,能够在标准的通用硬件和操作系统上,为高频交易、工业自动化、机器人控制等场景提供微秒级的

weixin_34413802的博客 395

使用 Ftrace 分析函数性能

作者简介:伟林,中年码农,从事过电信、手机、安全、芯片等行业,目前依旧从事Linux方向开发工作,个人爱好Linux相关知识分享。0.背景ftrace的功能非常强大,可以在系统的各个关键点上采集数据用以追踪系统的运行情况。既支持预设的静态插桩点(trace event),也支持每个函数的动态插桩(function tracer)。还可以利用动态插桩来测量函数的执行时间(f...

Peter的专栏 1189

【实时Linux核心技术:从概念到实战】06:编写你的第一个实时线程:API详解与注意事项

在汽车电子、工业控制与机器人等对确定性要求极高的领域,Linux默认调度策略难以满足硬实时需求。本文手把手教你基于POSIX实时扩展编写具备微秒级低抖动的实时线程:从正确配置`pthread`属性(含极易被忽略的`PTHREAD_EXPLICIT_SCHED`陷阱)、采用`clock_nanosleep`实现无漂移周期控制,到`mlockall`防页面换出、CPU亲和性绑核、堆栈预热等系统级调优。文章附完整可运行代码及`cyclictest`性能对比,并梳理9个线上高频致命错误与规避方案,助你构建稳定可信的

专注于人工智能、软件开发、工控自动化、工厂数字化及智能化等领域,希望和大家共同进步! 258

linux trace机制分析 - 2.ftrace使用

文章目录简介使用 简介 ftrace通常被理解为是一个linux内置的函数追踪器,可以很方便的帮助开发者内核函数的运行情况,通常被用来在分析函数延迟和系统性能。实际上ftrace不仅仅是一个函数追踪器,更准确的说ftrace是一个综合了多种手段的trace框架。比如分析延迟跟踪检查、中断的禁用与执行、任务的调度与抢占等等。 ftrace一个最常用的用法是事件追踪,通过内核中几百个event point 可以通过tracefs查看内核的运行情况。 使用 在内核编译配置中开启ftracer Kernel h

BaiTianio的博客 2897

IVP 2021 UNETR: Transformers for 3D Medical Image Segmentation

动机 具有跟踪和扩展路径的全卷积神经网络(FCNNs)(如编码器和解码器),近年来在各种医学图像分割应用中显示出了突出的地位。在这些体系结构中,编码器通过学习全局上下文表示来扮演一个不可或缺的角色,该全局上下文表示将进一步用于解码器的语义输出预测。 卷积层作为FCNN的主要构造块,其局部性限制了此类网络学习长时空间依赖的能力。 在NLP领域,基于transformer的模型已经在各种任务中实现了最先进的基准。transformer中的自我注意机制使他们能够动态地突出词序列的重要特征并学习其长时依赖关系。这

smile909的博客 1979

高通 Gen AI 推理扩展 (GENIE)(5):实战指南与Engine API深度解析

本文深入解析高通Gen AI推理扩展(GENIE)的Engine API核心功能与实战应用,涵盖引擎实例化、多引擎动态切换、性能调优及跨平台部署等关键技巧。通过详细代码示例和性能数据,帮助开发者高效利用GENIE进行AI推理,提升应用性能与稳定性。

weixin_29065959的博客 465

linux内核:ftrace——追踪内核行为

描述了ftrace的使用方法,trace-cmd常用方法

Leo-夜空的博客 2325

硬件-电源-智能保护:eFuse选型与实战设计全攻略

本文深入解析eFuse在硬件电源设计中的关键作用与实战选型技巧。作为智能保护器件,eFuse能提供毫安级精度的过流保护、多重故障防护和智能恢复功能,比传统保险丝快1000倍以上。文章详细介绍了eFuse的工作原理、五大选型维度和PCB布局要点,帮助工程师有效预防电源异常导致的硬件损坏,适用于AI服务器、工业控制等场景。

weixin_30879169的博客 200

linux trace学习(二)——trace使用

备注:1. Kernel版本:4.19.1232. 使用工具:Source Insight 4.0。

枫潇潇 9614

【DRAM存储器五十一】LPDDR5介绍--DFE与Pre-Emphasis

简述LPDDR5的DFE和预加重应用设置

highman110的博客 707

#systemverilog# 之 event region 和 timeslot 仿真调度(三)Verdi 辅助查看

通过今天的学习,我们知道:event region 是event sequence 的全集。我们打开了 event region feature 也就开启了event sequence 的功能。

那么菜的博客 1366

linux内核内置trace工具使用简介

使用内核内置的trace工具,可以跟踪具体某个进程的内核函数调用过程,并且可以统计到每个函数的运行时间,对优化系统性能有一定的帮助 内核trace工具使用方法如下: 1.修改内核配置,支持trace Kernel hacking  --->  [*] Tracers  --->[*]   Kernel Function Tracer                              

专注于嵌入式开发 1万+

【模型之美】3、文本聚类与智能摘要,让AI帮你做个总结

在信息过载时代,文本聚类与摘要技术成为关键工具。本文提出了一种结合OpenAI Embedding与传统机器学习的方法,实现从文本预处理到智能分析的完整流程。技术方案分为三阶段:1)通过Embedding将文本转化为语义向量;2)采用K-Means等算法进行无监督聚类;3)利用大模型生成可解释的标签和摘要。实验证明,该方法能有效提升信息处理效率,如在新闻聚合系统中自动分类10,000篇文章,准确率较传统方法提高15%。方案包含实用代码示例,涵盖文本清洗、向量化、聚类优化等关键技术细节,可应用于对话记忆、舆情

2025博客之星Top81。专注AI工程化与架构实战。从分布式思维到模型部署,用工程化视角为你厘清AI落地的真实路径。 1440

kernel ftrace使用示例

使能 kernel ftrace 需要打开下面编译宏 CONFIG_FTRACE=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_FUNCTION_TRACER=y CONFIG_FUNCTION_GRAPH_TRACER=y CONFIG_IRQSOFF_TRACER=y CONFIG_SCHED_TRACER=y CONFIG_ENABLE_DE

weixin_40419181的博客 701

华为OD机试双机位C卷 - 高矮个子排队 (C++/Py/Java/Js/Go/C)

华为OD机试双机位C卷 高矮个子排队,提供C语言、Java、C++、Python、Go、JavaScript源码实现,并提供完整思路讲解。华为OD机试 - 高矮个子排队。

qq_45776114的博客 2160

YOLOv8改进目录一览 | 涉及卷积层、轻量化、注意力、损失函数、Backbone、SPPF、Neck、检测头等全方位改进

1️⃣本专栏已更新150多种不同的改进方法,所使用并改进的每一个模块均包含详细的模块分析、原理讲解、个人总结、多种改进方式以及完整的修改流程,所有改进100%可直接运行。2️⃣所有改进点均为近三年顶会,顶刊提出的先进算法,将其融入到中,紧跟学术热点,适应不同的任务场景。3️⃣团队内发表数篇SCI论文,熟悉完整的发表流程,订阅专栏即可进群享受模型训练、模型改进、论文写作、投稿选刊,从入门到论文的各种答疑内容。4️⃣专栏内容会持续更新,最近更新时间:2024-12-24。项目介绍在大家购买专栏后,加入学

Limiiiing的博客 1万+

Linux内核深度剖析:四层动态切片实战方法论

Linux内核不是黑箱,而是可观测、可干预的精密系统。理解其核心机制需跨越源码与运行态、抽象与硬件、配置与行为三大断层。本文基于真实云原生故障案例(如soft lockup、P99延迟抖动),聚焦内核剖析与动态观测两大关键技术能力,解析启动流程、调度器行为、内存映射和中断生命周期四大可观测切面。通过QEMU+GDB单步跟踪、ftrace函数级追踪、page_owner内存溯源、trace-cmd中断时序分析等工程化手段,将内核从理论概念转化为终端可验证的实时状态。适用于性能优化、SRE排障与嵌入式移植等一线

weixin_33701617的博客 382

jdk-8u144-linux-x64.tar.gz.

该资源为搭建Linux系统Java环境用,资源的后缀为tar.gz格式的压缩包

Linux-RT核心原理与硬实时系统搭建实战

实时操作系统(RTOS)的本质是时间行为的可预测性与确定性,而非单纯追求执行速度。Linux-RT通过PREEMPT_RT补丁重构内核抢占机制、中断处理和调度策略,将通用Linux转化为支持硬实时(Hard Real-Time)的执行环境——其关键能力包括微秒级中断延迟、全可抢占内核、优先级继承及CPU亲和性控制。该技术广泛应用于工业自动化(如EtherCAT主站)、电力保护、实时图像处理与FPGA协同边缘AI等对时间约束极为严苛的场景。当系统要求控制周期≤1ms且抖动容忍度<1%时,Linux-RT成为不

weixin_34268169的博客 416

Probability and Statistics - 4th edition

Probability and Statistics - 4th edition

pscad IEEE标准系统搭建说明

pscad IEEE标准系统搭建说明,包含典型的IEEE 9节点、14节点、30节点、39节点和118节点系统。内含标准测试系统的原始数据和潮流结果。

Linux内核进程唤醒机制:wake_up与wake_up_process详解

进程唤醒是操作系统调度的核心机制,直接影响系统响应速度和吞吐量。Linux内核通过wait_queue实现高效的进程休眠/唤醒管理,其中wake_up()和wake_up_process()是最关键的唤醒接口。wake_up_process实现精准的单进程唤醒,适合实时任务调度;而wake_up则采用广播模式唤醒整个等待队列,常见于IO完成处理。理解其底层通过try_to_wake_up()的通用路径(包含内存屏障、自旋锁和调度类回调)对开发高性能驱动和系统程序至关重要。在数据库连接池等实际场景中,合理选择

weixin_30340617的博客 398

Linux内核实时化:低延迟与可抢占补丁的原理、对比与工程实践

实时操作系统(RTOS)的核心在于提供确定性的任务调度与极低的响应延迟,以满足工业控制、汽车电子等对时序要求严苛的场景。其技术原理主要围绕如何保证高优先级任务能够及时抢占CPU执行,从而将内核延迟控制在微秒级。这一技术价值在于,它使得通用、生态丰富的Linux系统能够应用于硬实时领域,极大地拓展了其应用边界。在嵌入式开发中,面对实时性需求,开发者常需在内核层面进行优化,其中低延迟补丁(Low-Latency Patch)与可抢占内核补丁(Preemptable Kernel Patch)是两种经典的基础方案

weixin_33895695的博客 375

SoC到操作系统五层架构:嵌入式系统全栈实操指南

现代嵌入式系统并非简单的硬件+软件堆叠,而是由SoC、固件、设备驱动、内核和操作系统构成的五层协同体系。这一分层本质是对物理世界不确定性(如温漂、时序偏差、协议歧义)的逐级防御设计,每层承担特定维度的抽象与管控责任。SoC已演进为可编程硬件平台,其内部NPU、VPU、ISP等单元支持寄存器级功能重构;固件则超越启动代码角色,成为ACPI表生成、电源管理、安全服务等关键能力的提供者。理解该架构对解决真实工程问题至关重要——例如0.3%的固件时钟偏差可导致SD卡初始化失败率飙升至17%,而PCIe热插拔异常往往

weixin_30627341的博客 389

Linux 内核 RCU CPU Stall 检测器完全指南:成因、参数调优与 “Splat“ 日志解读

本文基于 Linux 内核官方文档 [Documentation/RCU/stallwarn.rst](https://link.gitcode.com/i/f9f073a2cf5fc3552414240b38780a64) 编写,完整覆盖 RCU CPU stall 警告的触发成因、`rcupdate` 模块的调优参数(Kconfig 与 sysfs 双层机制)、以及各类内核日志 "splat"

gitblog_00152的博客 263

OS17.【Linux】进程基础知识(1)

介绍Linux进程的基本概念和管理方法。主要内容包括:1.进程的浅层定义 - 正在运行的程序;2.查看进程的多种方法(ps、top、/proc目录);3.进程管理实践(查看、终止进程);4.进程控制块(PCB)的组成和作用;5.Linux内核中task_struct结构体的详细解析。文章通过代码示例和命令演示,帮助读者理解进程标识符(PID)、工作目录、程序计数器等核心概念,并深入分析了操作系统如何通过PCB组织和管理进程。最后提供了Linux内核源代码中task_struct的完整定义,为后续学

zhangcoder的博客 记录学习轨迹~ 1042

中断上半部响应时间优化:从测量到根因定位的完整指南

中断响应时间是衡量实时系统性能的关键指标之一,其本质是从硬件中断信号到达中断控制器到CPU执行注册处理函数之间的延迟,与中断处理时间有着本质区别。理解中断屏蔽窗口、锁竞争、中断控制器配置等因素对响应时间的影响,是进行性能优化的前提。在多核嵌入式系统中,合理的irq affinity配置、精简临界区、以及选择适当的上下半部机制,能有效降低p99与max延迟,避免FIFO溢出和存储超时。本文从测量手段到根因定位,结合实际案例,系统阐述影响中断上半部响应时间的典型瓶颈,并给出可落地的优化方案。

weixin_34234829的博客 455

linux_2.6.35_wait和wakeup源码阅读

写在这里只当是为了备份一下,只是匆匆的浏览了一下源码,如果您想获得些什么深刻的东西,这个肯定是没有了,就是流水帐一个。 正文: 在wait.h中,可以看到定义了许多wait_up相关的宏 #define wake_up(x)            __wake_up(x, TASK_NORMAL, 1, NULL) #define wake_up_nr(x, nr)

1260

【linux kernel】linux内核如何唤醒线程

linux内核如何唤醒线程 //本文代码片段出自linux内核版本:4.1.15 linux内核唤醒线程主要使用wake_up_process()。 一、wake_up_process()分析 在linux内核中,唤醒线程由wake_up_process()函数实现。其定义在(/kernel/sched/core.c): int wake_up_process(struct task_struct *p) { WARN_ON(task_is_stopped_or_traced(p)); return t

iriczhao的博客 4034
上一篇: Linux Kernel threaded irq
下一篇: MIPI-CSI2
windtakers
博客等级 码龄17年 94粉丝 197原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值