linux kernel ftrace 之wakeup tracer and wakeup_rt tracer

Linux内核调试利器:ftrace原理与应用详解 函数追踪是Linux内核调试的核心技术之一,通过在运行时动态修改代码实现低开销监控。ftrace作为内核原生追踪框架,采用nop指令替换和环形缓冲区等机制,既保证了性能又提供了丰富的调试信息。在系统性能优化、内核问题定位等场景中,ftrace能够精确追踪函数调用关系、执行耗时等关键指标。特别是结合动态过滤器和事件触发功能,开发者可以针对特定进程或代码路径进行深度分析。本文以调度延迟和内存分配问题为例,展示了如何利用function_graph追踪器和wakeup_rt追踪器解决实际问题,同时对比了ftrac 阅读详情

对于 ftrace 的function tracer/function graph tracer 用的比较多,本文介绍 wakeup tracer/wakeup_rt tracer。

开始介绍之前,先扯两句别的,在一开始想了解wakeup/wakeup_rt tracer 的时候,因为内核Documents 中有ftrace的文档介绍,
直接在工程中的ftrace.txt 看了一下wakeup/wakeup_rt tracer 的介绍,当场看糊涂了,写的完全驴唇不对马嘴,工程中的kernel 是3.4的,
当时就怀疑Documents 应该是没跟上代码更新,看了一下linus torvals tree, 果然更新了。建议直接看最新版本的介绍:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/trace/ftrace.txt


1.wakeup/wakeup_rt tracer 是什么?

下面有详细的英文介绍,简单来说,wakeup/wakeup_rt tracer 是ftrace 的一个tracing utilities,是两种tracer.
区别在于wakeup tracer 是Trace 所有的进程中(实时/非实时进程)的schedule latency, 而wakeup_rt 只Trace 实时进程的最大schedule latency.


  "wakeup"
Traces and records the max latency that it takes for
the highest priority task to get scheduled after
it has been woken up.
        Traces all tasks as an average developer would expect.


  "wakeup_rt"
        Traces and records the max latency that it takes for just
        RT tasks (as the current "wakeup" does). This is useful
        for those interested in wake up timings of RT tasks.


2. wakeup tracer How-to



$ adb shell cat /sys/kernel/debug/tracing/available_tracers
function_graph wakeup_rt wakeup preemptirqsoff preemptoff irqsoff function nop


$ adb shell "echo wakeup > /sys/kernel/debug/tracing/current_tracer"
$ adb shell "echo 0 > /sys/kernel/debug/tracing/tracing_max_latency"
$ adb shell "echo 1 > /sys/kernel/debug/tracing/tracing_on"
$ adb shell cat /sys/kernel/debug/tracing/trace




# tracer: wakeup
#
# wakeup latency trace v1.1.5 on 3.4.0-gd26777b-00428-gd66afe1-dirty
# --------------------------------------------------------------------
# latency: 21865 us, #11408/11408, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:2)//wakeup latency 是21865us
#    -----------------
#    | task: PowerManagerSer-943 (uid:1000 nice:-4 policy:0 rt_prio:0)//trace 的进程是PowerManagerSer-943
#    -----------------
#//下面是详细的trace,向你解释为什么PowerManagerSer-943 进程从wake up 到真正scheduled in 得以执行用了21865us.
 //第一列是执行wakeup PowerManagerSer-943 进程的时候正在运行的进程,后面是运行于哪个CPU,中断开关情况,need-resched flag 的情况,
 //中断服务程序中,还是软中断中,抢占深度
 //当你看到第一列中的进程优先级并不高时,PowerManagerSer-943 为啥得不到执行,原因无非就是中断被关了,无法调度,或是抢占深度count 置位,无法
 //抢占。
#                  _------=> CPU#            
#                 / _-----=> irqs-off        
#                | / _----=> need-resched    
#                || / _---=> hardirq/softirq 
#                ||| / _--=> preempt-depth   
#                |||| /     delay             
#  cmd     pid   ||||| time  |   caller      
#     \   /      |||||  \    |   /           
   <...>-1108    0d..5    6us+:   1108:120:R   + [000]   943:116:R PowerManagerSer
   <...>-1108    0d..5    9us+: 0
   <...>-1108    0d..4   11us : check_preempt_curr <-ttwu_do_wakeup
   <...>-1108    0d..4   12us : check_preempt_wakeup <-check_preempt_curr
   <...>-1108    0d..4   13us : update_curr <-check_preempt_wakeup
   <...>-1108    0d..4   15us : wakeup_preempt_entity <-check_preempt_wakeup
   <...>-1108    0d..4   16us : _raw_spin_unlock <-try_to_wake_up
   <...>-1108    0d..4   17us+: sub_preempt_count <-_raw_spin_unlock
   <...>-1108    0d..3   19us+: ttwu_stat <-try_to_wake_up
...


kworker/-31782   0d..2 21843us+: update_stats_wait_end <-set_next_entity
kworker/-31782   0d..2 21849us : _raw_spin_unlock_irq <-__schedule
kworker/-31782   0...2 21850us+: sub_preempt_count <-_raw_spin_unlock_irq
kworker/-31782   0...2 21852us : __schedule
kworker/-31782   0...2 21853us :  31782:120:S ==> [000]   943:116:R PowerManagerSer


没一列的详述:
  cmd: The name of the process in the trace.


  pid: The PID of that process.


  CPU#: The CPU which the process was running on.


  irqs-off: 'd' interrupts are disabled. '.' otherwise.
   Note: If the architecture does not support a way to
 read the irq flags variable, an 'X' will always
 be printed here.


  need-resched:
'N' both TIF_NEED_RESCHED and PREEMPT_NEED_RESCHED is set,
'n' only TIF_NEED_RESCHED is set,
'p' only PREEMPT_NEED_RESCHED is set,
'.' otherwise.


  hardirq/softirq:
'H' - hard irq occurred inside a softirq.
'h' - hard irq is running
's' - soft irq is running
'.' - normal context.


  preempt-depth: The level of preempt_disabled


The above is mostly meaningful for kernel developers.


  time: When the latency-format option is enabled, the trace file
output includes a timestamp relative to the start of the
trace. This differs from the output when latency-format
is disabled, which includes an absolute timestamp.


  delay: This is just to help catch your eye a bit better. And
needs to be fixed to be only relative to the same CPU.
The marks are determined by the difference between this
current trace and the next trace.
 '$' - greater than 1 second
 '#' - greater than 1000 microsecond
 '!' - greater than 100 microsecond
 '+' - greater than 10 microsecond
 ' ' - less than or equal to 10 microsecond.


  The rest is the same as the 'trace' file.


  Note, the latency tracers will usually end with a back trace
  to easily find where the latency occurred.


3.wakeup_rt tracer How-to



$ adb shell cat /sys/kernel/debug/tracing/available_tracers
function_graph wakeup_rt wakeup preemptirqsoff preemptoff irqsoff function nop


$ adb shell "echo wakeup_rt > /sys/kernel/debug/tracing/current_tracer"
$ adb shell "echo 0 > /sys/kernel/debug/tracing/tracing_max_latency"
$ adb shell "echo 1 > /sys/kernel/debug/tracing/tracing_on"
$ adb shell cat /sys/kernel/debug/tracing/trace


# tracer: wakeup_rt
#
# wakeup_rt latency trace v1.1.5 on 3.4.0-gd26777b-00428-gd66afe1-dirty
# --------------------------------------------------------------------
# latency: 1973 us, #967/967, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:1)
#    -----------------
#    | task: mdss_fb0-319 (uid:0 nice:0 policy:1 rt_prio:16)
#    -----------------
#
#                  _------=> CPU#            
#                 / _-----=> irqs-off        
#                | / _----=> need-resched    
#                || / _---=> hardirq/softirq 
#                ||| / _--=> preempt-depth   
#                |||| /     delay             
#  cmd     pid   ||||| time  |   caller      
#     \   /      |||||  \    |   /           
surfacef-253     2d..4    2us+:    253:112:R   + [000]   319: 83:R mdss_fb0
surfacef-253     2d..4    8us+: 0
RenderTh-2982    0dN.3   16us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   19us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   21us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   23us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   25us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   27us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   29us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   31us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   33us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   34us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   36us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   38us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   40us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   42us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   44us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   45us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   47us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   49us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   51us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   53us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   55us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   57us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   59us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   60us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   63us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   64us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   66us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   68us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   70us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   71us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   73us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   76us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   77us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   79us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1   81us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1   83us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1   84us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1   86us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2   88us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2   90us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3   92us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3   95us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2   96us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2   98us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  100us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  102us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  103us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  106us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  108us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  110us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  112us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  114us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  116us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  117us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  119us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  121us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  123us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  124us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  127us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  129us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  131us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  133us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  135us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  136us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  138us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  140us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  141us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  143us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  145us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  147us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  149us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  150us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  152us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  154us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  156us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  158us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  160us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  162us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  164us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  165us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  167us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  169us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  171us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  173us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  176us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  178us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  180us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  182us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  183us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  185us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  187us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  189us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  192us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  193us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  196us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  197us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  199us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  201us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  203us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  205us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  208us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  209us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  211us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  213us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  215us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  217us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  218us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  220us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  224us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  226us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  228us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  229us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  231us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  233us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  235us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  237us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  239us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  241us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  243us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  244us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  246us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  248us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  250us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  252us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  253us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  258us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  259us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  261us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  263us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  265us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  267us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  269us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  271us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  273us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  275us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  276us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  278us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  280us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  282us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  284us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  286us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  289us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  290us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  292us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  294us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  296us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  298us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  299us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  301us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  305us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  307us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  308us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  310us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  312us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  314us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  316us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  317us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  321us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  323us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  325us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  327us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  329us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  331us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  332us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  335us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  337us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  339us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  341us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  343us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  344us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  346us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  348us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  350us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  353us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  355us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  357us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  359us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  360us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  362us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  364us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  366us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  370us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  372us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  374us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  375us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  377us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  380us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  381us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  384us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  386us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  388us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  393us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  395us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  397us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  403us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  405us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  407us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  408us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  410us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  412us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  414us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  423us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  425us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  427us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  429us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  431us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  432us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  434us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  436us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  438us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  440us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  442us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  443us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  445us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  447us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  449us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  450us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  452us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  454us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  456us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  458us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  460us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  461us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  463us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  466us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  467us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  469us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  471us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  473us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  475us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  476us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  478us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  481us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  482us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  484us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  486us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  488us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  490us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  492us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  493us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  495us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  497us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  499us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  501us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  503us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  504us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  506us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  508us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  510us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  511us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  515us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  516us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  518us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  520us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  522us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  523us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  525us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  527us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  531us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  532us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  534us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  536us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  538us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  540us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  542us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  543us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  546us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  548us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  550us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  551us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  553us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  555us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  557us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  558us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  562us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  564us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  566us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  567us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  569us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  571us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  573us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  574us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  576us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  578us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  580us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  582us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  583us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  585us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  587us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  589us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  591us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  592us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  594us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  596us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  598us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  601us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  603us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  604us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  607us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  608us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  611us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  613us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  615us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  617us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  618us+: task_sched_runtime <-thread_group_cputime
RenderTh-2982    0dN.1  620us+: task_rq_lock <-task_sched_runtime
RenderTh-2982    0dN.1  622us+: _raw_spin_lock_irqsave <-task_rq_lock
RenderTh-2982    0dN.1  624us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dN.2  628us+: _raw_spin_lock <-task_rq_lock
RenderTh-2982    0dN.2  629us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dN.3  631us+: _raw_spin_unlock <-task_sched_runtime
RenderTh-2982    0dN.3  633us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dN.2  635us+: _raw_spin_unlock_irqrestore <-task_sched_runtime
RenderTh-2982    0dN.2  637us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dN.1  639us+: __rcu_read_unlock <-thread_group_cputime
RenderTh-2982    0dN.1  640us+: nsecs_to_jiffies <-thread_group_times
RenderTh-2982    0dN.1  643us+: nsecs_to_jiffies64 <-nsecs_to_jiffies
RenderTh-2982    0dN.1  646us+: _raw_spin_unlock_irq <-do_sys_times
RenderTh-2982    0dN.1  648us+: gic_handle_irq <-__irq_svc
RenderTh-2982    0dN.1  651us+: handle_IPI <-gic_handle_irq
RenderTh-2982    0dN.1  653us+: scheduler_ipi <-handle_IPI
RenderTh-2982    0dN.1  656us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0dN.1  659us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0dN.1  661us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0dN.1  663us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0dN.1  665us+: irq_enter <-handle_IRQ
RenderTh-2982    0dN.1  667us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0dN.1  669us+: add_preempt_count <-irq_enter
RenderTh-2982    0dNh1  671us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0dNh1  673us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0dNh1  676us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0dNh1  678us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0dNh1  680us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2  682us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0dNh2  684us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0dNh2  685us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1  687us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0dNh1  691us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0dNh1  693us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0dNh1  695us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0dNh1  697us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0dNh1  699us+: adreno_read <-adreno_regread
RenderTh-2982    0dNh1  701us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1  705us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0dNh1  707us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0dNh1  709us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2  711us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2  713us+: __queue_work <-queue_work_on
RenderTh-2982    0dNh2  715us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0dNh2  716us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh3  718us+: get_cwq <-__queue_work
RenderTh-2982    0dNh3  720us+: insert_work <-__queue_work
RenderTh-2982    0dNh3  724us+: get_pool_nr_running <-insert_work
RenderTh-2982    0dNh3  725us+: wake_up_worker <-insert_work
RenderTh-2982    0dNh3  727us+: wake_up_process <-wake_up_worker
RenderTh-2982    0dNh3  729us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0dNh3  731us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0dNh3  733us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh4  735us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0dNh4  737us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0dNh4  739us+: __rcu_read_lock <-select_task_rq_fair
RenderTh-2982    0dNh4  741us+: source_load <-select_task_rq_fair
RenderTh-2982    0dNh4  743us+: target_load <-select_task_rq_fair
RenderTh-2982    0dNh4  745us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  747us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  749us+: cpu_avg_load_per_task <-select_task_rq_fair
RenderTh-2982    0dNh4  751us+: idle_cpu <-select_task_rq_fair
RenderTh-2982    0dNh4  753us+: __rcu_read_unlock <-select_task_rq_fair
RenderTh-2982    0dNh4  755us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0dNh4  757us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0dNh4  758us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh5  760us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0dNh5  762us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  764us+: activate_task <-ttwu_activate
RenderTh-2982    0dNh5  765us+: enqueue_task <-activate_task
RenderTh-2982    0dNh5  767us+: update_rq_clock <-enqueue_task
RenderTh-2982    0dNh5  770us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0dNh5  772us+: update_curr <-enqueue_task_fair
RenderTh-2982    0dNh5  774us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0dNh5  776us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0dNh5  778us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0dNh5  780us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0dNh5  782us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0dNh5  784us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0dNh5  787us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh6  789us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0dNh6  792us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0dNh6  793us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh5  796us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0dNh5  797us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0dNh5  799us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  802us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0dNh5  804us+: resched_task <-check_preempt_curr
RenderTh-2982    0dNh5  806us+: smp_send_reschedule <-resched_task
RenderTh-2982    0dNh5  808us+: gic_raise_softirq <-smp_send_reschedule
RenderTh-2982    0dNh5  812us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0dNh5  813us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh4  815us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0dNh4  817us+: __rcu_read_lock <-ttwu_stat
RenderTh-2982    0dNh4  820us+: __rcu_read_unlock <-ttwu_stat
RenderTh-2982    0dNh4  821us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0dNh4  823us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh3  825us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0dNh3  827us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh2  829us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1  831us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0dNh1  833us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0dNh1  835us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2  837us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2  839us+: __queue_work <-queue_work_on
RenderTh-2982    0dNh2  841us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0dNh2  842us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh3  844us+: get_cwq <-__queue_work
RenderTh-2982    0dNh3  846us+: insert_work <-__queue_work
RenderTh-2982    0dNh3  848us+: get_pool_nr_running <-insert_work
RenderTh-2982    0dNh3  849us+: wake_up_worker <-insert_work
RenderTh-2982    0dNh3  851us+: wake_up_process <-wake_up_worker
RenderTh-2982    0dNh3  853us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0dNh3  854us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0dNh3  856us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh4  858us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0dNh4  860us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0dNh4  862us+: __rcu_read_lock <-select_task_rq_fair
RenderTh-2982    0dNh4  864us+: source_load <-select_task_rq_fair
RenderTh-2982    0dNh4  866us+: target_load <-select_task_rq_fair
RenderTh-2982    0dNh4  868us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  869us+: effective_load <-select_task_rq_fair
RenderTh-2982    0dNh4  871us+: cpu_avg_load_per_task <-select_task_rq_fair
RenderTh-2982    0dNh4  874us+: idle_cpu <-select_task_rq_fair
RenderTh-2982    0dNh4  876us+: __rcu_read_unlock <-select_task_rq_fair
RenderTh-2982    0dNh4  878us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0dNh4  880us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0dNh4  881us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh5  883us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0dNh5  885us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  887us+: activate_task <-ttwu_activate
RenderTh-2982    0dNh5  889us+: enqueue_task <-activate_task
RenderTh-2982    0dNh5  891us+: update_rq_clock <-enqueue_task
RenderTh-2982    0dNh5  893us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0dNh5  895us+: update_curr <-enqueue_task_fair
RenderTh-2982    0dNh5  897us+: update_min_vruntime <-update_curr
RenderTh-2982    0dNh5  900us+: cpuacct_charge <-update_curr
RenderTh-2982    0dNh5  902us+: __rcu_read_lock <-cpuacct_charge
RenderTh-2982    0dNh5  903us+: __rcu_read_unlock <-cpuacct_charge
RenderTh-2982    0dNh5  905us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0dNh5  907us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0dNh5  909us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0dNh5  911us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0dNh5  913us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0dNh5  914us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0dNh5  917us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh6  919us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0dNh6  921us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0dNh6  923us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh5  925us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0dNh5  927us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0dNh5  929us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0dNh5  934us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0dNh5  936us+: check_preempt_wakeup <-check_preempt_curr
RenderTh-2982    0dNh5  938us+: update_curr <-check_preempt_wakeup
RenderTh-2982    0dNh5  940us+: wakeup_preempt_entity <-check_preempt_wakeup
RenderTh-2982    0dNh5  942us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0dNh5  944us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh4  946us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0dNh4  948us+: __rcu_read_lock <-ttwu_stat
RenderTh-2982    0dNh4  951us+: __rcu_read_unlock <-ttwu_stat
RenderTh-2982    0dNh4  953us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0dNh4  955us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh3  957us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0dNh3  959us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh2  961us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1  965us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0dNh1  967us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0dNh1  969us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1  972us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0dNh1  974us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0dNh1  976us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2  982us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0dNh2  984us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0dNh2  986us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1  987us+: irq_exit <-handle_IRQ
RenderTh-2982    0dNh1  989us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.2  991us+: idle_cpu <-irq_exit
RenderTh-2982    0dN.2  993us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0dN.2  995us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.1  997us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0dN.1  999us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0dN.1 1001us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0dN.1 1003us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0dN.1 1004us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0dN.1 1006us+: irq_enter <-handle_IRQ
RenderTh-2982    0dN.1 1008us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0dN.1 1010us+: add_preempt_count <-irq_enter
RenderTh-2982    0dNh1 1012us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0dNh1 1014us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0dNh1 1016us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0dNh1 1018us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0dNh1 1020us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1022us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1023us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0dNh2 1025us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1027us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0dNh1 1030us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1031us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0dNh1 1033us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0dNh1 1035us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0dNh1 1036us+: adreno_read <-adreno_regread
RenderTh-2982    0dNh1 1038us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1042us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0dNh1 1044us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0dNh1 1046us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1048us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1050us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1052us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0dNh1 1053us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0dNh1 1055us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1057us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1058us+: __queue_work <-queue_work_on
RenderTh-2982    0dNh2 1061us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0dNh2 1063us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh3 1064us+: get_cwq <-__queue_work
RenderTh-2982    0dNh3 1066us+: insert_work <-__queue_work
RenderTh-2982    0dNh3 1068us+: get_pool_nr_running <-insert_work
RenderTh-2982    0dNh3 1070us+: wake_up_worker <-insert_work
RenderTh-2982    0dNh3 1072us+: wake_up_process <-wake_up_worker
RenderTh-2982    0dNh3 1073us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0dNh3 1075us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0dNh3 1077us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0dNh4 1079us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0dNh4 1081us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh3 1083us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0dNh3 1084us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0dNh2 1087us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1088us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0dNh1 1090us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0dNh1 1092us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1095us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1097us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0dNh1 1098us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1100us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1102us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1104us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1106us+: irq_exit <-handle_IRQ
RenderTh-2982    0dNh1 1108us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.2 1110us+: idle_cpu <-irq_exit
RenderTh-2982    0dN.2 1112us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0dN.2 1113us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.1 1115us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0dN.1 1118us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0dN.1 1120us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0dN.1 1122us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0dN.1 1124us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0dN.1 1126us+: irq_enter <-handle_IRQ
RenderTh-2982    0dN.1 1127us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0dN.1 1129us+: add_preempt_count <-irq_enter
RenderTh-2982    0dNh1 1132us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0dNh1 1133us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0dNh1 1135us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0dNh1 1137us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0dNh1 1139us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1141us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1143us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0dNh2 1145us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1147us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0dNh1 1149us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1150us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0dNh1 1152us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0dNh1 1154us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0dNh1 1155us+: adreno_read <-adreno_regread
RenderTh-2982    0dNh1 1159us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1163us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0dNh1 1164us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0dNh1 1166us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1168us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1170us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1171us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0dNh1 1173us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0dNh1 1175us+: add_preempt_count <-queue_work
RenderTh-2982    0dNh2 1177us+: queue_work_on <-queue_work
RenderTh-2982    0dNh2 1179us+: sub_preempt_count <-queue_work
RenderTh-2982    0dNh1 1181us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0dNh1 1182us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0dNh1 1184us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0dNh1 1187us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0dNh1 1191us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0dNh1 1193us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0dNh2 1194us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1197us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0dNh2 1198us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0dNh1 1200us+: irq_exit <-handle_IRQ
RenderTh-2982    0dNh1 1202us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.2 1204us+: idle_cpu <-irq_exit
RenderTh-2982    0dN.2 1206us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0dN.2 1208us+: sub_preempt_count <-irq_exit
RenderTh-2982    0dN.1 1210us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0.N.1 1212us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0.N.. 1214us+: __schedule <-preempt_schedule
RenderTh-2982    0.N.. 1216us+: add_preempt_count <-__schedule
RenderTh-2982    0.N.1 1218us+: rcu_note_context_switch <-__schedule
RenderTh-2982    0.N.1 1220us+: rcu_sched_qs <-rcu_note_context_switch
RenderTh-2982    0dN.1 1222us+: rcu_preempt_qs <-rcu_note_context_switch
RenderTh-2982    0.N.1 1224us+: _raw_spin_lock_irq <-__schedule
RenderTh-2982    0dN.1 1225us+: add_preempt_count <-_raw_spin_lock_irq
RenderTh-2982    0dN.2 1227us+: update_rq_clock <-__schedule
RenderTh-2982    0dN.2 1229us+: put_prev_task_fair <-__schedule
RenderTh-2982    0dN.2 1231us+: update_curr <-put_prev_task_fair
RenderTh-2982    0dN.2 1233us+: calc_delta_mine <-update_curr
RenderTh-2982    0dN.2 1235us+: update_min_vruntime <-update_curr
RenderTh-2982    0dN.2 1237us+: cpuacct_charge <-update_curr
RenderTh-2982    0dN.2 1240us+: __rcu_read_lock <-cpuacct_charge
RenderTh-2982    0dN.2 1242us+: __rcu_read_unlock <-cpuacct_charge
RenderTh-2982    0dN.2 1244us+: __enqueue_entity <-put_prev_task_fair
RenderTh-2982    0dN.2 1246us+: pick_next_task_stop <-__schedule
RenderTh-2982    0dN.2 1248us+: pick_next_task_rt <-__schedule
RenderTh-2982    0dN.2 1250us+: dequeue_pushable_task <-pick_next_task_rt
RenderTh-2982    0d..2 1253us+: _raw_spin_unlock_irq <-__schedule
RenderTh-2982    0d..2 1256us+: gic_handle_irq <-__irq_svc
RenderTh-2982    0d..2 1259us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0d..2 1260us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0d..2 1262us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0d..2 1264us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0d..2 1265us+: irq_enter <-handle_IRQ
RenderTh-2982    0d..2 1267us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0d..2 1269us+: add_preempt_count <-irq_enter
RenderTh-2982    0d.h2 1271us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0d.h2 1272us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0d.h2 1274us+: handle_percpu_devid_irq <-generic_handle_irq
RenderTh-2982    0d.h2 1276us+: arch_timer_handler_cp15 <-handle_percpu_devid_irq
RenderTh-2982    0d.h2 1278us+: hrtimer_interrupt <-arch_timer_handler_cp15
RenderTh-2982    0d.h2 1280us+: ktime_get <-hrtimer_interrupt
RenderTh-2982    0d.h2 1282us+: arch_counter_read <-ktime_get
RenderTh-2982    0d.h2 1283us+: arch_counter_get_cntpct <-arch_counter_read
RenderTh-2982    0d.h2 1285us+: _raw_spin_lock <-hrtimer_interrupt
RenderTh-2982    0d.h2 1287us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h3 1289us+: __run_hrtimer <-hrtimer_interrupt
RenderTh-2982    0d.h3 1292us+: __remove_hrtimer <-__run_hrtimer
RenderTh-2982    0d.h3 1295us+: _raw_spin_unlock <-__run_hrtimer
RenderTh-2982    0d.h3 1296us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h2 1298us+: tick_sched_timer <-__run_hrtimer
RenderTh-2982    0d.h2 1300us+: ktime_get <-tick_sched_timer
RenderTh-2982    0d.h2 1302us+: arch_counter_read <-ktime_get
RenderTh-2982    0d.h2 1303us+: arch_counter_get_cntpct <-arch_counter_read
RenderTh-2982    0d.h2 1305us+: update_process_times <-tick_sched_timer
RenderTh-2982    0d.h2 1307us+: account_process_tick <-update_process_times
RenderTh-2982    0d.h2 1309us+: account_system_time <-account_process_tick
RenderTh-2982    0d.h2 1311us+: __rcu_read_lock <-account_system_time
RenderTh-2982    0d.h2 1313us+: __rcu_read_unlock <-account_system_time
RenderTh-2982    0d.h2 1315us+: run_local_timers <-update_process_times
RenderTh-2982    0d.h2 1316us+: hrtimer_run_queues <-run_local_timers
RenderTh-2982    0d.h2 1318us+: raise_softirq <-run_local_timers
RenderTh-2982    0d.h2 1320us+: __raise_softirq_irqoff <-raise_softirq
RenderTh-2982    0d.h2 1322us+: rcu_check_callbacks <-update_process_times
RenderTh-2982    0d.h2 1323us+: rcu_is_cpu_rrupt_from_idle <-rcu_check_callbacks
RenderTh-2982    0d.h2 1325us+: rcu_bh_qs <-rcu_check_callbacks
RenderTh-2982    0d.h2 1327us+: rcu_preempt_qs <-rcu_check_callbacks
RenderTh-2982    0d.h2 1329us+: rcu_pending <-rcu_check_callbacks
RenderTh-2982    0d.h2 1331us+: __rcu_pending <-rcu_pending
RenderTh-2982    0d.h2 1332us+: cpu_needs_another_gp <-__rcu_pending
RenderTh-2982    0d.h2 1335us+: __rcu_pending <-rcu_pending
RenderTh-2982    0d.h2 1337us+: cpu_needs_another_gp <-__rcu_pending
RenderTh-2982    0d.h2 1338us+: __rcu_pending <-rcu_pending
RenderTh-2982    0d.h2 1340us+: cpu_needs_another_gp <-__rcu_pending
RenderTh-2982    0d.h2 1342us+: raise_softirq <-rcu_check_callbacks
RenderTh-2982    0d.h2 1343us+: __raise_softirq_irqoff <-raise_softirq
RenderTh-2982    0d.h2 1345us+: printk_tick <-update_process_times
RenderTh-2982    0d.h2 1347us+: scheduler_tick <-update_process_times
RenderTh-2982    0d.h2 1349us+: _raw_spin_lock <-scheduler_tick
RenderTh-2982    0d.h2 1351us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h3 1353us+: update_rq_clock <-scheduler_tick
RenderTh-2982    0d.h3 1355us+: update_cpu_load <-scheduler_tick
RenderTh-2982    0d.h3 1357us+: sched_avg_update <-update_cpu_load
RenderTh-2982    0d.h3 1359us+: task_tick_rt <-scheduler_tick
RenderTh-2982    0d.h3 1360us+: update_curr_rt <-task_tick_rt
RenderTh-2982    0d.h3 1362us+: cpuacct_charge <-update_curr_rt
RenderTh-2982    0d.h3 1364us+: __rcu_read_lock <-cpuacct_charge
RenderTh-2982    0d.h3 1366us+: __rcu_read_unlock <-cpuacct_charge
RenderTh-2982    0d.h3 1368us+: sched_avg_update <-update_curr_rt
RenderTh-2982    0d.h3 1369us+: _raw_spin_lock <-update_curr_rt
RenderTh-2982    0d.h3 1371us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h4 1373us+: balance_runtime <-update_curr_rt
RenderTh-2982    0d.h4 1376us+: _raw_spin_unlock <-update_curr_rt
RenderTh-2982    0d.h4 1377us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h3 1380us+: _raw_spin_unlock <-scheduler_tick
RenderTh-2982    0d.h3 1381us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h2 1383us+: idle_cpu <-scheduler_tick
RenderTh-2982    0d.h2 1385us+: trigger_load_balance <-scheduler_tick
RenderTh-2982    0d.h2 1386us+: raise_softirq <-trigger_load_balance
RenderTh-2982    0d.h2 1388us+: __raise_softirq_irqoff <-raise_softirq
RenderTh-2982    0d.h2 1390us+: idle_cpu <-trigger_load_balance
RenderTh-2982    0d.h2 1392us+: __rcu_read_lock <-trigger_load_balance
RenderTh-2982    0d.h2 1394us+: __rcu_read_unlock <-trigger_load_balance
RenderTh-2982    0d.h2 1396us+: run_posix_cpu_timers <-update_process_times
RenderTh-2982    0d.h2 1398us+: hrtimer_forward <-tick_sched_timer
RenderTh-2982    0d.h2 1401us+: ktime_add_safe <-hrtimer_forward
RenderTh-2982    0d.h2 1403us+: ktime_add_safe <-hrtimer_forward
RenderTh-2982    0d.h2 1404us+: _raw_spin_lock <-__run_hrtimer
RenderTh-2982    0d.h2 1406us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.h3 1408us+: enqueue_hrtimer <-__run_hrtimer
RenderTh-2982    0d.h3 1411us+: _raw_spin_unlock <-hrtimer_interrupt
RenderTh-2982    0d.h3 1413us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.h2 1414us+: tick_program_event <-hrtimer_interrupt
RenderTh-2982    0d.h2 1416us+: clockevents_program_event <-tick_program_event
RenderTh-2982    0d.h2 1418us+: ktime_get <-clockevents_program_event
RenderTh-2982    0d.h2 1420us+: arch_counter_read <-ktime_get
RenderTh-2982    0d.h2 1422us+: arch_counter_get_cntpct <-arch_counter_read
RenderTh-2982    0d.h2 1423us+: arch_timer_set_next_event_cp15 <-clockevents_program_event
RenderTh-2982    0d.h2 1425us+: gic_eoi_irq <-handle_percpu_devid_irq
RenderTh-2982    0d.h2 1428us+: irq_exit <-handle_IRQ
RenderTh-2982    0d.h2 1429us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d..3 1431us+: __do_softirq <-irq_exit
RenderTh-2982    0d..3 1433us+: __local_bh_disable <-__do_softirq
RenderTh-2982    0d.s3 1436us+: gic_handle_irq <-__irq_svc
RenderTh-2982    0d.s3 1438us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0d.s3 1440us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0d.s3 1442us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0d.s3 1443us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0d.s3 1445us+: irq_enter <-handle_IRQ
RenderTh-2982    0d.s3 1447us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0d.s3 1449us+: add_preempt_count <-irq_enter
RenderTh-2982    0d.H3 1451us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0d.H3 1453us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0d.H3 1455us+: handle_percpu_devid_irq <-generic_handle_irq
RenderTh-2982    0d.H3 1457us+: arch_timer_handler_cp15 <-handle_percpu_devid_irq
RenderTh-2982    0d.H3 1459us+: gic_eoi_irq <-handle_percpu_devid_irq
RenderTh-2982    0d.H3 1461us+: irq_exit <-handle_IRQ
RenderTh-2982    0d.H3 1462us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s4 1465us+: idle_cpu <-irq_exit
RenderTh-2982    0d.s4 1467us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0d.s4 1469us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s3 1474us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0d.s3 1480us+: irq_find_mapping <-gic_handle_irq
RenderTh-2982    0d.s3 1485us+: irq_domain_legacy_revmap <-irq_find_mapping
RenderTh-2982    0d.s3 1488us+: handle_IRQ <-gic_handle_irq
RenderTh-2982    0d.s3 1490us+: perf_mon_interrupt_in <-handle_IRQ
RenderTh-2982    0d.s3 1498us+: irq_enter <-handle_IRQ
RenderTh-2982    0d.s3 1503us+: rcu_irq_enter <-irq_enter
RenderTh-2982    0d.s3 1509us+: add_preempt_count <-irq_enter
RenderTh-2982    0d.H3 1519us+: generic_handle_irq <-handle_IRQ
RenderTh-2982    0d.H3 1525us+: irq_to_desc <-generic_handle_irq
RenderTh-2982    0d.H3 1528us+: handle_fasteoi_irq <-generic_handle_irq
RenderTh-2982    0d.H3 1533us+: _raw_spin_lock <-handle_fasteoi_irq
RenderTh-2982    0d.H3 1535us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H4 1537us+: handle_irq_event <-handle_fasteoi_irq
RenderTh-2982    0d.H4 1539us+: _raw_spin_unlock <-handle_irq_event
RenderTh-2982    0d.H4 1540us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H3 1542us+: handle_irq_event_percpu <-handle_irq_event
RenderTh-2982    0d.H3 1545us+: kgsl_irq_handler <-handle_irq_event_percpu
RenderTh-2982    0d.H3 1547us+: adreno_irq_handler <-kgsl_irq_handler
RenderTh-2982    0d.H3 1548us+: a3xx_irq_handler <-adreno_irq_handler
RenderTh-2982    0d.H3 1550us+: adreno_regread <-a3xx_irq_handler
RenderTh-2982    0d.H3 1552us+: adreno_read <-adreno_regread
RenderTh-2982    0d.H3 1553us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0d.H3 1557us+: a3xx_cp_callback <-a3xx_irq_handler
RenderTh-2982    0d.H3 1558us+: queue_work <-a3xx_cp_callback
RenderTh-2982    0d.H3 1560us+: add_preempt_count <-queue_work
RenderTh-2982    0d.H4 1562us+: queue_work_on <-queue_work
RenderTh-2982    0d.H4 1564us+: __queue_work <-queue_work_on
RenderTh-2982    0d.H4 1566us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0d.H4 1567us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H5 1569us+: get_cwq <-__queue_work
RenderTh-2982    0d.H5 1571us+: insert_work <-__queue_work
RenderTh-2982    0d.H5 1573us+: get_pool_nr_running <-insert_work
RenderTh-2982    0d.H5 1574us+: wake_up_worker <-insert_work
RenderTh-2982    0d.H5 1576us+: wake_up_process <-wake_up_worker
RenderTh-2982    0d.H5 1578us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0d.H5 1580us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0d.H5 1581us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H6 1583us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0d.H6 1585us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H7 1587us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0d.H7 1589us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H6 1590us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0d.H6 1593us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0d.H6 1595us+: __rcu_read_lock <-select_task_rq_fair
RenderTh-2982    0d.H6 1597us+: source_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1599us+: target_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1601us+: effective_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1603us+: effective_load <-select_task_rq_fair
RenderTh-2982    0d.H6 1604us+: cpu_avg_load_per_task <-select_task_rq_fair
RenderTh-2982    0d.H6 1606us+: idle_cpu <-select_task_rq_fair
RenderTh-2982    0d.H6 1608us+: __rcu_read_unlock <-select_task_rq_fair
RenderTh-2982    0d.H6 1610us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0d.H6 1612us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0d.H6 1613us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H7 1615us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0d.H7 1617us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.H7 1618us+: activate_task <-ttwu_activate
RenderTh-2982    0d.H7 1620us+: enqueue_task <-activate_task
RenderTh-2982    0d.H7 1622us+: update_rq_clock <-enqueue_task
RenderTh-2982    0d.H7 1624us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0d.H7 1627us+: update_curr <-enqueue_task_fair
RenderTh-2982    0d.H7 1628us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0d.H7 1630us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0d.H7 1632us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0d.H7 1634us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0d.H7 1636us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0d.H7 1637us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0d.H7 1639us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H8 1642us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0d.H8 1644us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0d.H8 1646us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H7 1648us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0d.H7 1649us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0d.H7 1651us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.H7 1654us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0d.H7 1658us+: resched_task <-check_preempt_curr
RenderTh-2982    0d.H7 1660us+: smp_send_reschedule <-resched_task
RenderTh-2982    0d.H7 1662us+: gic_raise_softirq <-smp_send_reschedule
RenderTh-2982    0d.H7 1665us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0d.H7 1667us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H6 1668us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0d.H6 1670us+: __rcu_read_lock <-ttwu_stat
RenderTh-2982    0d.H6 1672us+: __rcu_read_unlock <-ttwu_stat
RenderTh-2982    0d.H6 1674us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0d.H6 1676us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H5 1678us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0d.H5 1680us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H4 1681us+: sub_preempt_count <-queue_work
RenderTh-2982    0d.H3 1683us+: adreno_dispatcher_schedule <-a3xx_cp_callback
RenderTh-2982    0d.H3 1685us+: queue_work <-adreno_dispatcher_schedule
RenderTh-2982    0d.H3 1686us+: add_preempt_count <-queue_work
RenderTh-2982    0d.H4 1688us+: queue_work_on <-queue_work
RenderTh-2982    0d.H4 1690us+: __queue_work <-queue_work_on
RenderTh-2982    0d.H4 1692us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0d.H4 1694us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H5 1695us+: get_cwq <-__queue_work
RenderTh-2982    0d.H5 1697us+: insert_work <-__queue_work
RenderTh-2982    0d.H5 1699us+: get_pool_nr_running <-insert_work
RenderTh-2982    0d.H5 1701us+: wake_up_worker <-insert_work
RenderTh-2982    0d.H5 1702us+: wake_up_process <-wake_up_worker
RenderTh-2982    0d.H5 1704us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0d.H5 1705us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0d.H5 1707us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.H6 1709us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0d.H6 1711us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H5 1712us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0d.H5 1714us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.H4 1716us+: sub_preempt_count <-queue_work
RenderTh-2982    0d.H3 1718us+: adreno_regwrite <-a3xx_irq_handler
RenderTh-2982    0d.H3 1720us+: kgsl_trace_regwrite <-adreno_regwrite
RenderTh-2982    0d.H3 1722us+: uncached_logk_pc_idx <-uncached_logk_pc
RenderTh-2982    0d.H3 1725us+: note_interrupt <-handle_irq_event_percpu
RenderTh-2982    0d.H3 1727us+: _raw_spin_lock <-handle_irq_event
RenderTh-2982    0d.H3 1728us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.H4 1730us+: gic_eoi_irq <-handle_fasteoi_irq
RenderTh-2982    0d.H4 1732us+: _raw_spin_unlock <-handle_fasteoi_irq
RenderTh-2982    0d.H4 1734us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.H3 1736us+: irq_exit <-handle_IRQ
RenderTh-2982    0d.H3 1738us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s4 1740us+: idle_cpu <-irq_exit
RenderTh-2982    0d.s4 1741us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0d.s4 1743us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d.s3 1745us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0..s3 1748us+: run_timer_softirq <-__do_softirq
RenderTh-2982    0..s3 1749us+: hrtimer_run_pending <-run_timer_softirq
RenderTh-2982    0..s3 1752us+: _raw_spin_lock_irq <-run_timer_softirq
RenderTh-2982    0d.s3 1754us+: add_preempt_count <-_raw_spin_lock_irq
RenderTh-2982    0d.s4 1757us+: _raw_spin_unlock_irq <-run_timer_softirq
RenderTh-2982    0..s4 1758us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0..s3 1761us+: delayed_work_timer_fn <-run_timer_softirq
RenderTh-2982    0..s3 1762us+: get_work_cwq <-delayed_work_timer_fn
RenderTh-2982    0..s3 1764us+: __queue_work <-delayed_work_timer_fn
RenderTh-2982    0..s3 1766us+: get_gcwq <-__queue_work
RenderTh-2982    0..s3 1768us+: _raw_spin_lock_irqsave <-__queue_work
RenderTh-2982    0d.s3 1769us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.s4 1771us+: get_cwq <-__queue_work
RenderTh-2982    0d.s4 1773us+: insert_work <-__queue_work
RenderTh-2982    0d.s4 1775us+: get_pool_nr_running <-insert_work
RenderTh-2982    0d.s4 1777us+: wake_up_worker <-insert_work
RenderTh-2982    0d.s4 1779us+: wake_up_process <-wake_up_worker
RenderTh-2982    0d.s4 1780us+: try_to_wake_up <-wake_up_process
RenderTh-2982    0d.s4 1782us+: _raw_spin_lock_irqsave <-try_to_wake_up
RenderTh-2982    0d.s4 1784us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.s5 1787us+: task_waking_fair <-try_to_wake_up
RenderTh-2982    0d.s5 1789us+: select_task_rq_fair <-try_to_wake_up
RenderTh-2982    0d.s5 1791us+: cpus_share_cache <-try_to_wake_up
RenderTh-2982    0d.s5 1793us+: _raw_spin_lock <-try_to_wake_up
RenderTh-2982    0d.s5 1795us+: add_preempt_count <-_raw_spin_lock
RenderTh-2982    0d.s6 1796us+: ttwu_do_activate.constprop.90 <-try_to_wake_up
RenderTh-2982    0d.s6 1798us+: ttwu_activate <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.s6 1800us+: activate_task <-ttwu_activate
RenderTh-2982    0d.s6 1801us+: enqueue_task <-activate_task
RenderTh-2982    0d.s6 1804us+: update_rq_clock <-enqueue_task
RenderTh-2982    0d.s6 1806us+: enqueue_task_fair <-enqueue_task
RenderTh-2982    0d.s6 1807us+: update_curr <-enqueue_task_fair
RenderTh-2982    0d.s6 1809us+: update_cfs_load <-enqueue_task_fair
RenderTh-2982    0d.s6 1811us+: account_entity_enqueue <-enqueue_task_fair
RenderTh-2982    0d.s6 1813us+: update_cfs_shares <-enqueue_task_fair
RenderTh-2982    0d.s6 1815us+: __enqueue_entity <-enqueue_task_fair
RenderTh-2982    0d.s6 1817us+: sched_update_nr_prod <-enqueue_task_fair
RenderTh-2982    0d.s6 1820us+: _raw_spin_lock_irqsave <-sched_update_nr_prod
RenderTh-2982    0d.s6 1821us+: add_preempt_count <-_raw_spin_lock_irqsave
RenderTh-2982    0d.s7 1824us+: nr_iowait_cpu <-sched_update_nr_prod
RenderTh-2982    0d.s7 1825us+: _raw_spin_unlock_irqrestore <-sched_update_nr_prod
RenderTh-2982    0d.s7 1827us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.s6 1829us+: wq_worker_waking_up <-ttwu_activate
RenderTh-2982    0d.s6 1831us+: kthread_data <-wq_worker_waking_up
RenderTh-2982    0d.s6 1833us+: ttwu_do_wakeup <-ttwu_do_activate.constprop.90
RenderTh-2982    0d.s6 1837us+: check_preempt_curr <-ttwu_do_wakeup
RenderTh-2982    0d.s6 1839us+: _raw_spin_unlock <-try_to_wake_up
RenderTh-2982    0d.s6 1840us+: sub_preempt_count <-_raw_spin_unlock
RenderTh-2982    0d.s5 1842us+: ttwu_stat <-try_to_wake_up
RenderTh-2982    0d.s5 1845us+: _raw_spin_unlock_irqrestore <-try_to_wake_up
RenderTh-2982    0d.s5 1847us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0d.s4 1849us+: _raw_spin_unlock_irqrestore <-__queue_work
RenderTh-2982    0..s4 1850us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0..s3 1852us+: _raw_spin_lock_irq <-run_timer_softirq
RenderTh-2982    0d.s3 1854us+: add_preempt_count <-_raw_spin_lock_irq
RenderTh-2982    0d.s4 1856us+: _raw_spin_unlock_irq <-run_timer_softirq
RenderTh-2982    0..s4 1857us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0..s3 1859us+: rcu_bh_qs <-__do_softirq
RenderTh-2982    0..s3 1861us+: run_rebalance_domains <-__do_softirq
RenderTh-2982    0..s3 1863us+: rebalance_domains <-run_rebalance_domains
RenderTh-2982    0..s3 1865us+: update_shares <-rebalance_domains
RenderTh-2982    0..s3 1868us+: __rcu_read_lock <-update_shares
RenderTh-2982    0..s3 1870us+: __rcu_read_unlock <-update_shares
RenderTh-2982    0..s3 1872us+: __rcu_read_lock <-rebalance_domains
RenderTh-2982    0..s3 1874us+: msecs_to_jiffies <-rebalance_domains
RenderTh-2982    0..s3 1876us+: load_balance <-rebalance_domains
RenderTh-2982    0..s3 1878us+: find_busiest_group <-load_balance
RenderTh-2982    0..s3 1881us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1883us+: target_load <-find_busiest_group
RenderTh-2982    0..s3 1885us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1888us+: update_group_power <-find_busiest_group
RenderTh-2982    0..s3 1889us+: msecs_to_jiffies <-update_group_power
RenderTh-2982    0..s3 1891us+: scale_rt_power <-update_group_power
RenderTh-2982    0..s3 1895us+: source_load <-find_busiest_group
RenderTh-2982    0..s3 1896us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1899us+: source_load <-find_busiest_group
RenderTh-2982    0..s3 1901us+: idle_cpu <-find_busiest_group
RenderTh-2982    0..s3 1904us+: __rcu_read_unlock <-rebalance_domains
RenderTh-2982    0..s3 1906us+: rcu_bh_qs <-__do_softirq
RenderTh-2982    0..s3 1907us+: rcu_process_callbacks <-__do_softirq
RenderTh-2982    0..s3 1910us+: __rcu_process_callbacks <-rcu_process_callbacks
RenderTh-2982    0..s3 1911us+: force_quiescent_state <-__rcu_process_callbacks
RenderTh-2982    0..s3 1913us+: rcu_process_gp_end.isra.23 <-__rcu_process_callbacks
RenderTh-2982    0..s3 1917us+: check_for_new_grace_period <-__rcu_process_callbacks
RenderTh-2982    0..s3 1918us+: cpu_needs_another_gp <-__rcu_process_callbacks
RenderTh-2982    0..s3 1920us+: __rcu_process_callbacks <-rcu_process_callbacks
RenderTh-2982    0..s3 1922us+: force_quiescent_state <-__rcu_process_callbacks
RenderTh-2982    0..s3 1924us+: rcu_process_gp_end.isra.23 <-__rcu_process_callbacks
RenderTh-2982    0..s3 1926us+: check_for_new_grace_period <-__rcu_process_callbacks
RenderTh-2982    0..s3 1927us+: cpu_needs_another_gp <-__rcu_process_callbacks
RenderTh-2982    0..s3 1929us+: __rcu_process_callbacks <-rcu_process_callbacks
RenderTh-2982    0..s3 1932us+: rcu_process_gp_end.isra.23 <-__rcu_process_callbacks
RenderTh-2982    0d.s3 1934us+: _raw_spin_trylock <-rcu_process_gp_end.isra.23
RenderTh-2982    0d.s3 1936us+: add_preempt_count <-_raw_spin_trylock
RenderTh-2982    0d.s4 1938us+: _raw_spin_unlock_irqrestore <-rcu_process_gp_end.isra.23
RenderTh-2982    0..s4 1939us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0..s3 1941us+: check_for_new_grace_period <-__rcu_process_callbacks
RenderTh-2982    0d.s3 1943us+: _raw_spin_trylock <-check_for_new_grace_period
RenderTh-2982    0d.s3 1944us+: add_preempt_count <-_raw_spin_trylock
RenderTh-2982    0d.s4 1946us+: _raw_spin_unlock_irqrestore <-check_for_new_grace_period
RenderTh-2982    0d.s4 1948us+: sub_preempt_count <-_raw_spin_unlock_irqrestore
RenderTh-2982    0..s3 1950us+: cpu_needs_another_gp <-__rcu_process_callbacks
RenderTh-2982    0..s3 1952us+: rcu_bh_qs <-__do_softirq
RenderTh-2982    0d.s3 1953us+: __local_bh_enable <-__do_softirq
RenderTh-2982    0d.s3 1955us+: sub_preempt_count <-__local_bh_enable
RenderTh-2982    0d..3 1957us+: idle_cpu <-irq_exit
RenderTh-2982    0d..3 1959us+: rcu_irq_exit <-irq_exit
RenderTh-2982    0d..3 1961us+: sub_preempt_count <-irq_exit
RenderTh-2982    0d..2 1962us+: perf_mon_interrupt_out <-handle_IRQ
RenderTh-2982    0...2 1965us+: sub_preempt_count <-_raw_spin_unlock_irq
RenderTh-2982    0...2 1968us : __schedule
RenderTh-2982    0...2 1970us :   2982:116:R ==> [000]   319: 83:R mdss_fb0
ftrace与trace_cmd自动化布署方式 trace_cmd ftrace使用 阅读详情

相关推荐

Ubuntu实时内核部署与深度调优:从延迟尖峰到稳定低延迟的实战指南

在追求极致性能的实时计算领域,Linux内核的调度策略是决定系统响应确定性的核心。通用Linux内核采用的完全公平调度器(CFS)虽然保证了整体吞吐量和公平性,但其非抢占式的设计内核段和不可预测的中断处理,容易导致关键任务被延迟,形成影响系统稳定性的延迟尖峰。为解决这一问题,Linux实时内核(PREEMPT_RT)通过将大部分内核代码变为可抢占式,并引入优先级继承等机制,从根本上提升了系统的实时响应能力。其技术价值在于,能够在标准的通用硬件和操作系统上,为高频交易、工业自动化、机器人控制等场景提供微秒级的

weixin_34413802的博客 395

使用 Ftrace 分析函数性能

作者简介:伟林,中年码农,从事过电信、手机、安全、芯片等行业,目前依旧从事Linux方向开发工作,个人爱好Linux相关知识分享。0.背景ftrace的功能非常强大,可以在系统的各个关键点上采集数据用以追踪系统的运行情况。既支持预设的静态插桩点(trace event),也支持每个函数的动态插桩(function tracer)。还可以利用动态插桩来测量函数的执行时间(f...

Peter的专栏 1189

【实时Linux核心技术:从概念到实战】06:编写你的第一个实时线程:API详解与注意事项

在汽车电子、工业控制与机器人等对确定性要求极高的领域,Linux默认调度策略难以满足硬实时需求。本文手把手教你基于POSIX实时扩展编写具备微秒级低抖动的实时线程:从正确配置`pthread`属性(含极易被忽略的`PTHREAD_EXPLICIT_SCHED`陷阱)、采用`clock_nanosleep`实现无漂移周期控制,到`mlockall`防页面换出、CPU亲和性绑核、堆栈预热等系统级调优。文章附完整可运行代码及`cyclictest`性能对比,并梳理9个线上高频致命错误与规避方案,助你构建稳定可信的

专注于人工智能、软件开发、工控自动化、工厂数字化及智能化等领域,希望和大家共同进步! 258

linux trace机制分析 - 2.ftrace使用

文章目录简介使用 简介 ftrace通常被理解为是一个linux内置的函数追踪器,可以很方便的帮助开发者内核函数的运行情况,通常被用来在分析函数延迟和系统性能。实际上ftrace不仅仅是一个函数追踪器,更准确的说ftrace是一个综合了多种手段的trace框架。比如分析延迟跟踪检查、中断的禁用与执行、任务的调度与抢占等等。 ftrace一个最常用的用法是事件追踪,通过内核中几百个event point 可以通过tracefs查看内核的运行情况。 使用 在内核编译配置中开启ftracer Kernel h

BaiTianio的博客 2897

手把手教你用STM32H743+FreeRTOS+LWIP搭建物联网网关(附完整工程)

本文详细介绍了如何使用STM32H743微控制器结合FreeRTOS实时操作系统和LWIP协议栈构建高性能物联网网关。内容涵盖硬件配置、FreeRTOS任务设计、LWIP协议栈优化及物联网协议集成,提供完整工程示例,助力开发者快速实现稳定可靠的网络通信枢纽。

weixin_42623010的博客 503

成分数据的闭合效应对地球化学数据的影响及其解决方案

封闭效应 (Closure Effect),也称为常数和效应 (Constant Sum Effect),是地球化学数据分析中一个需要特别注意的统计现象。它的产生是由于我们通常使用成分数据 (Compositional Data)百分比 (%)百万分率 (ppm)由于这些数据表示的是各组分占总体的比例,它们的总和必然是一个常数 (例如 100% 或 1,000,000 ppm)。这意味着每个元素的含量不是独立的,而是与其他元素的含量相互制约。改变其中一个元素的含量必然会影响其他元素的相对比例。

qq_59349063的博客 1538

linux内核:ftrace——追踪内核行为

描述了ftrace的使用方法,trace-cmd常用方法

Leo-夜空的博客 2325

OSGEarth三维可视化避坑指南:如何高效加载SHP/TIF地理数据?

本文详细介绍了OSGEarth在三维地球可视化中高效加载SHP/TIF地理数据的实战策略,包括空间参考系配置、性能优化、二三维联动等关键技巧。通过具体案例和代码示例,帮助开发者解决坐标偏移、性能卡顿等常见问题,提升三维GIS开发效率。

weixin_29216957的博客 222

linux trace学习(二)——trace使用

备注:1. Kernel版本:4.19.1232. 使用工具:Source Insight 4.0。

枫潇潇 9614

IMU卡尔曼滤波方法详细介绍

本文介绍了基于IMU和卡尔曼滤波的状态估计方法。系统采用9维状态向量(位置、速度、加速度计零偏),通过牛顿运动学建立状态方程,并利用外部位置观测进行校正。详细推导了状态转移矩阵和控制输入矩阵的形式,阐述了预测和更新的计算步骤。同时说明了过程噪声Q和观测噪声R的设置原则,其中Q反映模型不确定性,R取决于传感器精度。该方法可有效融合IMU测量与外部观测数据,实现位置、速度和偏置的联合估计。

haing2019的博客 757

AVM环视:系统搭建整体流程

文章目录前言一、畸变矫正鱼眼相机二、透视变换三、图像拼接四、图像融合总结 前言 由于正在做AVM环视,所以记录一下整个过程,归纳为一个系列文章,并对期间遇到的错误进行归纳整理。本文主要概括了AVM环视的大致流程和逻辑,不涉及代码部分,仅供思路参考,其中涉及的具体技术会在后期整理为小章节进行分析。 一、畸变矫正 鱼眼相机 做环视基本上使用到的是鱼眼照片,因为鱼眼相机的视角范围可以达到220°到230°,所以一个相机就可以采集到一个面的全部信息,例如汽车具有四个面,那么在汽车的前后左右分别安装一个鱼眼相机.

qq_34468890的博客 6536

linux内核内置trace工具使用简介

使用内核内置的trace工具,可以跟踪具体某个进程的内核函数调用过程,并且可以统计到每个函数的运行时间,对优化系统性能有一定的帮助 内核trace工具使用方法如下: 1.修改内核配置,支持trace Kernel hacking  --->  [*] Tracers  --->[*]   Kernel Function Tracer                              

专注于嵌入式开发 1万+

FM1288回声消除芯片调试方案 - 综述

专栏文章概览

优快云mianfeixiazai001的博客 1714

kernel ftrace使用示例

使能 kernel ftrace 需要打开下面编译宏 CONFIG_FTRACE=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_FUNCTION_TRACER=y CONFIG_FUNCTION_GRAPH_TRACER=y CONFIG_IRQSOFF_TRACER=y CONFIG_SCHED_TRACER=y CONFIG_ENABLE_DE

weixin_40419181的博客 701

白话机器学习算法理论+实战番外篇之Xgboost

1. 写在前面 如果想从事数据挖掘或者机器学习的工作,掌握常用的机器学习算法是非常有必要的,在这简单的先捋一捋, 常见的机器学习算法: 监督学习算法:逻辑回归,线性回归,决策树,朴素贝叶斯,K近邻,支持向量机,集成算法Adaboost等 无监督算法:聚类,降维,关联规则, PageRank等 我前面已经尝试用最白话的语言完成了一个白话机器学习算法理论+实战系列,这个系列已经基本包含了上面这些算...

Miracle8070 1万+

Sigrity System Explorer ViaWizard模式进行过孔建模和仿真分析操作指导

点击TVac_lower tdvac_lower,自动跳转到2D时域曲线,查看最小低电平穿过VIL(ac)max电压的延时,结果如下。点击TVac_high tdvac_high,自动跳转到2D时域曲线,查看最小高电平穿过VIH(ac)min电压的延时,结果如下。点击max Vix_Rise,自动跳转到2D时域曲线,查看上升边PN交叉点与参考电压VREF(dc)的偏差,结果如下。Threshold:阈值,软件自带3个模板,default,Tr01,Tr02,本例选择Default。

weixin_54787054的博客 571

Linux内核深度剖析:四层动态切片实战方法论

Linux内核不是黑箱,而是可观测、可干预的精密系统。理解其核心机制需跨越源码与运行态、抽象与硬件、配置与行为三大断层。本文基于真实云原生故障案例(如soft lockup、P99延迟抖动),聚焦内核剖析与动态观测两大关键技术能力,解析启动流程、调度器行为、内存映射和中断生命周期四大可观测切面。通过QEMU+GDB单步跟踪、ftrace函数级追踪、page_owner内存溯源、trace-cmd中断时序分析等工程化手段,将内核从理论概念转化为终端可验证的实时状态。适用于性能优化、SRE排障与嵌入式移植等一线

weixin_33701617的博客 382

VB摄像头监控

该VB模块可对摄像头捕获的画面进行录像等,可在创建的窗口进行控制

Linux-RT核心原理与硬实时系统搭建实战

实时操作系统(RTOS)的本质是时间行为的可预测性与确定性,而非单纯追求执行速度。Linux-RT通过PREEMPT_RT补丁重构内核抢占机制、中断处理和调度策略,将通用Linux转化为支持硬实时(Hard Real-Time)的执行环境——其关键能力包括微秒级中断延迟、全可抢占内核、优先级继承及CPU亲和性控制。该技术广泛应用于工业自动化(如EtherCAT主站)、电力保护、实时图像处理与FPGA协同边缘AI等对时间约束极为严苛的场景。当系统要求控制周期≤1ms且抖动容忍度<1%时,Linux-RT成为不

weixin_34268169的博客 416

RU DOS版、EFI版32bit/64bit BIOS常用工具

DOS版、EFI版的RU工具,可用来查看IO、MEM、PCI、SMBIOS等资源。亲测在EFI SHELL下可用。

win10 跳过U8环境检测的IE WebControls.rar

win10 安装u8的时候 提示IE WebControls 不通过,这个工具可以欺骗u8的检查。 跳过U8环境检测的IE WebControls。提示导入失败后,重新检测通过。

Linux内核进程唤醒机制:wake_up与wake_up_process详解

进程唤醒是操作系统调度的核心机制,直接影响系统响应速度和吞吐量。Linux内核通过wait_queue实现高效的进程休眠/唤醒管理,其中wake_up()和wake_up_process()是最关键的唤醒接口。wake_up_process实现精准的单进程唤醒,适合实时任务调度;而wake_up则采用广播模式唤醒整个等待队列,常见于IO完成处理。理解其底层通过try_to_wake_up()的通用路径(包含内存屏障、自旋锁和调度类回调)对开发高性能驱动和系统程序至关重要。在数据库连接池等实际场景中,合理选择

weixin_30340617的博客 398

假人有限元模型

假人模型,车辆碰撞假人有限元模型,适合做乘员安全的研究者利用

Linux内核实时化:低延迟与可抢占补丁的原理、对比与工程实践

实时操作系统(RTOS)的核心在于提供确定性的任务调度与极低的响应延迟,以满足工业控制、汽车电子等对时序要求严苛的场景。其技术原理主要围绕如何保证高优先级任务能够及时抢占CPU执行,从而将内核延迟控制在微秒级。这一技术价值在于,它使得通用、生态丰富的Linux系统能够应用于硬实时领域,极大地拓展了其应用边界。在嵌入式开发中,面对实时性需求,开发者常需在内核层面进行优化,其中低延迟补丁(Low-Latency Patch)与可抢占内核补丁(Preemptable Kernel Patch)是两种经典的基础方案

weixin_33895695的博客 376

SoC到操作系统五层架构:嵌入式系统全栈实操指南

现代嵌入式系统并非简单的硬件+软件堆叠,而是由SoC、固件、设备驱动、内核和操作系统构成的五层协同体系。这一分层本质是对物理世界不确定性(如温漂、时序偏差、协议歧义)的逐级防御设计,每层承担特定维度的抽象与管控责任。SoC已演进为可编程硬件平台,其内部NPU、VPU、ISP等单元支持寄存器级功能重构;固件则超越启动代码角色,成为ACPI表生成、电源管理、安全服务等关键能力的提供者。理解该架构对解决真实工程问题至关重要——例如0.3%的固件时钟偏差可导致SD卡初始化失败率飙升至17%,而PCIe热插拔异常往往

weixin_30627341的博客 389

Linux 内核 RCU CPU Stall 检测器完全指南:成因、参数调优与 “Splat“ 日志解读

本文基于 Linux 内核官方文档 [Documentation/RCU/stallwarn.rst](https://link.gitcode.com/i/f9f073a2cf5fc3552414240b38780a64) 编写,完整覆盖 RCU CPU stall 警告的触发成因、`rcupdate` 模块的调优参数(Kconfig 与 sysfs 双层机制)、以及各类内核日志 "splat"

gitblog_00152的博客 263

OS17.【Linux】进程基础知识(1)

介绍Linux进程的基本概念和管理方法。主要内容包括:1.进程的浅层定义 - 正在运行的程序;2.查看进程的多种方法(ps、top、/proc目录);3.进程管理实践(查看、终止进程);4.进程控制块(PCB)的组成和作用;5.Linux内核中task_struct结构体的详细解析。文章通过代码示例和命令演示,帮助读者理解进程标识符(PID)、工作目录、程序计数器等核心概念,并深入分析了操作系统如何通过PCB组织和管理进程。最后提供了Linux内核源代码中task_struct的完整定义,为后续学

zhangcoder的博客 记录学习轨迹~ 1042

中断上半部响应时间优化:从测量到根因定位的完整指南

中断响应时间是衡量实时系统性能的关键指标之一,其本质是从硬件中断信号到达中断控制器到CPU执行注册处理函数之间的延迟,与中断处理时间有着本质区别。理解中断屏蔽窗口、锁竞争、中断控制器配置等因素对响应时间的影响,是进行性能优化的前提。在多核嵌入式系统中,合理的irq affinity配置、精简临界区、以及选择适当的上下半部机制,能有效降低p99与max延迟,避免FIFO溢出和存储超时。本文从测量手段到根因定位,结合实际案例,系统阐述影响中断上半部响应时间的典型瓶颈,并给出可落地的优化方案。

weixin_34234829的博客 455

linux_2.6.35_wait和wakeup源码阅读

写在这里只当是为了备份一下,只是匆匆的浏览了一下源码,如果您想获得些什么深刻的东西,这个肯定是没有了,就是流水帐一个。 正文: 在wait.h中,可以看到定义了许多wait_up相关的宏 #define wake_up(x)            __wake_up(x, TASK_NORMAL, 1, NULL) #define wake_up_nr(x, nr)

1260

【linux kernel】linux内核如何唤醒线程

linux内核如何唤醒线程 //本文代码片段出自linux内核版本:4.1.15 linux内核唤醒线程主要使用wake_up_process()。 一、wake_up_process()分析 在linux内核中,唤醒线程由wake_up_process()函数实现。其定义在(/kernel/sched/core.c): int wake_up_process(struct task_struct *p) { WARN_ON(task_is_stopped_or_traced(p)); return t

iriczhao的博客 4034
上一篇: Linux Kernel threaded irq
下一篇: MIPI-CSI2
windtakers
博客等级 码龄17年 94粉丝 197原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值