1 Kprobe-based Event Tracer
2 =========================
4 Documentation is written by Masami Hiramatsu
9 This tracer is similar to the events tracer which is based on Tracepoint
10 infrastructure. Instead of Tracepoint, this tracer is based on kprobes(kprobe
11 and kretprobe). It probes anywhere where kprobes can probe(this means, all
12 functions body except for __kprobes functions).
14 Unlike the function tracer, this tracer can probe instructions inside of
15 kernel functions. It allows you to check which instruction has been executed.
17 Unlike the Tracepoint based events tracer, this tracer can add and remove
18 probe points on the fly.
20 Similar to the events tracer, this tracer doesn't need to be activated via
21 current_tracer, instead of that, just set probe points via
22 /sys/kernel/debug/tracing/kprobe_events. And you can set filters on each
23 probe events via /sys/kernel/debug/tracing/events/kprobes/<EVENT>/filter.
26 Synopsis of kprobe_events
27 -------------------------
28 p[:EVENT] SYMBOL[+offs|-offs]|MEMADDR [FETCHARGS] : Set a probe
29 r[:EVENT] SYMBOL[+0] [FETCHARGS] : Set a return probe
32 SYMBOL[+offs|-offs] : Symbol+offset where the probe is inserted.
33 MEMADDR : Address where the probe is inserted.
35 FETCHARGS : Arguments.
36 %REG : Fetch register REG
37 sN : Fetch Nth entry of stack (N >= 0)
38 sa : Fetch stack address.
39 @ADDR : Fetch memory at ADDR (ADDR should be in kernel)
40 @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
41 aN : Fetch function argument. (N >= 0)(*)
42 rv : Fetch return value.(**)
43 ra : Fetch return address.(**)
44 +|-offs(FETCHARG) : fetch memory at FETCHARG +|- offs address.(***)
46 (*) aN may not correct on asmlinkaged functions and at the middle of
48 (**) only for return probe.
49 (***) this is useful for fetching a field of data structures.
52 Per-Probe Event Filtering
53 -------------------------
54 Per-probe event filtering feature allows you to set different filter on each
55 probe and gives you what arguments will be shown in trace buffer. If an event
56 name is specified right after 'p:' or 'r:' in kprobe_events, the tracer adds
57 an event under tracing/events/kprobes/<EVENT>, at the directory you can see
58 'id', 'enabled', 'format' and 'filter'.
61 You can enable/disable the probe by writing 1 or 0 on it.
64 It shows the format of this probe event. It also shows aliases of arguments
65 which you specified to kprobe_events.
68 You can write filtering rules of this event. And you can use both of aliase
69 names and field names for describing filters.
74 To add a probe as a new event, write a new definition to kprobe_events
77 echo p:myprobe do_sys_open a0 a1 a2 a3 > /sys/kernel/debug/tracing/kprobe_events
79 This sets a kprobe on the top of do_sys_open() function with recording
80 1st to 4th arguments as "myprobe" event.
82 echo r:myretprobe do_sys_open rv ra >> /sys/kernel/debug/tracing/kprobe_events
84 This sets a kretprobe on the return point of do_sys_open() function with
85 recording return value and return address as "myretprobe" event.
86 You can see the format of these events via
87 /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format.
89 cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format
93 field:unsigned short common_type; offset:0; size:2;
94 field:unsigned char common_flags; offset:2; size:1;
95 field:unsigned char common_preempt_count; offset:3; size:1;
96 field:int common_pid; offset:4; size:4;
97 field:int common_tgid; offset:8; size:4;
99 field: unsigned long ip; offset:16;tsize:8;
100 field: int nargs; offset:24;tsize:4;
101 field: unsigned long arg0; offset:32;tsize:8;
102 field: unsigned long arg1; offset:40;tsize:8;
103 field: unsigned long arg2; offset:48;tsize:8;
104 field: unsigned long arg3; offset:56;tsize:8;
106 alias: a0; original: arg0;
107 alias: a1; original: arg1;
108 alias: a2; original: arg2;
109 alias: a3; original: arg3;
111 print fmt: "%lx: 0x%lx 0x%lx 0x%lx 0x%lx", ip, arg0, arg1, arg2, arg3
114 You can see that the event has 4 arguments and alias expressions
117 echo > /sys/kernel/debug/tracing/kprobe_events
119 This clears all probe points. and you can see the traced information via
120 /sys/kernel/debug/tracing/trace.
122 cat /sys/kernel/debug/tracing/trace
125 # TASK-PID CPU# TIMESTAMP FUNCTION
127 <...>-1447 [001] 1038282.286875: do_sys_open+0x0/0xd6: 0x3 0x7fffd1ec4440 0x8000 0x0
128 <...>-1447 [001] 1038282.286878: sys_openat+0xc/0xe <- do_sys_open: 0xfffffffffffffffe 0xffffffff81367a3a
129 <...>-1447 [001] 1038282.286885: do_sys_open+0x0/0xd6: 0xffffff9c 0x40413c 0x8000 0x1b6
130 <...>-1447 [001] 1038282.286915: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a
131 <...>-1447 [001] 1038282.286969: do_sys_open+0x0/0xd6: 0xffffff9c 0x4041c6 0x98800 0x10
132 <...>-1447 [001] 1038282.286976: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a
135 Each line shows when the kernel hits a probe, and <- SYMBOL means kernel
136 returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel
137 returns from do_sys_open to sys_open+0x1b).