1 Kprobe-based Event Tracer
2 =========================
4 Documentation is written by Masami Hiramatsu
9 This tracer is similar to the events tracer which is based on Tracepoint
10 infrastructure. Instead of Tracepoint, this tracer is based on kprobes(kprobe
11 and kretprobe). It probes anywhere where kprobes can probe(this means, all
12 functions body except for __kprobes functions).
14 Unlike the function tracer, this tracer can probe instructions inside of
15 kernel functions. It allows you to check which instruction has been executed.
17 Unlike the Tracepoint based events tracer, this tracer can add and remove
18 probe points on the fly.
20 Similar to the events tracer, this tracer doesn't need to be activated via
21 current_tracer, instead of that, just set probe points via
22 /sys/kernel/debug/tracing/kprobe_events. And you can set filters on each
23 probe events via /sys/kernel/debug/tracing/events/kprobes/<EVENT>/filter.
26 Synopsis of kprobe_events
27 -------------------------
28 p[:EVENT] SYMBOL[+offs]|MEMADDR [FETCHARGS] : Set a probe
29 r[:EVENT] SYMBOL[+0] [FETCHARGS] : Set a return probe
31 EVENT : Event name. If omitted, the event name is generated
32 based on SYMBOL+offs or MEMADDR.
33 SYMBOL[+offs] : Symbol+offset where the probe is inserted.
34 MEMADDR : Address where the probe is inserted.
36 FETCHARGS : Arguments. Each probe can have up to 128 args.
37 %REG : Fetch register REG
38 sN : Fetch Nth entry of stack (N >= 0)
39 sa : Fetch stack address.
40 @ADDR : Fetch memory at ADDR (ADDR should be in kernel)
41 @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
42 aN : Fetch function argument. (N >= 0)(*)
43 rv : Fetch return value.(**)
44 ra : Fetch return address.(**)
45 +|-offs(FETCHARG) : fetch memory at FETCHARG +|- offs address.(***)
47 (*) aN may not correct on asmlinkaged functions and at the middle of
49 (**) only for return probe.
50 (***) this is useful for fetching a field of data structures.
53 Per-Probe Event Filtering
54 -------------------------
55 Per-probe event filtering feature allows you to set different filter on each
56 probe and gives you what arguments will be shown in trace buffer. If an event
57 name is specified right after 'p:' or 'r:' in kprobe_events, the tracer adds
58 an event under tracing/events/kprobes/<EVENT>, at the directory you can see
59 'id', 'enabled', 'format' and 'filter'.
62 You can enable/disable the probe by writing 1 or 0 on it.
65 This shows the format of this probe event. It also shows aliases of arguments
66 which you specified to kprobe_events.
69 You can write filtering rules of this event. And you can use both of aliase
70 names and field names for describing filters.
73 This shows the id of this probe event.
77 You can check the total number of probe hits and probe miss-hits via
78 /sys/kernel/debug/tracing/kprobe_profile.
79 The first column is event name, the second is the number of probe hits,
80 the third is the number of probe miss-hits.
85 To add a probe as a new event, write a new definition to kprobe_events
88 echo p:myprobe do_sys_open a0 a1 a2 a3 > /sys/kernel/debug/tracing/kprobe_events
90 This sets a kprobe on the top of do_sys_open() function with recording
91 1st to 4th arguments as "myprobe" event.
93 echo r:myretprobe do_sys_open rv ra >> /sys/kernel/debug/tracing/kprobe_events
95 This sets a kretprobe on the return point of do_sys_open() function with
96 recording return value and return address as "myretprobe" event.
97 You can see the format of these events via
98 /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format.
100 cat /sys/kernel/debug/tracing/events/kprobes/myprobe/format
104 field:unsigned short common_type; offset:0; size:2;
105 field:unsigned char common_flags; offset:2; size:1;
106 field:unsigned char common_preempt_count; offset:3; size:1;
107 field:int common_pid; offset:4; size:4;
108 field:int common_tgid; offset:8; size:4;
110 field: unsigned long ip; offset:16;tsize:8;
111 field: int nargs; offset:24;tsize:4;
112 field: unsigned long arg0; offset:32;tsize:8;
113 field: unsigned long arg1; offset:40;tsize:8;
114 field: unsigned long arg2; offset:48;tsize:8;
115 field: unsigned long arg3; offset:56;tsize:8;
117 alias: a0; original: arg0;
118 alias: a1; original: arg1;
119 alias: a2; original: arg2;
120 alias: a3; original: arg3;
122 print fmt: "%lx: 0x%lx 0x%lx 0x%lx 0x%lx", ip, arg0, arg1, arg2, arg3
125 You can see that the event has 4 arguments and alias expressions
128 echo > /sys/kernel/debug/tracing/kprobe_events
130 This clears all probe points. and you can see the traced information via
131 /sys/kernel/debug/tracing/trace.
133 cat /sys/kernel/debug/tracing/trace
136 # TASK-PID CPU# TIMESTAMP FUNCTION
138 <...>-1447 [001] 1038282.286875: do_sys_open+0x0/0xd6: 0x3 0x7fffd1ec4440 0x8000 0x0
139 <...>-1447 [001] 1038282.286878: sys_openat+0xc/0xe <- do_sys_open: 0xfffffffffffffffe 0xffffffff81367a3a
140 <...>-1447 [001] 1038282.286885: do_sys_open+0x0/0xd6: 0xffffff9c 0x40413c 0x8000 0x1b6
141 <...>-1447 [001] 1038282.286915: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a
142 <...>-1447 [001] 1038282.286969: do_sys_open+0x0/0xd6: 0xffffff9c 0x4041c6 0x98800 0x10
143 <...>-1447 [001] 1038282.286976: sys_open+0x1b/0x1d <- do_sys_open: 0x3 0xffffffff81367a3a
146 Each line shows when the kernel hits a probe, and <- SYMBOL means kernel
147 returns from SYMBOL(e.g. "sys_open+0x1b/0x1d <- do_sys_open" means kernel
148 returns from do_sys_open to sys_open+0x1b).