Systemtap does't show all the local variables in a kernel function

Question

I'm trying to debug some code in the kernel using systemtap. I need to print the value of local variables in that function but it looks like systemtap can only see the function arguments and not the local variables defined in that function. Here is my probe script.

probe kernel.function("tcp_write_xmit") {
    if( execname() == "bw_client"){
            printf(
               "tcp_write_xmit skb len %d\n",
               $skb
            );
    }
}

When I run this, I get the following error

semantic error: failed to retrieve location attribute for 'skb'
[man error::dwarf] (dieoffset: 0x5bd30b4): identifier 
'$skb' at /home/cca-user/systaptest/txprobe.stp:37:6
source:                    $skb
                           ^
Pass 2: analysis failed.  [man error::pass2]

However the function tcp_write_xmit clearly has skb

Using the -L option to print available variables gives me 5 variables which are the function arguments but none of the local variables are seen.

root@i-sahmed-node2: /mnt/linux-3.13.0 # stap -L
'kernel.function("tcp_write_xmit")'
kernel.function("tcp_write_xmit@/build/buildd/linux-3.13.0/net/ipv4
/tcp_output.c:1832") $sk:struct sock* $mss_now:unsigned int $nonagle:int  
$push_one:int $gfp:gfp_t

Here is the kernel and the systemtap version that I am running

root@i-sahmed-node2: /mnt/linux-3.13.0 # stap --version
Systemtap translator/driver (version 2.3/0.158, Debian version 2.3-1ubuntu1 (trusty))
Copyright (C) 2005-2013 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 NSS TR1_UNORDERED_MAP NLS

root@i-sahmed-node2: /mnt/linux-3.13.0 # uname -a
Linux i-sahmed-node2 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Any Ideas?

Answer 1

You are using .function probe. In this probe only parameters passed to the functions are visible. If you need to examine a local variable then you need to use .statement probe.

More info about probing

Answer 2

If you need to examine a local variable then you need to use .statement probe. An example is given as bellow:

# script
$ cat trace_statement.stp
probe kernel.statement("*@kernel/events/core.c:11758") {
    printf("func name: %s\n\n", ppfunc())
    printf("local variables: %s\n\n", $$locals)
    printf("pretty print pmu structure: %s\n\n", $pmu$)
    //printf("%s\n", kernel_string(@var("pmu")->name))
    printf("pmu name field: %s\n\n", $pmu->name$)
}


# output
$ stap trace_statement.stp

func name: __do_sys_perf_event_open

local variables: group_leader=? output_event=0x0 event=0xffff001803fa6000 sibling=? attr={...} ctx=? gctx=? event_file=0x0 group={...} task=? pmu=0xffff0008025d6000 event_fd=0x3 move_group=0x0 err=? f_flags=0x80002 cgroup_fd=?

pretty print pmu structure: {.entry={...}, .module=0x0, .dev=0xffff0008025d6c00, .attr_groups=0xffff0008025d6270, .attr_update=0x0, .name="armv8_pmuv3", .type=8, .capabilities=64, .pmu_disable_count=0xffff8000113c6628, .pmu_cpu_context=0xffff8000113c6630, .exclusive_cnt={...}, .task_ctx_nr=0, .hrtimer_interval_ms=4, .nr_addr_filters=0, .pmu_enable=0xffff8000108c9f30, .pmu_disable=0xffff8000108c9e70, .event_init=0xffff8000108c9c64, .event_mapped=0x0, .event_unmapped=0x0, .add=0xffff8000108ca1c0, .del=0xffff8000108ca600, .start=0xffff8

pmu name field: "armv8_pmuv3"

probe kernel.statement("*@kernel/events/core.c:11758") probes kernel file core.c in line 11756.

$$vars includes each variable in scope at the probe point and $$locals expands to a subset of $$vars containing only the local variables.

We can access the local variable pmu by its name directly. The $ suffix could be used to pretty print the data structure.

The -> operator can be chained to look at data structures contained within data structures and follow pointers to other data structures. Therefore we use $pmu->name to access name filed of the data structure.

You can refer to https://sourceware.org/systemtap/SystemTap_Beginners_Guide/ to see more details.

Answer 3

Try stap -L 'kernel.statement("tcp_write_xmit@*:*")' to see what statements and variables at each are still retained in your debuginfo. Ensure your kernel doesn't hamstring gcc with harmful stuff like KCFLAGS += -fno-var-tracking-assignments.