CentOS 5.1

GNU/Linux	CentOS 5.1
	lket(5)

LKET

NAME
DESCRIPTION
BINARY TRACING
EVENT REGISTER
CUSTOMIZED TRACE DATA
EXAMPLES
EVENT HOOKS AND TRACE DATA FORMAT
SEE ALSO

NAME

LKET − Linux Kernel Event Trace tool based on SystemTap

DESCRIPTION

The Linux Kernel Event Trace (LKET) tool is an extension to the tapsets library available on SystemTap. Its goal is to utilize the dynamic probing capabilities provided through SystemTap to create a set of standard hooks that probe pre-defined places in the kernel. It can be used to collect important information that can be used as a starting point to analyze a performance problem in the system.

The LKET tapsets are designed to only trace the events selected by the user. Once the data has been collected, it is then post-processed according to the need of the user. Trace data can be processed in various different ways to generate simple to complex reports.

BINARY TRACING

By default, LKET will log the trace data in binary format.

To get a better performance for binary tracing, the "−b" option should be turned on for stap and thus −M option has to be added to stop staprun merging per-cpu files.

You could use the command lket−b2a to convert the binary trace data generated by LKET into readable data in ascii format.

lket−b2a uses the pre-cpu binary trace data files(stpd_cpu*) as inputs, and generates an output file named lket.out. or dump the trace data into MySQL database. See lket−b2a(1) manual page for more detail.

If you want LKET to log trace data in ASCII format directly, you should:

stap −D ASCII_TRACE ...

*Notes* that in order to make LKET able to work in binary tracing mode, all strings logged by LKET should be NULL-terminated, which means you have to use "%0s" instead of "%s" for both user appended extra printing statements and _lket_trace() which is called in LKET tapsets.

EVENT REGISTER

LKET provides a way to log the metadata of the trace data by events registering.

Two functions is provided:

void _register_sys_event (char *event_desc, int grpid, int hookid, char *fmt, char *field_name)

register_user_event(grpid:long, hookid:long, fmt:string, names:string)

event_desc is a string representation of the event, e.g: syscall.entry, scsi.iocompleted.

grpid and hookid is the groupid and hookid of the event to be registered.

fmt contains a set of fomat tokens seperated by ":". The valid format tokens are: UINT8, UINT16, UINT32, UINT64 and STRING which represents 8-bit, 16-bit, 32-bit, 64-bit binary data and NULL-terminated respectively.

names contains a set of names seperated by ":". The names contains in names should match the format tokens contains in fmt

_register_sys_event is a c function which is used to register the newly added trace hooks in LKET tapsets. For example, supposing you want to add a new event hook to trace the entry of sys_open, and you want this event hook to log the fd, flag and mode paremeters for you. You should add:

_register_sys_event("iosyscall.open.entry",
_GROUP_IOSYSCALL,
_HOOKID_IOSYSCALL_OPEN_ENTRY,
"STRING:INT32:INT32",
"filename:flags:mode");

into the function register_sys_events in LKET/register_event.stp

register_user_event is a SystemTap script function which is used for user to add extra trace data for a event hook. See the section CUSTOMIZED TRACE DATA for more detail

CUSTOMIZED TRACE DATA

LKET provides a set of event hooks that log the predefined trace data for you, but LKET also make you able to log extra trace data for a event.

LKET provides a way to do this without modifying the tapset of that event hook. You can simply use printf to trace extra data. For example, supposing you want to trace sk_buff−>mac_len and sk_buff−>priority besides the sk_buff−>len, sk_buff−>protocol and sk_buff−>truesize for the netdev event hooks:

probe register_event
{
register_user_event(GROUP_NETDEV, HOOKID_NETDEV_TRANSMIT,
"INT32:INT32", "mac_len:priority")
}
probe addevent.netdev.transmit
{
printf("%4b%4b", $skb−>mac_len, $skb−>priority)
}

EXAMPLES

Here are some examples of using LKET:
Trace all events provided by LKET:

stap −e "probe addevent.* {}" −bM

Trace all available events by skipping those unavaiabled on current
system:

stap −e "probe addevent.* ? {}" −bM

Trace all system calls:

stap −e "probe addevent.syscall {}" −bM

Trace the entry of all system calls:

stap −e "probe addevent.syscall.entry {}" −bM

Trace netdev transmition and log extra data of mac_len and priority:

stap −e "probe addevent.netdev.transmit { printf(\"%4b%4b\", $skb−>mac_len, $skb−>priority) }" −bM

You can press "Ctrl+c" to stop the tracing. Then you will find there are one or more per-cpu data files (stpd_cpu*) on current directory. You can use lket−b2a to convert these binary trace data files into readable ascii format or dump them into database. See lket−b2a(1) man page for more detail.

EVENT HOOKS AND TRACE DATA FORMAT

The following sections enumerate the variety of event hooks implemented in LKET and their trace data format. The trace data generated by different event hooks contain common data as well as some data specific to that event hook.

the INT8, INT16, INT32, INT64 and STRING appeared in trace data format represents 8-bit, 16-bit, 32-bit, 64-bit binary data and NULL-terminated string respectively.

The data common(i.e. common_data in the following subsecions) to all event hooks is:

timestamp(INT64),(tid<<32|pid)(INT64),(ppid<<32|groupID<<24|hookID<<16|cpu_id<<8)(INT64)

Each event hook group is a collection of those hooks that have similarities of what they could trace. And the ID of each event hook (HookID) is defined in the context of its corresponding group.

EVENT REGISTER
Event register is not actually an event. It is used to log the metadata of the trace data, including the extra trace data appended by user. See EVENT REGISTER and CUSTOMIZED TRACE DATA for more details.
register_sys_event

This is a function used to register event hooks available in LKET. It should be called from register_event.stp:register_sys_events().

register_user_event

This is a function used to log the metadata of the extra trace data appended by user for a specific event. It should be called in the probe register_event

SYSTEM CALLS
You could use addevent.syscall to trace the entry and return of all system calls. It contains two sub event hooks:
addevent.syscall.entry

Trace entry of all system calls.

Data format is:

common_data, syscall_name(STRING)

addevent.syscall.return

Trace return of all system calls.

Data format is:

common_data, syscall_name(STRING)

PROCESS CREATION
This group contains three sub event hooks. All of them are turned on by default. You can use the flags stoptrace_fork and stoptrace_exec to stop tracing fork/execve in your script, e.g.:

probe begin
{
stoptrace_fork = 1
stoptrace_exec = 1
}

process_snapshot()

This event hook isn’t a probe definition but a function. It is called by LKET silently to take a snapshot of all running processes.

Data format is:

common_data, tid(INT32), pid(INT32), ppid(INT32), process_name(STRING)

lket_internal.process.fork

Trace fork of processes

Data format is:

common_data, new_tid(INT32), new_pid(INT32), ppid(INT32)

lket_internal.process.execve

Trace execve of new processes

Data format is:

common_data, tid(INT32), pid(INT32), ppid(INT32), new_process_name(STRING)

SIGNAL
You could use addevent.signal to trace signal activities. It contains the following events:
addevent.signal.send.entry

Trace when a signal is sent to a process

Data format is: common_data, sig(INT8), shared(INT8), send2queue(INT8), pid(INT32)

addevent.signal.send.return

Trace when returning from sending signal