3. KTF kernel specific features

Running tests, examining results via debugfs

We provide debugfs interfaces for examining the results of the last test run and for running tests which do not require configuration specification. ktf testsets can be run via:

cat /sys/kernel/debug/ktf/run/<testset>

Individual tests can be run via

cat /sys/kernel/debug/ktf/run/<testset>-tests/<test>

Results can be displayed for the last run via

cat /sys/kernel/debug/ktf/results/<testset>

Individual tests can be run via

cat /sys/kernel/debug/ktf/results/<testset>-tests/<test>

These interfaces eliminate the use of the netlink socket API and provide a simple way to keep track of test failures. It can be useful to log into a machine and examine what tests were run without having console history available, and in particular:

cat /sys/kernel/debug/ktf/run/*

...is a useful way of running all KTF tests.

Reference to module internal symbols

When working with unit tests, the need to access non-public interfaces often arises. In general non-public interfaces is of course not intended to be used by outside programs, but a test framework is somewhat special here in that it is often necessary or desirable to unit test internal data structures or algorithms even if they are not exposed. The program under test may be a complicated beast by itself and merely exercising the public interfaces may not be flexible enough to stress the internal code. Even if it is possible to get the necessary “pressure” from the outside like that, it might be much more challenging or require a lot more work.

The usual method to gain access to internal interfaces is to be part of the internals. To some extent this is the way a lot of the kernel testing utilities operate. The obvious advantages of this is that the test code ‘automatically’ follows the module and it’s changes. The disadvantage is that test code is tightly integrated with the code itself. One important goal with KTF is to make it possible to write detailed and sophisticated test code which does not affect the readability or complexity of the tested code.

KTF contains a small python program, resolve, which parses a list of symbol names on the form:

#module first_module
#header first_header.h
private_symbol1
private_symbol2
...
#header second_header.h
#module next_module
...

The output is a header file and a struct containing function pointers and some convenience macro definitions to make it possible to ‘use’ the internal functions just as one would if within the module. This logic is based on kallsyms, and would of course only work if that functionality is enabled in the kernel KTF compiles against.

Requesting callbacks when a certain function gets called/returns

Tap into function entry using KTF entry probes. Many tests need to move beyond kernel APIs and ensure that side effects (logging a message etc) occur. A good way to do this is to probe entry of relevant functions. In order to do so in KTF you need to:

  • define an entry probe function with the same return value and arguments as the function to be probed
  • within the body of the entry probe function, ensure return is wrapped with KTF_ENTRY_PROBE_RETURN(<return value>);
  • entry probes need to registered for use and de-registered when done via KTF_[UN]REGISTER_ENTRY_PROBE(<kernel function name>).

See example h4.c in examples/ for a simple case where we probe printk() and ensure it is called.

Sometimes is is also useful to check that an intermediate function is returning an expected value. Return probes can be used to register/probe function return. In order to probe function return:

  • define a return probe point; i.e KTF_RETURN_PROBE(<kernel function>)
  • within the body of the return probe the return value can be retrieved via KTF_RETURN_VALUE(). Type will obviously depend on the function probed so should be cast if dereferencing is required.
  • return probes need to be registered for use and unregistered when done via KTF_[UN]REGISTER_RETURN_PROBE(<kernel function name>).

See example h4.c in examples/ for a simple case where we verify return value of printk().

Coverage analytics

While other coverage tools exist, they generally involve gcc-level support which is required at compile-time. KTF offers kernel module coverage support via kprobes instead. Tests can enable/disable coverage on a per-module basis, and coverage data can be retrieved via:

# more /sys/kernel/debug/ktf/coverage

For a given module we show how many of its functions were called versus the total, e.g.:

# cat /sys/kernel/debug/ktf/coverage
MODULE               #FUNCTIONS    #CALLED
selftest             14            1

We see 1 out of 14 functions was called when coverage was enabled.

We can also see how many times each function was called:

MODULE          FUNCTION                   COUNT
selftest        myelem_free                0
selftest        myelem_cmp                 0
selftest        ktf_return_printk          0
selftest        cov_counted                1
selftest        dummy                      0

In addition, we can track memory allocated via kmem_cache_alloc()/kmalloc() originating from module functions we have enabled coverage for. This allows us to track memory associated with the module specifically to find leaks etc. If memory tracking is enabled, /sys/kernel/debug/ktf/coverage will show outstanding allocations - the stack at allocation time; the memory address and size.

Coverage can be enabled via the “ktfcov” utility. Syntax is as follows:

ktfcov [-d module] [-e module [-m]]

“-e” enables coverage for the specified module; “-d” disables coverage. “-m” in combination with “-e” enables memory tracking for the module under test.