LTP Results
LTP Test Results
One of the more common ways of testing the Linux kernel is to use the Linux Test Project (LTP) testsuite. LTP contains many different tests and testsuites, a number of which are often not applicable in a given environment or test setup.
Running LTP in a system constructed using buildroot typically involves the following steps:
- When configuring buildroot (using e.g. make menuconfig), under Package Selection for the target, unselect Hide applications that are provided by busybox, select bash, and select ltp-testsuite. Under later versions of LTP that can generate HTML, also go deeper in Interpreter languages / Scripting and select microperl. Then make buildroot as usual. On the target, the LTP testsuite ends up in /root/ltp-full-<VERSION>.
- LTP is generally much too big (~150 to 350 MB) to put in a root filesystem that is linked into the kernel (such as initramfs). So you'll generally need to put your root filesystem somewhere else, such as mounting over NFS.
- Run LTP. This can be as simple as: cd /root/ltp-full-20070228 ; ./runltp although you'll typically want to pass runltp a few options to specify output files, tests to run, etc.
In some older or bleeding-edge/unstable environments, some tests may crash the kernel or just hang. You may need to comment out these tests from the relevant test series under /runtest/, or simply not request that test when running LTP, to complete running the tests.
Results for the second snapshot (of July 2008)
Here are some unofficial LTP results run on May 14, 2009 (and a more accurate report from a patched copy of LTP's genhtml) for the Linux for Xtensa snapshot of July 11, 2008.
The version of LTP used by buildroot, as of that snapshot, is 20070228. However, that version does not generate HTML output. This test run was made using LTP version 20090430 instead, to get HTML formatted results. A number of buildroot files were updated to support this version of LTP (see below).
LTP version 20090430 runs a few more tests by default relative to version 20070228. Many of these new tests require a much larger set of packages installed on the target than the bare minimum root filesystem one gets by default following buildroot build instructions. Some of these tests are even specific to the x86 architecture. Thus, instead of running LTP with a default set of tests, LTP was run with a specific set that excludes most of the new ones (but includes timer).
Also note that a number of kernel bugs were fixed since that snapshot, as reflected in various kernel git trees on linux-xtensa.org . Three of these caused the kernel to crash or hang, and were commented out to obtain these test results: mm01 in runtest/mm, sem01 in runtest/ipc, and hackbench01 in runtest/sched.
Here is how LTP was invoked for this run:
cd /root/ltp-full-20090430 time ./runltp -l 20090514c.log -g 20090514c.html -o 20090514c.out -f syscalls,fs,fsx,mm,ipc,sched,math,nptl,pty,timers
You may also be interested to note that due to certain bugs in LTP itself, LTP HTML output erroneously reports certain tests as passing when they are not (regardless of architecture). Careful reading of the Test-Output column (in the absence of a clear PASS line therein) usually indicates the actual result.
Buildroot patches for LTP 20090430
- Remove package/ltp-testsuite/ltp-testsuite-sh-is-not-C-code.patch
- Edit package/ltp-testsuite/ltp-testsuite.mk:
6c6,7 < LTP_TESTSUITE_VERSION:=20070228 --- > LTP_TESTSUITE_VERSION:=20090430 > #LTP_TESTSUITE_VERSION:=20070228 17a19 > ltp-testsuite-newer.patch \
- Edit package/ltp-testsuite/ltp-testsuite.patch:
40,41c40,41 < @@ -20,7 +20,7 @@ < LDLIBS += -L../../../../lib -lltp --- > @@ -23,7 +23,7 @@ > %_64: CFLAGS += -D_FILE_OFFSET_BITS=64 44,45c44,45 < -TARGETS = $(patsubst %.c,%,$(SRCS)) < +TARGETS =#$(patsubst %.c,%,$(SRCS)) --- > -TARGETS += $(patsubst %.c,%,$(SRCS)) > +TARGETS +=#$(patsubst %.c,%,$(SRCS)) 52c52 < @@ -85,9 +85,7 @@ --- > @@ -90,9 +90,7 @@ 63c63 < @@ -96,10 +94,6 @@ --- > @@ -101,10 +99,6 @@
- Add package/ltp-testsuite/ltp-testsuite-newer.patch (this disables a number of tests that don't build because they rely on recent system calls that did not exist in uClibc as of the July 2008 snapshot):
diff -ur ltp-full-20090430/testcases/kernel/syscalls/remap_file_pages/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/remap_file_pages/Makefile --- ltp-full-20090430/testcases/kernel/syscalls/remap_file_pages/Makefile 2009-05-12 10:44:15.000000000 -0700 +++ ltp-full-20090430-patched/testcases/kernel/syscalls/remap_file_pages/Makefile 2009-05-12 10:40:28.000000000 -0700 @@ -20,7 +20,7 @@ LDLIBS += -L../../../../lib -lltp SRCS = $(wildcard *.c) -TARGETS = $(patsubst %.c,%,$(SRCS)) +TARGETS = #$(patsubst %.c,%,$(SRCS)) all: $(TARGETS) diff -ur ltp-full-20090430/testcases/kernel/syscalls/signalfd/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd/Makefile --- ltp-full-20090430/testcases/kernel/syscalls/signalfd/Makefile 2008-11-11 00:31:27.000000000 -0800 +++ ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd/Makefile 2009-05-12 10:55:16.000000000 -0700 @@ -19,7 +19,7 @@ LDLIBS += -L../../../../lib -lltp SRCS = $(wildcard *.c) -TARGETS = $(patsubst %.c,%,$(SRCS)) +TARGETS = #$(patsubst %.c,%,$(SRCS)) all: $(TARGETS) diff -ur ltp-full-20090430/testcases/kernel/syscalls/signalfd4/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd4/Makefile --- ltp-full-20090430/testcases/kernel/syscalls/signalfd4/Makefile 2009-01-16 02:07:55.000000000 -0800 +++ ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd4/Makefile 2009-05-12 10:55:54.000000000 -0700 @@ -20,7 +20,7 @@ LDLIBS += -L../../../../lib -lltp SRCS = $(wildcard *.c) -TARGETS = $(patsubst %.c,%,$(SRCS)) +TARGETS = #$(patsubst %.c,%,$(SRCS)) all: $(TARGETS) diff -ur ltp-full-20090430/testcases/kernel/syscalls/splice/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/splice/Makefile --- ltp-full-20090430/testcases/kernel/syscalls/splice/Makefile 2009-05-12 11:12:23.000000000 -0700 +++ ltp-full-20090430-patched/testcases/kernel/syscalls/splice/Makefile 2009-05-12 11:15:27.000000000 -0700 @@ -20,7 +20,7 @@ LDLIBS += -L../../../../lib -lltp SRCS = $(wildcard *.c) -TARGETS = $(patsubst %.c,%,$(SRCS)) +TARGETS = #$(patsubst %.c,%,$(SRCS)) all: $(TARGETS) Only in ltp-full-20090430/testcases/kernel/syscalls/splice: splice01 diff -ur ltp-full-20090430/testcases/kernel/syscalls/tee/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/tee/Makefile --- ltp-full-20090430/testcases/kernel/syscalls/tee/Makefile 2009-05-12 11:12:52.000000000 -0700 +++ ltp-full-20090430-patched/testcases/kernel/syscalls/tee/Makefile 2009-05-12 11:12:46.000000000 -0700 @@ -20,7 +20,7 @@ LDLIBS += -L../../../../lib -lltp SRCS = $(wildcard *.c) -TARGETS = $(patsubst %.c,%,$(SRCS)) +TARGETS = #$(patsubst %.c,%,$(SRCS)) all: $(TARGETS) diff -ur ltp-full-20090430/testcases/kernel/syscalls/vmsplice/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/vmsplice/Makefile --- ltp-full-20090430/testcases/kernel/syscalls/vmsplice/Makefile 2009-05-12 11:13:17.000000000 -0700 +++ ltp-full-20090430-patched/testcases/kernel/syscalls/vmsplice/Makefile 2009-05-12 11:13:09.000000000 -0700 @@ -21,7 +21,7 @@ LDLIBS += -L../../../../lib -lltp SRCS = $(wildcard *.c) -TARGETS = $(patsubst %.c,%,$(SRCS)) +TARGETS = #$(patsubst %.c,%,$(SRCS)) all: $(TARGETS) diff -ur ltp-full-20090430/testcases/network/lib6/Makefile ltp-full-20090430-patched/testcases/network/lib6/Makefile --- ltp-full-20090430/testcases/network/lib6/Makefile 2009-01-22 01:26:10.000000000 -0800 +++ ltp-full-20090430-patched/testcases/network/lib6/Makefile 2009-05-12 11:45:19.000000000 -0700 @@ -24,8 +24,8 @@ LOADLIBES+= -L../../../lib runcc.a -lltp -lpthread SRCS=$(wildcard *.c) -TARGETS= asapi_01 asapi_02 asapi_03 asapi_04 asapi_05 asapi_06 asapi_07 \ - getaddrinfo_01 in6_01 in6_02 +TARGETS= asapi_01 asapi_02 asapi_03 asapi_07 \ + in6_02 all: $(TARGETS)