Marc: Include link to more accurate LTP results HTML page

2009-05-20T23:44:03Z

Include link to more accurate LTP results HTML page

← Older revision		Revision as of 23:44, 20 May 2009
Line 13:		Line 13:
	== Results for the second snapshot (of July 2008) ==		== Results for the second snapshot (of July 2008) ==

	Here are some [http://www.linux-xtensa.org/ltp/20090514c.html unofficial LTP results run on May 14, 2009]		Here are some [http://www.linux-xtensa.org/ltp/20090514c.html unofficial LTP results run on May 14, 2009] (and a [http://www.linux-xtensa.org/ltp/20090514c3.html more accurate report] from a patched copy of LTP's genhtml)
	for the Linux for Xtensa [http://www.linux-xtensa.org/download.html snapshot] of July 11, 2008.		for the Linux for Xtensa [http://www.linux-xtensa.org/download.html snapshot] of July 11, 2008.

Line 65:		Line 65:
	---		---
	> @@ -101,10 +99,6 @@		> @@ -101,10 +99,6 @@
	~~68c68~~
	~~< -#define MAX_SWAPFILES 32~~
	~~---~~
	~~> -#define MAX_SWAPFILES 32~~

	* Add <tt>package/ltp-testsuite/ltp-testsuite-newer.patch</tt> (this disables a number of tests that don't build because they rely on recent system calls that did not exist in uClibc as of the July 2008 snapshot):		* Add <tt>package/ltp-testsuite/ltp-testsuite-newer.patch</tt> (this disables a number of tests that don't build because they rely on recent system calls that did not exist in uClibc as of the July 2008 snapshot):

Marc: Add preliminary LTP test results for the July 2008 snapshot

2009-05-19T23:21:54Z

Add preliminary LTP test results for the July 2008 snapshot

New page

= LTP Test Results =

One of the more common ways of testing the Linux kernel is to use the [http://ltp.sourceforge.net/ Linux Test Project (LTP)] testsuite. LTP contains many different tests and testsuites, a number of which are often not applicable in a given environment or test setup.

Running LTP in a system constructed using [[Buildroot_Build_Instructions| buildroot]] typically involves the following steps:

* When configuring buildroot (using e.g. <tt>make menuconfig</tt>), under '''Package Selection for the target''', unselect '''Hide applications that are provided by busybox''', select '''<tt>bash</tt>''', and select '''<tt>ltp-testsuite</tt>'''. Under later versions of LTP that can generate HTML, also go deeper in '''Interpreter languages / Scripting''' and select '''<tt>microperl</tt>'''. Then make buildroot as usual. On the target, the LTP testsuite ends up in <tt>/root/ltp-full-</tt>''<VERSION>''.
* LTP is generally much too big (~150 to 350 MB) to put in a root filesystem that is linked into the kernel (such as initramfs). So you'll generally need to put your root filesystem somewhere else, such as [[Buildroot_Build_Instructions#Mounting_the_Root_Filesystem_Over_NFS| mounting over NFS]].
* Run LTP. This can be as simple as: <tt>cd /root/ltp-full-20070228 ; ./runltp</tt> although you'll typically want to pass <tt>runltp</tt> a few options to specify output files, tests to run, etc.

In some older or bleeding-edge/unstable environments, some tests may crash the kernel or just hang. You may need to comment out these tests from the relevant test series under <tt>/runtest/</tt>, or simply not request that test when running LTP, to complete running the tests.

== Results for the second snapshot (of July 2008) ==

Here are some [http://www.linux-xtensa.org/ltp/20090514c.html unofficial LTP results run on May 14, 2009]
for the Linux for Xtensa [http://www.linux-xtensa.org/download.html snapshot] of July 11, 2008.

The version of LTP used by buildroot, as of that snapshot, is 20070228. However, that version does not generate HTML output. This test run was made using LTP version 20090430 instead, to get HTML formatted results. A number of buildroot files were updated to support this version of LTP (see below).

LTP version 20090430 runs a few more tests by default relative to version 20070228. Many of these new tests require a much larger set of packages installed on the target than the bare minimum root filesystem one gets by default following [[Buildroot_Build_Instructions| buildroot build instructions]]. Some of these tests are even specific to the x86 architecture. Thus, instead of running LTP with a default set of tests, LTP was run with a specific set that excludes most of the new ones (but includes <tt>timer</tt>).

Also note that a number of kernel bugs were fixed since that snapshot, as reflected in various kernel git trees on linux-xtensa.org . Three of these caused the kernel to crash or hang, and were commented out to obtain these test results: <tt>mm01</tt> in <tt>runtest/mm</tt>, <tt>sem01</tt> in <tt>runtest/ipc</tt>, and <tt>hackbench01</tt> in <tt>runtest/sched</tt>.

Here is how LTP was invoked for this run:

cd /root/ltp-full-20090430
time ./runltp -l 20090514c.log -g 20090514c.html -o 20090514c.out -f syscalls,fs,fsx,mm,ipc,sched,math,nptl,pty,timers

You may also be interested to note that due to certain bugs in LTP itself, LTP HTML output erroneously reports certain tests as passing when they are not (regardless of architecture). Careful reading of the Test-Output column (in the absence of a clear PASS line therein) usually indicates the actual result.

=== Buildroot patches for LTP 20090430 ===

* Remove <tt>package/ltp-testsuite/ltp-testsuite-sh-is-not-C-code.patch</tt>
* Edit <tt>package/ltp-testsuite/ltp-testsuite.mk</tt>:

6c6,7
< LTP_TESTSUITE_VERSION:=20070228
---
> LTP_TESTSUITE_VERSION:=20090430
> #LTP_TESTSUITE_VERSION:=20070228
17a19
> ltp-testsuite-newer.patch \

* Edit <tt>package/ltp-testsuite/ltp-testsuite.patch</tt>:

40,41c40,41
< @@ -20,7 +20,7 @@
< LDLIBS += -L../../../../lib -lltp
---
> @@ -23,7 +23,7 @@
> %_64: CFLAGS += -D_FILE_OFFSET_BITS=64
44,45c44,45
< -TARGETS = $(patsubst %.c,%,$(SRCS))
< +TARGETS =#$(patsubst %.c,%,$(SRCS))
---
> -TARGETS += $(patsubst %.c,%,$(SRCS))
> +TARGETS +=#$(patsubst %.c,%,$(SRCS))
52c52
< @@ -85,9 +85,7 @@
---
> @@ -90,9 +90,7 @@
63c63
< @@ -96,10 +94,6 @@
---
> @@ -101,10 +99,6 @@
68c68
< -#define MAX_SWAPFILES 32
---
> -#define MAX_SWAPFILES 32

* Add <tt>package/ltp-testsuite/ltp-testsuite-newer.patch</tt> (this disables a number of tests that don't build because they rely on recent system calls that did not exist in uClibc as of the July 2008 snapshot):

diff -ur ltp-full-20090430/testcases/kernel/syscalls/remap_file_pages/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/remap_file_pages/Makefile
--- ltp-full-20090430/testcases/kernel/syscalls/remap_file_pages/Makefile 2009-05-12 10:44:15.000000000 -0700
+++ ltp-full-20090430-patched/testcases/kernel/syscalls/remap_file_pages/Makefile 2009-05-12 10:40:28.000000000 -0700
@@ -20,7 +20,7 @@
LDLIBS += -L../../../../lib -lltp

SRCS = $(wildcard *.c)
-TARGETS = $(patsubst %.c,%,$(SRCS))
+TARGETS = #$(patsubst %.c,%,$(SRCS))

all: $(TARGETS)

diff -ur ltp-full-20090430/testcases/kernel/syscalls/signalfd/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd/Makefile
--- ltp-full-20090430/testcases/kernel/syscalls/signalfd/Makefile 2008-11-11 00:31:27.000000000 -0800
+++ ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd/Makefile 2009-05-12 10:55:16.000000000 -0700
@@ -19,7 +19,7 @@
LDLIBS += -L../../../../lib -lltp

SRCS = $(wildcard *.c)
-TARGETS = $(patsubst %.c,%,$(SRCS))
+TARGETS = #$(patsubst %.c,%,$(SRCS))

all: $(TARGETS)

diff -ur ltp-full-20090430/testcases/kernel/syscalls/signalfd4/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd4/Makefile
--- ltp-full-20090430/testcases/kernel/syscalls/signalfd4/Makefile 2009-01-16 02:07:55.000000000 -0800
+++ ltp-full-20090430-patched/testcases/kernel/syscalls/signalfd4/Makefile 2009-05-12 10:55:54.000000000 -0700
@@ -20,7 +20,7 @@
LDLIBS += -L../../../../lib -lltp

SRCS = $(wildcard *.c)
-TARGETS = $(patsubst %.c,%,$(SRCS))
+TARGETS = #$(patsubst %.c,%,$(SRCS))

all: $(TARGETS)

diff -ur ltp-full-20090430/testcases/kernel/syscalls/splice/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/splice/Makefile
--- ltp-full-20090430/testcases/kernel/syscalls/splice/Makefile 2009-05-12 11:12:23.000000000 -0700
+++ ltp-full-20090430-patched/testcases/kernel/syscalls/splice/Makefile 2009-05-12 11:15:27.000000000 -0700
@@ -20,7 +20,7 @@
LDLIBS += -L../../../../lib -lltp

SRCS = $(wildcard *.c)
-TARGETS = $(patsubst %.c,%,$(SRCS))
+TARGETS = #$(patsubst %.c,%,$(SRCS))

all: $(TARGETS)

Only in ltp-full-20090430/testcases/kernel/syscalls/splice: splice01
diff -ur ltp-full-20090430/testcases/kernel/syscalls/tee/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/tee/Makefile
--- ltp-full-20090430/testcases/kernel/syscalls/tee/Makefile 2009-05-12 11:12:52.000000000 -0700
+++ ltp-full-20090430-patched/testcases/kernel/syscalls/tee/Makefile 2009-05-12 11:12:46.000000000 -0700
@@ -20,7 +20,7 @@
LDLIBS += -L../../../../lib -lltp

SRCS = $(wildcard *.c)
-TARGETS = $(patsubst %.c,%,$(SRCS))
+TARGETS = #$(patsubst %.c,%,$(SRCS))

all: $(TARGETS)

diff -ur ltp-full-20090430/testcases/kernel/syscalls/vmsplice/Makefile ltp-full-20090430-patched/testcases/kernel/syscalls/vmsplice/Makefile
--- ltp-full-20090430/testcases/kernel/syscalls/vmsplice/Makefile 2009-05-12 11:13:17.000000000 -0700
+++ ltp-full-20090430-patched/testcases/kernel/syscalls/vmsplice/Makefile 2009-05-12 11:13:09.000000000 -0700
@@ -21,7 +21,7 @@
LDLIBS += -L../../../../lib -lltp

SRCS = $(wildcard *.c)
-TARGETS = $(patsubst %.c,%,$(SRCS))
+TARGETS = #$(patsubst %.c,%,$(SRCS))

all: $(TARGETS)

diff -ur ltp-full-20090430/testcases/network/lib6/Makefile ltp-full-20090430-patched/testcases/network/lib6/Makefile
--- ltp-full-20090430/testcases/network/lib6/Makefile 2009-01-22 01:26:10.000000000 -0800
+++ ltp-full-20090430-patched/testcases/network/lib6/Makefile 2009-05-12 11:45:19.000000000 -0700
@@ -24,8 +24,8 @@
LOADLIBES+= -L../../../lib runcc.a -lltp -lpthread

SRCS=$(wildcard *.c)
-TARGETS= asapi_01 asapi_02 asapi_03 asapi_04 asapi_05 asapi_06 asapi_07 \
- getaddrinfo_01 in6_01 in6_02
+TARGETS= asapi_01 asapi_02 asapi_03 asapi_07 \
+ in6_02

all: $(TARGETS)

LTP Results - Revision history

Marc: Include link to more accurate LTP results HTML page

Marc: Add preliminary LTP test results for the July 2008 snapshot