Linux Kernel

Getting linux kernel sources

Now that linux mainline has up-to-date xtensa support kernel.org is the primary place to go for xtensa linux. Depending on whether kernel tree with full history is needed for development or not either release tarball should be downloaded or git tree cloned.

• latest linux release archive: 'mainline' at the https://www.kernel.org/ points to it. May be buggy though.
• latest stable release archive: 'stable' at the https://www.kernel.org/ points to it. The best choice usually.
• mainline linux git tree: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/ . Use git clone with one of the URLs at that page to clone it.
• stable linux git tree: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/ . Use git clone with one of the URLs at that page to clone it.

Using tip of the cloned git tree for work is usually a bad idea, basing development branches on known releases (marked by tags) would give more predictable results.

Adding specific processor configuration

Note that the Xtensa port of Linux uses the terminology variant to describe a processor configuration. The steps involved adding a processor configuration are:

1. Create a new directory for the processor variant and copy a set of configuration files that describes the processor to that directory.
2. When configuring the kernel set .config variable CONFIG_XTENSA_VARIANT_CUSTOM_NAME to the name of used variant.

All processor configuration specific sources and header files are located in arch/xtensa/variants/variant-name for source files, and arch/xtensa/variants/variant-name/include/variant for header files. The include directory typically contains at least the following three files that describe the processor configuration:

• tie.h - describing custom defined TIE registers
• tie-asm.h - macros to access additional TIE registers
• core.h - describing various processor configurations, such as cache sizes, register options, etc.

All processor configurations provide a set of these files as part of the overlay provided by Tensilica.

If we want to add another processor, we have to create the following directory and copy the three configurations file to that directory:

mkdir -p arch/xtensa/variants/MyCore/include/variant
cp tie.h tie-asm.h core.h arch/xtensa/variants/MyCore/include/variant

Configuring and building the kernel

To build the kernel a toolchain for the target variant is needed: gcc (it doesn't need to have libc) and binutils. Currently the kernel can not be built with xcc. The toolchain may be built with crosstool-NG or buildroot.

Define CROSS_COMPILE=cross-compiler-spec environment variable to specify cross-compiler to be used. cross-compiler-spec is a path to cross-compiler binaries plus cross-compiler name up to its last component (e.g. if your cross-compiler is /opt/xtensa/tools/fsf/bin/xtensa-fsf-elf-gcc then specify CROSS_COMPILE=/opt/xtensa/tools/fsf/bin/xtensa-fsf-elf- to use it).

The kernel may be built in the same directory as its source (in-tree) or in a separate directory (out-of-tree). Building out-of-tree is usually cleaner and involves less actions when several configurations need to be built. Adding parameter O=build-dir-path to make controls out-of-tree kernel configuration/building.

Configuring the kernel

Configuring the kernel creates build infrastructure and .config file in the build directory. .config file may be written from scratch, copied from the outside source or taken from a predefined configuration and edited. For a list of predefined configurations see contents of directory arch/xtensa/configs. It's a good idea to run configuration after copying external or predefined config, as newer kernel releases may have added/removed or changed configuration parameters and the config file may need to be updated.

• to use external .config just copy it to the build directory
• to use predefined .config run ARCH=xtensa CROSS_COMPILE=cross-compiler-spec make O=build-dir-path predefined_config_name
• to edit existing configuration run ARCH=xtensa CROSS_COMPILE=cross-compiler-spec make O=build-dir-path menuconfig.

To search for a specific configuration item in curses interface use '/'. The following configuration parameters are important/interesting/worth checking:

• CONFIG_XTENSA_VARIANT_CUSTOM: must be set when a custom core variant is used
• CONFIG_XTENSA_VARIANT_CUSTOM_NAME: must specify custom core variant name
• CONFIG_XTENSA_VARIANT_MMU: if set to 'N' will build noMMU kernel even when core variant actually supports MMU. Should always be set for MMUv2 cores.
• CONFIG_HAVE_SMP: must always be set correctly, as presence of MX interrupt distributor affects external interrupts numbering.
• CONFIG_INITIALIZE_XTENSA_MMU_INSIDE_VMLINUX: controls whether uImage will reinitialize memory mapping. Should be set for noMMU and MMUv3 cores, unless MMU is fully initialized by uBoot. Ignored for MMUv2 cores. Has no effect for Image.elf, as it has to initialize MMU either in reset vector or in vmlinux.
• XTENSA_PLATFORM_*: must specify target platform. xtensa currently does not support multiplatform kernels.
• CONFIG_CMDLINE: default kernel command line. Gets overridden by command line specified in the device tree (built-in or externally loaded) or passed from the bootloader.
• BUILTIN_DTB: default kernel device tree. Only used when device tree is not supplied by the bootloader.

• CONFIG_DEVTMPFS: enables kernel self-maintained /dev filesystem.
• CONFIG_SERIAL_OF_PLATFORM: enables UART driver that may be instantiated from the device tree.
• CONFIG_LD_NO_RELAX: disables link-time relaxation at kernel build. This is useful against linker relaxation bugs and currently speeds up the build significantly.

Building the kernel

• to build kernel and modules run ARCH=xtensa CROSS_COMPILE=cross-compiler-spec make O=build-dir-path all
• to build only the kernel run ARCH=xtensa CROSS_COMPILE=cross-compiler-spec make O=build-dir-path zImage
• to install kernel modules into modules-dir run ARCH=xtensa CROSS_COMPILE=cross-compiler-spec INSTALL_MOD_PATH=modules-dir make O=build-dir-path modules_install
• all commands supported by kernel build system may be listed with ARCH=xtensa make help

The resulting binaries are located relatively to the build-dir-path:

• vmlinux: kernel ELF image, useful for debugging.
• arch/xtensa/boot/Image.elf: kernel with reset vector handler, may be used with xt-ocd or QEMU.
• arch/xtensa/boot/uImage: kernel image for uBoot, may as well be used with QEMU.

Built modules are located in their respective directories.

Booting the kernel

ISS platform

Kernels built for ISS are only bootable on Xtensa ISS (xt-run) or on QEMU. Neither xt-run nor QEMU can pass parameters to the ISS kernel, so normally it should have everything built-in: kernel command line and possibly initramfs. The easiest way to generate initramfs cpio archive is by setting BR2_TARGET_ROOTFS_CPIO to y in the buildroot configuration.

The kernel can also be configured to use simdisk, which is a host file with a file system image for the guest: see kernel configuration parameters CONFIG_BLK_DEV_SIMDISK and CONFIG_SIMDISK0_FILENAME. If CONFIG_SIMDISK0_FILENAME is a relative path it is relative to the current directory of the running simulator. The easiest way to generate file system image is by setting BR2_TARGET_ROOTFS_EXT2 to y in the buildroot configuration.

Running linux in Xtensa ISS:

 $ xt-run --turbo --memlimit=128 build-dir-path/arch/xtensa/boot/Image.elf

Since xt-run doesn't manage terminal settings, ^C and other terminal control codes interrupt xt-run instead of its guest. stty raw may be used to set up terminal not to terminate xt-run.

Running linux in QEMU: see QEMU reference page.

XTFPGA platform

XTFPGA is a collective name for FPGA boards capable of running Xtensa bitstream: LX60, LX110, LX200, ML605 and KC705.

Kernels built for XTFPGA may be booted on FPGA board and on QEMU (see QEMU reference page on how to do it).

There are two supported ways of booting linux on FPGA board: with xt-ocd and with uBoot.

Booting with xt-ocd

To use xt-ocd Xtensa tools need to be installed, boot from flash disabled on the board (refer to the specific board prototyping guide for the location of corresponding DIP switch), JTAG probe connected to the board and topology.xml file set up. Please refer to the Tensilica On Chip Debugging Guide (onchip_debug_guide.pdf) for the list of supported probes and setup instructions.

Sample topology.xml for the KC705 daughterboard built-in USB JTAG probe and single core bitstream:

<configuration>
  <controller id='Controller0' module='ft2232' probe='Dual RS232-HS' speed='10MHz' examine='1'/>
  <driver id='XtensaDriver0' module='xtensa' step-intr='mask,stepover,setps' />
  <chain controller='Controller0'>
    <tap id='TAP0' irwidth='5' />
  </chain>
  <system module='jtag'>
    <component id='Component0' tap='TAP0' config='trax' />
  </system>
  <device id='Xtensa0' component='Component0' driver='XtensaDriver0' />
  <application id='GDBStub' module='gdbstub' port='22000'>
    <target device='Xtensa0' />
  </application>
</configuration>

Running xt-ocd:

 $ xt-ocd -c topology.xml

Booting kernel on XTFPGA boards with xt-ocd + xt-gdb:

 $ xt-gdb --xtensa-core=xtensa-core-name build-dir-path/arch/xtensa/boot/Image.elf
 (xt-gdb) target remote xt-ocd-address-and-port 
 (xt-gdb) symbol-file build-dir-path/vmlinux
 (xt-gdb) reset
 (xt-gdb) load
 (xt-gdb) c

Booting with uBoot

Please see U-Boot page for instructions on installing uBoot onto the board.

uBoot can load kernel uImage from FLASH or from the network via TFTP.

Loading kernel from FLASH

Kernel may be stored as raw data in a separate FLASH partition, or it may be stored in a FLASH file system and loaded as a file. The following steps load kernel from UBIFS volume on a FLASH partition:

• make sure that mtdparts environment variable is set correctly. If it's not, use 'mtdparts default' command to reset MTD partitions configuration to default, or 'mtdparts add' to add partitions manually. Example of correct default settings for KC705:

 U-Boot> mtdparts
 
 device nor0 <f0000000.flash>, # parts = 4
  #: name                size            offset          mask_flags
  0: data                0x06000000      0x00000000      0
  1: boot loader area    0x00800000      0x06000000      0
  2: kernel image        0x017e0000      0x06800000      0
  3: boot environment    0x00020000      0x07fe0000      0
 
 active partition: nor0,0 - (data) 0x06000000 @ 0x00000000
 
 defaults:
 mtdids  : nor0=f0000000.flash
 mtdparts: mtdparts=f0000000.flash:0x06000000(data),0x00800000(boot loader area),0x017e0000(kernel image),0x00020000(boot environment)

• attach MTD partition to UBI with 'ubi part mtd-partition-name':

 U-Boot> ubi part data
 UBI: attaching mtd1 to ubi0
 UBI: physical eraseblock size:   131072 bytes (128 KiB)
 UBI: logical eraseblock size:    130944 bytes
 UBI: smallest flash I/O unit:    1
 UBI: VID header offset:          64 (aligned 64)
 UBI: data offset:                128
 UBI: attached mtd1 to ubi0
 UBI: MTD device name:            "mtd=0"
 UBI: MTD device size:            96 MiB
 UBI: number of good PEBs:        768
 UBI: number of bad PEBs:         0
 UBI: max. allowed volumes:       128
 UBI: wear-leveling threshold:    4096
 UBI: number of internal volumes: 1
 UBI: number of user volumes:     1
 UBI: available PEBs:             0
 UBI: total number of reserved PEBs: 768
 UBI: number of PEBs reserved for bad PEB handling: 0
 UBI: max/mean erase counter: 8/4

• mount UBI volume that contains kernel uImage with 'ubifsmount ubi:volume-name':

 U-Boot> ubifsmount ubi:data
 UBIFS: mounted UBI device 0, volume 0, name "data"
 UBIFS: mounted read-only
 UBIFS: file system size:   98731776 bytes (96417 KiB, 94 MiB, 754 LEBs)
 UBIFS: journal size:       4975872 bytes (4859 KiB, 4 MiB, 38 LEBs)
 UBIFS: media format:       w4/r0 (latest is w4/r0)
 UBIFS: default compressor: LZO
 UBIFS: reserved for root:  4879407 bytes (4765 KiB)

• load kernel image from the mounted file system with 'ubifsload address path-to-uImage':

 U-Boot> ubifsload 0x02000000 uImage
 Loading file 'uImage' to addr 0x02000000 with size 2850403 (0x002b7e63)...
 Done

Loading kernel from TFTP

In order for uBoot to be able to load uImage, device tree blob and initramfs they need to be copied somewhere under the tftproot and network must be configured in uBoot.

Rootfs

Rootfs in initial RAMFS

Initial RAMFS is a CPIO archive that the kernel may use to populate a tmpfs and then use as a root file system. It may be integrated into the kernel image (see kernel .config variable CONFIG_INITRAMFS_SOURCE) or loaded into memory separately by the bootloader. In the latter case its load address is passed to the kernel in the boot parameters block.

When Buildroot is used to generate root file system it can produce cpio image. When Buildroot is also used to build the kernel it can be directed to build the root file system into the kernel. See Buildroot .config symbols BR2_TARGET_ROOTFS_CPIO and BR2_TARGET_ROOTFS_INITRAMFS.

Rootfs in FLASH (UBIFS)

The easiest way to generate UBIFS image is by setting BR2_TARGET_ROOTFS_UBIFS to y in the buildroot config. Buildroot will also need the following file system parameters, that can be queried with mtdinfo tool:

• BR2_TARGET_ROOTFS_UBIFS_LEBSIZE: logical erase block (LEB) size (130944 for this device)
• BR2_TARGET_ROOTFS_UBIFS_MINIOSIZE: minimum I/O unit size (1 for this device)
• BR2_TARGET_ROOTFS_UBIFS_MAXLEBCNT: maximum logical eraseblock count (this device has 768, 764 are usable, as reported by ubiattach).

 # mtdinfo -a -u

Count of MTD devices:           4
Present MTD devices:            mtd0, mtd1, mtd2, mtd3
Sysfs interface supported:      yes

mtd0
Name:                           data
Type:                           nor
Eraseblock size:                131072 bytes, 128.0 KiB
Amount of eraseblocks:          768 (100663296 bytes, 96.0 MiB)
Minimum input/output unit size: 1 byte
Sub-page size:                  1 byte
Character device major/minor:   90:0
Bad blocks are allowed:         false
Device is writable:             true
Default UBI VID header offset:  64
Default UBI data offset:        128
Default UBI LEB size:           130944 bytes, 127.9 KiB
Maximum UBI volumes count:      128

To format FLASH partition and put UBIFS image into it the following command sequence may be used:

• run ubiformat:

 # ubiformat /dev/mtd0
 ubiformat: mtd0 (nor), size 100663296 bytes (96.0 MiB), 768 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 1 bytes
 libscan: scanning eraseblock 767 -- 100 % complete  
 ubiformat: 768 eraseblocks have valid erase counter, mean value is 2
 ubiformat: formatting eraseblock 767 -- 100 % complete  

• attach formatted device to UBI (this will create /dev/ubi0 device):

 # ubiattach -p /dev/mtd0
 UBI device number 0, total 768 LEBs (100564992 bytes, 95.9 MiB), available 764 LEBs (100041216 bytes, 95.4 MiB), LEB size 130944 bytes (127.9 KiB)

• create UBI volumes on formatted device (this will create /dev/ubi0_0 device, AKA ubi0:rootfs):

 # ubimkvol /dev/ubi0 -m -N rootfs
 Set volume size to 100041216
 Volume ID 0, size 764 LEBs (100041216 bytes, 95.4 MiB), LEB size 130944 bytes (127.9 KiB), dynamic, name "rootfs", alignment 1

• put UBI rootfs image so that xtensa linux could access it (in this example kernel is booted with NFS root and UBI image goes to /mnt/rootfs.ubifs)
• run ubiupdatevol to copy file system image to the specific UBI volume:

 # ubiupdatevol /dev/ubi0_0 /mnt/rootfs.ubifs

• now the volume may be mounted:

 # mount -t ubifs /dev/ubi0_0 /mnt

• to use that volume as rootfs add the following parameters to the kernel command line (note that 0 in the 'ubi.mtd=0' is the number of mtd partition to bind to ubi0, and 'rootfs' in the ubi0:rootfs below is the volume name):

 ubi.mtd=0 root=ubi0:rootfs rootfstype=ubifs

Further reading:

• http://www.linux-mtd.infradead.org/doc/ubifs.html
• http://www.linux-mtd.infradead.org/faq/ubifs.html
• https://www.kernel.org/doc/Documentation/filesystems/ubifs.txt

Features by release

Linux 5.19

• support coprocessors on SMP.
• support protection faults on noMMU.
• support full tickless operation.
• KCSAN.
• hibernation.

Linux 5.18

• support compiler plugins (with gcc-12 and newer).
• support for kernel ABI selection.
• removed dependency on compiler's libgcc.

Linux 5.16

• support for cores without windowed registers.
• call0 ABI kernel.

Linux 5.13

• GDBIO semihosting.

Linux 5.9

• syscall audit.
• seccomp filter.

Linux 5.6

• exception vectors can be attached to the rest of the kernel or placed at specific address.

Linux 5.5

• eXecute In Place (XIP) kernel.

Linux 5.4

• support for call0 ABI userspace.

Linux 5.3

• 'virt' board (sim with PCI, corresponding board model available in QEMU since v4.2).

Linux 5.2

• support for cores with exclusive access option.
• basic support for cores with MPU.

Linux 5.1

• generic spinlock/rwlock.

Linux 5.0

• memtest.
• Jump labels.
• tracehook.
• syscall tracepoints.
• multiple coprocessor fixes.
• SMP fixes.

Linux 4.16

• KASAN.
• SSP.
• allocation of DMA buffers from high memory.
• support DMA_ATTR_NO_KERNEL_MAPPING.
• kernel is built with -mtext-section-literals and -mno-serialize-volatile.

Linux 4.9

• New kernel memory layouts: with 256MBytes and 512MBytes of low memory.
• Configurable KSEG physical memory address and kernel load address.
• Custom sysmem boot time memory allocator is replaced with memblock.
• KMEMLEAK.
• reserved-memory DT node.
• Cadence Configurable System Platform (CSP).

Linux 4.6

• Hardware breakpoints/watchpoints.
• ioremap for memory outside KIO region.
• Big endian: serial, network, i2c work.

Linux 4.4

• DMA to high memory.

Linux 4.3

• Hardware perf counters and perf support.

Linux 4.1

• ALSA playback driver for XTFPGA.

Linux 3.19

• noMMU.

Linux 3.17

• Highmem on cores with aliasing cache.
• New (simpler) method of configuring for specific core variant.

Linux 3.16

• Stability: xtensa linux may be rebuilt natively.

Linux 3.15

• Highmem.
• KC705 board.

Linux 3.14

• SMP.
• Futex.
• register spilling overhaul.
• OpenCores ethernet works when connected to gigabit network.
• c67x00 USB host is usable on XTFPGA.
• TAP network works in ISS.

Linux 3.12

• Kernel preemption.

Linux 3.11

• Multihit exceptions fixed.
• Static function tracer.

Linux 3.10

• MMUv3 cores.
• Lockdep/stack tracing.

Linux 3.9

• TLS (THREADPTR register).
• Oprofile.
• ISS simdisk.

Linux 3.8

• Device trees.
• uImage generation.
• IRQ subsystem overhaul (IRQ domains, medium-priority interrupts).
• XTFPGA boards.
• s32c1i-based atomic/bit operations and spinlocks.

Linux 3.7

• Overall architecture port overhaul, mainline is alive again, builds and runs.
• There were dragons prior to that release.

Kernel loadable modules using FLIX with L32R

The kernel module loader (for the Xtensa architecture) supports only the limited set of relocations needed for module loading. This includes, for example, relocation of literals referenced by L32R instructions. This loader is not given specific information about any custom TIE configured in the processor, and thus is currently not able to relocate L32R instructions encoded in a FLIX bundle.

To ensure kernel modules load properly, they need to avoid L32R instructions within FLIX instruction bundles.

Existing kernel code does not generate such instructions, so it is not susceptible. However, it is possible to generate FLIX instructions using assembly code, or using the Tensilica XCC compiler. In the case of assembly code, simply avoid using L32R or MOVI instructions within a FLIX instruction bundle, unless a MOVI is known to fit without expanding ("relaxing") into an L32R instruction. In the case of XCC, pass the -mno-flix to the compiler (xt-xcc) to avoid generating FLIX bundles.

Note: The use of Tensilica software tools (including XCC) for Linux development is described in the Tensilica OSKit Guide, available from Tensilica.

Debugging Kernel Loadable Modules

Debugging loadable modules isn't easy, a few pointers mentioned here might save you some time. There is a mechanism that Amit Kalie mentioned on his KGDB webpage on loading the module symbols into gdb while loading a module Debuging init_module.

The insmod command has a --map option that dumps the module component offsets needed by Amit's approach. Unfortunately the initial Xtensa buildroot snapshots have a version of insmod that doesn't provide this option. Also, while debugging the kernel it's not convenient to run insmod on the console of the target.

A workaround to this problem is to hack the kernel module.c file and enable debugging.

   #if 1
   #define DEBUGP printk
   #else
   #define DEBUGP(fmt , a...)
   #endif

This will enable module.c to print the module section addresses. You then just list them in the gdb add-symbol-file command using that command's section (-s) options. Note that the .text section is specified without using the -s option: its address is given alone before any -s option.

   final section addresses:                  [Kernel Console Log/Output]
       0xc02a9098 .note.gnu.build-id
       0xc02b1000 .init.literal
       0xc02a7000 .text
       0xc02b11d8 .init.text
       0xc02a90bc .rodata
       0xc02a92c4 .rodata.str1.4
       0xc02ac27c .data
       0xc02ac640 __param
       0xc02ac668 .gnu.linkonce.this_module
       0xc02ac76c .bss
       0xc02aaf0c .symtab
       0xc02ab8fc .strtab

   add-symbol-file /export/src/Transwitch/a2000_src/a2000_src/Linux_atlanta/ctlm-cpe/drivers/ctlmeth/ctlmeth.ko   \
       0xc02a7000                           \
       -s .data 0xc02ac27c                  \
       -s .init.text 0xc02b11d8             \
       -s .bss 0xc02ac76c                   \
       -s .init.literal 0xc02b1000          \
       -s  .rodata 0xc02a90bc

You likely want to compile your modules unoptimized and unstripped to facilitate debugging and provide the path to the module source to gdb with the dir command. For example:

   dir /home/pdelaney/src/Transwitch/a2000_src/a2000_src/Linux_atlanta/ctlm-cpe/drivers/ctlmeth/

Debugging on ISS

 $ xt-gdb build-dir-path/arch/xtensa/boot/Image.elf
 (xt-gdb) target sim --turbo --memlimit=128
 (xt-gdb) symbol-file build-dir-path/vmlinux
 (xt-gdb) run