Saturday, June 9, 2012

How to recompile the linux kernel for specific hardware in Debian 6

I have Debian 6 installed on my mini-ITX system.

I want to try recompiling the kernel to speed up boot and reduce the footprint on the slow, old system. Eventually, I want to use the lessons learned to recompile Ubuntu 12.04 to work on the system. Currently, 12.04 will not boot, since the old VIA 32-bit processor does not fully support x686 nor pae.

If the kernel and modules are small enough, and customized for the equipment, we can even get rid of the initrd at boot. That will also speed up the boot a lot!

Step 1: Collect CPU and module information 

We can compile the CPU for the specific processor and modules we actually use. On such a resource-constrained system, getting rid of a lot of unused bloat is welcome.

The following command shows that we have a VIA Samuel 2 processor, 533MHz, 32-bit system.

$ cat /proc/cpuinfo 
processor : 0
vendor_id : CentaurHauls
cpu family : 6
model  : 7
model name : VIA Samuel 2
stepping : 3
cpu MHz  : 533.370
cache size : 64 KB
fdiv_bug : no
hlt_bug  : no
f00f_bug : no
coma_bug : no
fpu  : yes
fpu_exception : yes
cpuid level : 1
wp  : yes
flags  : fpu de tsc msr cx8 mtrr pge mmx 3dnow
bogomips : 1066.74
clflush size : 32
cache_alignment : 32
address sizes : 32 bits physical, 32 bits virtual
power management:

The following command lists most of the modules used (modules used during boot then unloaded do not show up here).

$ lsmod
Module                  Size  Used by
fuse                   43213  1 
snd_via82xx            15020  0 
gameport                6105  1 snd_via82xx
snd_ac97_codec         79248  1 snd_via82xx
ac97_bus                 710  1 snd_ac97_codec
snd_mpu401_uart         4155  1 snd_via82xx
snd_rawmidi            12737  1 snd_mpu401_uart
snd_pcm                46098  2 snd_via82xx,snd_ac97_codec
snd_seq_device          3661  1 snd_rawmidi
snd_timer              12304  1 snd_pcm
via_ircc               13188  0 
snd                    34142  7 snd_via82xx,snd_ac97_codec,snd_mpu401_uart,snd_rawmidi,snd_pcm,snd_seq_device,snd_timer
vt8231                  8678  0 
irda                   71690  1 via_ircc
snd_page_alloc          5021  2 snd_via82xx,snd_pcm
i2c_viapro              4387  0 
shpchp                 21168  0 
parport_pc             15747  0 
evdev                   5520  4 
parport                22170  1 parport_pc
button                  3578  0 
processor              23003  1 
pcspkr                  1235  0 
serio_raw               2912  0 
i2c_core               12739  1 i2c_viapro
soundcore               3294  1 snd
crc_ccitt               1039  1 irda
pci_hotplug            18493  1 shpchp
ext3                   92928  3 
jbd                    28166  1 ext3
mbcache                 3482  1 ext3
sd_mod                 25961  5 
crc_t10dif              1012  1 sd_mod
ata_generic             2239  0 
fan                     2590  0 
pata_via                5721  4 
uhci_hcd               16021  0 
libata                117427  2 ata_generic,pata_via
via_rhine              14427  0 
thermal                 9210  0 
ehci_hcd               28859  0 
thermal_sys             9346  3 processor,fan,thermal
scsi_mod              105921  2 sd_mod,libata
mii                     2654  1 via_rhine
usbcore                99061  3 uhci_hcd,ehci_hcd
nls_base                4445  1 usbcore

The following command tells us that the current version of the kernel in use is 2.6.32-5-486:

$ uname -a
Linux mini-deb6 2.6.32-5-486 #1 Sun May 6 03:29:22 UTC 2012 i686 GNU/Linux

Step 2: Install the kernel source

The kernel source code is needed. "Recompiling" a kernel actually just means compiling a new version of the kernel from source. The process is covered very well in this forum post. We will install all the needed packages for the recompile, install the linux source code too /usr/src, and copy the existing old kernel config so we have a nice baseline if we mess it up.

# apt-get install ncurses-dev kernel-package cpio initramfs-tools linux-base module-init-tools udev linux-source-2.6.32-5-486
# cd /usr/src
# tar -xjf linux-source-*.tar.bz2
# ln -s linux-source-* linux
# cd linux
# cp .config .config-original-2.6.32

One tiny extra step - we'll be manually rebooting at the end of the process, so I reset my grub timeout to 15 seconds so I have time to select the kernel I want without rushing. Grub will grab this new value when it automatically updates in Step 4.

# nano /etc/default/grub
(change the line GRUB TIMEOUT= )

Step 3: Edit the config file, then compile

Now we're ready to modify the .config file.

# make menuconfig 

Menuconfig takes about an hour, perhaps more. To build a really minimal kernel, go through and switch all of the options off. Next, go back and look at the list of modules. Search for each module (search is the forward-slash key '/'), and turn each back on. Make sure each is compiled into the kernel, not added as a module. Some options require you to search for prerequisites and turn those on first. Like I said, it took me over an hour to work through all the modules on the list.

When finished, save the config. Then we're ready to compile:

# make-kpkg clean
# make-kpkg --append-to-version=-custom1 kernel_image kernel_headers

Iterate the version number (-custom1 to -custom2) or you'll lose track of progress very quickly! If you want an initrd, add --initrd to the line.

Compiling on my slow machine takes about 2.5 hours. I run make-kpkg inside a screen session. I can ssh in, start compiling, detach the screen, and return in a few hours to see the result.

Step 4: Install the new kernel

The make-kpkg command creates a lovely new package for us in /usr/src. Install it.

# ls -l /usr/src
total 97892
lrwxrwxrwx  1 root root       19 Jun  2 19:20 linux -> linux-source-2.6.32
-rw-r--r--  1 root root  6060196 Jun  2 13:56 linux-headers-2.6.32-custom1_2.6.32-custom1-10.00.Custom_i386.deb
-rw-r--r--  1 root root  4170548 Jun  2 13:47 linux-image-2.6.32-custom1_2.6.32-custom1-10.00.Custom_i386.deb
drwxr-xr-x 25 root root     4096 Jun  9 14:02 linux-source-2.6.32
-rw-r--r--  1 root root 65220157 May  5 18:19 linux-source-2.6.32.tar.bz2
drwx------  2 root root    16384 May 27 17:36 lost+found

# dpkg -i  /usr/src/linux-image-2.6.32-custom1_2.6.32-custom1-10.00.Custom_i386.deb

The install will also create an updated initrd (if you asked for one) and add the appropriate grub entries. If you changed the Grub Timeout value in Step 2, the update will reflect that, too.

Change one setting in the new /etc/grub/grub.cfg. Normally, grub tells the kernel to load the boot drive read-only, then initrd unmounts it and remounts it read-write. Since we don't have an initrd, we must change that grub behavior. For the first few iterations of kernel testing, this doesn't matter - it wont affect anything until you reach busybox prompts because the root filesystem cannot be loaded:

# nano /etc/grub/grub.cfg

Look for the root=/dev/sda1 ro stanza in the last line of the appropriate menuentry, and change ro (read only) to rw (read/write). Here's an example of an unchanged menuentry:

menuentry 'Debian GNU/Linux, with Linux 2.6.32-custom8' --class debian --class gnu-linux --class gnu --class os {
        insmod part_gpt
        insmod ext2
        set root='(hd0,gpt1)'
        search --no-floppy --fs-uuid --set 89b0fb32-b7e3-46e8-878b-5da4cd69fbdd
        echo    'Loading Linux 2.6.32-custom8 ...'
        linux   /boot/vmlinuz-2.6.32-custom8 root=/dev/sda1 ro console=tty1 console=ttyS0,115200n8

Reboot into the new kernel.

# shutdown -r now

Step 5: Lather, Rinse, Repeat

Keep the hardware handy - you need to see the console boot messages, and you need access to the power switch.

The first few times you try this, the kernel is likely to fail. Either you forgot to add a module, or you set something wrong. Record the error off the screen, and jump into a search engine to see what it means.

Kernel failures fall into several broad categories. Each is usually due to the same cause - a misconfigured kernel .config file. Happily, these are sequential. As you move down the list, you're getting closer to final success.
- The easiest to diagnose is the simple refusal to start. The kernel fails a basic start test, and tells you in plain language that it won't start because it doesn't support this processor or the processor doesn't support this service, etc.
- Once the kernel passes those start tests, the next set of failures are kernel panics. This means that the kernel has loaded, but some piece of hardware or initrd instruuction or filesystem has mystified it, so it barfs a bunch of diagnostic information and freezes.
- Once the kernel is over its panic attacks, the next set of failures are dropping to the busybox prompt when the root filesystem fails to be recognized or loaded. This usually means either a misconfigured root= or UUID= in grub, or a missing or incorrect filesystem module.
- Finally, the system boots and works...mostly. Some application or piece of hardware doesn't work, though.

Reboot into the original working kernel (hard reboot, using the power switch).

Remove the non-working kernel. This will also remove the appropriate initrd (if any), and appropriate grub entries.

# apt-get remove linux-image-2.6.32-custom1

Return to Step 3, change the config file to fix the problem, and compile again.

Step 6: The new kernel works!

Once the new kernel works, clean up. Backup the working config file, remove the source code package, and delete the 'linux' symlink so a future recompile doesn't accidentally overwrite this working version.

# cp .config .config-2.6.32-custom5   # or whatever custom iteration number you want
# apt-get remove linux-source-2.6.32
# cd /usr/src
# rm linux

Now you can brag to your friends about how easy it was...

Let's take a quick look at the old and new kernels side-by-side:

$ ls -l /boot
total 17052
-rw-r--r-- 1 root root  110922 May  6 07:49 config-2.6.32-5-486
-rw-r--r-- 1 root root   55311 Jun 10 08:54 config-2.6.32-custom8
drwxr-xr-x 3 root root    4096 Jun 10 11:48 grub
-rw-r--r-- 1 root root 8722934 May 27 15:54 initrd.img-2.6.32-5-486
-rw-r--r-- 1 root root 1249814 May  6 07:49
-rw-r--r-- 1 root root 1333763 Jun 10 11:00
-rw-r--r-- 1 root root 2192352 May  6 07:48 vmlinuz-2.6.32-5-486
-rw-r--r-- 1 root root 3734224 Jun 10 11:00 vmlinuz-2.6.32-custom8

The actual kernel, vmlinuz, is 1.5MB (70%) larger, but no longer has an initrd (saves 8.7 MB) or /lib/modules directory (saves 67 kB). Boot time savings seems negligible so far

Here is the final working config file.

Tuesday, June 5, 2012

External VGA monitor and Xrandr in XFCE

Finally, around Ubuntu 11.10 or so, my laptop's VGA output became usable.

Hooray for all those kernel hackers!

Now, the VGA monitor is detected during an existing X session (instead of restarting X), it doesn't flicker anymore, and XFCE's settings show it right away.

1) Plug in the monitor and turn it on. XFCE can already be running.

2) Go to Settings Manager --> Display, click on the monitor, and click the "use this output" checkbox.

[Optional] 3) Go to Settings Manager --> Desktop, and choose a distinctive desktop for the monitor.

[Optional] 4) To use the monitors side-by-side (extended mode) instead of the default mirrored mode, open a terminal window and try:
$ xrandr
Screen 0: minimum 320 x 200, current 1366 x 1536, maximum 8192 x 8192
VGA-0 connected 1024x768+0+768 (normal left inverted right x axis y axis) 304mm x 228mm
   1024x768       74.9*    75.1     70.1     71.8     60.0  
   832x624        74.6  
   640x480        72.8     75.0     66.7     60.0  
   640x400        70.1  
LVDS connected 1366x768+0+0 (normal left inverted right x axis y axis) 344mm x 193mm
   1366x768       60.0*+
   1152x768       59.8  
   1024x768       59.9  
   720x480        59.7  
   640x480        59.4

From this command, you can see that the displays are named LVDS (laptop) at 1366x768 and VGA-0 (ext monitor) at 1024.768. Let's move VGA-0 to the right of LVDS:

$ xrandr --output VGA-0 --auto --right-of LVDS

The command also works with --left-of, --above, and --below. Xrandr changes like this are not persistent across restarts, so redo it every time, put it in a bash alias, or script it into the startup.

Now I have a second (smaller) screen to the right.

Saturday, June 2, 2012

How to use a null-modem serial cable

This will be useful when I need access to a headless server (or any other headless machine) with my laptop.

For example, my little mini-ITX headless server needs occasional terminal input, and leaving my big screen and keyboard nearby is taking up valuable space that I prefer to devote to more important laundry or loose change or perhaps a book or two.

Desktop clutter comparison: extra monitor & keyboard vs serial cable & laptop

Headless Mini-ITX board booted
to Debian 6 (LXDE)
using native keyboard and monitor.
Headless Mini-ITX board booted
to GRUB, seen through serial port.
Laptop acting as terminal


The server has a good, old-fashioned RS-232, 9-pin serial port for the console connection. One client also has an RS-232 port. A different client (laptop) has only USB ports.

  • $3.49 for a 6-foot serial null-modem cable, female RS-232 to female RS-232.

  • $3.19 for a 3-foor USB-to-male RS-232 adapter cable. This is for later when I try from my laptop.

  • Connectivity Test of the null-modem cable between two machines with serial ports. This test proves that the cable is really a null-modem cable (not a data cable) and working, that the serial ports are working, and that both systems are configured correctly,  If possible, avoid using the USB-to-serial adapter for this test - it adds another variable unnecessarily.

    Plug the cable into a serial port on each machine. Designate one machine Server and the other machine Client. It really does not matter which.

    On the Server , open the serial port, and leave it open.
    # getty -L ttyS0 115200 vt100

    On the Client with a serial port, install the screen application, then use it at a terminal emulator to connect over the serial port.
    # apt-get install screen
    # screen /dev/ttyS0 115200
      (Hit return once or twice)

    When finished with screen, quit using the command (CTRL+A , then backslash \)

    You should be able to login to the Server from the Client. If you have problems:
    • Make sure you didn't start multiple screen sessions (they interfere)
    • Make sure neither machine is running both getty and screen on the same port (they interfere). One machine should run getty, the other should run screen.
    • Make sure both sides are using the same speed (115200). This modem setup does not auto-negotiate speeds!
    • Try varying which service starts first. After each attempt, kill both the screen and getty services (but only the getty running on that port!)

    Configuring the Server

    If the console is available during a reboot, then you don't need that external monitor or keyboard anymore! A nearby laptop or desktop can serve (this saves me a lot of valuable space). In order to be useful at reboot time, the bootloader and startup need to be configured to use the serial port.

    I'm assuming the first serial port (serial 0 = COM1 = /dev/ttyS0)

    1) Add the serial port to the bootloader.
       If using grub, see the well-written instructions on the Ubuntu Wiki.
       If the bootloader is Syslinux, add the following line to /boot/syslinux.cfg. It must be the first line.
    serial 0 115200

    2) This server runs Debian 6, which uses sysvinit (for newer, upstart-based systems, see these instructions). Edit the /etc/inittab file. Add the following line near Line 55, with the other getty respawn lines:
    T0:2345:respawn:/sbin/getty -L 115200 ttyS0 vt100

    Reload the inittab using the command init q

    3) Edit the /etc/securetty file. In Debian 6, this isn't necessary, the proper ttyS0 entry to create a serial port already exists.

    Even better instructions
    Thorough instructions, including bootloaders
    Syslinux instructions

    Configuring the Client. Not much to do here, since screen is already installed. It works just like the test. If the client does not have a serial port, use a serial-to-USB adapter as shown below.
    # screen /dev/ttyS0 115200      # Null-modem serial-to-serial
    # screen /dev/ttyUSB0 115200    # Using a serial-to-USB adapter

    I found greater success if the server getty is started first, then the client screen. Including when trying to catch the grub boot screen - push the power-on button on the server, then hit [return] on the client to launch screen, then slowly press a few arrow keys and space bar until the menu appears. Sometimes the connection just doesn't work, and after a pause (grub waiting), you hear disk activity (kernel loading). Then just poweroff the server and try again.

    Switching client and server. Console setup is one-way only, because screen and getty cannot control the same port at the same time. If inittab on the server causes getty to respawn, then edit (comment out) and reload inittab to kill that getty.

    Running screen as user instead of root. Screen might refuse to start for a user if the serial port is owned by root. Running as a user is preferable if you'll be using the serial port a lot.

    Create a new group, add the group to the port, change permissions on the port to match the group, and add the user to the group.

    # ls -l /dev/ttyS0
    crw------- 1 root root 4, 64 Jun 1 04:25 /dev/ttyS0
    # groupadd serialport
    # chgrp serialport
    # chmod 0060 /dev/ttys0
    # ls -l /dev/ttyS0
    c---rw---- 1 root serialport 4, 64 Jun 1 04:27 /dev/ttyS0
    # usermod <my-name> --append --groups serialport

    Log out, log back in, now user can access screen.

    $ screen /dev/ttyS0 115200

    Capturing console output to a text file. This is handy for debugging and keeping notes. Screen doesn't really seem to have this capability. But hooray for the shell environment - you can use the 'tee' command. The following will save a textfile that gets overwritten each session, so feel free to change the name each time.

    $ screen /dev/ttyS0 115200 | tee /path/to/console-recording-textfile