Starting a Simutrans server

I love Simutrans. It's a great game.

Simutrans has a multiplayer version, if somebody hosts a server. I want to host a server version compatible with an Ubuntu release.

• Will it overload my server's CPU and memory?
• Is my network connection adequate?

Let's find out.
These instructions are for my Debian 6 server:

The server version has no graphics - that saves a lot of CPU overhead! But it must be compiled from source. Instructions are on the Simutrans Wiki, and even better instructions are on Timothy's blog post.

The basic theory is that we:

a) Determine the version of Simutrans to install. 

b) Do a complete client-install of the correct version, including base, binary, config, and pakset. 

c) Download the source code. 

d) Compile a server-only binary. 

e) Replace the original client binary with the compiled server binary. 

f) Edit the config file.

A Simutrans install consists of four items.

1. The executable/binary.
2. Simutrans-base, all those supporting languages and config files that the binary needs to work.
3. The config file, simutrans/simuconf.tab, included with simutrans-base.
4. The pakset, containing all the factories, vehicles, and other themes.

Simutrans has two paksets in Ubuntu: pak64 (the default) and pak128.britain. These instructions assume only pak64.




1) Create the build environment (as root):

# apt-get install build-essential subversion libbz2-dev zlib1g-dev unzip
# exit      # Drop root privileges - do the actual build as a user, not root

The build-essential package includes the rmadison command, used in the next step.

2) Pick the version to install. Client and server should be the same release. See the Simutrans Forum release announcements to match up releases with SVN revisions. I want this server to work with Ubuntu 11.10.

• Check the Ubuntu 11.10 version of the Simutrans binary and the pak64 theme:
  

  $ rmadison simutrans | grep oneiric
   simutrans |  110.0.1-3 | oneiric/universe | source, amd64, armel, i386, powerpc
  $ rmadison simutrans-pak64 | grep oneiric
   simutrans-pak64 |  110.0.1-1 | oneiric/universe | source, all

  (Or check packages.ubuntu.com)
  The version numbers are before the dash, so both are Version 110.0.1

• Check the corresponding Debian version on my server, to see if Simutrans must be installed manually or by package:

  rmadison -u debian simutrans | grep squeeze
   simutrans | 102.2.2~ds1-1 | squeeze | source, amd64, armel, i386, ia64, kfreebsd-amd64, kfreebsd-i386, mips, mipsel, powerpc, s390, sparc

  Since we want to install Version 110.0.1, we cannot use the Debian package (102.2.2) - we must install manually.

• Match the version number (110.0.1) in the Simutrans Forum release announcements:
  Version 110.0.1 executable download URL.
  Version 110.0.1 pak64 download URL.
  Farther down in the same release announcement, you see the SVN revision number: r4359
  

  Release of 110.0.1 (r4359 on 12-Mar-2011):

3) Create the simutrans folder, complete with locations for the downloads, and a working file for compiling the new binary.

$ cd ~
$ mkdir simutrans
$ mkdir simutrans/110.0.1
$ mkdir simutrans/110.0.1/downloads
$ mkdir simutrans/110.0.1/svn

4) Download and unpack simutrans, including the base, binary, config file, and theme pak.

$ cd ~/simutrans/110.0.1/downloads
$ wget http://sourceforge.net/projects/simutrans/files/simutrans/110-0-1/simulinux-110-0-1.zip
$ unzip simulinux-110-0-1.zip -d ~
$ wget http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
$ unzip simupak64-110-0-1.zip -d ~
$ cd ~

NOTE: The pakset location is ~/simutrans/pak. Addons to the paksets in version 111.0 (r4911) is ~/simutrans/pak. Addons to the pakset in version 111.1 (r5115) and later is ~/simutrans/addons/pak.

We now have a functioning simutrans install...if there's an X server and other supporting software. On a server, there probably isn't. You can launch Simutrans using the binary simutrans/simutrans for a great client experience.

Next, we will create the compiled server-optimized binary.

5) Download the Simutrans source:

$ cd simutrans/110.0.1/svn

-r is the revision number from Step 1.
username is anon, password is blank (hit Return).

$ svn co --username anon -r 4359 svn://tron.homeunix.org/simutrans/simutrans/trunk

6) Apply the Debian patch:
In this post, I discovered and triaged Launchpad Bug #931181, which makes non-Debian versions of Simutrans incompatible with Debian versions due to a differing sha-1 implementation.

We need to download and apply the Debian patch to make our server compatible with Debian/Ubuntu clients.

$ cd ~/simutrans/110.0.1/svn/trunk
$ wget http://anonscm.debian.org/gitweb/?p=pkg-games/simutrans.git;a=blob_plain;f=debian/patches/sha1-replacement.diff
$ rm utils/sha1.*
$ patch Makefile debian_patches_sha1-replacement.diff 
$ patch utils/checksum.h debian_patches_sha1-replacement.diff
$ nano utils/sha1.cc   # Create new file from the patch
$ nano utils/sha1.h    # Create new file from the patch

7) Edit the makefile configuration:

$ cp config.template config.default
$ nano config.default    # Use any editor, it needn't be nano

# Uncomment the following lines (and ONLY these lines):
BACKEND = posix
COLOUR_DEPTH = 0
OSTYPE = linux
DEBUG = 3
OPTIMISE = 1
WITH_REVISION = 1

8) Compile the binary:

$ make
$ strip sim   #Reduce binary size from 17.3 MB to 2.5 MB

9) Add the compiled binary to the simutrans install. We'll call this new binary 'simd110'. Note that the original (client) binary can be left in place if you wish, since it's called 'simutrans' but it won't be very useful.

$ cd ~
$ ln simutrans/110.0.1/svn/simutrans/simutrans/trunk/sim simutrans/simd110

If you want to save space, you can copy the binary instead of linking it, and delete the old client binary and the entire svn directory. I recommend testing before deleting all your work!

10) Edit the config file

$nano simutrans/config/simuconf.tab

#################################program stuff##################################
singleuser_install = 1
pak_file_path = pak/

###################################network stuff##############################
server_frames_ahead = 4
server_frames_per_step = 4
server_frames_between_checks = 256   #Smaller!
server_announce_interval = 900
server_dns = myserver.domain.net
server_name = Ubuntu 11.10 Compatible
server_comments = If you use Ubuntu 11.10 (Oneiric), this is compatible with the Simutrans version in the Software Center
server_email = me@myemail.domain.net
server_pakurl = http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
server_infurl = http://myserver.domain.net/server-info.html
pause_server_no_clients = 1

11) Use the old client (non-server) binary to create the initial map. The headless server version has no way to choose a map pattern or create a new map, and you have no way to look at it to see if you like it! Save the map as "servergame.sve" inside the simutrans directory.

12) Test the server binary. To do this, you need a client install *and* a server install with identical paksets but preferably different config files. The best test is, of course, from a working stock Ubuntu 11.10 install.

Start the server on port 13353 with

$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3 -load servergame.sve

Try connecting from the client to the server a few times.
Check the server's simutrans directory to ensure the game is saved upon each connect/disconnect.

Try killing the server and restarting it:

$ pkill simd110
$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3

You now have a fully-operational simutrans server, compatible with the desired Ubuntu version of Simutrans. This is not a complete setup - it's best to configure the server with a special non-shell-access user so neither root nor a real user are running the server. I'll talk about that in a different post.

Testing a Simutrans Server

If you're experimenting with Simutrans' multiplayer, you will eventually want to try starting your own server. Here's what you need to know.

First, the normal installed version of Simutrans works as either a client and a server.

For example, on a Linux system:

     To start Simutrans normally (client mode):

     $ simutrans

     To start Simutrans as a server:

     $ simutrans -server

Second, you can run a client and a server on the same machine. To connect the client to the server, use the loopback address (127.0.0.1) or the machine's IP address in the client's connect dialog.

Now we get into pesky little details:

- The client and server must both be using the same pakset version and addons (if any). The server and client compare hashes of each pakset and addon item, and refuse to connect if every hash doesn't match.

- The client and server should (not must) be using the same game version, or within a release or two. Network play is still pretty new, improvements are still being added, and backward compatibility with older versions is not guaranteed.

Installing a TP-Link TL-WN722N (Atheros AR9721 802.11n) wireless dongle into Debian 6

My stock command-line Debian 6 (squeeze) install needs a few additional packages to handle a wireless dongle.

According to the Debian wiki, this chipset requires the firmware-atheros package, which is non-free.

Also the kernel needs to be 2.6.39 or higher...so stock squeeze (2.6.32) won't work, but the version in squeeze-backports (2.6.39) will work. Alternately, I could upgrade to Wheezy (the future Debian 7.0) or roll-my-own kernel.

Edit /etc/apt/sources.list to add the non-free archive and backports.

deb http://ftp.us.debian.org/debian/ squeeze main non-free
deb http://security.debian.org/ squeeze/updates main non-free
deb http://ftp.us.debian.org/debian/ squeeze-updates main non-free
deb http://backports.debian.org/debian-backports squeeze-backports main non-free

Begin using non-free and backports by updating the package cache:

# apt-get update

Install the packages needed to get wireless working in general, and this dongle in particular:

# apt-get install wireless-tools usbutils firmware-atheros

For backported packages, must specify. Use the Debian package search tool to locate the correct package name in the backport archive. For my 486-compatible system, it's:

# apt-get -t squeeze-backports install linux-image-2.6.39-bpo.2-486

Finally, create a wireless interface entry in /etc/network/interfaces

# The wireless interface
auto wlan0
iface wlan0 inet dhcp
    wireless-essid YourNetwork
    wireless-mode Managed

Since I use the syslinux bootloader (instead of grub) to get around the limitations of my 10-year-old motherboard, the new /boot/initrd.img generated by the new kernel must be manually copied over to my /boot partition. The following commands are specific to my setup only! IGNORE THIS SECTION IF YOU USE GRUB!

Remove unused grub stuff:

rm -r /boot/grub
rm /boot/System.map-2.6.39-bpo.2-486
rm /boot/config-2.6.39-bpo.2-486

Backup the old bootables:

# mv /boot/vmlinuz /boot/vmlinuz.old
# mv /boot/initrd.img /boot/initrd.old

Change the name of the new bootlables to 8-and-3 compatible so syslinux can use them:

mv /boot/initrd.img-2.6.39-bpo.2-486 /boot/initrd.img
mv /boot/vmlinuz-2.6.39-bpo.2-486 /boot/vmlinuz

Reboot, and suddenly the wireless works...until a bug or two intercede.

The dongle keeps crashing due to a possible kernel bug (or perhaps several). Interestingly, I also seemed to have two dhclient processes running. So lots of narrowing down to do...

1) Two dhclient processes upon boot (killed one manually)
2) Kernel bug that kills the dongle with dmesg: phy0: wlan0: No probe response from AP
3) Kernel or crda bug that thinks I'm in China (CN) instead of the US (US). The preferred fix is to install the crda package (not available for Debian 6.0). Alternate fix is to add a file /etc/modprobe.d/regdom.conf, containing the line options cfg80211 ieee80211_regdom=US (source)

Let's reboot.

Aha. After a reboot, ps -e | grep dh shows two dhclient processes. I killed the higher-numbered (later-started) one...which killed both of them and respawned a fresh dhclient process. So there's a bug: dhclient is getting spawned twice upon boot. But it's not related to the crash - that's a separate bug....

Repairing a Vicks Warm Mist V745A humidifier

Room humidifiers are not difficult. They hum or hiss and water vapor comes out. They glug occasionally as the tank empties.

But not our new one. No glug, no hiss.
Before returning it, let's see it it's easily fixable.

The lack of glug is the key clue - the valve isn't letting water into the heating chamber.

Here's the valve. It's inside the tank-fill-cap.

Valve doesn't work

Correct

Looks like it was assembled improperly - the stem fills the rubber hole, not allowing water through...but looks are deceiving. I suspect the rubber seal isn't supposed to be inside the hole at all, but a sealing cap on top of the hole.

Use the stem to gently poke the rubber seal out of the hole, and suddenly the seal's size and shape make more
sense. And it even seats on the stem properly.

Best of all, now it even works.

Local file cache to speed up debootstrap

For my server project, I'm using debootstrap. Here's how to speed up debootstrap by using the existing local cache instead of re-downloading hundreds of MB of packages. Using debootstrap requires a little preparation, and installation of the 'apt-move' package.

Apt-move is a nifty little tool that moves (or copies) existing packages out of your cache and into a debian archive format. Apt-move creates all three elements of the archive: The various directories full of packages, the Release file, and the Packages file.

This is only a useful technique if your package cache is complete. If you have cleaned it out since your last major upgrade, then this technique may fail. It's easy to regenerate the cache, though the large download is probably what you were trying to avoid!

1) Prepare the local mirror. For this example, let's use 6.0 (squeeze):

Create a place to host the mirror
# mkdir /var/local/mirrors

     Install the apt-move package. 
# apt-get install apt-move

     Edit the configuration file:
# nano /etc/apt-move.conf
LOCALDIR=/var/local/mirrors
DIST=squeeze
COPYONLY=yes

     Create the mirror
# apt-move update

You can see the mirror in /var/local/mirrors

2) Use the new mirror to debootstrap a new chroot environment

Create the new environment
# mkdir /var/new-env

     Debootstrap from the local mirror
# debootstrap squeeze /var/new-env file:///var/local/mirrors

Environment complete, ready to chroot into.

3) If debootstrap fails, it's probably due to a file missing from the cache. There are two ways to regenerate.

The hard way is to work error by error:
1) Run debootstrap, and identifying the missing file from the error message.
2) Use dpkg -S filename to identify the package name.
3) Use apt-get --reinstall install packagename to add it to the package cache.
4) Use apt-move update to move the package to the mirror.
5) Run debootstrap again.

The easy way is to use debootstrap to recreate all the missing files, and copy them into the cache:

# mkdir /tmp/cache
# debootstrap --arch=i386 --download-only squeeze /tmp/cache
# mv /tmp/cache/var/cache/apt/archives/* /var/cache/apt/
# rm -rf /tmp/cache
# apt-move update

The amazing simplicity of sshfs

So I'm out for a few minutes, waiting in a lounge during a children's choir rehearsal. And I want something off my home server.

sudo apt-get install sshfs
ssh_port=12345                  # Use your own port number, of course
server_name=myserver.dyndns.org # Use your own server, of course
user_name=myname                # Your username on the server
local_mount_point=/path/to/local/mountpoint
mkdir ${local_mount_point}
fwknop -A 'tcp/${ssh_port}' -s -D ${server_name}
sshfs ${user_name}@${server_name}:/path/on/server ${local_mount_point} -p ${ssh_port}

And SHAZAM! Remote server path shows up in File Manager.

No mucking with Samba or PPTP or other strangeness. Just

sshfs me@myserver.dyndns.org:/mnt/Common/ /tmp/server/ -p 12345

To unmount:

fusermount -u ${local_mount_point}

Creating a bootable USB drive when the BIOS supports only USB-ZIP

Mini-ITX Epia 5000-L motherboard, ready to boot.
The red USB stick is the boot medium,
1GB USB stick with a complete Debian 6 install
masquerading as a 250MB USB-ZIP drive.

My used mini-ITX motherboard has a BIOS limitation that prevents it from booting from ordinary hybrid-iso images. The BIOS doesn't recognize USB-HDD (the popular type used today), but only older USB-FDD (USB floppy drive, limited to 1.4MB) or USB-ZIP (USB Zip Drive, limited to 250MB).

Who today has useful systems that fit in 250MB?

This is a demonstration of how to use a chainloader to get around the 250MB restriction. Here is how to build a working full install of Debian 6 on a bootable USB Stick. It's a full install (not a live-install that uses a ramdisk), so all the slow read/writes to the USB stick will make the system very slow and wear out the USB stick prematurely. Again, this is just a demo of how to use a chainloader to get around that 250MB restriction.

1) Create the Linux system

This is the long and complicated part. Use debootstrap to create a complete system somewhere else. I explained how I did it in this post for an SD card, and it's almost exactly the same.

Creating the complete system in, say, /var/usb-env should include creating /boot/initrd.img.version# and vmlinuz.version# , but not grub. We will use initrd.img and vmlinuz in Step #3, and this demo uses the simpler syslinux instead of grub.

2) Prepare the USB Stick

To be understood by the old BIOS, the USB stick needs an MBR up front, followed by a FAT boot partition of less than 250MB and labeled as the 4th partition (though it's really the first partition). After that, we can do whatever we like.

The best tool for creating a USB-ZIP bootable partition is 'mkdiskimage', part of the syslinux package. Everything below should be done as root to the unmounted USB-stick:

MAKE SURE you are using the right /dev/DEVICE
    /dev/sdb was mine, but probably won't be yours!

# mkdiskimage -Mz4 -i usb-zip /dev/sdb 15 64 32
    -M and 15: Create a 15 MB partition (I only needed 12 MB)
    -z4: Create a zip-disk geometry on partition 4 (normal for zip disks)
    -i usb-zip: Name of the partition. Use any name you wish
    Of course, your /dev/DEVICENODE may vary. BE SURE you are using the right node!
    15 64 32: 15 MB (see -M), 64 heads, 32 sectors (62 and 32 are required for zip)

After the USB-ZIP partition is made, you can do the rest of the partitions in your favorite editor. I used Gparted to create a 700MB Linux partition and the remaining space on the device as a swap partition.

3) Install files onto the boot partition

The install files consist of the bootloader, and the Linux /boot/initrd.img.version# and vmlinuz.version# files. The bootloader simply tells the system to load the linux kernel (vmlinuz), then to load the temporary system startup files (initrd.img) that the kernel can understand. One of the appended command options tells initrd where to find the final system root mountpoint on the other partition.

There are many bootloaders that can be used here, including grub. For simplicity, I'm using syslinux instead of grub. I don't need all of grub's configuration options - I just want the bootloader to immediately start loading the kernel and initrd.

For syslinux, we only need four files on the boot partition: The syslinux binary, syslinux.cfg, vmlinuz, and initrd.img. We also need to know the UUID of the other partitions for the syslinux.cfg file.

Once again, MAKE SURE you are using the right /dev/DEVICE
    /dev/sdb was mine, but probably won't be yours!

    Check the USB stick partitions before mounting
# fsck -f /dev/sdb1  # Blank linux, -f forces a check
# fsck -f /dev/sdb4  # Blank boot

    Get the UUIDs of the various partitions
# blikd   # Get the UUIDs of the newly created partitions

    Install Syslinux 
# syslinux --install /dev/sdb4

    Mount the boot partition
# mount -t vfat /dev/sdb4 /mnt

    Copy vmlinuz and initrd.img from the prepared system.
    Rename the files to comply with MSDOS 8.3 format
# cp /var/usb-env/vmlinuz.version /mnt/vmlinuz
# cp /var/usb-env/initrd.img.version /mnt/initrd.img

    Create the syslinux.cfg file
# nano /mnt/syslinux.cfg    # Use any editor you wish

##### Begin File
Default USB-Stick

display syslinux.msg
F1      syslinux.f1

prompt 40
timeout 30

Label USB-Stick
  kernel vmlinuz
  append initrd=initrd.img root=UUID=>UUID of the root partition<
##### End file

    Unount the boot partition
# umount /mnt

4) Copy the linux system onto the USB Stick

Once again, MAKE SURE you are using the right /dev/DEVICE
    /dev/sdb was mine, but probably won't be yours!

    Mount the Linux partition
# mount /dev/sdb1 /mnt

    Copy the system files
# cp -r /var/usb-env/{bin,etc,home,lib,media,mnt,opt,root,sbin,selinux,srv,tmp,usr,var} /mnt/

    Create the system mountpoints
# mkdir /mnt/{boot,dev,proc,sys}

    Edit /etc/fstab to add the correct UUIDs for the root, boot, and swap partitions
    and remove the cruft from the creating-system disks.

If you need to discover a UUID, use the 'blkid' command.
# nano /mnt/etc/fstab     # Use any editor you wish, of course

#####Begin File
# <file system> <mount point> <type> <options> <dump>  <pass>
UUID=<UUID of the main system> / ext3 defaults,errors=remount-ro,noatime 0 1
UUID=<UUID of the boot partition> /boot vfat defaults,errors=remount-ro,noatime 0 2 
UUID=<UUID of the swap partition> none  swap sw 0 0 
#####End File

    Unmount the linux partition, check the USB stick for damage once more
# umount /mnt
# fsck -f /dev/sdb4
# fsck -f /dev/sdb1

Successful boot to Debian 6 login prompt!

5) You're Done

Unplug the USB stick, walk over to the system you want to boot, plug in the USB stick, and....well, boot.

Installing Debian 6 onto an SD card, installing into an old VIA EPIA-5000L motherboard

In this post I discussed my new-to-me fanless motherboard, and how I got it to turn on/off, and test the boot.

Next step is to install a hard drive and install Debian 6.

Hardware

This hardware will eventually become a router or light-duty server, so the OS doesn't need huge storage. I used:

• SD-to-IDE adapter (ebay, $8.88)
• The adapter reqires a FDD (floppy disk) power connector. Since my power supply doesn't have that connector, I picked up a molex (IDE HDD power) to FDD adapter (amazon, $3.92)
• 8GB SD card (left over from a now-dead camera). An SD card can also be plugged in to my Ubuntu 11.10 laptop. This turned out to be critically useful for me

Install Failures

Using the same method as this post to make bootable USB drives, I tried the latest Debian 6 businesscard and netinstall disks. Both failed at various points. Sometimes the iso was corrupted during download. Sometimes there was a failure writing the new partition table. Sometimes the table was successfully written, but the new partition couldn't be mounted for install. Sometimes the install was successful, but grub failed to install.

One by one, I winnowed through the issues. MD5 for the iso downloads. Pull the card and reformat it in my laptop to fix the partition table. Finally, I got consistent failures: The businesscard iso failed during the install debootstrap, and the netinstall iso failed to create an initramfs and grub.

The businesscard iso could successfully drop to a working shell, so I used these instructions to manually install debootstrap and see if I could bypass the installer failures. This worked well, though slowly (writing to the SD card is much slower than to a real hard drive), and my many mistakes after this point were a learning experience in filesystem corruption, chroot, initramfs, and grub.

fsck, chroot, initramfs, and grub

Booting to a live environment, dropping to a shell, debootstrapping to an installed system, and then rebooting to test it...only to discover it's unbootable...take a long time for each cycle, mostly due to the SD card slow write speed, and the SD card's seeming love of getting it's filesystem corrupted somewhere in the process each time.

I sped up the process and injected filesystem checks by moving most of it onto a Debian VM on my laptop. The SD card mounts as a USB drive, so the VM must be USB-capable. Lesson Learned: Before mounting/unmounting the card, fsck it! Every time! This is how I got rid of the filesystem corruption, which seemed to be happening from something unclean during shutdown (almost true - I'll tell you the real story later). On some occasions the filesystem was so messed up that I simply wiped it and copied a new system over from the VM.

Now, I could reliably build a system that would boot...but grub would complain or initramfs would drop to a shell. On two occasions it successfully booted, then froze (and corrupted the filesystem). Testing with variations in /boot/grub/grub.cfg fixed grub to boot reliably...only to later learn that grub 'error' wasn't an error at all, but a different configuration problem I had caused somewhere else.

That left a frustrating initramfs refusal to mount the card. I could mount it manually, but that resulted in kernel panic. After a lot of digging through initramfs scripts, I discovered a packaging bug - initramfs relies on klibc-tools...but the dependency is not listed in Debian 6! (It is correct in Ubuntu 10.10). One apt-get later and the system booted reliably every time...but the filesystem was frequently corrupting.

Apt activities that triggered man-db began returning "fopen: Permission denied" errors. Fixed with chown -R man:root /var/cache/man


And the result...

I finally determined that the frequent filesystem corruptions were due to a defective SD-to-IDE adapter. I tested the card reader on my laptop - no problems, so it wasn't the laptop, the card, or my handling. The proof was when the SD card was freshly installed, fsck'd before removal from the laptop, fsck'd again upon insertion to the Epia system, and run for an hour. Missing and corrupted file errors began to proliferate. Still need to check the RAM.


Lessons Learned
1) fsck the SD card before every mount and after every unmount
2) Manually check for the existence of /boot/grub/grub.cfg, /boot/grub/normal.mod, and /sbin/init
3) Use blkid to get the UUID of the root partition. Check that against /boot/grub/grub.cfg

Steps to succesfully creating the SD card
Without wandering down my many blind paths, here is the final process to install a bootable Debian 6 onto an SD card.

1) Use a VM (I used Virtualbox) with USB access on a fast machine (my laptop).
2) Within the VM, create a chroot environment and use debootstrap to create the new linux system that will be copied onto the card.
3) Prepare the SD card, copy the environment, and install grub.
4) Ensure a good system after the first boot from the card.

Creating a VM is beyond the scope of this discussion.

Creating the new linux system happens entirely on the VM. Create the base debian install

# mkdir /var/card-env
# debootstrap --arch=i386 squeeze /var/card-env

Depending on your network connection, debootstrap may take a long time.


Mount the system-required stuff

# mkdir /var/card-env/proc
# mount --bind /proc /var/card-env/proc
# mkdir /var/card-env/dev
# mount --bind /dev /var/card-env/dev
# mount --bind /dev/pts /var/card-env/dev-pts
# mkdir /var/card-env/sys
# mount --bind /sys /var/card-env/sys

Copy useful stuff before chrooting


These files must be edited later in the process

# cp /etc/fstab /var/card-env/etc/
# cp /etc/network/interfaces /var/card-env/etc/network/
# cp /etc/hosts /var/card-env/etc/

Chroot into the card-env

# chroot /var/card-env

Set up the package manager. Do these in the chroot environment.

# echo "deb-src http://ftp.us.debian.org/debian squeeze main" >> /etc/apt/sources.list
# echo "deb http://security.debian.org/ squeeze/updates main" >> /etc/apt/sources.list
# echo "deb-src http://security.debian.org/ squeeze/updates main" >> /etc/apt/sources.list
# apt-get update

If any apt-get errors pop up, take time and fix them now. Do these in the chroot environment.

If any command-not-found errors occur:

Use 'which commandname' to find the full path
for example, 'which date' returns '/bin/date'
Use dpkg -S full-path to find the package name
for example, 'dpkg -S /bin/date' returns 'coreutils'
Reinstall that package
for example, 'apt-get remove coreutils && apt-get install coreutils'

Edit hostname, hosts, and interfaces. Do these in the chroot environment

# echo MyNewHostName > /etc/hostname
# nano /etc/hosts                # Change the hostname
# nano /etc/network/interfaces   # Reduce interfaces to just to loopback device
                               # udev will add the rest anyway upon boot

Add your root and user accounts. Do these in the chroot environment

# passwd root
# adduser MyName

Add essential elements to the new filesystem. Do these in the chroot environment

# apt-get install locales
# dpkg-reconfigure locales     # Choose a UTF-8 in your preferred language
# apt-cache search linux-image # Choose a kernel that matches your architecture
# apt-get install klibc-tools initramfs-tools linux-image-VERSION-i486 grub-pc

Take a few minutes and troubleshoot any kernel-install or initramfs issues. Do this in the chroot environment. Grub will have a lot of incorrect information, and we'll fix that after we plug in the SD card.

Add desired optional packages. Do this in the chroot environment

# apt-get install openssh-server     # For example

Exit the chroot environment, then unmount the system-essentials

# exit
# umount /var/card-env/proc
# umount /var/card-env/dev-pts
# umount /var/card-env/dev
# umount /var/card-env/sys

You now have a complete Debian environment (except for grub and fstab), ready to copy from the VM onto the SD card.

Plug in the SD card to the VM machine, and add the SD card (as a USB) to the VM. Leave it unmounted. udev assigned my card as /dev/sdb.

Partition the card as desired. I created /dev/sdb1 (bootable ext3), and /dev/sdb5 (swap). Do these steps as root. Card must be plugged in but unmounted for these steps:

# mke2fs -j /dev/sdb1   # Assuming you haven't formatted it yet
# mkswap /sev/sdb5   # Assuming you haven't formatted it yet

Mount the card

# fsck /dev/sdb1
# mount /dev/sdb1 /mnt

Change grub's device map to the SD Card

# ls /dev/disk/by_id    # Note the SD Card's ID string
# nano /var/card-env/boot/grub/device.map  # Change HD0 to the SD card

Change grub's config file and fstab to the SD Card partition

# blkid        # Note the linux partition's UUID string
# nano /var/card-env/boot/grub/grub.cfg    # Change all the UUID strings to the SD Card Partition
# nano /var/card-env/etc/fstab             # Change the UUID strings on the root and swap lines

Copy the environment over to the card

# cp /var/card-env/* /mnt/
# grub-install --root-directory /mnt /dev/sdb

Final checks

# blkid   # Note the UUID of the card boot partition
# nano /mnt/boot/grub/grub.cfg   # Check for the same UUID
# ls -l /mnt/sbin/init    # Check for existence
# ls /mnt/boot            # Check for an initrd.img file and a vmlinuz file

Unmount and fsck the SD card

# umount /mnt
# fsck /dev/sdb1

You now have a complete bootable Debian 6 installed on the SD card, ready to try booting from.

During the first boot from the SD card, do the following:

# update-initramfs -u  # Get rid of "Can't find disk" errors
# tune2fs -c 1 /dev/sda # Set the flag to run fsck every boot until further notice
# ifconfig -a   # Note the MAC address and interface of the ethernet port

Keep the current system interfaces, delete leftovers from the laptop environment

# nano /etc/udev/rules.d/70-persistent-net-rules

Bring up the network and prepare apt-get

# ifconfig up eth1  # Your eth may differ
# dhclient eth1
# apt-get update
# apt-get upgrade

Add essential system components

# apt-get install rsyslog