Simutrans packaging bug

After a bit of scratching my head, I finally figured out that Simutrans 110.0.1 (the version in Ubuntu 11.10) and the Debian-packaged version that is actually in Ubuntu have a weird incompatibility.

About a year ago, Simutrans added a network-multiplayer-server feature. But Debian/Ubuntu-packaged clients can only communicate with Debian/Ubuntu-packaged servers. Similarly, manually-installed clients can only talk to manually installed servers.

Well, that's certainly not intended behavior - all the servers should be able to talk to all the clients, regardless of whether they were packaged or not!

So I filed LP #931181, because the Debian Games Team needs to know.
And I opened a thread on the developer's forum.

Clue: One developer immediately noted that it seemed to be a checksum issue. (Thanks, Dwachs!)

Well, let's check that. Let's take one item in the pakset, building.PostOffice.pak, and see how it checksum:

Let's hash it manually first:

$ openssl sha1 ~/simutrans110/simutrans/pak/building.PostOffice.pak 
SHA1(/home/me/simutrans110/simutrans/pak/building.PostOffice.pak)= 8b1b5caa71013474039f357094cd1716bd3800bd

$ openssl sha1 /usr/share/games/simutrans/pak/building.PostOffice.pak 
SHA1(/usr/share/games/simutrans/pak/building.PostOffice.pak)= 8b1b5caa71013474039f357094cd1716bd3800bd

So the those pakset items (and, by extension, probably the entire pakset) are identical.

Next, let's check the logs of success and failure:

Succesful (Ubuntu 11.10) server: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

Succesful (Ubuntu 11.10) client: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

Failed (Manually-installed) server: Message: pakset_info_t::debug: PostOffice -> sha1 = 263C0CB8C858272A12FF54FC64F57292AF8ADAB7

Failed (Ubuntu 11.10) client: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

It seems that Dwachs put me on the right track. Given identical files to checksum, the two binaries checksum differently, and both do it differently than Ubuntu's default method, even though each claim to be using the same sha1!


Next, let's look for the changed sha-1 patch at the Debian Games Team git repository.

Aha! February 16, 2011: Add replacement for non-free SHA-1 implementation
Aha! February 16, 2011: get-orig-source: Remove non-free SHA-1 implementation.

The patch, is clearly a replacement for the original Simutrans trunk/utils/sha1.cc
And, looking at the current tree, the patch is still applied.

Er, did I just walk into a year-old dispute between Debian and Simutrans devs?

Next, I need to prove that the patch is the culprit. I need to apply the patch to a stock simutrans source, compile it, run it as a server (in debug mode), successfully join the game from a debian-install client.


Happily, I already have a stock 110.0.1 source, setup, and pakset in my simutrans110 directory, from earlier troubleshooting. Full instructions on how to compile are in this post.

Let's build the unpatched (stock) version first. This should be compatible with a manual install, and incompatible with the Debian version.

$ cp simutrans110 r4359-stock
$ cd r4359-stock/svn/trunk
$ make clean
$ make
$ strip sim
$ cp sim ~/r4359-stock/simutrans/sim-unpatched

Now, let's build a patched version. This should be compatible with the Debian version, and incompatible with a manual install.

$ sudo apt-get install libssl-dev
$ cp simutrans110 r4359-patched
$ cp Desktop/debian_patches_sha1-replacement.diff r4359-patched/svn/trunk
$ cd r4359-patched/svn/trunk

  --- Apply the patch ---
$ rm utils/sha1.*
$ patch Makefile debian_patches_sha1-replacement.diff 
$ patch utils/checksum.h debian_patches_sha1-replacement.diff
$ nano utils/sha1.cc
$ nano utils/sha1.h

$ make clean
$ make
$ strip sim
$ cp sim ~/r4359-stock/simutrans/sim-patched

Finally, we test.

Ubuntu 11.10's package install of Simutrans 110.0.1 can join a game served by sim-patched, but not a game served by sim-unpatched.

A manually-installed version of 110.0.1 is, as expected, the reverse. It can join a game served by sim-unpatched, but not a game served by sim-patched.

Starting a Simutrans server

I love Simutrans. It's a great game.

Simutrans has a multiplayer version, if somebody hosts a server. I want to host a server version compatible with an Ubuntu release.

• Will it overload my server's CPU and memory?
• Is my network connection adequate?

Let's find out.
These instructions are for my Debian 6 server:

The server version has no graphics - that saves a lot of CPU overhead! But it must be compiled from source. Instructions are on the Simutrans Wiki, and even better instructions are on Timothy's blog post.

The basic theory is that we:

a) Determine the version of Simutrans to install. 

b) Do a complete client-install of the correct version, including base, binary, config, and pakset. 

c) Download the source code. 

d) Compile a server-only binary. 

e) Replace the original client binary with the compiled server binary. 

f) Edit the config file.

A Simutrans install consists of four items.

1. The executable/binary.
2. Simutrans-base, all those supporting languages and config files that the binary needs to work.
3. The config file, simutrans/simuconf.tab, included with simutrans-base.
4. The pakset, containing all the factories, vehicles, and other themes.

Simutrans has two paksets in Ubuntu: pak64 (the default) and pak128.britain. These instructions assume only pak64.




1) Create the build environment (as root):

# apt-get install build-essential subversion libbz2-dev zlib1g-dev unzip
# exit      # Drop root privileges - do the actual build as a user, not root

The build-essential package includes the rmadison command, used in the next step.

2) Pick the version to install. Client and server should be the same release. See the Simutrans Forum release announcements to match up releases with SVN revisions. I want this server to work with Ubuntu 11.10.

• Check the Ubuntu 11.10 version of the Simutrans binary and the pak64 theme:
  

  $ rmadison simutrans | grep oneiric
   simutrans |  110.0.1-3 | oneiric/universe | source, amd64, armel, i386, powerpc
  $ rmadison simutrans-pak64 | grep oneiric
   simutrans-pak64 |  110.0.1-1 | oneiric/universe | source, all

  (Or check packages.ubuntu.com)
  The version numbers are before the dash, so both are Version 110.0.1

• Check the corresponding Debian version on my server, to see if Simutrans must be installed manually or by package:

  rmadison -u debian simutrans | grep squeeze
   simutrans | 102.2.2~ds1-1 | squeeze | source, amd64, armel, i386, ia64, kfreebsd-amd64, kfreebsd-i386, mips, mipsel, powerpc, s390, sparc

  Since we want to install Version 110.0.1, we cannot use the Debian package (102.2.2) - we must install manually.

• Match the version number (110.0.1) in the Simutrans Forum release announcements:
  Version 110.0.1 executable download URL.
  Version 110.0.1 pak64 download URL.
  Farther down in the same release announcement, you see the SVN revision number: r4359
  

  Release of 110.0.1 (r4359 on 12-Mar-2011):

3) Create the simutrans folder, complete with locations for the downloads, and a working file for compiling the new binary.

$ cd ~
$ mkdir simutrans
$ mkdir simutrans/110.0.1
$ mkdir simutrans/110.0.1/downloads
$ mkdir simutrans/110.0.1/svn

4) Download and unpack simutrans, including the base, binary, config file, and theme pak.

$ cd ~/simutrans/110.0.1/downloads
$ wget http://sourceforge.net/projects/simutrans/files/simutrans/110-0-1/simulinux-110-0-1.zip
$ unzip simulinux-110-0-1.zip -d ~
$ wget http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
$ unzip simupak64-110-0-1.zip -d ~
$ cd ~

NOTE: The pakset location is ~/simutrans/pak. Addons to the paksets in version 111.0 (r4911) is ~/simutrans/pak. Addons to the pakset in version 111.1 (r5115) and later is ~/simutrans/addons/pak.

We now have a functioning simutrans install...if there's an X server and other supporting software. On a server, there probably isn't. You can launch Simutrans using the binary simutrans/simutrans for a great client experience.

Next, we will create the compiled server-optimized binary.

5) Download the Simutrans source:

$ cd simutrans/110.0.1/svn

-r is the revision number from Step 1.
username is anon, password is blank (hit Return).

$ svn co --username anon -r 4359 svn://tron.homeunix.org/simutrans/simutrans/trunk

6) Apply the Debian patch:
In this post, I discovered and triaged Launchpad Bug #931181, which makes non-Debian versions of Simutrans incompatible with Debian versions due to a differing sha-1 implementation.

We need to download and apply the Debian patch to make our server compatible with Debian/Ubuntu clients.

$ cd ~/simutrans/110.0.1/svn/trunk
$ wget http://anonscm.debian.org/gitweb/?p=pkg-games/simutrans.git;a=blob_plain;f=debian/patches/sha1-replacement.diff
$ rm utils/sha1.*
$ patch Makefile debian_patches_sha1-replacement.diff 
$ patch utils/checksum.h debian_patches_sha1-replacement.diff
$ nano utils/sha1.cc   # Create new file from the patch
$ nano utils/sha1.h    # Create new file from the patch

7) Edit the makefile configuration:

$ cp config.template config.default
$ nano config.default    # Use any editor, it needn't be nano

# Uncomment the following lines (and ONLY these lines):
BACKEND = posix
COLOUR_DEPTH = 0
OSTYPE = linux
DEBUG = 3
OPTIMISE = 1
WITH_REVISION = 1

8) Compile the binary:

$ make
$ strip sim   #Reduce binary size from 17.3 MB to 2.5 MB

9) Add the compiled binary to the simutrans install. We'll call this new binary 'simd110'. Note that the original (client) binary can be left in place if you wish, since it's called 'simutrans' but it won't be very useful.

$ cd ~
$ ln simutrans/110.0.1/svn/simutrans/simutrans/trunk/sim simutrans/simd110

If you want to save space, you can copy the binary instead of linking it, and delete the old client binary and the entire svn directory. I recommend testing before deleting all your work!

10) Edit the config file

$nano simutrans/config/simuconf.tab

#################################program stuff##################################
singleuser_install = 1
pak_file_path = pak/

###################################network stuff##############################
server_frames_ahead = 4
server_frames_per_step = 4
server_frames_between_checks = 256   #Smaller!
server_announce_interval = 900
server_dns = myserver.domain.net
server_name = Ubuntu 11.10 Compatible
server_comments = If you use Ubuntu 11.10 (Oneiric), this is compatible with the Simutrans version in the Software Center
server_email = me@myemail.domain.net
server_pakurl = http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
server_infurl = http://myserver.domain.net/server-info.html
pause_server_no_clients = 1

11) Use the old client (non-server) binary to create the initial map. The headless server version has no way to choose a map pattern or create a new map, and you have no way to look at it to see if you like it! Save the map as "servergame.sve" inside the simutrans directory.

12) Test the server binary. To do this, you need a client install *and* a server install with identical paksets but preferably different config files. The best test is, of course, from a working stock Ubuntu 11.10 install.

Start the server on port 13353 with

$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3 -load servergame.sve

Try connecting from the client to the server a few times.
Check the server's simutrans directory to ensure the game is saved upon each connect/disconnect.

Try killing the server and restarting it:

$ pkill simd110
$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3

You now have a fully-operational simutrans server, compatible with the desired Ubuntu version of Simutrans. This is not a complete setup - it's best to configure the server with a special non-shell-access user so neither root nor a real user are running the server. I'll talk about that in a different post.

Testing a Simutrans Server

If you're experimenting with Simutrans' multiplayer, you will eventually want to try starting your own server. Here's what you need to know.

First, the normal installed version of Simutrans works as either a client and a server.

For example, on a Linux system:

     To start Simutrans normally (client mode):

     $ simutrans

     To start Simutrans as a server:

     $ simutrans -server

Second, you can run a client and a server on the same machine. To connect the client to the server, use the loopback address (127.0.0.1) or the machine's IP address in the client's connect dialog.

Now we get into pesky little details:

- The client and server must both be using the same pakset version and addons (if any). The server and client compare hashes of each pakset and addon item, and refuse to connect if every hash doesn't match.

- The client and server should (not must) be using the same game version, or within a release or two. Network play is still pretty new, improvements are still being added, and backward compatibility with older versions is not guaranteed.

Installing a TP-Link TL-WN722N (Atheros AR9721 802.11n) wireless dongle into Debian 6

My stock command-line Debian 6 (squeeze) install needs a few additional packages to handle a wireless dongle.

According to the Debian wiki, this chipset requires the firmware-atheros package, which is non-free.

Also the kernel needs to be 2.6.39 or higher...so stock squeeze (2.6.32) won't work, but the version in squeeze-backports (2.6.39) will work. Alternately, I could upgrade to Wheezy (the future Debian 7.0) or roll-my-own kernel.

Edit /etc/apt/sources.list to add the non-free archive and backports.

deb http://ftp.us.debian.org/debian/ squeeze main non-free
deb http://security.debian.org/ squeeze/updates main non-free
deb http://ftp.us.debian.org/debian/ squeeze-updates main non-free
deb http://backports.debian.org/debian-backports squeeze-backports main non-free

Begin using non-free and backports by updating the package cache:

# apt-get update

Install the packages needed to get wireless working in general, and this dongle in particular:

# apt-get install wireless-tools usbutils firmware-atheros

For backported packages, must specify. Use the Debian package search tool to locate the correct package name in the backport archive. For my 486-compatible system, it's:

# apt-get -t squeeze-backports install linux-image-2.6.39-bpo.2-486

Finally, create a wireless interface entry in /etc/network/interfaces

# The wireless interface
auto wlan0
iface wlan0 inet dhcp
    wireless-essid YourNetwork
    wireless-mode Managed

Since I use the syslinux bootloader (instead of grub) to get around the limitations of my 10-year-old motherboard, the new /boot/initrd.img generated by the new kernel must be manually copied over to my /boot partition. The following commands are specific to my setup only! IGNORE THIS SECTION IF YOU USE GRUB!

Remove unused grub stuff:

rm -r /boot/grub
rm /boot/System.map-2.6.39-bpo.2-486
rm /boot/config-2.6.39-bpo.2-486

Backup the old bootables:

# mv /boot/vmlinuz /boot/vmlinuz.old
# mv /boot/initrd.img /boot/initrd.old

Change the name of the new bootlables to 8-and-3 compatible so syslinux can use them:

mv /boot/initrd.img-2.6.39-bpo.2-486 /boot/initrd.img
mv /boot/vmlinuz-2.6.39-bpo.2-486 /boot/vmlinuz

Reboot, and suddenly the wireless works...until a bug or two intercede.

The dongle keeps crashing due to a possible kernel bug (or perhaps several). Interestingly, I also seemed to have two dhclient processes running. So lots of narrowing down to do...

1) Two dhclient processes upon boot (killed one manually)
2) Kernel bug that kills the dongle with dmesg: phy0: wlan0: No probe response from AP
3) Kernel or crda bug that thinks I'm in China (CN) instead of the US (US). The preferred fix is to install the crda package (not available for Debian 6.0). Alternate fix is to add a file /etc/modprobe.d/regdom.conf, containing the line options cfg80211 ieee80211_regdom=US (source)

Let's reboot.

Aha. After a reboot, ps -e | grep dh shows two dhclient processes. I killed the higher-numbered (later-started) one...which killed both of them and respawned a fresh dhclient process. So there's a bug: dhclient is getting spawned twice upon boot. But it's not related to the crash - that's a separate bug....

Repairing a Vicks Warm Mist V745A humidifier

Room humidifiers are not difficult. They hum or hiss and water vapor comes out. They glug occasionally as the tank empties.

But not our new one. No glug, no hiss.
Before returning it, let's see it it's easily fixable.

The lack of glug is the key clue - the valve isn't letting water into the heating chamber.

Here's the valve. It's inside the tank-fill-cap.

Valve doesn't work

Correct

Looks like it was assembled improperly - the stem fills the rubber hole, not allowing water through...but looks are deceiving. I suspect the rubber seal isn't supposed to be inside the hole at all, but a sealing cap on top of the hole.

Use the stem to gently poke the rubber seal out of the hole, and suddenly the seal's size and shape make more
sense. And it even seats on the stem properly.

Best of all, now it even works.

Local file cache to speed up debootstrap

For my server project, I'm using debootstrap. Here's how to speed up debootstrap by using the existing local cache instead of re-downloading hundreds of MB of packages. Using debootstrap requires a little preparation, and installation of the 'apt-move' package.

Apt-move is a nifty little tool that moves (or copies) existing packages out of your cache and into a debian archive format. Apt-move creates all three elements of the archive: The various directories full of packages, the Release file, and the Packages file.

This is only a useful technique if your package cache is complete. If you have cleaned it out since your last major upgrade, then this technique may fail. It's easy to regenerate the cache, though the large download is probably what you were trying to avoid!

1) Prepare the local mirror. For this example, let's use 6.0 (squeeze):

Create a place to host the mirror
# mkdir /var/local/mirrors

     Install the apt-move package. 
# apt-get install apt-move

     Edit the configuration file:
# nano /etc/apt-move.conf
LOCALDIR=/var/local/mirrors
DIST=squeeze
COPYONLY=yes

     Create the mirror
# apt-move update

You can see the mirror in /var/local/mirrors

2) Use the new mirror to debootstrap a new chroot environment

Create the new environment
# mkdir /var/new-env

     Debootstrap from the local mirror
# debootstrap squeeze /var/new-env file:///var/local/mirrors

Environment complete, ready to chroot into.

3) If debootstrap fails, it's probably due to a file missing from the cache. There are two ways to regenerate.

The hard way is to work error by error:
1) Run debootstrap, and identifying the missing file from the error message.
2) Use dpkg -S filename to identify the package name.
3) Use apt-get --reinstall install packagename to add it to the package cache.
4) Use apt-move update to move the package to the mirror.
5) Run debootstrap again.

The easy way is to use debootstrap to recreate all the missing files, and copy them into the cache:

# mkdir /tmp/cache
# debootstrap --arch=i386 --download-only squeeze /tmp/cache
# mv /tmp/cache/var/cache/apt/archives/* /var/cache/apt/
# rm -rf /tmp/cache
# apt-move update

The amazing simplicity of sshfs

So I'm out for a few minutes, waiting in a lounge during a children's choir rehearsal. And I want something off my home server.

sudo apt-get install sshfs
ssh_port=12345                  # Use your own port number, of course
server_name=myserver.dyndns.org # Use your own server, of course
user_name=myname                # Your username on the server
local_mount_point=/path/to/local/mountpoint
mkdir ${local_mount_point}
fwknop -A 'tcp/${ssh_port}' -s -D ${server_name}
sshfs ${user_name}@${server_name}:/path/on/server ${local_mount_point} -p ${ssh_port}

And SHAZAM! Remote server path shows up in File Manager.

No mucking with Samba or PPTP or other strangeness. Just

sshfs me@myserver.dyndns.org:/mnt/Common/ /tmp/server/ -p 12345

To unmount:

fusermount -u ${local_mount_point}