Showing posts with label 931181. Show all posts
Showing posts with label 931181. Show all posts

Tuesday, May 1, 2012

More about Bug 931181



Back in this post, I discussed Launchpad Bug #931181, a SHA1 implementation mistake that broke Debian/Ubuntu Simutrans multiplayer support. So Ubuntu and Debian players could share a game, but cold not connect to any other non-Debian server.

16 Feb 2011, the original patch that broke network play was applied.
12 Feb 2012, I reported the bug.
19 Feb 2012, a Simutrans dev noticed that the issue seemed to be endianness in the SHA1 hash used to communicate paksets between client and server.
21 Mar 2012, Version 111.2.2 of Simutrans is released
29 Apr 2012, the Debian package maintainer created a patch to fix the bug, and applied it to the 111.2.2 upload.

Today, Debian sid and testing (wheezy) and Ubuntu 12.10 have the patched version...but it still doesn't work.

For a clean set of tests, I installed a Debian 7 (testing) VM.

1) Test the Packages: Install the stock Debian Testing packages of simutrans 111.2.2 and simutrans-pak64 111.2 right out of the repos.

# apt-get install simutrans simutrans-pak64
$ simutrans -debug 3 &> s-package.log


Result: Fails to connect to the a.64.simutrans.entropy.me.uk server game. The UI claims the same pakset incompatibility as before. Here's the logfile.

As a check, let's take a look at one SHA1 hash:

PostOffice -> sha1 = 263C0CB8C858272A12FF54FC64F57292AF8ADAB7



2) Test the manual install: Install the upstream Simutrans versions of Simutrans 111.2.2 and Pak64 from the release announcement on the upstream website.

$ mkdir manual
$ cd manual
$ wget http://sourceforge.net/projects/simutrans/files/simutrans/111-2-2/simulinux-111-2-2.zip
$ wget http://sourceforge.net/projects/simutrans/files/pak64/111-2/simupak64-111-2.zip
$ unzip simulinux-111-2-2.zip
$ unzip simupak64-111-2.zip
$ chmod -R +Xr simutrans/pak       # This fixes the pakset's permissions
$ simutrans/simutrans -debug 3 &> ~/s-manual.log

Result: Succesful connection to the a.64.simutrans.entropy.me.uk server game. Here's the logfile.

Let's compare SHA1 hashes to see if the original issue (Launchpad Bug #931181, SHA1 endinaness ) has been fixed, or if this is a separate issue...

PostOffice -> sha1 = 263C0CB8C858272A12FF54FC64F57292AF8ADAB7

The hashes are the same. The original SHA1 issue has been fixed. There is a different bug or misconfiguration.

3) Is the Pakset good? Let's eliminate a possibility -- the packaged Pak64. Use the manually-installed Simutrans from #2 with the packaged Pak64 from #1.

$ simutrans/simutrans -debug 3 -object /usr/share/games/simutrans/pak &> ~/s-crossover.log

Result: Succesful connection to the a.64.simutrans.entropy.me.uk server game. Here's the logfile. The packaged pakset is not the problem. It's the correct version, and compiled properly. The problem is in Simutrans itself somewhere....

Monday, February 20, 2012

Simutrans packaging bug

After a bit of scratching my head, I finally figured out that Simutrans 110.0.1 (the version in Ubuntu 11.10) and the Debian-packaged version that is actually in Ubuntu have a weird incompatibility.

About a year ago, Simutrans added a network-multiplayer-server feature. But Debian/Ubuntu-packaged clients can only communicate with Debian/Ubuntu-packaged servers. Similarly, manually-installed clients can only talk to manually installed servers.

Well, that's certainly not intended behavior - all the servers should be able to talk to all the clients, regardless of whether they were packaged or not!

So I filed LP #931181, because the Debian Games Team needs to know.
And I opened a thread on the developer's forum.

Clue: One developer immediately noted that it seemed to be a checksum issue. (Thanks, Dwachs!)

Well, let's check that. Let's take one item in the pakset, building.PostOffice.pak, and see how it checksum:

Let's hash it manually first:
$ openssl sha1 ~/simutrans110/simutrans/pak/building.PostOffice.pak 
SHA1(/home/me/simutrans110/simutrans/pak/building.PostOffice.pak)= 8b1b5caa71013474039f357094cd1716bd3800bd

$ openssl sha1 /usr/share/games/simutrans/pak/building.PostOffice.pak 
SHA1(/usr/share/games/simutrans/pak/building.PostOffice.pak)= 8b1b5caa71013474039f357094cd1716bd3800bd
So the those pakset items (and, by extension, probably the entire pakset) are identical.

Next, let's check the logs of success and failure:

Succesful (Ubuntu 11.10) server: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

Succesful (Ubuntu 11.10) client: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

Failed (Manually-installed) server: Message: pakset_info_t::debug: PostOffice -> sha1 = 263C0CB8C858272A12FF54FC64F57292AF8ADAB7

Failed (Ubuntu 11.10) client: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

It seems that Dwachs put me on the right track. Given identical files to checksum, the two binaries checksum differently, and both do it differently than Ubuntu's default method, even though each claim to be using the same sha1!


Next, let's look for the changed sha-1 patch at the Debian Games Team git repository.

Aha! February 16, 2011: Add replacement for non-free SHA-1 implementation
Aha! February 16, 2011: get-orig-source: Remove non-free SHA-1 implementation.

The patch, is clearly a replacement for the original Simutrans trunk/utils/sha1.cc
And, looking at the current tree, the patch is still applied.

Er, did I just walk into a year-old dispute between Debian and Simutrans devs?

Next, I need to prove that the patch is the culprit. I need to apply the patch to a stock simutrans source, compile it, run it as a server (in debug mode), successfully join the game from a debian-install client.


Happily, I already have a stock 110.0.1 source, setup, and pakset in my simutrans110 directory, from earlier troubleshooting. Full instructions on how to compile are in this post.

Let's build the unpatched (stock) version first. This should be compatible with a manual install, and incompatible with the Debian version.
$ cp simutrans110 r4359-stock
$ cd r4359-stock/svn/trunk
$ make clean
$ make
$ strip sim
$ cp sim ~/r4359-stock/simutrans/sim-unpatched

Now, let's build a patched version. This should be compatible with the Debian version, and incompatible with a manual install.
$ sudo apt-get install libssl-dev
$ cp simutrans110 r4359-patched
$ cp Desktop/debian_patches_sha1-replacement.diff r4359-patched/svn/trunk
$ cd r4359-patched/svn/trunk

  --- Apply the patch ---
$ rm utils/sha1.*
$ patch Makefile debian_patches_sha1-replacement.diff 
$ patch utils/checksum.h debian_patches_sha1-replacement.diff
$ nano utils/sha1.cc
$ nano utils/sha1.h

$ make clean
$ make
$ strip sim
$ cp sim ~/r4359-stock/simutrans/sim-patched

Finally, we test.

Ubuntu 11.10's package install of Simutrans 110.0.1 can join a game served by sim-patched, but not a game served by sim-unpatched.

A manually-installed version of 110.0.1 is, as expected, the reverse. It can join a game served by sim-unpatched, but not a game served by sim-patched.

Thursday, January 26, 2012

Starting a Simutrans server

I love Simutrans. It's a great game.





Simutrans has a multiplayer version, if somebody hosts a server. I want to host a server version compatible with an Ubuntu release.
  • Will it overload my server's CPU and memory?
  • Is my network connection adequate?
Let's find out.
These instructions are for my Debian 6 server:

The server version has no graphics - that saves a lot of CPU overhead! But it must be compiled from source. Instructions are on the Simutrans Wiki, and even better instructions are on Timothy's blog post.

The basic theory is that we:
    a) Determine the version of Simutrans to install. 
    b) Do a complete client-install of the correct version, including base, binary, config, and pakset. 
    c) Download the source code. 
    d) Compile a server-only binary. 
    e) Replace the original client binary with the compiled server binary. 
    f) Edit the config file.

A Simutrans install consists of four items.
  1. The executable/binary.
  2. Simutrans-base, all those supporting languages and config files that the binary needs to work.
  3. The config file, simutrans/simuconf.tab, included with simutrans-base.
  4. The pakset, containing all the factories, vehicles, and other themes.

Simutrans has two paksets in Ubuntu: pak64 (the default) and pak128.britain. These instructions assume only pak64.




1) Create the build environment (as root):
# apt-get install build-essential subversion libbz2-dev zlib1g-dev unzip
# exit      # Drop root privileges - do the actual build as a user, not root
The build-essential package includes the rmadison command, used in the next step.

2) Pick the version to install. Client and server should be the same release. See the Simutrans Forum release announcements to match up releases with SVN revisions. I want this server to work with Ubuntu 11.10.
  • Check the Ubuntu 11.10 version of the Simutrans binary and the pak64 theme:
    $ rmadison simutrans | grep oneiric
     simutrans |  110.0.1-3 | oneiric/universe | source, amd64, armel, i386, powerpc
    $ rmadison simutrans-pak64 | grep oneiric
     simutrans-pak64 |  110.0.1-1 | oneiric/universe | source, all
    (Or check packages.ubuntu.com)
    The version numbers are before the dash, so both are Version 110.0.1

  • Check the corresponding Debian version on my server, to see if Simutrans must be installed manually or by package:
    rmadison -u debian simutrans | grep squeeze
     simutrans | 102.2.2~ds1-1 | squeeze | source, amd64, armel, i386, ia64, kfreebsd-amd64, kfreebsd-i386, mips, mipsel, powerpc, s390, sparc
    Since we want to install Version 110.0.1, we cannot use the Debian package (102.2.2) - we must install manually.


3) Create the simutrans folder, complete with locations for the downloads, and a working file for compiling the new binary.
$ cd ~
$ mkdir simutrans
$ mkdir simutrans/110.0.1
$ mkdir simutrans/110.0.1/downloads
$ mkdir simutrans/110.0.1/svn

4) Download and unpack simutrans, including the base, binary, config file, and theme pak.
$ cd ~/simutrans/110.0.1/downloads
$ wget http://sourceforge.net/projects/simutrans/files/simutrans/110-0-1/simulinux-110-0-1.zip
$ unzip simulinux-110-0-1.zip -d ~
$ wget http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
$ unzip simupak64-110-0-1.zip -d ~
$ cd ~

NOTE: The pakset location is ~/simutrans/pak. Addons to the paksets in version 111.0 (r4911) is ~/simutrans/pak. Addons to the pakset in version 111.1 (r5115) and later is ~/simutrans/addons/pak.

We now have a functioning simutrans install...if there's an X server and other supporting software. On a server, there probably isn't. You can launch Simutrans using the binary simutrans/simutrans for a great client experience.

Next, we will create the compiled server-optimized binary.

5) Download the Simutrans source:
$ cd simutrans/110.0.1/svn

-r is the revision number from Step 1.
username is anon, password is blank (hit Return).
$ svn co --username anon -r 4359 svn://tron.homeunix.org/simutrans/simutrans/trunk

6) Apply the Debian patch:
In this post, I discovered and triaged Launchpad Bug #931181, which makes non-Debian versions of Simutrans incompatible with Debian versions due to a differing sha-1 implementation.

We need to download and apply the Debian patch to make our server compatible with Debian/Ubuntu clients.
$ cd ~/simutrans/110.0.1/svn/trunk
$ wget http://anonscm.debian.org/gitweb/?p=pkg-games/simutrans.git;a=blob_plain;f=debian/patches/sha1-replacement.diff
$ rm utils/sha1.*
$ patch Makefile debian_patches_sha1-replacement.diff 
$ patch utils/checksum.h debian_patches_sha1-replacement.diff
$ nano utils/sha1.cc   # Create new file from the patch
$ nano utils/sha1.h    # Create new file from the patch

7) Edit the makefile configuration:
$ cp config.template config.default
$ nano config.default    # Use any editor, it needn't be nano

# Uncomment the following lines (and ONLY these lines):
BACKEND = posix
COLOUR_DEPTH = 0
OSTYPE = linux
DEBUG = 3
OPTIMISE = 1
WITH_REVISION = 1

8) Compile the binary:
$ make
$ strip sim   #Reduce binary size from 17.3 MB to 2.5 MB

9) Add the compiled binary to the simutrans install. We'll call this new binary 'simd110'. Note that the original (client) binary can be left in place if you wish, since it's called 'simutrans' but it won't be very useful.
$ cd ~
$ ln simutrans/110.0.1/svn/simutrans/simutrans/trunk/sim simutrans/simd110
If you want to save space, you can copy the binary instead of linking it, and delete the old client binary and the entire svn directory. I recommend testing before deleting all your work!

10) Edit the config file
$nano simutrans/config/simuconf.tab
#################################program stuff##################################
singleuser_install = 1
pak_file_path = pak/

###################################network stuff##############################
server_frames_ahead = 4
server_frames_per_step = 4
server_frames_between_checks = 256   #Smaller!
server_announce_interval = 900
server_dns = myserver.domain.net
server_name = Ubuntu 11.10 Compatible
server_comments = If you use Ubuntu 11.10 (Oneiric), this is compatible with the Simutrans version in the Software Center
server_email = me@myemail.domain.net
server_pakurl = http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
server_infurl = http://myserver.domain.net/server-info.html
pause_server_no_clients = 1

11) Use the old client (non-server) binary to create the initial map. The headless server version has no way to choose a map pattern or create a new map, and you have no way to look at it to see if you like it! Save the map as "servergame.sve" inside the simutrans directory.

12) Test the server binary. To do this, you need a client install *and* a server install with identical paksets but preferably different config files. The best test is, of course, from a working stock Ubuntu 11.10 install.

Start the server on port 13353 with
$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3 -load servergame.sve

Try connecting from the client to the server a few times.
Check the server's simutrans directory to ensure the game is saved upon each connect/disconnect.

Try killing the server and restarting it:
$ pkill simd110
$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3

You now have a fully-operational simutrans server, compatible with the desired Ubuntu version of Simutrans. This is not a complete setup - it's best to configure the server with a special non-shell-access user so neither root nor a real user are running the server. I'll talk about that in a different post.