Showing posts with label Simutrans. Show all posts
Showing posts with label Simutrans. Show all posts

Tuesday, May 1, 2012

More about Bug 931181



Back in this post, I discussed Launchpad Bug #931181, a SHA1 implementation mistake that broke Debian/Ubuntu Simutrans multiplayer support. So Ubuntu and Debian players could share a game, but cold not connect to any other non-Debian server.

16 Feb 2011, the original patch that broke network play was applied.
12 Feb 2012, I reported the bug.
19 Feb 2012, a Simutrans dev noticed that the issue seemed to be endianness in the SHA1 hash used to communicate paksets between client and server.
21 Mar 2012, Version 111.2.2 of Simutrans is released
29 Apr 2012, the Debian package maintainer created a patch to fix the bug, and applied it to the 111.2.2 upload.

Today, Debian sid and testing (wheezy) and Ubuntu 12.10 have the patched version...but it still doesn't work.

For a clean set of tests, I installed a Debian 7 (testing) VM.

1) Test the Packages: Install the stock Debian Testing packages of simutrans 111.2.2 and simutrans-pak64 111.2 right out of the repos.

# apt-get install simutrans simutrans-pak64
$ simutrans -debug 3 &> s-package.log


Result: Fails to connect to the a.64.simutrans.entropy.me.uk server game. The UI claims the same pakset incompatibility as before. Here's the logfile.

As a check, let's take a look at one SHA1 hash:

PostOffice -> sha1 = 263C0CB8C858272A12FF54FC64F57292AF8ADAB7



2) Test the manual install: Install the upstream Simutrans versions of Simutrans 111.2.2 and Pak64 from the release announcement on the upstream website.

$ mkdir manual
$ cd manual
$ wget http://sourceforge.net/projects/simutrans/files/simutrans/111-2-2/simulinux-111-2-2.zip
$ wget http://sourceforge.net/projects/simutrans/files/pak64/111-2/simupak64-111-2.zip
$ unzip simulinux-111-2-2.zip
$ unzip simupak64-111-2.zip
$ chmod -R +Xr simutrans/pak       # This fixes the pakset's permissions
$ simutrans/simutrans -debug 3 &> ~/s-manual.log

Result: Succesful connection to the a.64.simutrans.entropy.me.uk server game. Here's the logfile.

Let's compare SHA1 hashes to see if the original issue (Launchpad Bug #931181, SHA1 endinaness ) has been fixed, or if this is a separate issue...

PostOffice -> sha1 = 263C0CB8C858272A12FF54FC64F57292AF8ADAB7

The hashes are the same. The original SHA1 issue has been fixed. There is a different bug or misconfiguration.

3) Is the Pakset good? Let's eliminate a possibility -- the packaged Pak64. Use the manually-installed Simutrans from #2 with the packaged Pak64 from #1.

$ simutrans/simutrans -debug 3 -object /usr/share/games/simutrans/pak &> ~/s-crossover.log

Result: Succesful connection to the a.64.simutrans.entropy.me.uk server game. Here's the logfile. The packaged pakset is not the problem. It's the correct version, and compiled properly. The problem is in Simutrans itself somewhere....

How to build a Simutrans .deb package manually

Simutrans, as I've written before, is a fun game.

Currently it's in active development, with new features and bug fixes arriving several times each year.  But Ubuntu's repository system is based on six-month snapshots; it cannot keep up with the pace of Simutrans releases.

That causes a big problem for multiplayer games: The server requires clients to be on a similar pakset and release, and the server operators usually keep up with the releases. So Ubuntu users are locked out of multiplayer games - the versions in the Ubuntu repos are simply too old.

So I want roll my own .deb of the latest version of Simutrans. This is the first step - eventually a PPA of the backported latest release would be cool.



1) Create a VM for the build environment. For this, I used a VM of my current install, Ubuntu 11.10. But you can use anything back to Debian 6, of course.

The VM serves several purposes. It provides the appropriate build environment for each version of Ubuntu. It also provides a safe place to test the final package installation. And, of course, it protects my main system.

Beyond the basic setup, you should probably clean up the desktop shortcut bar; you don't need Software Center or LibreOffice, but you do need Terminal on there. You should also set Software Sources to automatically download-and-apply security updates, you can safely ignore all non-security updates.

Also, install the following packages needed for a useful Simutrans build environment:
$ sudo apt-get install build-essential devscripts debhelper libsdl1.2-dev 
libsdl-mixer1.2-dev zlib1g-dev imagemagick libpng-dev libbz2-dev libssl-dev subversion

Do NOT install the simutrans package from the Ubuntu repositories. If you do install it, for example to confirm that it works, remove it.


2) Prepare the working directory for Simutrans and download the sources

First, let's set up the Simutrans working directory

~$ mkdir Simutrans_Development
~$ cd Simutrans_Development
~/Simutrans_Development$ wget http://ftp.us.debian.org/debian/pool/main/s/simutrans/simutrans_111.2.2-1.debian.tar.xz 
~/Simutrans_Development$ tar -xvf simutrans_111.2.2-1.debian.tar.xz 

Increment the version number by editing the changelog in each Debian directory.

~/Pak64_Development$ cd ~/Simutrans_Development
~/Simutrans_Development$ dch -v 111.2.2-0ppa1

simutrans (111.2.1-0ppa1) oneiric; urgency=low

  * New upstream release

 -- me <me@example.com>  Wed, 25 Apr 2012 07:23:32 -0500

Edit the Debian rules file to reflect the most recent upstream release, then download a new source code tarball, uncompress the tarball, and copy the debian folder into the new folder.

~/Pak64_Development$ cd ~/Simutrans_Development
~/Simutrans_Developmen$ nano debian/rules

SVNREV  = 5583
VERSION = 111.2.2


~/Simutrans_Development$ make -f debian/rules get-orig-source
tar -xvf simutrans_111.2.2.orig.tar.zx
cp -r debian/ simutrans-111.2.2/

Finally, a progress check: The directory should look something like this:

~$ ls Simutrans_Development/
debian             simutrans_111.2.2-1.debian.tar.xz
simutrans-111.2.2  simutrans_111.2.2-1.orig.tar.xz


3) Prepare the working directory for Pak64 and download the sources

~$ mkdir Pak64_Development
~/Simutrans_Development$ cd ~/Pak64_Development
~/Pak64_Development$ wget http://ftp.us.debian.org/debian/pool/main/s/simutrans-pak64/simutrans-pak64_111.2-1.debian.tar.xz
~/Pak64_Development$ tar -xvf simutrans_111.2-1.debian.tar.xz

Increment the version number by editing the changelog in each Debian directory.

~/Simutrans_Development$ cd ~/Pak64_Development
~/Pak64_Development$ dch -v 111.2-0ppa1

simutrans-pak64 (111.2-0ppa1) oneiric; urgency=low

  * New upstream release

 -- me <me@example.com>  Wed, 25 Apr 2012 07:23:32 -0500

Edit the Debian rules file to reflect the most recent upstream release, then download a new source code tarball, uncompress the tarball, and copy the debian folder into the new folder.

~/Simutrans_Development$ cd ~/Pak64_Development
~/Pak64_Development$ nano debian/rules

SVNREV  = 811
VERSION = 111.2
UPSTREAM_SVNREV = 810M # see README.source

~/Pak64_Development$ make -f debian/rules get-orig-source
tar -xvf simutrans-pak64_111.2.orig.tar.xz
cp -r debian/ simutrans-pak64-111.2/

Finally, a progress check: The directory should look something like this:

~$ ls Pak64_Development/
debian                                 simutrans-pak64-111.2
simutrans-pak64_111.2-1.debian.tar.xz  simutrans-pak64_111.2.orig.tar.xz


4) Build Simutrans

We're ready to try building the package. This is where the troubleshooting really starts.

~$ cd Simutrans_Development/simutrans-111.2.2
~/Simutrans_Development/simutrans-111.2.2$ debuild -us -uc

First  Problem: debhelper 9.
 
dpkg-checkbuilddeps: Unmet build dependencies: debhelper (>= 9)
dpkg-buildpackage: warning: Build dependencies/conflicts unsatisfied; aborting.
dpkg-buildpackage: warning: (Use -d flag to override.)
debuild: fatal error at line 1348:
dpkg-buildpackage -rfakeroot -D -us -uc failed

The debhelper package was originally version 7.x. According to the git log, it incremented up to 9 in March 2012, long after the build environment (that's why we review the log!). We can try the -h flag to avoid this...or unapply the patch that incremented debhelper.
 
~/Simutrans_Development/simutrans-111.2.1$ debuild -us -uc -d

Second Problem: Lintian errors.
 
Now running lintian...
W: simutrans source: newer-standards-version 3.9.3 (current is 3.9.2)
W: simutrans: binary-without-manpage usr/games/simutrans-nettool
Finished running lintian.

Lintian errors are a good sign - it means everything else worked! Both warnings in this case can be safely ignored...though a good citizen would also file bugs with patches to fix those warnings.

Take a look at the Simutrans_Development directory now. See all those ready-to-install deb packages?
 
ian@VM-11-10:~/Simutrans_Development$ ls
debian                                simutrans_111.2.2-0ppa1_i386.deb
simutrans-111.2.2                     simutrans_111.2.2.debian.tar.gz
simutrans_111.2.2-1.debian.tar.xz     simutrans_111.2.2.orig.tar.xz
simutrans_111.2.2-1.dsc               simutrans-data_111.2.2-0ppa1_all.deb
simutrans_111.2.2-0ppa1_i386.build    simutrans-makeobj_111.2.2-0ppa1_i386.deb
simutrans_111.2.2-0ppa1_i386.changes

Do not install the simutrans_111.2.1-1_i386.deb or simutrans-data_111.2.1-1_all.deb packages yet. If you try, they will install a pak64 package from the repositories instead of the one you are about to build.

5) Install makeobj

makeobj is a package (simutrans-makeobj_111.2.1-1_i386.deb) that is built with the Simutrans game, and used in turn to create the paks. Install the deb that was just created before building pak64.

~/Simutrans_Development$ sudo dpkg -i simutrans-makeobj_111.2.2-0ppa1_i386.deb 

6) Build Pak64


We're ready to try building the pak package.

~$ cd Pak64_Development/simupak-64
~/Pak64_Development/simupak64$ debuild -us -uc

Problem: Makefile patches failed

The file debian/patches/buildsystem has several patches that get applied to the makefile. As the makefile changes over time, the patch program may not find the right place to apply some patches. Those changes to the patch must be applied manually.

patching file Makefile
Hunk #2 FAILED at 46.
Hunk #3 succeeded at 65 (offset 2 lines).
Hunk #4 FAILED at 84.
2 out of 4 hunks FAILED -- saving rejects to file Makefile.rej

Hunk #2 was a straightforward change to one line (adding the word "zip").
Hunk #4 had the old version number instead of the new.

Upon the second try, the pak64 build was successful. You can see the deb package in the directory listing:

~$ ls Pak64_Development/
debian                                     simutrans-pak64_111.2.1-1.dsc
simutrans-pak64_111.2-0ppa1.debian.tar.xz  simutrans-pak64_111.2.1-1_i386.build
simutrans-pak64-111.2                      simutrans-pak64_111.2.1-1_i386.changes
simutrans-pak64_111.2-0ppa1_all.deb        simutrans-pak64_111.2-1.orig.tar.xz
simutrans-pak64_111.2-1.debian.tar.xz      simutrans-pak64_111.2-1.orig.tar.xz

7) Test Install

When installing package-by-package, make sure to install in the right order!

sudo dpkg -i Pak64_Development/simutrans-pak64_111.2-0ppa1_all.deb
sudo dpkg -i Simutrans_Development/simutrans-data_111.2.2-0ppa1_all.deb 
sudo dpkg -i Simutrans_Development/simutrans_111.2.2-0ppa1_i386.deb


I fire up Simutrans...and it works!

8) Removing Simutrans

When the test is complete, and I want to remove simutrans:
sudo dpkg -r simutrans
sudo dpkg -r simutrans-data
sudo dpkg -r simutrans-pak64

Monday, February 20, 2012

Simutrans packaging bug

After a bit of scratching my head, I finally figured out that Simutrans 110.0.1 (the version in Ubuntu 11.10) and the Debian-packaged version that is actually in Ubuntu have a weird incompatibility.

About a year ago, Simutrans added a network-multiplayer-server feature. But Debian/Ubuntu-packaged clients can only communicate with Debian/Ubuntu-packaged servers. Similarly, manually-installed clients can only talk to manually installed servers.

Well, that's certainly not intended behavior - all the servers should be able to talk to all the clients, regardless of whether they were packaged or not!

So I filed LP #931181, because the Debian Games Team needs to know.
And I opened a thread on the developer's forum.

Clue: One developer immediately noted that it seemed to be a checksum issue. (Thanks, Dwachs!)

Well, let's check that. Let's take one item in the pakset, building.PostOffice.pak, and see how it checksum:

Let's hash it manually first:
$ openssl sha1 ~/simutrans110/simutrans/pak/building.PostOffice.pak 
SHA1(/home/me/simutrans110/simutrans/pak/building.PostOffice.pak)= 8b1b5caa71013474039f357094cd1716bd3800bd

$ openssl sha1 /usr/share/games/simutrans/pak/building.PostOffice.pak 
SHA1(/usr/share/games/simutrans/pak/building.PostOffice.pak)= 8b1b5caa71013474039f357094cd1716bd3800bd
So the those pakset items (and, by extension, probably the entire pakset) are identical.

Next, let's check the logs of success and failure:

Succesful (Ubuntu 11.10) server: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

Succesful (Ubuntu 11.10) client: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

Failed (Manually-installed) server: Message: pakset_info_t::debug: PostOffice -> sha1 = 263C0CB8C858272A12FF54FC64F57292AF8ADAB7

Failed (Ubuntu 11.10) client: Message: pakset_info_t::debug: PostOffice -> sha1 = B80C3C262A2758C8FC54FF129272F564B7DA8AAF

It seems that Dwachs put me on the right track. Given identical files to checksum, the two binaries checksum differently, and both do it differently than Ubuntu's default method, even though each claim to be using the same sha1!


Next, let's look for the changed sha-1 patch at the Debian Games Team git repository.

Aha! February 16, 2011: Add replacement for non-free SHA-1 implementation
Aha! February 16, 2011: get-orig-source: Remove non-free SHA-1 implementation.

The patch, is clearly a replacement for the original Simutrans trunk/utils/sha1.cc
And, looking at the current tree, the patch is still applied.

Er, did I just walk into a year-old dispute between Debian and Simutrans devs?

Next, I need to prove that the patch is the culprit. I need to apply the patch to a stock simutrans source, compile it, run it as a server (in debug mode), successfully join the game from a debian-install client.


Happily, I already have a stock 110.0.1 source, setup, and pakset in my simutrans110 directory, from earlier troubleshooting. Full instructions on how to compile are in this post.

Let's build the unpatched (stock) version first. This should be compatible with a manual install, and incompatible with the Debian version.
$ cp simutrans110 r4359-stock
$ cd r4359-stock/svn/trunk
$ make clean
$ make
$ strip sim
$ cp sim ~/r4359-stock/simutrans/sim-unpatched

Now, let's build a patched version. This should be compatible with the Debian version, and incompatible with a manual install.
$ sudo apt-get install libssl-dev
$ cp simutrans110 r4359-patched
$ cp Desktop/debian_patches_sha1-replacement.diff r4359-patched/svn/trunk
$ cd r4359-patched/svn/trunk

  --- Apply the patch ---
$ rm utils/sha1.*
$ patch Makefile debian_patches_sha1-replacement.diff 
$ patch utils/checksum.h debian_patches_sha1-replacement.diff
$ nano utils/sha1.cc
$ nano utils/sha1.h

$ make clean
$ make
$ strip sim
$ cp sim ~/r4359-stock/simutrans/sim-patched

Finally, we test.

Ubuntu 11.10's package install of Simutrans 110.0.1 can join a game served by sim-patched, but not a game served by sim-unpatched.

A manually-installed version of 110.0.1 is, as expected, the reverse. It can join a game served by sim-unpatched, but not a game served by sim-patched.

Thursday, January 26, 2012

Starting a Simutrans server

I love Simutrans. It's a great game.





Simutrans has a multiplayer version, if somebody hosts a server. I want to host a server version compatible with an Ubuntu release.
  • Will it overload my server's CPU and memory?
  • Is my network connection adequate?
Let's find out.
These instructions are for my Debian 6 server:

The server version has no graphics - that saves a lot of CPU overhead! But it must be compiled from source. Instructions are on the Simutrans Wiki, and even better instructions are on Timothy's blog post.

The basic theory is that we:
    a) Determine the version of Simutrans to install. 
    b) Do a complete client-install of the correct version, including base, binary, config, and pakset. 
    c) Download the source code. 
    d) Compile a server-only binary. 
    e) Replace the original client binary with the compiled server binary. 
    f) Edit the config file.

A Simutrans install consists of four items.
  1. The executable/binary.
  2. Simutrans-base, all those supporting languages and config files that the binary needs to work.
  3. The config file, simutrans/simuconf.tab, included with simutrans-base.
  4. The pakset, containing all the factories, vehicles, and other themes.

Simutrans has two paksets in Ubuntu: pak64 (the default) and pak128.britain. These instructions assume only pak64.




1) Create the build environment (as root):
# apt-get install build-essential subversion libbz2-dev zlib1g-dev unzip
# exit      # Drop root privileges - do the actual build as a user, not root
The build-essential package includes the rmadison command, used in the next step.

2) Pick the version to install. Client and server should be the same release. See the Simutrans Forum release announcements to match up releases with SVN revisions. I want this server to work with Ubuntu 11.10.
  • Check the Ubuntu 11.10 version of the Simutrans binary and the pak64 theme:
    $ rmadison simutrans | grep oneiric
     simutrans |  110.0.1-3 | oneiric/universe | source, amd64, armel, i386, powerpc
    $ rmadison simutrans-pak64 | grep oneiric
     simutrans-pak64 |  110.0.1-1 | oneiric/universe | source, all
    (Or check packages.ubuntu.com)
    The version numbers are before the dash, so both are Version 110.0.1

  • Check the corresponding Debian version on my server, to see if Simutrans must be installed manually or by package:
    rmadison -u debian simutrans | grep squeeze
     simutrans | 102.2.2~ds1-1 | squeeze | source, amd64, armel, i386, ia64, kfreebsd-amd64, kfreebsd-i386, mips, mipsel, powerpc, s390, sparc
    Since we want to install Version 110.0.1, we cannot use the Debian package (102.2.2) - we must install manually.


3) Create the simutrans folder, complete with locations for the downloads, and a working file for compiling the new binary.
$ cd ~
$ mkdir simutrans
$ mkdir simutrans/110.0.1
$ mkdir simutrans/110.0.1/downloads
$ mkdir simutrans/110.0.1/svn

4) Download and unpack simutrans, including the base, binary, config file, and theme pak.
$ cd ~/simutrans/110.0.1/downloads
$ wget http://sourceforge.net/projects/simutrans/files/simutrans/110-0-1/simulinux-110-0-1.zip
$ unzip simulinux-110-0-1.zip -d ~
$ wget http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
$ unzip simupak64-110-0-1.zip -d ~
$ cd ~

NOTE: The pakset location is ~/simutrans/pak. Addons to the paksets in version 111.0 (r4911) is ~/simutrans/pak. Addons to the pakset in version 111.1 (r5115) and later is ~/simutrans/addons/pak.

We now have a functioning simutrans install...if there's an X server and other supporting software. On a server, there probably isn't. You can launch Simutrans using the binary simutrans/simutrans for a great client experience.

Next, we will create the compiled server-optimized binary.

5) Download the Simutrans source:
$ cd simutrans/110.0.1/svn

-r is the revision number from Step 1.
username is anon, password is blank (hit Return).
$ svn co --username anon -r 4359 svn://tron.homeunix.org/simutrans/simutrans/trunk

6) Apply the Debian patch:
In this post, I discovered and triaged Launchpad Bug #931181, which makes non-Debian versions of Simutrans incompatible with Debian versions due to a differing sha-1 implementation.

We need to download and apply the Debian patch to make our server compatible with Debian/Ubuntu clients.
$ cd ~/simutrans/110.0.1/svn/trunk
$ wget http://anonscm.debian.org/gitweb/?p=pkg-games/simutrans.git;a=blob_plain;f=debian/patches/sha1-replacement.diff
$ rm utils/sha1.*
$ patch Makefile debian_patches_sha1-replacement.diff 
$ patch utils/checksum.h debian_patches_sha1-replacement.diff
$ nano utils/sha1.cc   # Create new file from the patch
$ nano utils/sha1.h    # Create new file from the patch

7) Edit the makefile configuration:
$ cp config.template config.default
$ nano config.default    # Use any editor, it needn't be nano

# Uncomment the following lines (and ONLY these lines):
BACKEND = posix
COLOUR_DEPTH = 0
OSTYPE = linux
DEBUG = 3
OPTIMISE = 1
WITH_REVISION = 1

8) Compile the binary:
$ make
$ strip sim   #Reduce binary size from 17.3 MB to 2.5 MB

9) Add the compiled binary to the simutrans install. We'll call this new binary 'simd110'. Note that the original (client) binary can be left in place if you wish, since it's called 'simutrans' but it won't be very useful.
$ cd ~
$ ln simutrans/110.0.1/svn/simutrans/simutrans/trunk/sim simutrans/simd110
If you want to save space, you can copy the binary instead of linking it, and delete the old client binary and the entire svn directory. I recommend testing before deleting all your work!

10) Edit the config file
$nano simutrans/config/simuconf.tab
#################################program stuff##################################
singleuser_install = 1
pak_file_path = pak/

###################################network stuff##############################
server_frames_ahead = 4
server_frames_per_step = 4
server_frames_between_checks = 256   #Smaller!
server_announce_interval = 900
server_dns = myserver.domain.net
server_name = Ubuntu 11.10 Compatible
server_comments = If you use Ubuntu 11.10 (Oneiric), this is compatible with the Simutrans version in the Software Center
server_email = me@myemail.domain.net
server_pakurl = http://sourceforge.net/projects/simutrans/files/pak64/110-0-1/simupak64-110-0-1.zip
server_infurl = http://myserver.domain.net/server-info.html
pause_server_no_clients = 1

11) Use the old client (non-server) binary to create the initial map. The headless server version has no way to choose a map pattern or create a new map, and you have no way to look at it to see if you like it! Save the map as "servergame.sve" inside the simutrans directory.

12) Test the server binary. To do this, you need a client install *and* a server install with identical paksets but preferably different config files. The best test is, of course, from a working stock Ubuntu 11.10 install.

Start the server on port 13353 with
$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3 -load servergame.sve

Try connecting from the client to the server a few times.
Check the server's simutrans directory to ensure the game is saved upon each connect/disconnect.

Try killing the server and restarting it:
$ pkill simd110
$ simutrans/simd110 -server 13353 -lang en -nosound -nomidi -noaddons -log 1 -debug 3

You now have a fully-operational simutrans server, compatible with the desired Ubuntu version of Simutrans. This is not a complete setup - it's best to configure the server with a special non-shell-access user so neither root nor a real user are running the server. I'll talk about that in a different post.

Wednesday, January 25, 2012

Testing a Simutrans Server

If you're experimenting with Simutrans' multiplayer, you will eventually want to try starting your own server. Here's what you need to know.

First, the normal installed version of Simutrans works as either a client and a server.

For example, on a Linux system:

     To start Simutrans normally (client mode):
     $ simutrans

     To start Simutrans as a server:
     $ simutrans -server


Second, you can run a client and a server on the same machine. To connect the client to the server, use the loopback address (127.0.0.1) or the machine's IP address in the client's connect dialog.

Now we get into pesky little details:

- The client and server must both be using the same pakset version and addons (if any). The server and client compare hashes of each pakset and addon item, and refuse to connect if every hash doesn't match.

- The client and server should (not must) be using the same game version, or within a release or two. Network play is still pretty new, improvements are still being added, and backward compatibility with older versions is not guaranteed.