Upstart Jobs at login

Login is not the same is startup. Let's just get that out of the way first.

• Startup is the time between boot and the login screen. It's the habitat of system jobs.
• Login is the time after you enter your password. It's the habitat of user jobs.

The easy way to run a task at login is to run a script from your .bashrc.
And the (deceptively not-) easy way to run a task at logout is to run a script from your .bash_logout

But today we're not doing it the easy way. Today we're going to use dbus and Upstart.

Emitting Upstart Signals from your .bashrc

It's terribly easy.

1) Emit a user-level Upstart signal by adding a line to .bashrc:

# Upstart signal that .bashrc is running
initctl emit "I_AM_LOGGING_IN"

2) Add a user-level Upstart job to ~.config/upstart/ for one user, or to /usr/share/upstart/sessions/ for all users:

# /home/$USER/.config/upstart/login_test.conf
description "login test"
start on I_AM_LOGGING_IN            # Start criteria
exec /bin/date > /tmp/login_test    # Do something

3) Open a new terminal window (to load the new .bashrc). When you open the window, the Upstart job creates the tempfile at /tmp/login_test.

Clean up: Restore your bashrc, and delete the sample Upstart job.

Can I emit system-level Upstart signals from .bashrc?

Not directly. The script runs as a user, not as root.

You can use a secondary method of triggering system-level Upstart signals, like sending a Dbus signal, or manipulating a file, or connecting to a socket.

Can I emit Upstart signals from .bash_logout?

No.

Using initctl emit in .bash_logout will merely result in an error. The user-level Upstart daemon seems to be terminated before .bash_logout is run. The command will return a cryptic "Rejected send message"error from PID 1 (system Upstart). Since .bash_logout is not running as root, it cannot emit system-level signals.

Also, GUI terminal programs do not not run .bash_logout, unless you specify compatibility (with a flag) when you start.

That easy way of doing login actions is still too hard

Boy, are you difficult to please.

Okay, there is an even easier way, but it's more complicated to explain: Instead of .bashrc emitting an Upstart event, let Upstart listen for a dbus signal.

Here is an example of the dbus message that occurs when I login via SSH to a new session. This signal is emitted by systend-logind every time a new TTY, ssh, or X-based GUI login occurs.

The signal is not emitted when you are in a GUI environment and simply open a terminal window - that's not a login, that's a spawn of your already-existing GUI environment:

signal sender=:1.3 -> dest=(null destination) serial=497 
  path=/org/freedesktop/login1; interface=org.freedesktop.login1.Manager;
  member=SessionNew
    string "4"
    object path "/org/freedesktop/login1/session/_34"

The important elements are the source, the "SessionNew" signal, and the path of the new session.

Aside, let's query systemd-logind to find if the login is to a TTY, X session, or SSH. logind has lots of useful information about each session:

$ dbus-send --system                              \ 
            --dest=org.freedesktop.login1         \
            --print-reply                         \
            --type=method_call                    \
            /org/freedesktop/login1/session/_34   \ # Path from the signal
            org.freedesktop.DBus.Properties.Get   \
            string:org.freedesktop.login1.Session \
            string:Service
method return sender=:1.3 -> dest=:1.211 reply_serial=2
   variant       string "sshd"

It's right. I did connect using ssh.

Now let's construct an Upstart job that runs when I login via a TTY, X Session, or SSH. We will use Upstart's built-in dbus listener.

# /home/$USER/.config/upstart/login_test.conf
description "login test"
start on dbus SIGNAL=SessionNew     # Listen for the dbus Signal
exec /bin/date > /tmp/login_test    # Do something

• Now, whenever you login to a TTY, X session, or SSH session, the job will run.
• If your job needs to tell the difference between those sessions, you know how to find out using dbus.
• If *everybody* needs the job, place it in /usr/share/upstart/sessions/ instead of each user's .config/upstart/

What about super-easy logout jobs?

Logout jobs are harder, and generally not recommended. Not super-easy. They are hard because you can't guarantee they will run. Maybe the user will hold down the power button. Or use the "shutdown -h now" command. Or the power supply sent a message that the battery only has 60 seconds of life left. Or the user absolutely cannot miss that bus....

Here's the dbus signal that systemd-logind emits when a TTY, X, or SSH user session ends:

signal sender=:1.21 -> dest=(null destination) serial=286 
  path=/org/freedesktop/Accounts/User1000; 
  interface=org.freedesktop.Accounts.User; member=Changed

All this tells me is that User1000 now has a different number of sessions running. Maybe it's a login (yes, it emits the same signal upon login). Maybe it's a logout.

Sure, we can do a login-and-logout Upstart job...

# /home/$USER/.config/upstart/login_test.conf
description "login and logout test"
start on dbus SIGNAL=Changed INTERFACE=org.freedesktop.Accounts.User
exec /bin/date > /tmp/login_test

...but then you need logic to figure out who logged in or logged out, and whether it's an event you care about. Certainly doable, but probably not worthwhile for most users.

In other words, if you want to backup-at-logout, you need to structure it as a backup-then-logout sequence. Logout is not an appropriate trigger to start the sequence...from the system's point of view.

But I really want to do a job a logout!

Okay, here's how to do a job when you log out of the GUI environment. Logging out is the trigger. This won't work for SSH or TTY sessions.

The Upstart jobs /etc/init/lightdm.conf and/etc/init/gdm.conf emit a system-level "desktop-shutdown" signal when the X server is stopped. You can use that job as your start criteria.

# /etc/init/logoff_test.conf
description "logout test"
setuid some_username        # Your script probably doesn't need to run as root
start on stopping lightdm   # Run *before* it is stopped
exec /bin/date > /tmp/logoff_test

Searching for the right Upstart signal or job

If you want to use Upstart to start/stop a job on any of the not-obvious triggers (like "startup"), then you need to do some digging to find the right trigger.

initctl show-config

Be careful, there are TWO sets of Upstart jobs: System jobs and user jobs. Use sudo to distinguish between them.

$ sudo initctl show-config dbus   # Use sudo for system-level jobs
dbus
  start on local-filesystems
  stop on deconfiguring-networking

$ initctl show-config dbus        # Omit sudo for user-level jobs
dbus
  start on starting xsession-init 

Searching for a job or a signal using grep

The initctl show-config command without any job name prints all the jobs. That means you can use grep on the full list. Here is an example of using grep to look for all root jobs that care about the system "startup" signal:

$ sudo initctl show-config | grep -B8 startup
  start on (starting mountall or (runlevel [016] and ((desktop-shutdown or stopped xdm) or stopped uxlaunch)))
resolvconf
  start on mounted MOUNTPOINT=/run
  stop on runlevel [06]
ssh
  start on runlevel [2345]
  stop on runlevel [!2345]
udev-fallback-graphics
  start on (startup and (((graphics-device-added PRIMARY_DEVICE_FOR_DISPLAY=1 or drm-device-added PRIMARY_DEVICE_FOR_DISPLAY=1) or stopped udevtrigger) or container))
--
mountall
  emits virtual-filesystems
  emits local-filesystems
  emits remote-filesystems
  emits all-swaps
  emits filesystem
  emits mounting
  emits mounted
  start on startup
--
acpid
  start on runlevel [2345]
  stop on runlevel [!2345]
checkfs.sh
  start on mounted MOUNTPOINT=/
checkroot-bootclean.sh
  start on mounted MOUNTPOINT=/
kmod
  start on (startup and started udev)
--
  start on runlevel S
  stop on runlevel [!S]
wait-for-state
  stop on (started $WAIT_FOR or stopped $WAIT_FOR)
flush-early-job-log
  start on filesystem
friendly-recovery
  emits recovery
  emits startup
--
  start on runlevel [2345]
  stop on runlevel [!2345]
socket-test
  start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1
tty2
  start on (runlevel [23] and ((not-container or container CONTAINER=lxc) or container CONTAINER=lxc-libvirt))
  stop on runlevel [!23]
udevtrigger
  start on ((startup and started udev) and not-container)
--
  emits not-container
  start on mounted MOUNTPOINT=/run
mounted-dev
  start on mounted MOUNTPOINT=/dev
tty3
  start on (runlevel [23] and ((not-container or container CONTAINER=lxc) or container CONTAINER=lxc-libvirt))
  stop on runlevel [!23]
udev-finish
  start on ((((startup and filesystem) and started udev) and stopped udevtrigger) and stopped udevmonitor)
alsa-state
  start on runlevel [2345]
cryptdisks-udev
  start on block-device-added ID_FS_USAGE=crypto
hostname
  start on startup
--
network-interface
  emits net-device-up
  emits net-device-down
  emits static-network-up
  start on net-device-added
  stop on net-device-removed INTERFACE=$INTERFACE
plymouth-ready
  emits plymouth-ready
  start on (startup or started plymouth-splash)
--
  start on (started plymouth and ((graphics-device-added PRIMARY_DEVICE_FOR_DISPLAY=1 or drm-device-added PRIMARY_DEVICE_FOR_DISPLAY=1) or stopped udev-fallback-graphics))
plymouth-upstart-bridge
  start on (started dbus or runlevel [06])
  stop on stopping plymouth
tty1
  start on (stopped rc RUNLEVEL=[2345] and ((not-container or container CONTAINER=lxc) or container CONTAINER=lxc-libvirt))
  stop on runlevel [!2345]
udevmonitor
  start on (startup and starting udevtrigger)

We found one job that emits startup (friendly-recovery).
We found seven jobs that listen for it: udev-fallback-graphics, mountall, kmod, udevtrigger, hostname, plymouth-ready, and udevmonitor

Searching for a signal using upstart-monitor

The upstart-monitor application is a handy GUI and command-line tool to listen to all the signal chatter in Upstart. The application is provided by the upstart-monitor package in the Ubuntu repositories. A bug in 13.10 prevents it from running on a non-GUI system like Ubuntu Server, but it's also easy to fix the bug yourself...

Here are the signals emitted by Upstart when I switch over to a TTY, login, wait ten seconds, and then logout. This isn't an example of monitoring logins (do that using consolekit or logind) - this is an example of monitoring the Upstart signals emitted by a change in tty2.

$ upstart-monitor --no-gui --destination=system-bus
# Upstart Event Monitor (console mode)
#
# Connected to D-Bus system bus
#
# Columns: time, event and environment

2014-01-03 23:23:43.013436 stopping JOB='tty2' INSTANCE='' RESULT='ok'
2014-01-03 23:23:43.020309 starting JOB='tty2' INSTANCE=''
2014-01-03 23:23:43.031193 starting JOB='startpar-bridge' INSTANCE='tty2--started'
2014-01-03 23:23:43.033055 started JOB='startpar-bridge' INSTANCE='tty2--started'
2014-01-03 23:23:43.040671 stopping JOB='startpar-bridge' INSTANCE='tty2--started' RESULT='ok'
2014-01-03 23:23:43.042496 stopped JOB='startpar-bridge' INSTANCE='tty2--started' RESULT='ok'
2014-01-03 23:23:43.044271 started JOB='tty2' INSTANCE=''
^C

You can see the progression of signals: starting, started, stopping, stopped.
You can also see the nesting of jobs. startpar-bridge starts on starting tty2, and runs it's entire task of starting-started-stopping-stopped for tty2 to transition from starting to started.

If you want to trigger a job when tty2 is starting or started, you now know the signals that get emitted. Your job can listen for those signals.

Drawing out relationships using dotfiles

Dot diagram of Upstart user jobs

The initctl2dot application creates dotfile graphics of xdot application. initctl2dot is included with the upstart package, part of all Ubuntu installations (even ubuntu-minimal). xdot is a separate package available in the Ubuntu repositories (Software Center).


As the name implies, initctl2dot's input is initctl's output.You can manually trim an initctl show-show-config output, and input that to initctl2dot if you really want a specific diagram.

You can easily diagram and display the entire system job tree...though it's perhaps less useful than you may expect:

$ initctl2dot --system --outfile /tmp/upstart_root_tree.dot
$ xdot /tmp/upstart_root_tree.dot

You can also diagram the user job tree:

$ initctl2dot --user --outfile /tmp/upstart_user_tree.dot
$ xdot /tmp/upstart_user_tree.dot

Limiting the dotfile size

The initctl2dot manpage includes options for showing/hiding various relationship types (emit, start on, stop on, etc) for clarity.

Another handy option is the --restrict-to-jobs flag, to draw much smaller charts.

For example, let's diagram the system "startup" signal relationships we already discovered using grep:

$ initctl2dot --system --outfile /tmp/upstart_startup_tree.dot \
              --restrict-to-jobs=friendly-recovery,udev-fallback-graphics,\
                                 mountall,kmod,udevtrigger,hostname,\
                                 plymouth-ready,udevmonitor 
$ xdot /tmp/upstart_startup_tree.dot 

And there you have it. How to search system jobs and user jobs for useful signals, and how to easily diagram out the relationships among signals and jobs.

Python http.server and upstart-socket-bridge

In a previous post, I showed how to make the Python 3 socketserver.TCPServer class compatible with upstart-socket-bridge by overriding the server_bind() and server_activate() methods.

Python's http.server module builds on socketserver. Let's see if we can similarly make http.server compatible with upstart-socket-bridge.

About http.server

The http.server module is intended to be a simple way to create webservers. Most of the module is devoted to classes that handle incoming and outgoing data. Only one class, HTTPServer, handles the networking stuff.

---

Example 1

Here is the first example http.server.SimpleHTTPRequestHandler in the Python documentation:

import http.server
import socketserver

PORT = 8000

Handler = http.server.SimpleHTTPRequestHandler

httpd = socketserver.TCPServer(("", PORT), Handler)

print("serving at port", PORT)
httpd.serve_forever()

When you run this code, and point a web browser to port 8000, the code serves the current working directory, with links to files and subdirectories.

Convert Example 1 to work with upstart-socket-bridge

The example server is only seven lines.

Line 3, the PORT, is no longer needed. Upstart will define the port.

Line 5, the socketserver.TCPServer line, will be the biggest change. We need to define a new class based on TCPServer, and override two methods. The is exactly what we did in the previous post.

Line 6, the print statement, can be deleted. When the job is started by Upstart, there is no display - nowhere to print to. An important side effect of starting the job using Upstart is that the Present Working Directory is / (root), unless you specify otherwise in the /etc/init config file that starts the job.

Because of the change to Line 5, the final result is 6 more lines...not bad, and now it works with upstart-socket-bridge.

import http.server
import socketserver

class MyServer(socketserver.TCPServer):
    def server_bind(self):
        """ Replace the socket FD with the Upstart-provided FD"""
        fd = int(os.environ["UPSTART_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

    def server_activate(self):
        pass            # Upstart takes care of listen()

Handler = http.server.SimpleHTTPRequestHandler
server = MyServer(None, Handler)
server.serve_forever()

As always, the script is triggered by an appropriate Upstart job.

Test the script, in this case, by pointing a web browser at the port specified in the Upstart job.

---

Example 2

This example is also from the Python http.server documentation:

def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
    server_address = ('', 8000)
    httpd = server_class(server_address, handler_class)
    httpd.serve_forever()

It's not a standalone example, but instead an abstract example of how to use the module in a larger application.

Let's reformat it into a working example:

import http.server
server_address = ('', 8000)
handler_class=http.server.BaseHTTPRequestHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Now the server runs...sort of. It gives us a 501 error message. Let's add a class to the handler that reads the path and gives a real, valid response. (Source)

import http.server

class MyHandler(http.server.BaseHTTPRequestHandler):
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["<html><head><title>The Title</title></head>",
                   "<body>This is a test.<br />",
                   "You accessed path: ", self.path, "<br />",
                   "</body></html>"]
        self.wfile.write("".join(content).encode("UTF-8"))


server_address = ('', 8000)
handler_class=MyHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Okay, now this is a good working example of a working http.server. It really shows how http.server hides all the complexity of networking on one simple .server line and focuses all your effort on content in the handler. It's clear how you would parse the URL input using self.path. It's clear how you can create and send content using self.wfile.

Convert Example #2 to work with upstart-socket-bridge

There are two classes at work: http.server.BaseHTTPRequestHandler handles content doesn't care about the networking. http.server.HTTPServer handles netwokring and doesn't care about content.

Good news: HTTPServer is based on socketserver.TCPServer, which we already know how to patch!

The changes I made:

1) I want it to launch at each connection, exchange data once, and then terminate. Each connection will launch a separate instance. We no longer want the service to serve_forever().

httpd.serve_forever()    # Old
httpd.handle_request()   # New

2) Let's make it compatible with all three inits: sysvinit (deamon), Upstart, and systemd.

import os

if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \   # Upstart
    or "LISTEN_FDS" in os.environ.keys():     # systemd
        httpd = MyServer(None, MyHandler)     # Need a custom Server class
        httpd.handle_request()                # Run once
    else:                                     # sysvinit
        server_address = ('', 8000)
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()                 # Run forever

3) Add a custom handler that overrides server_bind() and server_activate() in both the socketserver and http.server.HTTPServer modules. This is the secret sauce that makes Upstart compatibility work:

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    def server_bind(self):
        # Get the File Descriptor from an Upstart-created environment variable
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])      # Upstart
        else:
            fd = int(os.environ["LISTEN_FDS"])       # Systemd
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # Only http.server.CGIHTTPRequestHandler uses these
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port

4) Add the Upstart job to monitor the port:

# /etc/init/socket-test.py
description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=8000 ADDR=127.0.0.1
setuid your_username               # Does not need to run as root
exec /usr/bin/python3 /tmp/socket-server.py

And the final product looks like:

#!/usr/bin/python3

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    """
    This class overrides two methods in the socketserver module:
        socketserver __init__ uses both server_bind() and server_activate()
    This class overrides one method in the http.server.HTTPServer class:
        HTTPServer uses both socketserver __init__ and it's own custom
        server_bind
    These overrides makes it compatible with Upstart and systemd socket-bridges
    Warning: It won't bind() or listen() to a socket anymore
    """
    def server_bind(self):
        """
        Get the File Descriptor from an Upstart-created environment variable
        instead of binding or listening to a socket.
        """
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])
        else:
            fd = int(os.environ["LISTEN_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # From http.server:
        # http.server.CGIHTTPRequestHandler uses these.
        # Other handler classes don't use these.
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port


    def server_activate(self):
        """
        This socketserver method sends listen(), so it needs to be overridden
        """
        pass


class MyHandler(http.server.BaseHTTPRequestHandler):
    """
    A very simple custom handler.
    It merely reads the URL and responds with the path.
    This shows how you read data from a GET, and send a response. 
    """
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["",
                   "This is a test.
",
                   "You accessed path: ", self.path, "
",
                   ""]
        self.wfile.write("".join(content).encode("UTF-8"))


if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \     # Upstart
    or "LISTEN_FDS" in os.environ.keys():       # systemd
        httpd = MyServer(None, MyHandler)       # Use fd to get connection
        httpd.handle_request()                  # Handle once, then terminate
    else:
        server_address = ('', 8000)             # sysvinit, classic bind()
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()

Test the service:

• The Python 3 script and Upstart job both use Port 8000.
• Save the Python 3 script. Make it executable.
• Save the Upstart job. Change the Upstart job to Port 8001. Make sure it points to the Python 3 script.

Webserver daemon using sysvinit - run forever:

• Run the Python 3 script.
• Start a web browser, and point it to http://localhost:8000/test/string
• The browser should show a response
• Kill the Python 3 script

Web service using upstart - run once:

• Don't start the Python 3 script. Upstart will do it for you.
• Start a web browser, and point it to http://localhost:8001/test/string
• The browser should show a response.

Python socketserver and upstart-socket-bridge

In a previous post, I described how to use Upstart to monitor a port, trigger a Python script upon connection, and then let the Python application send and receive data over that port.

In a server application, Upstart can take over the bind() and listen() elements of the server daemon, sometimes eliminating the need for a daemon at all.

Here is an example of a Python socketserver, and how it changes when using Upstart.

Some complexity is reduced: The handler is simpler, it no longer needs to fork() or serve_forever(). Each connection triggers the application, so the application need only handle a single read()/write() cycle, then terminate.

Some complexity is increased: We need to override two methods of a socketserver class to make it upstart-socket-bridge compatible.

---

Here is the original socketserver.TCPServer serve_forever example, from the Python documentation:

import socketserver

class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print("{} wrote:".format(self.client_address[0]))
        print(self.data)
        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    HOST, PORT = "localhost", 9999

    # Create the server, binding to localhost on port 9999
    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

    # Activate the server; this will keep running until you
    # interrupt the program with Ctrl-C
    server.serve_forever()

It's easy to test. Let's save the file as /tmp/socket-server-original.py

$ python3 /tmp/socket-server-original.py     # On Terminal #1

$ echo "Test String" | nc localhost 9999     # On Terminal #2
TEST STRING

The example works.

---

Now, let's adapt it for upstart-socket-bridge.

It takes a bit of reading through the socketserver source code to figure it out:

TCPServer's __init__ calls good old bind() and listen() using it's own methods server_bind() and server_activate(), respectively.

Both of those methods can be overridden. Better yet, they use socket module calls (socket.bind() and socket.listen()) that we already know how to replace with socket.fromfd().

So we can get TCPServer to work by overriding the two methods like this:

import os, socket

class MyServer(socketserver.TCPServer):
    """
    This class overrides two method in socketserver.TCPServer 
    and makes it Upstart-compatible
    """
    def server_bind(self):
        """ Replace the socket-created file descriptor (fd) with the Upstart-provided fd"""
        fd = int(os.environ["UPSTART_FDS"]
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

def server_activate(self): """This means listen(), but we don't want it do do anything""" pass


Changing the TCPServer class slightly changes the way way we activate.
We must call the new class instead of TCPServer, and we no longer need the address since bind() no longer does anything:

    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)  # OLD
    server = MyServer(None, MyTCPHandler)                        # NEW

The code that handles the request, generates the response, and sends the response has two changes. Printing on the terminal won't work, since Upstart doesn't use a terminal to start the process. So we'll delete the printing lines:

        print("{} wrote:".format(self.client_address[0]))
        print(self.data)

Finally, one more change. The script can quit after the response has been sent:

    server.serve_forever()      # OLD
    server.handle_request()     # NEW

---

When we put it all together, here is the new version that works with upstart-socket-bridge:

import socketserver, os, socket

class MyServer(socketserver.TCPServer):
    """
    This class overrides two method in socketserver.TCPServer 
    and makes it Upstart-compatible
    """
    def server_bind(self):
        """ Replace the socket FD with the Upstart-provided FD"""
        fd = int(os.environ["UPSTART_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

    def server_activate(self):
        pass            # Upstart takes care of listen()


class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()

        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    # Create the server using the file descriptor from Upstart
    server = MyServer(None, MyTCPHandler)

    # Run once, then terminate
    server.handle_request()

Let's save the file as /tmp/socket-server-new.py
We'll need to adapt (or create) /etc/init/socket-test.conf Upstart job file.

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1
setuid myusername   # Use *your* user name. Doesn't need to be root
exec /usr/bin/python3 /tmp/socket-server-new.py

And let's give it a try on the terminals:

$ python3 /tmp/socket-server-new.py           # On Terminal #1

$ echo "Test String" | nc localhost 34567     # On Terminal #2
TEST STRING

Overriding socketserver for upstart compatibility works.

---

After this test, cleanup is easy

$ sudo rm /etc/init/socket-test.conf
$ rm /tmp/socket-server-original.py /tmp/socket-server-new.py

upstart-socket-bridge

Upstart-socket-bridge is a lot like xinetd. They replace the need for some daemons by monitoring a port, and then launching the desired application when an inbound connection is detected. U-s-b is part of a family of Upstart services that replace many daemon monitoring and listening functions and hooks.

Unlike xinetd, services need to be customized (patched) to run with upstart-socket-bridge.

Documentation is quite sparse. Hence this blog post. That's not intended to criticize; it's really hard to write "good" documentation when you don't know the use cases or the experience level of the user. If you have experience with writing sockets in C, and understand what a file descriptor is and how to use one,  then the documentation is just fine. I didn't before I began this odyssey.

---

How do I make it work?

Here are three simple examples of how it works.
One uses shell script.
One uses Python3.
One uses C.

Hello, World! with shell script

The script that gets triggered by the port action. The port is just a trigger, no data gets exchanged on the port.

1) Let's create a shell script called test-script. This script merely prints out the Upstart-related environment variables into a file.

#!/bin/sh
outfile=/tmp/outfile
date > $outfile            # Timestamp
printenv | grep UPSTART >> $outfile
exit 0

2)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /bin/sh /tmp/test-script                         # Launch the service

3)  Let's run it. Connect to the port using netcat.

$ nc localhost 34567
^C       # End the process using CTRL+C

4)  Look at the result. Hey, look, it's all the environment variables we need!

$ cat /tmp/outfile

5)  Clean up:

$sudo rm /etc/init/socket-test.conf           # Disconnect the launch trigger
$rm /tmp/test-script /tmp/outfile             # Delete the test service

"Hello, World" service in Python 3

(UPDATED: Thanks to Dmitrijs Ledkovs for getting this to work!)

It's a simple echo server - the Python version of the C service below. It requires two files, the application and the Upstart .conf. It demonstrates how a service reads uses the port connection for a trigger and exchanging data.


1) Let's create the Python 3 file. Let's call it test-service.py

#!/usr/bin/python3
import os, socket

# Create the socket file descriptor from the the env var
sock_fd = socket.fromfd(int(os.environ["UPSTART_FDS"]),
                        socket.AF_INET, socket.SOCK_STREAM)

# Accept the connection, create a connection file descriptor
conn, addr = sock_fd.accept()

# Read
message = conn.recv(1024).decode('UTF-8')

# Manipulate data
reply = ("I got your message: " + message)

# Write
conn.send(reply.encode('UTF-8'))

# Finish
conn.close()

2)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /usr/bin/python3 /tmp/test-service.py            # Launch the service

3) Let's run it. Connect to the port using netcat, and then type in a string.

$ nc localhost 34567
Hello, World!                       # You type this in. Server read()s it.
I got your message: Hello, World!   # Server response.  Server write()s it.

4) Cleanup is simple. Simply delete the two files.

$ sudo rm /etc/init/socket-test.conf         # Disconnect the bridge
$ rm /tmp/test-service.py                    # Delete the test service

"Hello, World!" service in C

It's a simple echo server - the C version of the Python service above. It requires two files, the application and the Upstart .conf. It demonstrates how a service reads uses the port connection for a trigger and exchanging data.

1)  Let's create a C file. Let's call it test-service.c

#include <stdlib.h>
#include <netinet/in.h>
#include <string.h>

int main()
{
    /* Read the UPSTART_FDS env var to get the socket fd */
    char *name = "UPSTART_FDS";
    char *env = getenv (name);       // Read the environment variable
    int sock_fd = atoi(env);         // Socket file descriptor

    /* Don't need to do any of these normal socket tasks! Hooray!
    / int port_num;           
    / int sock_fd = socket(AF_INET, SOCK_STREAM, 0);  
    / memset((char *) &serv_addr, 0, sizeof(serv_addr));
    / serv_addr.sin_family = AF_INET;
    / serv_addr.sin_addr.s_addr = INADDR_ANY;
    / serv_addr.sin_port = htons(port_num);
    / struct sockaddr_in serv_addr
    / bind(sock_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr));
    / listen(sock_fd, 5)                                                 
    */

    /* Accept() the connection. Returns the second fd: 'conn_fd' */
    struct sockaddr_in cli_addr;   // Requires netinet/in.h
    int clilen = sizeof(cli_addr);
    int conn_fd = accept(sock_fd, (struct sockaddr *) &cli_addr, &clilen);

    /* Service is active. Read-from and write-to the connection fd */
    char response[276] = "I got your message: ";
    char buffer[256];
    memset((char *) &buffer, 0, sizeof(buffer));  
    read(conn_fd, buffer, 256);                   // Read from conn_fd
    strcat(response, buffer);                     
    write(conn_fd, response, strlen(response));   // Write to conn_fd

    /* Close the connection fd. Socket fd can be reused */
    close(conn_fd);
    return 0;
}

2)  Compile it using gcc, and output the compiled application as an executable called test-service. I put mine in /tmp to make cleanup easier. If you're familiar with gcc, the important element is that there are no flags and no libraries:

gcc -o test-service test-service.c

3)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /tmp/test-service                                # Launch the service

4) Let's run it. Connect to the port using netcat, and then type in a string.

$ nc localhost 34567
Hello, World!                       # You type this in. Server read()s it.
I got your message: Hello, World!   # Server response.  Server write()s it.

5) Cleanup is simple. Simply delete the three files.

$ sudo rm /etc/init/socket-test.conf         # Disconnect the bridge
$ rm /tmp/test-service.c /tmp/test/service   # Delete the test service

---

How does it work?

Here is the oversimplified explanation. Each stream of data whizzing round inside your system is tracked by the kernel. That tracking, sort of like an index or a pointer, is called a file descriptor (fd). A few fds are reserved (0=stdin, 1=stdout, 2=stderr) and you run into these in shell scripting or cron jobs.

A pipe or port or socket is just a way to tell the kernel that a stream of data output from Application A should be input to Application B. Let's look at it another way, and add that fd definition: An fd identifies a data stream output from A and input to B. The pipe/socket/port is a way to express how you want the fd set up.

Now the gritty stuff: A socket/port can have a single server and multiple clients attached. The server bind()s the port, and listen()s on the port for connections, and accept()s each connection from a client. Each connection to a client gets issues another file descriptor.

That's two file descriptors: The first defines the port/socket in general, and the second defines the specific connection between one client and the server.

Upstart tells your server application (since it's not actually serving, let's just call it the "service") the first file descriptor.

• Your service doesn't start a daemon at boot.
• Your service doesn't bind() to the socket/port. Upstart does that.
• Your service doesn't listen() on the socket/port. Upstart does that.
• Your service doesn't fork() for each connection. Upstart launches an instance of your service for each connection. Your service can terminate when the connection ends...if you want it to.
• Your service does accept() the connection from a client, communicate using the resulting file descriptor, and end when the connection close()s.

Let's try it with the example above:

1. Upstart and Service are running on Server. Client is running somewhere else - maybe it's also running on Server, maybe it's out on the network somewhere.
2. The file /etc/init/socket-test.conf tells Upstart to monitor port #34567 on behalf of test-service application. As currently written, it will begin monitoring at boot and stop monitoring at shutdown.
3. When Client --like netcat-- connect()s to port #34567, Upstart launches test-service application with a couple extra environment variables.
4. test-service reads the environment variables, including the file descriptor (fd).
5. test-service accept()s the connection on the file descriptor. This creates a second fd that Service can use to communicate.
6. When Client and test-service are done communicating, they close() the connection fd.
7. test-service can end. Upstart will restart it next time an inbound connection is received.

---

How do I make it work with a service I didn't write? (like my favorite Game Server or Media Server or Backup Server)

Maybe it will work, maybe it won't. There are a couple issues to consider. I don't see an easy, non-coding solution because we're talking about changing the nature of these services.

• Change from always-on to sometimes-on.
• Change to (save and) quit when the connection is done instead of listen()ing for another connection. I don't see any upstart trigger for a socket connection ending.
• Might make some service code simpler. No longer need to fork() or bind().
• Not portable to non-Upstart systems, so the daemon code remains. Adds a bit to code complexity and a new testing case.
• A different trigger (hardware, file change, etc) might be a better trigger than a connection to a socket. Upstart has bridges for those, too.

Cron to be deprecated in favor of Upstart in Ubuntu

Ubuntu's Upstart is an init daemon replacement, quite analagous to OS X's launchd. Launchd also replaced cron on OS X - and upstart plans to replace cron on Ubuntu. No telling when, but all my cron jobs will need to be reformatted.

Update: Sept 2011. Three years later and still waiting...