Ian's TechBlog: Python http.server and upstart-socket-bridge

In a previous post, I showed how to make the Python 3 socketserver.TCPServer class compatible with upstart-socket-bridge by overriding the server_bind() and server_activate() methods.

Python's http.server module builds on socketserver. Let's see if we can similarly make http.server compatible with upstart-socket-bridge.

About http.server

The http.server module is intended to be a simple way to create webservers. Most of the module is devoted to classes that handle incoming and outgoing data. Only one class, HTTPServer, handles the networking stuff.

Example 1

Here is the first example http.server.SimpleHTTPRequestHandler in the Python documentation:

import http.server
import socketserver

PORT = 8000

Handler = http.server.SimpleHTTPRequestHandler

httpd = socketserver.TCPServer(("", PORT), Handler)

print("serving at port", PORT)
httpd.serve_forever()

When you run this code, and point a web browser to port 8000, the code serves the current working directory, with links to files and subdirectories.

Convert Example 1 to work with upstart-socket-bridge

The example server is only seven lines.

Line 3, the PORT, is no longer needed. Upstart will define the port.

Line 5, the socketserver.TCPServer line, will be the biggest change. We need to define a new class based on TCPServer, and override two methods. The is exactly what we did in the previous post.

Line 6, the print statement, can be deleted. When the job is started by Upstart, there is no display - nowhere to print to. An important side effect of starting the job using Upstart is that the Present Working Directory is / (root), unless you specify otherwise in the /etc/init config file that starts the job.

Because of the change to Line 5, the final result is 6 more lines...not bad, and now it works with upstart-socket-bridge.

import http.server
import socketserver

class MyServer(socketserver.TCPServer):
    def server_bind(self):
        """ Replace the socket FD with the Upstart-provided FD"""
        fd = int(os.environ["UPSTART_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

    def server_activate(self):
        pass            # Upstart takes care of listen()

Handler = http.server.SimpleHTTPRequestHandler
server = MyServer(None, Handler)
server.serve_forever()

As always, the script is triggered by an appropriate Upstart job.

Test the script, in this case, by pointing a web browser at the port specified in the Upstart job.

Example 2

This example is also from the Python http.server documentation:

def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
    server_address = ('', 8000)
    httpd = server_class(server_address, handler_class)
    httpd.serve_forever()

It's not a standalone example, but instead an abstract example of how to use the module in a larger application.

Let's reformat it into a working example:

import http.server
server_address = ('', 8000)
handler_class=http.server.BaseHTTPRequestHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Now the server runs...sort of. It gives us a 501 error message. Let's add a class to the handler that reads the path and gives a real, valid response. (Source)

import http.server

class MyHandler(http.server.BaseHTTPRequestHandler):
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["<html><head><title>The Title</title></head>",
                   "<body>This is a test.<br />",
                   "You accessed path: ", self.path, "<br />",
                   "</body></html>"]
        self.wfile.write("".join(content).encode("UTF-8"))


server_address = ('', 8000)
handler_class=MyHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Okay, now this is a good working example of a working http.server. It really shows how http.server hides all the complexity of networking on one simple .server line and focuses all your effort on content in the handler. It's clear how you would parse the URL input using self.path. It's clear how you can create and send content using self.wfile.

Convert Example #2 to work with upstart-socket-bridge

There are two classes at work: http.server.BaseHTTPRequestHandler handles content doesn't care about the networking. http.server.HTTPServer handles netwokring and doesn't care about content.

Good news: HTTPServer is based on socketserver.TCPServer, which we already know how to patch!

The changes I made:

1) I want it to launch at each connection, exchange data once, and then terminate. Each connection will launch a separate instance. We no longer want the service to serve_forever().

httpd.serve_forever()    # Old
httpd.handle_request()   # New

2) Let's make it compatible with all three inits: sysvinit (deamon), Upstart, and systemd.

import os

if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \   # Upstart
    or "LISTEN_FDS" in os.environ.keys():     # systemd
        httpd = MyServer(None, MyHandler)     # Need a custom Server class
        httpd.handle_request()                # Run once
    else:                                     # sysvinit
        server_address = ('', 8000)
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()                 # Run forever

3) Add a custom handler that overrides server_bind() and server_activate() in both the socketserver and http.server.HTTPServer modules. This is the secret sauce that makes Upstart compatibility work:

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    def server_bind(self):
        # Get the File Descriptor from an Upstart-created environment variable
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])      # Upstart
        else:
            fd = int(os.environ["LISTEN_FDS"])       # Systemd
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # Only http.server.CGIHTTPRequestHandler uses these
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port

4) Add the Upstart job to monitor the port:

# /etc/init/socket-test.py
description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=8000 ADDR=127.0.0.1
setuid your_username               # Does not need to run as root
exec /usr/bin/python3 /tmp/socket-server.py

And the final product looks like:

#!/usr/bin/python3

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    """
    This class overrides two methods in the socketserver module:
        socketserver __init__ uses both server_bind() and server_activate()
    This class overrides one method in the http.server.HTTPServer class:
        HTTPServer uses both socketserver __init__ and it's own custom
        server_bind
    These overrides makes it compatible with Upstart and systemd socket-bridges
    Warning: It won't bind() or listen() to a socket anymore
    """
    def server_bind(self):
        """
        Get the File Descriptor from an Upstart-created environment variable
        instead of binding or listening to a socket.
        """
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])
        else:
            fd = int(os.environ["LISTEN_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # From http.server:
        # http.server.CGIHTTPRequestHandler uses these.
        # Other handler classes don't use these.
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port


    def server_activate(self):
        """
        This socketserver method sends listen(), so it needs to be overridden
        """
        pass


class MyHandler(http.server.BaseHTTPRequestHandler):
    """
    A very simple custom handler.
    It merely reads the URL and responds with the path.
    This shows how you read data from a GET, and send a response. 
    """
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["The Title",
                   "This is a test.
",
                   "You accessed path: ", self.path, "
",
                   ""]
        self.wfile.write("".join(content).encode("UTF-8"))


if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \     # Upstart
    or "LISTEN_FDS" in os.environ.keys():       # systemd
        httpd = MyServer(None, MyHandler)       # Use fd to get connection
        httpd.handle_request()                  # Handle once, then terminate
    else:
        server_address = ('', 8000)             # sysvinit, classic bind()
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()

Test the service:

The Python 3 script and Upstart job both use Port 8000.
Save the Python 3 script. Make it executable.
Save the Upstart job. Change the Upstart job to Port 8001. Make sure it points to the Python 3 script.

Webserver daemon using sysvinit - run forever:

Run the Python 3 script.
Start a web browser, and point it to http://localhost:8000/test/string
The browser should show a response
Kill the Python 3 script

Web service using upstart - run once:

Don't start the Python 3 script. Upstart will do it for you.
Start a web browser, and point it to http://localhost:8001/test/string
The browser should show a response.

Ian's TechBlog

Sunday, December 22, 2013

Python http.server and upstart-socket-bridge