upstart-socket-bridge

Upstart-socket-bridge is a lot like xinetd. They replace the need for some daemons by monitoring a port, and then launching the desired application when an inbound connection is detected. U-s-b is part of a family of Upstart services that replace many daemon monitoring and listening functions and hooks.

Unlike xinetd, services need to be customized (patched) to run with upstart-socket-bridge.

Documentation is quite sparse. Hence this blog post. That's not intended to criticize; it's really hard to write "good" documentation when you don't know the use cases or the experience level of the user. If you have experience with writing sockets in C, and understand what a file descriptor is and how to use one,  then the documentation is just fine. I didn't before I began this odyssey.

---

How do I make it work?

Here are three simple examples of how it works.
One uses shell script.
One uses Python3.
One uses C.

Hello, World! with shell script

The script that gets triggered by the port action. The port is just a trigger, no data gets exchanged on the port.

1) Let's create a shell script called test-script. This script merely prints out the Upstart-related environment variables into a file.

#!/bin/sh
outfile=/tmp/outfile
date > $outfile            # Timestamp
printenv | grep UPSTART >> $outfile
exit 0

2)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /bin/sh /tmp/test-script                         # Launch the service

3)  Let's run it. Connect to the port using netcat.

$ nc localhost 34567
^C       # End the process using CTRL+C

4)  Look at the result. Hey, look, it's all the environment variables we need!

$ cat /tmp/outfile

5)  Clean up:

$sudo rm /etc/init/socket-test.conf           # Disconnect the launch trigger
$rm /tmp/test-script /tmp/outfile             # Delete the test service

"Hello, World" service in Python 3

(UPDATED: Thanks to Dmitrijs Ledkovs for getting this to work!)

It's a simple echo server - the Python version of the C service below. It requires two files, the application and the Upstart .conf. It demonstrates how a service reads uses the port connection for a trigger and exchanging data.


1) Let's create the Python 3 file. Let's call it test-service.py

#!/usr/bin/python3
import os, socket

# Create the socket file descriptor from the the env var
sock_fd = socket.fromfd(int(os.environ["UPSTART_FDS"]),
                        socket.AF_INET, socket.SOCK_STREAM)

# Accept the connection, create a connection file descriptor
conn, addr = sock_fd.accept()

# Read
message = conn.recv(1024).decode('UTF-8')

# Manipulate data
reply = ("I got your message: " + message)

# Write
conn.send(reply.encode('UTF-8'))

# Finish
conn.close()

2)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /usr/bin/python3 /tmp/test-service.py            # Launch the service

3) Let's run it. Connect to the port using netcat, and then type in a string.

$ nc localhost 34567
Hello, World!                       # You type this in. Server read()s it.
I got your message: Hello, World!   # Server response.  Server write()s it.

4) Cleanup is simple. Simply delete the two files.

$ sudo rm /etc/init/socket-test.conf         # Disconnect the bridge
$ rm /tmp/test-service.py                    # Delete the test service

"Hello, World!" service in C

It's a simple echo server - the C version of the Python service above. It requires two files, the application and the Upstart .conf. It demonstrates how a service reads uses the port connection for a trigger and exchanging data.

1)  Let's create a C file. Let's call it test-service.c

#include <stdlib.h>
#include <netinet/in.h>
#include <string.h>

int main()
{
    /* Read the UPSTART_FDS env var to get the socket fd */
    char *name = "UPSTART_FDS";
    char *env = getenv (name);       // Read the environment variable
    int sock_fd = atoi(env);         // Socket file descriptor

    /* Don't need to do any of these normal socket tasks! Hooray!
    / int port_num;           
    / int sock_fd = socket(AF_INET, SOCK_STREAM, 0);  
    / memset((char *) &serv_addr, 0, sizeof(serv_addr));
    / serv_addr.sin_family = AF_INET;
    / serv_addr.sin_addr.s_addr = INADDR_ANY;
    / serv_addr.sin_port = htons(port_num);
    / struct sockaddr_in serv_addr
    / bind(sock_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr));
    / listen(sock_fd, 5)                                                 
    */

    /* Accept() the connection. Returns the second fd: 'conn_fd' */
    struct sockaddr_in cli_addr;   // Requires netinet/in.h
    int clilen = sizeof(cli_addr);
    int conn_fd = accept(sock_fd, (struct sockaddr *) &cli_addr, &clilen);

    /* Service is active. Read-from and write-to the connection fd */
    char response[276] = "I got your message: ";
    char buffer[256];
    memset((char *) &buffer, 0, sizeof(buffer));  
    read(conn_fd, buffer, 256);                   // Read from conn_fd
    strcat(response, buffer);                     
    write(conn_fd, response, strlen(response));   // Write to conn_fd

    /* Close the connection fd. Socket fd can be reused */
    close(conn_fd);
    return 0;
}

2)  Compile it using gcc, and output the compiled application as an executable called test-service. I put mine in /tmp to make cleanup easier. If you're familiar with gcc, the important element is that there are no flags and no libraries:

gcc -o test-service test-service.c

3)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /tmp/test-service                                # Launch the service

4) Let's run it. Connect to the port using netcat, and then type in a string.

$ nc localhost 34567
Hello, World!                       # You type this in. Server read()s it.
I got your message: Hello, World!   # Server response.  Server write()s it.

5) Cleanup is simple. Simply delete the three files.

$ sudo rm /etc/init/socket-test.conf         # Disconnect the bridge
$ rm /tmp/test-service.c /tmp/test/service   # Delete the test service

---

How does it work?

Here is the oversimplified explanation. Each stream of data whizzing round inside your system is tracked by the kernel. That tracking, sort of like an index or a pointer, is called a file descriptor (fd). A few fds are reserved (0=stdin, 1=stdout, 2=stderr) and you run into these in shell scripting or cron jobs.

A pipe or port or socket is just a way to tell the kernel that a stream of data output from Application A should be input to Application B. Let's look at it another way, and add that fd definition: An fd identifies a data stream output from A and input to B. The pipe/socket/port is a way to express how you want the fd set up.

Now the gritty stuff: A socket/port can have a single server and multiple clients attached. The server bind()s the port, and listen()s on the port for connections, and accept()s each connection from a client. Each connection to a client gets issues another file descriptor.

That's two file descriptors: The first defines the port/socket in general, and the second defines the specific connection between one client and the server.

Upstart tells your server application (since it's not actually serving, let's just call it the "service") the first file descriptor.

• Your service doesn't start a daemon at boot.
• Your service doesn't bind() to the socket/port. Upstart does that.
• Your service doesn't listen() on the socket/port. Upstart does that.
• Your service doesn't fork() for each connection. Upstart launches an instance of your service for each connection. Your service can terminate when the connection ends...if you want it to.
• Your service does accept() the connection from a client, communicate using the resulting file descriptor, and end when the connection close()s.

Let's try it with the example above:

1. Upstart and Service are running on Server. Client is running somewhere else - maybe it's also running on Server, maybe it's out on the network somewhere.
2. The file /etc/init/socket-test.conf tells Upstart to monitor port #34567 on behalf of test-service application. As currently written, it will begin monitoring at boot and stop monitoring at shutdown.
3. When Client --like netcat-- connect()s to port #34567, Upstart launches test-service application with a couple extra environment variables.
4. test-service reads the environment variables, including the file descriptor (fd).
5. test-service accept()s the connection on the file descriptor. This creates a second fd that Service can use to communicate.
6. When Client and test-service are done communicating, they close() the connection fd.
7. test-service can end. Upstart will restart it next time an inbound connection is received.

---

How do I make it work with a service I didn't write? (like my favorite Game Server or Media Server or Backup Server)

Maybe it will work, maybe it won't. There are a couple issues to consider. I don't see an easy, non-coding solution because we're talking about changing the nature of these services.

• Change from always-on to sometimes-on.
• Change to (save and) quit when the connection is done instead of listen()ing for another connection. I don't see any upstart trigger for a socket connection ending.
• Might make some service code simpler. No longer need to fork() or bind().
• Not portable to non-Upstart systems, so the daemon code remains. Adds a bit to code complexity and a new testing case.
• A different trigger (hardware, file change, etc) might be a better trigger than a connection to a socket. Upstart has bridges for those, too.

Handy Pipe and Socket Reference

I'm not going to explain sockets. There are too many good tutorials out there already.
Instead, I'm going to use sockets in a bunch of examples that build.

Ordinary Pipes
Named Pipes/Sockets
Unix Domain Sockets
TCP Sockets/Ports
upstart-socket-bridge

The Simplest

Pipes (|) and redirection (>). We already know how to do that in the shell.
These pipes connect related processes. For example, the children of a single terminal are related.

It's hard to pipe from shell to python (or the reverse) - it's possible, just not easy the first couple times you try.

Shell example:

$ echo Hello, Planet\! | sed 's/Planet/World/'
Hello, World!

Here is a good example of pipes in Python 3.

---

Named Pipes

Slightly more complicated (reference). These use a pipe file that synchronizes the data transfer when both processes are ready. A -> Pipe -> B. This is handy for inter-process communication that does not need to run through dbus, and saves a bit of overhead.

Names pipes are a shared file. They can have many (anonymous) writers, but only a single reader. Shell, Python, and C senders and listeners can interoperate (fairly) smoothly.

Named pipes are blocking. Start the reader first, and it will wait for the writer. Alternately, start the writer first, and it will wait for the reader. Once the writer has finished, the reader will also stop. One read() for one write(). A second read() requires a second write()...and a second write() requires a second read().

Named pipes in shell and C can be reused. Named pipes in Python cannot be reused after close() - they must be unlink() (destroyed) and remade afresh.

Conventionally, the listener is expected to create the pipe. But it's not required.

Processes that use named pipes work in the following order:

• Create the pipe (usually done by listener)
• Open the pipe (both sides)
• Write the pipe
• Writer close the pipe (allows the reader to see the data)
• Listener read the pipe
• Listener close the pipe
• Delete the pipe (usually done by the listener)

Creating and Deleting a Named Pipe:

Shell:

#!/bin/sh
mkfifo /tmp/testpipe        # Create pipe
rm /tmp/testpipe            # Delete pipe

Python3:

>>> import os
>>> os.mkfifo("/tmp/testpipe")    # Create pipe
>>> os.remove("/tmp/testpipe")    # Delete pipe 

C:

# include <sys/stat.h>
int main( ){
   int result = mkfifo ("/tmp/testpipe", S_IRUSR| S_IWUSR);  \\ Create Pipe
   unlink ("/tmp/testpipe");                                 \\ Delete Pipe
   return 0;
   }

Named Pipe Listeners:

 

Shell:

#!/bin/sh
pipe=/tmp/testpipe
trap "rm -f $pipe" EXIT
while true
do
    read line <$pipe   # Read the pipe
    echo $line
done

Python3:

>>> import os
>>> pipe = open("/tmp/testpipe", "r")
>>> pipe.read()     # You won't read anything until the sender close()
'Hello, World!'     # Data from pipe
>>> pipe.close()

C:

#include <fcntl.h>
#include <stdio.h>
int main()
{
    char buf[20];
    int fd = open("/tmp/testpipe", O_RDONLY);  // Open the pipe
    read(fd, buf, 20);                         // Read the message - unblock the writing process
    printf("Received: %s\n", buf);             // Display the message
    close(fd);                                 // Close the pipe
    return 0;
}

Named Pipe Senders:

Shell:

#!/bin/sh
pipe=/tmp/testpipe
[ ! -p $pipe ] && echo "Reader not running" && exit 1
[ "$1" ] && echo "$1" >$pipe || echo "Hello from $$" >$pipe

Python3:

>>> import os
>>> pipe = open("/tmp/testpipe", "w") # Hang here awaiting the receiver to read() once
>>> pipe.write("Hello, World!")
13
>>> pipe.close()    # Receiver read() is empty until this happens

C:

#include <fcntl.h>
int main()
{
    int fd = open("/tmp/testpipe", O_WRONLY);   // Open the pipe
    write(fd, "Hi", sizeof("Hi"));              // Write "Hi" to pipe
    close(fd);                                  // Close the pipe - allow the read
    return 0;
}

---

Unix Domain Sockets

Unix Domain Sockets is a fancy name for a fancy pipe. Like shell pipes, Unix Sockets use a pipe file...Well it *looks* like a pipe file. Actually, data is buffered instead of written to a file. The file is simply an easy way for the processes to find each other.

Unlike shell pipes, Unix sockets are bi-directional, and multiple writers are not anonymous. In Python, sockets are single-use: After a close() they cannot be reopened, and must be deleted and remade. Pipes can only be read() when the sender close(), but sockets can be read() for every send().

Since sockets are bi-directional, it's not a sender/receiver relationship anymore. It's a client/server relationship.

Processes using a Unix Domain Socket usually work in the following order:

• Create a file descriptor (fd) that defines the socket OR look up the fd of an existing socket.
• Server binds to the socket
• Server listens on the socket
• Client connects to the socket
• Server accepts the connection
• Both sides read/write
• Client closes socket (on Python, cannot be reopened - must be reconnected )
• Server closes socket (usually during service shutdown)
• Delete/unlink the file descriptor and socket (if it won't be reused)

Creating and Deleting a Unix Domain Socket:

Shell:

Shell creation can be automatic, for example using netcat:

$ nc -lU /tmp/sock           # Create socket, among other actions
$ rm /tmp/sock               # Delete socket

Python3:

>>> import socket, os
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)  
>>> tempsocket.bind("/tmp/sock")  # Create socket
>>> os.remove("/tmp/sock")        # Delete socket 

C:

#include <sys/socket.h>
#include <sys/un.h>
int main() {
  char *path = "/tmp/sock";
  struct sockaddr_un addr;
  char buf[100];
  int cl,rc;
  int fd = socket(AF_UNIX, SOCK_STREAM, 0);   // Create a file descriptor
  addr.sun_family = AF_UNIX;                  // It's a Unix socket
  strcpy(addr.sun_path, path);                // Set the path
  memset(&addr, 0, sizeof(addr));         // Fill the address array with zeroes (prevent garbage)
  bind(fd, (struct sockaddr*)&addr, sizeof(addr));    // Create socket (server)
  connect(fd, (struct sockaddr*)&addr, sizeof(addr)); // Connect (client)
  unlink(path);                               // Delete socket
  return 0;
}

Unix Domain Socket Servers (Bind to socket):

The server owns (binds) and monitors the socket, and is usually responsible for creating and deleting the socket. (C example)

Shell:

$ nc -lU /tmp/sock

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.bind("/tmp/sock")       # Listener is first process using socket
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept()   # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8')    # Listener blocks here awaiting send()
'Hello, World'                         # Try this a few times to capture each send()
>>> tempsocket.close()

C:

#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
int main() {
  char buf[100];
  int cl,rc;
  int fd = create_socket(socket_path)      // Creating the socket is not done here. See above
  listen(fd, 5);                // listen() is the step after bind()
  while (1) {
    cl = accept(fd, NULL, NULL);
    while ( (rc=read(cl,buf,sizeof(buf))) > 0) {
      printf("read %u bytes: %.*s\n", rc, rc, buf);
    }
    if (rc == 0) {
      printf("EOF\n");
      close(cl);
    }
  }
  return 0;
}

Unix Domain Socket Clients (Connect to socket):

Clients don't own (bind) the socket. Instead, they connect on-demand.

Shell:

$ nc -U /tmp/sock

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.connect("/tmp/sock")           # Listener must be active before sender
>>> tempsocket.send(data)                     # Try this a few times
13
>>> tempsocket.close()                        # End of stream

C:

#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
int main() {
  struct sockaddr_un addr;
  char buf[100];
  int rc;
  int fd = create_socket(socket path)  // Creating the socket is not done here

  // First step after connect()  
  while( (rc=read(STDIN_FILENO, buf, sizeof(buf))) > 0) write(fd, buf, rc);
  return 0;
}

Quick fun with netcat:

The easiest possible bi-directional socket.

$ nc -lU /tmp/sock   # Enter in Terminal #1
$ nc -U /tmp/sock    # Enter in Terminal #2

Type anything on either terminal.
Look! The two communicate!

Here is an example of using netcat to exchange files this way (from the man page)

$ nc -lU /tmp/sock > filename.out      # Terminal #1 (sender)
$ nc -U /tmp/sock < filename.in        # Terminal #2 (receiver)

---

TCP Sockets (Ports)

TCP ports are very much like Unix sockets. Instead of a file, the data stream is buffered. And Unix sockets are not networkable, but TCP ports are a foundation of the Internet. Ports use client/server, and the server owns (binds) the port that the client connects to. Sockets and Ports use the same shell application (netcat) and Python3 module (socket).

Since there is no file to create or delete, bind() and connect() simply use an IP address and port number.

Processes using a TCP port usually work in the following order (just like a Unix Domain Socket):

• Create a file descriptor (fd) that defines the port
• Server binds to the port
• Server listens on the port
• Client connects to the port
• Server accepts the connection
• Both sides read/write
• Client closes port
• Server closes port (usually during service shutdown)
• Delete/unlink the file descriptor and port (if it won't be reused)

TCP Port Services (Bind to port):

The server owns (binds) and monitors the port.

Shell:

$ nc -l 45678      # Port 45678

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.bind(("127.0.0.1", 45678))   # Tuple: (IP Addr, Port)
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept()   # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8')    # Listener blocks here awaiting send()
'Hello, World'                                  # Try it a few times to catch each send()
>>> tempsocket.close()

C:

#include <stdio.h>
#include <string.h>
#include <netinet/in.h>
int main()
{
     int sock_fd, conn_fd;   // Socket file descriptor and Connection file descriptor
     int port_num = 45678;
     socklen_t clilen;
     char buffer[256];
     struct sockaddr_in serv_addr, cli_addr;
     sockfd = socket(AF_INET, SOCK_STREAM, 0);        // Define socket type
     memset(&serv_addr, 0, sizeof(serv_addr));    // Set array to zeroes 
     serv_addr.sin_family = AF_INET;
     serv_addr.sin_addr.s_addr = INADDR_ANY;
     serv_addr.sin_port = htons(port_num);
     bind(sock_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr));   // Bind
     listen(sock_fd,5);                               // Listen
     clilen = sizeof(cli_addr);
     conn_fd = accept(sock_fd, 
                 (struct sockaddr *) &cli_addr, 
                 &clilen);                        // Accept
     memset(&buffer, 0, sizeof(buffer));          // Fill buffer with zeroes - prevent garbage
     read(conn_fd,buffer,255);                        // Read
     printf("Here is the message: %s\n",buffer);
     write(conn_fd,"I got your message",18);          // Write
     close(conn_fd);
     close(sock_fd);                                  // Close
     return 0; 
}

TCP Port Clients (Connect to port):

The client connects to the port on-demand.

Shell:

$ nc localhost 45678      # IP=loopback, Port #45678

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.connect(("127.0.0.1", 45678))   # Tuple: (IP Addr, Port)
>>> data = "Hello, World".encode("utf-8")
>>> tempsocket.send(data)          # Try this a few times
12
>>> tempsocket.close()

C:

#include <stdio.h>
#include <string.h>
#include <netdb.h>
int main(int argc, char *argv[])
{
    int sock_fd;
    int port_num = 45678;
    struct sockaddr_in serv_addr;
    struct hostent *server;
    char buffer[256];
    sock_fd = socket(AF_INET, SOCK_STREAM, 0);       // Define socket type
    server = gethostbyname(port_num);                // Lookup port owner
    memset(&serv_addr, 0, sizeof(serv_addr));    // Set array to zeroes
    serv_addr.sin_family = AF_INET;
    bcopy((char *)server->h_addr,
         (char *)&serv_addr.sin_addr.s_addr,
         server->h_length);
    serv_addr.sin_port = htons(port_num);
    connect(sock_fd,(struct sockaddr *) &serv_addr,sizeof(serv_addr)); // Connect
    printf("Please enter the message: ");
    memset(&buffer, 0, sizeof(buffer));
    fgets(buffer,255,stdin);                // Keyboard input
    write(sock_fd,buffer,strlen(buffer));   // Write to server
    memset(&buffer, 0, sizeof(buffer)); // Reuse buffer array - clear to prevent garbage
    read(sock_fd,buffer,255);               // Read response
    printf("%s\n",buffer);                  // Print response
    close(sock_fd);                         // Close
    return 0;
}

More fun with netcat

This is the basic way that webservers work.

$ echo "Hello, World" | nc -l 45678   # Enter in Terminal #1
$ nc localhost 45678                  # Enter in Terminal #2
                                      # OR open http://localhost:45678
                                      # in your favorite web browser