Simple geolocation in Ubuntu 14.04

Geolocation means 'figuring out where a spot on the Earth is'.

Usually, it's the even more limited question 'where am I?'

GeoClue

The default install of Ubuntu includes GeoClue, a dbus service that checks IP address and GPS data. Since 2012, when I last looked at GeoClue, it's changed a bit, and it has more backends available in the Ubuntu Repositories.

 

Privacy

Some commenters on the interwebs have claimed that GeoClue is privacy-intrusive. It's not. It merely tries to figure out your location, which can be handy for various services on your system. It doesn't share or send your location to anybody else.

dbus introspection and d-feet

You would expect that a dbus application like GeoClue would be visible using a dbus introspection tool like d-feet (provided by the d-feet package).

But there's a small twist: D-feet can only see dbus applications that are running. It can only see dbus applications that active, or are inactive daemons.

It's possible (and indeed preferable in many circumstances) to write a dbus application that is not a daemon - it starts at first connection, terminates when complete, and restarts at the next connection. D-feet cannot see these when they are not running.

Back in 2012, GeoClue was an always-on daemon, and always visible to d-feet.
But in 2014 GeoClue is (properly) no longer a daemon, and d-feet won't see GeoClue if it's not active.

This simply means we must trigger a connection to GeoClue to make it visible.
Below are two ways to do so: The geoclue-test-gui application, and a Python3 example.

---

geoclue-test-gui

One easy way to see GeoClue in action, and to make it visible to d-feet, is to use the geoclue-test-gui application (included in the geoclue-examples package)

$ sudo apt-get install geoclue-examples
$ geoclue-test-gui

---

GeoClue Python3 example

Once GeoClue is visible in d-feet (look in the 'session' tab), you can see the interfaces and try them out.

Here's an example of the GetAddress() and GetLocation() methods using Python3:

>>> import dbus

>>> dest           = "org.freedesktop.Geoclue.Master"
>>> path           = "/org/freedesktop/Geoclue/Master/client0"
>>> addr_interface = "org.freedesktop.Geoclue.Address"
>>> posn_interface = "org.freedesktop.Geoclue.Position"

>>> bus        = dbus.SessionBus()
>>> obj        = bus.get_object(dest, path)
>>> addr_iface = dbus.Interface(obj, addr_interface)
>>> posn_iface = dbus.Interface(obj, posn_interface)

>>> addr_iface.GetAddress()
(dbus.Int32(1404823176),          # Timestamp
 dbus.Dictionary({
     dbus.String('locality')   : dbus.String('Milwaukee'),
     dbus.String('country')    : dbus.String('United States'),
     dbus.String('countrycode'): dbus.String('US'),
     dbus.String('region')     : dbus.String('Wisconsin'), 
     dbus.String('timezone')   : dbus.String('America/Chicago')}, 
     signature=dbus.Signature('ss')),
 dbus.Struct(                 # Accuracy
     (dbus.Int32(3),
      dbus.Double(0.0),
      dbus.Double(0.0)),
      signature=None)
)

>>> posn_iface.GetPosition()
(dbus.Int32(3),               # Num of fields
 dbus.Int32(1404823176),      # Timestamp
 dbus.Double(43.0389),        # Latitude
 dbus.Double(-87.9065),       # Longitude
 dbus.Double(0.0),            # Altitude
 dbus.Struct((dbus.Int32(3),  # Accuracy
              dbus.Double(0.0),
              dbus.Double(0.0)),
              signature=None))

>>> addr_dict = addr_iface.GetAddress()[1]
>>> str(addr_dict['locality'])
'Milwaukee'

>>> posn_iface.GetPosition()[2]
dbus.Double(43.0389)
>>> posn_iface.GetPosition()[3]
dbus.Double(-87.9065)
>>> lat = float(posn_iface.GetPosition()[2])
>>> lon = float(posn_iface.GetPosition()[3])
>>> lat,lon
(43.0389, -87.9065)

Note: Geoclue's accuracy codes

---

Ubuntu GeoIP Service

When you run geoclue-test-gui, you discover that only one backend service is installed with the default install of Ubuntu - the Ubuntu GeoIP service.

The Ubuntu GeoIP service is provided by the geoclue-ubuntu-geoip package, and is included with the default install of Ubuntu 14.04. It simply queries an ubuntu.com server, and parses the XML response.

You can do it yourself, too:

$ wget -q -O - http://geoip.ubuntu.com/lookup

<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Ip>76.142.123.22</Ip>
  <Status>OK</Status>
  <CountryCode>US</CountryCode>
  <CountryCode3>USA</CountryCode3>
  <CountryName>United States</CountryName>
  <RegionCode>WI</RegionCode>
  <RegionName>Wisconsin</RegionName>
  <City>Milwaukee</City>
  <ZipPostalCode></ZipPostalCode>
  <Latitude>43.0389</Latitude>
  <Longitude>-87.9065</Longitude>
  <AreaCode>414</AreaCode>
  <TimeZone>America/Chicago</TimeZone>
</Response>

---

GeoIP

The default install of Ubuntu 14.04 also includes (the confusingly-named) GeoIP. While it has the prefix 'Geo', it's not a geolocator. It's completely unrelated to the Ubuntu GeoIP service. Instead, GeoIP is a database the IP addresses assigned to each country, provided by the geoip-database package. Knowing the country of origin of a packet or server or connection can be handy.

geoip-database has many bindings, including Python 2.7 (but sadly not Python 3). Easiest is the command line, provided by the additional geoip-bin package.

$ sudo apt-get install geoip-bin
$ geoiplookup 76.45.203.45
GeoIP Country Edition: US, United States

---

GeocodeGlib

Back in 2012, I compared the two methods of geolocation in Ubuntu: GeoClue and GeocodeGlib. GeocodeGlib was originally intended as a smaller, easier to maintain replacement for GeoClue. But as we have already seen, GeoClue has thrived instead of withering. The only two packages that seem to require GeocodeGlib in 14.04 are gnome-core-devel and gnome-clocks
GeocodeGlib, provided by the libgeocode-glib0 package, is no longer included with a default Ubuntu installation anymore, but it is easily available in the Software Center.

sudo apt-get install gir1.2-geocodeglib-1.0

That is the GTK introspection package for geocodeglib, and it pulls in libgeocode-glib0 as a dependency. The introspection package is necessary.

Useful documentation and code examples are non-existent. My python code sample from 2012 no longer works. It's easy to create a GeocodeGlib.Place() object, and to assign various values to it (town name, postal code, state), but I can't figure out how to get GeocoddeGlib to automatically determine and fill in other properties. So even though it seems maintained, I'm not recommending it as a useful geolocation service.

Intermediate GTFS: How to prevent stop_times.txt from hogging all your RAM

In a GTFS file for a large provider (example: Chicago), the stop_times.txt file can be huge.

Here's an example:

$ unzip -v 20140611.cta.gtfs

 Archive:  20140611.cta.gtfs
  Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
 --------  ------  ------- ---- ---------- ----- --------  ----
      235  Defl:N      153  35% 2014-06-10 22:49 6dfdcbd7  agency.txt
     4889  Defl:N      554  89% 2014-06-10 22:49 38544274  calendar.txt
     2139  Defl:N      335  84% 2014-06-10 22:49 f69f2cb4  calendar_dates.txt
    13000  Defl:N     2279  83% 2014-06-10 22:49 59bf453e  routes.txt
 29896363  Defl:N  7158190  76% 2014-06-10 22:54 21c7d003  shapes.txt
  1332298  Defl:N   264714  80% 2014-06-10 23:00 48a736df  stops.txt
355900629  Defl:N 57437701  84% 2014-06-10 23:24 e13831ff  stop_times.txt
     2514  Defl:N      575  77% 2014-06-10 23:24 51dca8cf  transfers.txt
  5654714  Defl:N   656156  88% 2014-06-10 23:25 7056aa15  trips.txt
       42  Defl:N       39   7% 2014-06-10 22:49 87676593  frequencies.txt
    18959  Defl:N     6180  67% 2011-06-20 11:07 bcf138d1  developers_license_agreement.htm
 --------          -------  ---                            -------
392825782         65526876  83%                            11 files

Uncompressed, the stop_times.txt file is 356 MB, 91% of the entire archive.

You can see right away that loading that behemoth into memory will cause a pause (30 seconds, on my hardware) that a user will probably notice. And the way Python dicts use memory, 356 MB of CSV data can easily explode into 3 or 4 GB of a comparable python dict.  And that leaves us with a problem: We can't load the file on-demand because it takes too darn long, we can't keep it stored in an easily-accessible form in memory because it's too darn big. While we could keep it in CSV format in RAM, that's not a fomat we can do anything with - every query would need to iterate through the entire file, and that's sort of the worst of both worlds.

---

So what can we do?

First, we can filter data immediately upon load, and discard big chunks of it right away. This means the structure of the program must change from:

cal = load_table(calendar.txt)
rou = load_table(routes.txt)
tri = load_table(trips.txt)
sto = load_table(stops.txt)
tim = load_table(stop_times.txt)
svcid = valid_service_ids_for_today(cal)
trips = trips_that_stop_there_today(tri, svc_id, tim)

to something more like:

cal = load_table(calendar.txt)
rou = load_table(routes.txt)
tri = load_table(trips.txt)
sto = load_table(stops.txt)
tim = load_and_filter_stop_times_table(stop_times.txt, filter_criteria)
svcid = valid_service_ids_for_today(cal)
trips = trips_that_stop_there_today(tri, svc_id, tim)

Instead of loading the entire table and holding it in memory, a customized loader filters the data right away, and chuck out the 90% we don't actually want.


Second, we can use Python's read size parameter to prevent a spike in RAM usage (and consequent slowdown of the entire system).

For example, normally you read data with something like:

with gtfs_file.open('stop_times.txt', mode='r') as infile:
    input = infile.read()
output = do_stuff_to(input.decode())
return output

And we can rearrange it to something like:

eof    = False
output = {}
with gtfs_file.open('stop_times.txt', mode='r') as infile:
    while eof == False:
        input_block  = infile.readlines(4096)
        if len(input_block) == 0:
            eof = True

        output_block = do_stuff_to(input_block)
        output.update(output_block)

return output

The EOF flag controls the process, and an empty read triggers the flag.
The input RAM requirements are tiny, since each block re-uses the same memory over and over. Only the filtered output grows.

The performance penalty of this kind of loading seems about 30% (10 seconds) longer than a single read(), but now the process can run happily in the background without slowing other processes. Filtering the huge file down to a manageable size and searchable format also make subsequent use of the data _much_ faster.

---

Working examples

Here's working sample code of a generic load_table(), which I called map_gtfs_table_to_dict()

Here's working sample code of a load_stop_times(), which I called map_stop_times_to_dict()


Let's use the generic sample code to load trips.txt from a CTA GTFS file:

>>> import map_gtfs_table_to_dict as load_table
>>> import zipfile
>>>
>>> # You must provide your own GTFS file, of course!
>>> gtfs_path = "/home/ian/gtfs/20140611.cta.gtfs"
>>> gtfs_file = zipfile.ZipFile(gtfs_path, mode='r')
>>> all_trips = load_table(gtfs_file, 'trips.txt')
>>>
>>> len(all_trips)
90428
>>>
>>> for trip_id in list(all_trips)[0:5]:
...      print(trip_id, all_trips[trip_id])
... 
45072053995 {'route_id': 'Y', 'direction_id': '0', 'wheelchair_accessible': '1',
'service_id': '104509', 'direction': '0', 'shape_id': '304500033',
'block_id': '45007747580', 'schd_trip_id': 'R501'}
437076941189 {'route_id': '147', 'direction_id': '1', 'wheelchair_accessible': '1',
'service_id': '43701', 'direction': 'North', 'shape_id': '4374519',
'block_id': '437008159276', 'schd_trip_id': '76941189'}
437076941185 {'route_id': '97', 'direction_id': '1', 'wheelchair_accessible': '1',
'service_id': '43701', 'direction': 'East', 'shape_id': '4374369',
'block_id': '437008158984', 'schd_trip_id': '76941185'}
437076941184 {'route_id': '97', 'direction_id': '1', 'wheelchair_accessible': '1',
'service_id': '43701', 'direction': 'East', 'shape_id': '4374369',
'block_id': '437008159008', 'schd_trip_id': '76941184'}
437076941186 {'route_id': '97', 'direction_id': '1', 'wheelchair_accessible': '1',
'service_id': '43701', 'direction': 'East', 'shape_id': '4374369',
'block_id': '437008158986', 'schd_trip_id': '76941186'}
>>> 

There we are, all 90,000 trips in a few seconds. Each line of the GTFS table broken into a dict for easy access.



Now, let's try using the same generic load function to load stop_times.txt

>>> import map_gtfs_table_to_dict as load_table
>>> import zipfile
>>>
>>> # You must provide your own GTFS file, of course!
>>> gtfs_path = "/home/ian/gtfs/20140611.cta.gtfs"
>>> gtfs_file = zipfile.ZipFile(gtfs_path, mode='r')
>>> all_stops = load_table(gtfs_file, 'stop_times.txt')
Traceback (most recent call last):
  File "", line 1, in 
  File "./test.py", line 56, in map_gtfs_table_to_dict
    line_dict[column] = line.split(',')[columns[column]].strip('" ')
MemoryError
>>> 

It took a few minutes to consume all that memory and fail, and everything else on the system slowed to a crawl.

Finally, let's use the custom load_stop_table() to load-and-filter stop_times.txt without reducing the rest of the system to a memory-starved crawl:

>>> import map_stop_times_to_dict as load_table
>>> import zipfile
>>>
>>> gtfs_path = "/home/ian/gtfs/20140611.cta.gtfs"
>>> gtfs_file = zipfile.ZipFile(gtfs_path, mode='r')
>>> stop_list = ['3661', '3698', '15026', '17433']
>>> filtered_stops = load_table(gtfs_file, stop_list)
4379
>>> for line in list(filtered_stops)[0:5]:
...     print(line, filtered_stops[line])
... 
4521987 {'stop_headsign': '79th Red Line', 'departure_time': '13:53:57',
'shape_dist_traveled': '22219', 'arrival_time': '13:53:57',
'pickup_type': '0', 'stop_id': '3661', 'stop_sequence': '35',
'trip_id': '437077075442'}
4538374 {'stop_headsign': '79th Red Line', 'departure_time': '15:04:01',
'shape_dist_traveled': '37847', 'arrival_time': '15:04:01',
'pickup_type': '0', 'stop_id': '3680', 'stop_sequence': '58',
'trip_id': '437077077228'}
4325384 {'stop_headsign': '127th/Lowe', 'departure_time': '17:26:00',
'shape_dist_traveled': '27644', 'arrival_time': '17:26:00',
'pickup_type': '0', 'stop_id': '5992', 'stop_sequence': '42',
'trip_id': '437077000628'}
475149 {'stop_headsign': '95th Red Line', 'departure_time': '18:01:18',
'shape_dist_traveled': '14028', 'arrival_time': '18:01:18',
'pickup_type': '0', 'stop_id': '15026', 'stop_sequence': '20',
'trip_id': '436073915060'}
4112403 {'stop_headsign': '95th Red Line', 'departure_time': '22:08:53',
'shape_dist_traveled': '29370', 'arrival_time': '22:08:53',
'pickup_type': '0', 'stop_id': '15026', 'stop_sequence': '42',
'trip_id': '437076995010'}
>>> 

It's still slow to load the entire file. On my equipment, I can load stop_times.txt in about 30 seconds this way. Varying the block size has some benefit, but you can play with that yourself.

In the final example, the 'line' variable -- the key to that subdict of one trip serving a stop -- is just a counter, the (approximate) CSV row number. stop_times.txt has no unique item on each line that can be used as a dict key.

tkinter clock

I did some playing around with tkinter after seeing this post.

Here is my solution: How to easily display a running clock (text, not graphical) using tkinter.

---

The Clock Class

A running clock requires two functions: One to create the tkinter widget, and one to continuously update the widget with the current time.

If we house these in a class, then creating a clock (or multiple clocks) gets much easier. All the complexity is hidden from the mainloop and the rest of your script.

A newly created Clock widget will *automatically* update itself. Both functions call tick(). Yeah, tick() calls itself upon completion. Cool little trick tkinter offers for precisely uses like this.

Here's an example class:

import tkinter
import time

class Clock():
    """ Class that contains the clock widget and clock refresh """

    def __init__(self, parent):
        """
        Create the clock widget
        It's an ordinary Label element
        """
        self.time = time.strftime('%H:%M:%S')
        self.widget = tkinter.Label(parent, text=self.time)
        self.widget.after(200, self.tick)      # Wait 200 ms, then run tick()


    def tick(self):
        """ Update the display clock """
        new_time = time.strftime('%H:%M:%S')
        if new_time != self.time:
            self.time = new_time
            self.widget.config(text=self.time)
        self.widget.after(200, self.tick)      # 200 =  millisecond delay
                                               #        before running tick() again 

And you implement it rather like this:

if __name__ == "__main__":
    """
    Create a tkinter window and populate it with elements
    One of those elements merely happens to include the clock.
    """

    # Generic window elements

    window = tkinter.Tk()
    frame  = tkinter.Frame(window, width=400, height=400 )
    frame.pack()

    # Add the frame elements, including the clock

    label1 = tkinter.Label(frame, text="Ordinary label")
    label1.pack()
    clock2  = Clock(frame)             # Create the clock widget
    clock2.widget.pack()               # Add the clock widget to frame
    label3 = tkinter.Label(frame, text="Ordinary label")
    label3.pack()

    window.mainloop()

We created and placed the clock2 widget almost like any other widget. We simply used the Clock class instead of the tkinter.Label class. There's no need to start() or stop() the clock widget - once created, it automatically updates itself.

We can place multiple clock widgets within the same frame without problems.

---

Advanced Placement

The Clock class works will all geometry managers, just like any widget:

    clock2.widget.pack()
    clock2.widget.grid(row=5,column=3)
    clock2.widget.place(x=150,y=200,anchor=center)

If we don't need to retain the 'clock' variable, or customize it's appearance, then we can simply place it like any other widget:

    tkinter.Label(frame, text="Ordinary label").pack()
    Clock(frame).widget.pack()

---

Customizing

One easy way to customize the clock's appearance is to use .configure
All Label configure options will work. The clock is simply a label.

    clock2 = Clock(frame)         # Create the clock widget
    clock2.widget.pack()          # Add the clock widget to frame
    clock2.widget.configure(bg='green',fg='blue',font=("helvetica",35))

---

Simplifying

There is one more simplification we can make, detailed as part of this tkinter example. We can change the class to inherit Label properties instead of creating a .widget attribute.If you're new to Python classes, the concept of inheriting from a class may take a few tries to wrap your brain around.

Attribute:

class Clock():
    def __init__(self, parent):
        self.widget = tkinter.Label(parent, text=self.time)

if __name__ == "__main__":
    foo()
    clock2.widget.do_something()
    more_foo()

Inherited:

class Clock3(tkinter.Label):
    def __init__(self, parent=None):
        tkinter.Label.__init__(self, parent)
        self.configure(text=self.time)

if __name__ == "__main__":
    foo()
    clock2.widget.do_something()
    more_foo()

---

Final result:

The final clock class looks like:

class Clock3(tkinter.Label):
    """ Class that contains the clock widget and clock refresh """

    def __init__(self, parent=None):
        tkinter.Label.__init__(self, parent)
        """
        Create and place the clock widget into the parent element
        It's an ordinary Label element.
        """
        self.time = time.strftime('%H:%M:%S')
        self.configure(text=self.time, bg='yellow')
        self.after(200, self.tick)


    def tick(self):
        """ Update the display clock every 200 milliseconds """
        new_time = time.strftime('%H:%M:%S')
        if new_time != self.time:
            self.time = new_time
            self.config(text=self.time)
        self.after(200, self.tick)

And you implement it rather like this:

if __name__ == "__main__":
    """
    Create a tkinter window and populate it with elements
    One of those elements merely happens to include the clock.
    """

    # Generic window elements

    window = tkinter.Tk()
    frame  = tkinter.Frame(window, width=400, height=400 )
    frame.pack()

    # Add the frame elements, including the clock

    label1 = tkinter.Label(frame, text="Ordinary label")
    label1.pack()
    clock2 = Clock(frame)             # Create the clock widget
    clock2.configure(bg='yellow')     # Customize the widget
    clock2.pack()                     # Add the clock widget to frame
    label3 = tkinter.Label(frame, text="Ordinary label")
    label3.pack()

    window.mainloop()

Importing GTFS files into SQLite

A great comment in my previous post about demystifying GTFS transit schedule data pointed out that the various files in a GTFS file are simply database tables. Each file can be imported into a relational database as a separate table, and queried using SQL instead of the custom scripts I used.

In fact, I found SQL to be faster and easier to maintain than the Python script...for a while. Eventually I rewrote my Python code, added a preprocessor, and found just as fast and easier to maintain than mixed Python/SQLite.
Nevertheless, thanks to Stefan, for the tip!

Here's a little more detail about exactly how to do it.

We will use the very simple, fast application SQLite for this, since our tables and queries will be rather simple and straightforward. Other possible databases include MongoDB and CouchDB. Indeed, for the very simple queries we used before, a series of good-old gdbm key-value databases could work.

---

Setup and importing GTFS tables into SQLite

In Ubuntu, installing SQLite3 is very simple:

sudo apt-get install sqlite3

Next, let's manually download the GTFS file for the Milwaukee County Transit System, uncompress it, create a new database, add a table to the database for the stops file, import stops file into the database, and save the database.

$ mkdir /home/me/GTFS                               # Create a working directory 
$ wget -O /home/me/GTFS/mcts.gtfs http://kamino.mcts.org/gtfs/google_transit.zip
                                                    # Download the GTFS file
$ unzip -d /home/me/GTFS /home/me/GTFS/mcts.gtfs    # Unzip the GTFS file
$ sqlite3

sqlite> attach /home/me/GTFS/mcts.db as mcts        # Create a new database
sqlite> create table stops(stop_id TEXT,stop_code TEXT,stop_name TEXT,

                           stop_desc TEXT,stop_lat REAL,stop_lon REAL,

                           zone_id NUMERIC,stop_url TEXT,timepoint NUMERIC);
sqlite> .separator ","                              # Tell SQLite that it's a CSV file
sqlite> .import /home/me/GTFS/stops.txt stops       # Import the file into a db table
sqlite> .dump                                       # Test the import
sqlite> delete from main.stops where stop_id like 'stop_id';  # Delete the header line
sqlite> select * from mcts.stops where stop_id == 5505;       # Test the import
sqlite> .backup mcts /home/me/GTFS/mcts.db          # Save the database
sqlite> .quit

Scripting imports

We can also script it. Here's a more robust script that creates multiple tables. The column names are explained on the first line of each table (which is why we must delete that line). The data types -TEXT, REAL, and NUMERIC-, and conversions from various Java, Python, C, and other datatypes are clearly explained in the SQLite documentation. The explanation of the field names, and expected datatype, is explained in the GTFS documentation. Each provider's GTFS file can include many optional fields, and may use different optional fields over time, so you are *likely* to need to tweak this script a bit to get it to work:

create table agency(agency_id TEXT,agency_name TEXT,agency_url TEXT,
                    agency_timezone TEXT,agency_lang TEXT, agency_phone TEXT);
create table calendar_dates(service_id TEXT,date NUMERIC,exception_type NUMERIC);
create table routes(route_id TEXT,agency_id TEXT,route_short_name TEXT,
                    route_long_name TEXT,route_desc TEXT,route_type NUMERIC,
                    route_url TEXT,route_color TEXT,route_text_color TEXT);
create table shapes(shape_id TEXT,shape_pt_lat REAL,shape_pt_lon REAL,
                    shape_pt_sequence NUMERIC);
create table stops(stop_id TEXT,stop_code TEXT,stop_name TEXT,
                   stop_desc TEXT,stop_lat REAL,stop_lon REAL,
                   zone_id NUMERIC,stop_url TEXT,timepoint NUMERIC);
create table stop_times(trip_id TEXT,arrival_time TEXT,departure_time TEXT,
                        stop_id TEXT,stop_sequence NUMERIC,stop_headsign TEXT,
                        pickup_type NUMERIC,drop_off_type NUMERIC);
create table trips(route_id TEXT,service_id TEXT,trip_id TEXT,
                   trip_headsign TEXT,direction_id NUMERIC,
                   block_id TEXT,shape_id TEXT);
.separator ','
.import /home/me/GTFS/agency.txt agency
.import /home/me/GTFS/calendar_dates.txt calendar_dates
.import /home/me/GTFS/routes.txt routes
.import /home/me/GTFS/shapes.txt shapes
.import /home/me/GTFS/stops.txt stops
.import /home/me/GTFS/stop_times.txt stop_times
.import /home/me/GTFS/trips.txt trips
delete from agency where agency_id like 'agency_id';
delete from calendar_dates where service_id like 'service_id';
delete from routes where route_id like 'route_id';
delete from shapes where shape_id like 'shape_id';
delete from stops where stop_id like 'stop_id';
delete from stop_times where trip_id like 'trip_id';
delete from trips where route_id like 'route_id';
select * from stops where stop_id == 5505;

And run that script using:

$ sqlite3 mcts.db < mcts_creator_script

---

Reading GTFS data from SQLite

Now, like in the previous GTFS post, let's find the next buses at the intersection of Howell and Oklahoma. There are four stops at that location: 658, 709, 5068, and 5152.


First, let's find the appropriate service codes for today's date:

# Query: The list of all service_ids for one date.
sqlite> SELECT service_id FROM calendar_dates WHERE date == 20140310;
14-MAR_CY-AON_0
[...long list...]
14-MAR_WN-PON_0

Scripting reads

These queries can also be scripted. Here's an example script that looks up the four stops we care about for a two-hour window:

-- Usage:  $ sqlite3 GTFS/mcts.db < GTFS/mcts_lookup.sh
-- Usage:  sqlite> .read GTFS/mcts_lookup.sh 

-- List of the valid service_id codes for the current date
CREATE VIEW valid_service_ids AS
   SELECT service_id 
   FROM calendar_dates 
   WHERE date == strftime('%Y%m%d', 'now', 'localtime')
   ;

SELECT stop_times.arrival_time, trips.route_id, trips.trip_headsign
   FROM trips, stop_times

   -- Match the trip_id field between the two tables
   WHERE stop_times.trip_id == trips.trip_id

   -- Limit selection to the stops we care about 
   AND stop_times.stop_id IN (658,709,5068,5152)

   -- Limit selection to service_ids for the correct day
   AND trips.service_id IN valid_service_ids

   -- Limit selection to the next hour from now
   AND stop_times.arrival_time > strftime(
                                 '%H:%M:%S', 'now', 'localtime', '-5 minutes')
   AND stop_times.arrival_time < strftime(
                                 '%H:%M:%S', 'now', 'localtime', '+1 hour')
   ORDER BY stop_times.arrival_time
   ;

-- Clean Up
DROP VIEW valid_service_ids;

And there are two ways to run the script:

sqlite&gt> .read lookup_script         # Within sqlite
$ sqlite3 mcts.db < lookup_script      # Shell script

---

Results

I found that importing GTFS files into SQLite requires a lot of memory and CPU...but is faster than Python, and the scripts are smaller and easier to maintain. SQLite is a good, fast processor or pre-processor.

File sizes:

• mcts.gtfs: 5.6M
• mcts.db: 79M
• mtcs.gtfs unzipped files: 86M

I think that preprocessing or shrinking those files will be important for low-power or low-bandwidth applications.

Query times:

Here are the query times for buses within a two-hour window at Howell and Oklahoma:

• Original Python: 13 sec
• SQLite: 2.5 sec
• Rewritten Python w/Preprocessor: 0.2 sec

Code size:

To do those queries,
The original Python 3 script: 206 lines.
SQLite script to import the GTFS file: 33 lines.
SQLite script to lookup the stops: 32 lines.

Amtrak real-time train data

Recently the US passenger rail operator Amtrak announced a nationwide real-time map of it's services.

That's great...until you try it.

The United States is big, even though the number of trains is fairly small. The number of Google map tiles is big. The polygons that show the routes are enormous. The nationwide train data is big.

That's a lot of data to load --and so slow to display-- if all I really want is realtime data on a single train (is my train late?).
 
After looking at the GETs emitted by all the (slow, unnecessary) javascript on that site, I figured out how to bypass it and pull just the table of current train status. It's 400kb, but it's a lot less than the over 1MB to go through the website and map....

https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-08584582962951999356/
features?version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&maxResults=250&
callback=jQuery19106722136430546803_1388112103417&dataType=jsonp&
jsonpCallback=retrieveTrainsData&contentType=application%2Fjson&_=1388112103419

The only fields required are 'version' and 'key' (which is different from the site cookie):

https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-08584582962951999356/
features?version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4

So we can easily download the current status table using wget:

$ wget -O train_status https://www.googleapis.com/mapsengine/v1/tables/
01382379791355219452-08584582962951999356/features?version=published\&
key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4\&jsonpCallback=retrieveTrainsData

Since parsing 400kb of JSON using shell script will be awkward and annoying, let's change over to Python3:

#/usr/bin/python3
import httplib2
url   = \
"""https://www.googleapis.com/mapsengine/v1/table
/01382379791355219452-08584582962951999356/features?version=published&
key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&jsonpCallback=retrieveTrainsData"""

h = httplib2.Http(None)
resp, content = h.request(url, "GET")

Let's tell Python that the content is a JSON string, and iterate through the list of trains looking for train number 7. That's a handy train because each run takes over 24 hours - it will always have a status.

import json
all_trains = json.loads(content.decode("UTF-8"))
for train in all_trains["features"]:
   if train["properties"]["TrainNum"] == "7":
       do something

After locating the train we care about, here is data about it:

latitude     = train["geometry"]["coordinates"][0]
longitude    = train["geometry"]["coordinates"][1]
speed        = train["properties"]["Velocity"]
report_time  = train["properties"]["LastValTS"]
next_station = train["properties"]["EventCode"]

Let's find a station (FAR) along the route. This is a little tougher, because Python's JSON module reads each Station as a string instead of a dict. We need to identify Station lines, and feed those through the JSON parser (again).

for prop in train["properties"].keys():
          if "Station" in prop:
              sta = json.loads(train["properties"][prop])
              if sta["code"] == "FAR":
                  # Schedule data
                  station_time_zone        = sta['tz']
                  scheduled_arrival_time   = sta['scharr'] # Only if a long stop
                  scheduled_departure_time = sta['schdep'] # All stations

                  # Past stations
                  actual_arrival_time      = sta['postarr']
                  actual_departure_time    = sta['postdep']
                  past_station_status      = sta['postcmnt']

                  # Future stations
                  estimated_arrival_time   = sta['estarr']
                  future_station_status    = sta['estarrcmnt']

The final Python script for Train 7 at FAR is:

#/usr/bin/python3

import httplib2
import json

url = "https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-08584582962951999356/features?version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&jsonpCallback=retrieveTrainsData"
h = httplib2.Http(None)
resp, content = h.request(url, "GET")

all_trains = json.loads(content.decode("UTF-8"))
for train in all_trains["features"]:
   if train["properties"]["TrainNum"] == "7":
      for prop in train["properties"].keys():
          if "Station" in prop:
              sta = json.loads(train["properties"][prop])
              if sta["code"] == "FAR":
                  print('Train number {}'.format(train["properties"]["TrainNum"]))
                  if 'schdep' in sta.keys():
                      print('Scheduled: {}'.format(sta['schdep']))
                  if 'postcmnt' in sta.keys():
                      print("Status: {}".format(sta['postcmnt']))
                  if 'estarrcmnt' in sta.keys():
                      print("Status: {}".format(sta['estarrcmnt']))

And the result looks rather like:

$ python3 Current\ train\ status.py 

Train number 7
Scheduled: 12/26/2013 03:35:00
Status: 1 HR 4 MI LATE

Train number 7
Scheduled: 12/27/2013 03:35:00

---

Two more items.

First, the Python script can be tweaked to show *all* trains for a station. Just remove the train number filter. To be useful, you might add some time comprehension, since some scheduled times can be a day in the future or past.

Second, the station information table has a similar static page: https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-17620014524089761219/features?&version=published&maxResults=1000&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&callback=jQuery19106722136430546803_1388112103417&dataType=jsonp&jsonpCallback=retrieveStationData&contentType=application%2Fjson&_=13881121034

As with the train status JSON application, many of the fields are dummies. This shorter URL works, too: https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-17620014524089761219/features?&version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&callback=jQuery19106722136430546803_1388112103417

Python http.server and upstart-socket-bridge

In a previous post, I showed how to make the Python 3 socketserver.TCPServer class compatible with upstart-socket-bridge by overriding the server_bind() and server_activate() methods.

Python's http.server module builds on socketserver. Let's see if we can similarly make http.server compatible with upstart-socket-bridge.

About http.server

The http.server module is intended to be a simple way to create webservers. Most of the module is devoted to classes that handle incoming and outgoing data. Only one class, HTTPServer, handles the networking stuff.

---

Example 1

Here is the first example http.server.SimpleHTTPRequestHandler in the Python documentation:

import http.server
import socketserver

PORT = 8000

Handler = http.server.SimpleHTTPRequestHandler

httpd = socketserver.TCPServer(("", PORT), Handler)

print("serving at port", PORT)
httpd.serve_forever()

When you run this code, and point a web browser to port 8000, the code serves the current working directory, with links to files and subdirectories.

Convert Example 1 to work with upstart-socket-bridge

The example server is only seven lines.

Line 3, the PORT, is no longer needed. Upstart will define the port.

Line 5, the socketserver.TCPServer line, will be the biggest change. We need to define a new class based on TCPServer, and override two methods. The is exactly what we did in the previous post.

Line 6, the print statement, can be deleted. When the job is started by Upstart, there is no display - nowhere to print to. An important side effect of starting the job using Upstart is that the Present Working Directory is / (root), unless you specify otherwise in the /etc/init config file that starts the job.

Because of the change to Line 5, the final result is 6 more lines...not bad, and now it works with upstart-socket-bridge.

import http.server
import socketserver

class MyServer(socketserver.TCPServer):
    def server_bind(self):
        """ Replace the socket FD with the Upstart-provided FD"""
        fd = int(os.environ["UPSTART_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

    def server_activate(self):
        pass            # Upstart takes care of listen()

Handler = http.server.SimpleHTTPRequestHandler
server = MyServer(None, Handler)
server.serve_forever()

As always, the script is triggered by an appropriate Upstart job.

Test the script, in this case, by pointing a web browser at the port specified in the Upstart job.

---

Example 2

This example is also from the Python http.server documentation:

def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
    server_address = ('', 8000)
    httpd = server_class(server_address, handler_class)
    httpd.serve_forever()

It's not a standalone example, but instead an abstract example of how to use the module in a larger application.

Let's reformat it into a working example:

import http.server
server_address = ('', 8000)
handler_class=http.server.BaseHTTPRequestHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Now the server runs...sort of. It gives us a 501 error message. Let's add a class to the handler that reads the path and gives a real, valid response. (Source)

import http.server

class MyHandler(http.server.BaseHTTPRequestHandler):
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["<html><head><title>The Title</title></head>",
                   "<body>This is a test.<br />",
                   "You accessed path: ", self.path, "<br />",
                   "</body></html>"]
        self.wfile.write("".join(content).encode("UTF-8"))


server_address = ('', 8000)
handler_class=MyHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Okay, now this is a good working example of a working http.server. It really shows how http.server hides all the complexity of networking on one simple .server line and focuses all your effort on content in the handler. It's clear how you would parse the URL input using self.path. It's clear how you can create and send content using self.wfile.

Convert Example #2 to work with upstart-socket-bridge

There are two classes at work: http.server.BaseHTTPRequestHandler handles content doesn't care about the networking. http.server.HTTPServer handles netwokring and doesn't care about content.

Good news: HTTPServer is based on socketserver.TCPServer, which we already know how to patch!

The changes I made:

1) I want it to launch at each connection, exchange data once, and then terminate. Each connection will launch a separate instance. We no longer want the service to serve_forever().

httpd.serve_forever()    # Old
httpd.handle_request()   # New

2) Let's make it compatible with all three inits: sysvinit (deamon), Upstart, and systemd.

import os

if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \   # Upstart
    or "LISTEN_FDS" in os.environ.keys():     # systemd
        httpd = MyServer(None, MyHandler)     # Need a custom Server class
        httpd.handle_request()                # Run once
    else:                                     # sysvinit
        server_address = ('', 8000)
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()                 # Run forever

3) Add a custom handler that overrides server_bind() and server_activate() in both the socketserver and http.server.HTTPServer modules. This is the secret sauce that makes Upstart compatibility work:

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    def server_bind(self):
        # Get the File Descriptor from an Upstart-created environment variable
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])      # Upstart
        else:
            fd = int(os.environ["LISTEN_FDS"])       # Systemd
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # Only http.server.CGIHTTPRequestHandler uses these
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port

4) Add the Upstart job to monitor the port:

# /etc/init/socket-test.py
description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=8000 ADDR=127.0.0.1
setuid your_username               # Does not need to run as root
exec /usr/bin/python3 /tmp/socket-server.py

And the final product looks like:

#!/usr/bin/python3

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    """
    This class overrides two methods in the socketserver module:
        socketserver __init__ uses both server_bind() and server_activate()
    This class overrides one method in the http.server.HTTPServer class:
        HTTPServer uses both socketserver __init__ and it's own custom
        server_bind
    These overrides makes it compatible with Upstart and systemd socket-bridges
    Warning: It won't bind() or listen() to a socket anymore
    """
    def server_bind(self):
        """
        Get the File Descriptor from an Upstart-created environment variable
        instead of binding or listening to a socket.
        """
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])
        else:
            fd = int(os.environ["LISTEN_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # From http.server:
        # http.server.CGIHTTPRequestHandler uses these.
        # Other handler classes don't use these.
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port


    def server_activate(self):
        """
        This socketserver method sends listen(), so it needs to be overridden
        """
        pass


class MyHandler(http.server.BaseHTTPRequestHandler):
    """
    A very simple custom handler.
    It merely reads the URL and responds with the path.
    This shows how you read data from a GET, and send a response. 
    """
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["",
                   "This is a test.
",
                   "You accessed path: ", self.path, "
",
                   ""]
        self.wfile.write("".join(content).encode("UTF-8"))


if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \     # Upstart
    or "LISTEN_FDS" in os.environ.keys():       # systemd
        httpd = MyServer(None, MyHandler)       # Use fd to get connection
        httpd.handle_request()                  # Handle once, then terminate
    else:
        server_address = ('', 8000)             # sysvinit, classic bind()
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()

Test the service:

• The Python 3 script and Upstart job both use Port 8000.
• Save the Python 3 script. Make it executable.
• Save the Upstart job. Change the Upstart job to Port 8001. Make sure it points to the Python 3 script.

Webserver daemon using sysvinit - run forever:

• Run the Python 3 script.
• Start a web browser, and point it to http://localhost:8000/test/string
• The browser should show a response
• Kill the Python 3 script

Web service using upstart - run once:

• Don't start the Python 3 script. Upstart will do it for you.
• Start a web browser, and point it to http://localhost:8001/test/string
• The browser should show a response.

Python socketserver and upstart-socket-bridge

In a previous post, I described how to use Upstart to monitor a port, trigger a Python script upon connection, and then let the Python application send and receive data over that port.

In a server application, Upstart can take over the bind() and listen() elements of the server daemon, sometimes eliminating the need for a daemon at all.

Here is an example of a Python socketserver, and how it changes when using Upstart.

Some complexity is reduced: The handler is simpler, it no longer needs to fork() or serve_forever(). Each connection triggers the application, so the application need only handle a single read()/write() cycle, then terminate.

Some complexity is increased: We need to override two methods of a socketserver class to make it upstart-socket-bridge compatible.

---

Here is the original socketserver.TCPServer serve_forever example, from the Python documentation:

import socketserver

class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print("{} wrote:".format(self.client_address[0]))
        print(self.data)
        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    HOST, PORT = "localhost", 9999

    # Create the server, binding to localhost on port 9999
    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

    # Activate the server; this will keep running until you
    # interrupt the program with Ctrl-C
    server.serve_forever()

It's easy to test. Let's save the file as /tmp/socket-server-original.py

$ python3 /tmp/socket-server-original.py     # On Terminal #1

$ echo "Test String" | nc localhost 9999     # On Terminal #2
TEST STRING

The example works.

---

Now, let's adapt it for upstart-socket-bridge.

It takes a bit of reading through the socketserver source code to figure it out:

TCPServer's __init__ calls good old bind() and listen() using it's own methods server_bind() and server_activate(), respectively.

Both of those methods can be overridden. Better yet, they use socket module calls (socket.bind() and socket.listen()) that we already know how to replace with socket.fromfd().

So we can get TCPServer to work by overriding the two methods like this:

import os, socket

class MyServer(socketserver.TCPServer):
    """
    This class overrides two method in socketserver.TCPServer 
    and makes it Upstart-compatible
    """
    def server_bind(self):
        """ Replace the socket-created file descriptor (fd) with the Upstart-provided fd"""
        fd = int(os.environ["UPSTART_FDS"]
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

def server_activate(self): """This means listen(), but we don't want it do do anything""" pass


Changing the TCPServer class slightly changes the way way we activate.
We must call the new class instead of TCPServer, and we no longer need the address since bind() no longer does anything:

    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)  # OLD
    server = MyServer(None, MyTCPHandler)                        # NEW

The code that handles the request, generates the response, and sends the response has two changes. Printing on the terminal won't work, since Upstart doesn't use a terminal to start the process. So we'll delete the printing lines:

        print("{} wrote:".format(self.client_address[0]))
        print(self.data)

Finally, one more change. The script can quit after the response has been sent:

    server.serve_forever()      # OLD
    server.handle_request()     # NEW

---

When we put it all together, here is the new version that works with upstart-socket-bridge:

import socketserver, os, socket

class MyServer(socketserver.TCPServer):
    """
    This class overrides two method in socketserver.TCPServer 
    and makes it Upstart-compatible
    """
    def server_bind(self):
        """ Replace the socket FD with the Upstart-provided FD"""
        fd = int(os.environ["UPSTART_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

    def server_activate(self):
        pass            # Upstart takes care of listen()


class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()

        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    # Create the server using the file descriptor from Upstart
    server = MyServer(None, MyTCPHandler)

    # Run once, then terminate
    server.handle_request()

Let's save the file as /tmp/socket-server-new.py
We'll need to adapt (or create) /etc/init/socket-test.conf Upstart job file.

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1
setuid myusername   # Use *your* user name. Doesn't need to be root
exec /usr/bin/python3 /tmp/socket-server-new.py

And let's give it a try on the terminals:

$ python3 /tmp/socket-server-new.py           # On Terminal #1

$ echo "Test String" | nc localhost 34567     # On Terminal #2
TEST STRING

Overriding socketserver for upstart compatibility works.

---

After this test, cleanup is easy

$ sudo rm /etc/init/socket-test.conf
$ rm /tmp/socket-server-original.py /tmp/socket-server-new.py

Handy Pipe and Socket Reference

I'm not going to explain sockets. There are too many good tutorials out there already.
Instead, I'm going to use sockets in a bunch of examples that build.

Ordinary Pipes
Named Pipes/Sockets
Unix Domain Sockets
TCP Sockets/Ports
upstart-socket-bridge

The Simplest

Pipes (|) and redirection (>). We already know how to do that in the shell.
These pipes connect related processes. For example, the children of a single terminal are related.

It's hard to pipe from shell to python (or the reverse) - it's possible, just not easy the first couple times you try.

Shell example:

$ echo Hello, Planet\! | sed 's/Planet/World/'
Hello, World!

Here is a good example of pipes in Python 3.

---

Named Pipes

Slightly more complicated (reference). These use a pipe file that synchronizes the data transfer when both processes are ready. A -> Pipe -> B. This is handy for inter-process communication that does not need to run through dbus, and saves a bit of overhead.

Names pipes are a shared file. They can have many (anonymous) writers, but only a single reader. Shell, Python, and C senders and listeners can interoperate (fairly) smoothly.

Named pipes are blocking. Start the reader first, and it will wait for the writer. Alternately, start the writer first, and it will wait for the reader. Once the writer has finished, the reader will also stop. One read() for one write(). A second read() requires a second write()...and a second write() requires a second read().

Named pipes in shell and C can be reused. Named pipes in Python cannot be reused after close() - they must be unlink() (destroyed) and remade afresh.

Conventionally, the listener is expected to create the pipe. But it's not required.

Processes that use named pipes work in the following order:

• Create the pipe (usually done by listener)
• Open the pipe (both sides)
• Write the pipe
• Writer close the pipe (allows the reader to see the data)
• Listener read the pipe
• Listener close the pipe
• Delete the pipe (usually done by the listener)

Creating and Deleting a Named Pipe:

Shell:

#!/bin/sh
mkfifo /tmp/testpipe        # Create pipe
rm /tmp/testpipe            # Delete pipe

Python3:

>>> import os
>>> os.mkfifo("/tmp/testpipe")    # Create pipe
>>> os.remove("/tmp/testpipe")    # Delete pipe 

C:

# include <sys/stat.h>
int main( ){
   int result = mkfifo ("/tmp/testpipe", S_IRUSR| S_IWUSR);  \\ Create Pipe
   unlink ("/tmp/testpipe");                                 \\ Delete Pipe
   return 0;
   }

Named Pipe Listeners:

 

Shell:

#!/bin/sh
pipe=/tmp/testpipe
trap "rm -f $pipe" EXIT
while true
do
    read line <$pipe   # Read the pipe
    echo $line
done

Python3:

>>> import os
>>> pipe = open("/tmp/testpipe", "r")
>>> pipe.read()     # You won't read anything until the sender close()
'Hello, World!'     # Data from pipe
>>> pipe.close()

C:

#include <fcntl.h>
#include <stdio.h>
int main()
{
    char buf[20];
    int fd = open("/tmp/testpipe", O_RDONLY);  // Open the pipe
    read(fd, buf, 20);                         // Read the message - unblock the writing process
    printf("Received: %s\n", buf);             // Display the message
    close(fd);                                 // Close the pipe
    return 0;
}

Named Pipe Senders:

Shell:

#!/bin/sh
pipe=/tmp/testpipe
[ ! -p $pipe ] && echo "Reader not running" && exit 1
[ "$1" ] && echo "$1" >$pipe || echo "Hello from $$" >$pipe

Python3:

>>> import os
>>> pipe = open("/tmp/testpipe", "w") # Hang here awaiting the receiver to read() once
>>> pipe.write("Hello, World!")
13
>>> pipe.close()    # Receiver read() is empty until this happens

C:

#include <fcntl.h>
int main()
{
    int fd = open("/tmp/testpipe", O_WRONLY);   // Open the pipe
    write(fd, "Hi", sizeof("Hi"));              // Write "Hi" to pipe
    close(fd);                                  // Close the pipe - allow the read
    return 0;
}

---

Unix Domain Sockets

Unix Domain Sockets is a fancy name for a fancy pipe. Like shell pipes, Unix Sockets use a pipe file...Well it *looks* like a pipe file. Actually, data is buffered instead of written to a file. The file is simply an easy way for the processes to find each other.

Unlike shell pipes, Unix sockets are bi-directional, and multiple writers are not anonymous. In Python, sockets are single-use: After a close() they cannot be reopened, and must be deleted and remade. Pipes can only be read() when the sender close(), but sockets can be read() for every send().

Since sockets are bi-directional, it's not a sender/receiver relationship anymore. It's a client/server relationship.

Processes using a Unix Domain Socket usually work in the following order:

• Create a file descriptor (fd) that defines the socket OR look up the fd of an existing socket.
• Server binds to the socket
• Server listens on the socket
• Client connects to the socket
• Server accepts the connection
• Both sides read/write
• Client closes socket (on Python, cannot be reopened - must be reconnected )
• Server closes socket (usually during service shutdown)
• Delete/unlink the file descriptor and socket (if it won't be reused)

Creating and Deleting a Unix Domain Socket:

Shell:

Shell creation can be automatic, for example using netcat:

$ nc -lU /tmp/sock           # Create socket, among other actions
$ rm /tmp/sock               # Delete socket

Python3:

>>> import socket, os
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)  
>>> tempsocket.bind("/tmp/sock")  # Create socket
>>> os.remove("/tmp/sock")        # Delete socket 

C:

#include <sys/socket.h>
#include <sys/un.h>
int main() {
  char *path = "/tmp/sock";
  struct sockaddr_un addr;
  char buf[100];
  int cl,rc;
  int fd = socket(AF_UNIX, SOCK_STREAM, 0);   // Create a file descriptor
  addr.sun_family = AF_UNIX;                  // It's a Unix socket
  strcpy(addr.sun_path, path);                // Set the path
  memset(&addr, 0, sizeof(addr));         // Fill the address array with zeroes (prevent garbage)
  bind(fd, (struct sockaddr*)&addr, sizeof(addr));    // Create socket (server)
  connect(fd, (struct sockaddr*)&addr, sizeof(addr)); // Connect (client)
  unlink(path);                               // Delete socket
  return 0;
}

Unix Domain Socket Servers (Bind to socket):

The server owns (binds) and monitors the socket, and is usually responsible for creating and deleting the socket. (C example)

Shell:

$ nc -lU /tmp/sock

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.bind("/tmp/sock")       # Listener is first process using socket
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept()   # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8')    # Listener blocks here awaiting send()
'Hello, World'                         # Try this a few times to capture each send()
>>> tempsocket.close()

C:

#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
int main() {
  char buf[100];
  int cl,rc;
  int fd = create_socket(socket_path)      // Creating the socket is not done here. See above
  listen(fd, 5);                // listen() is the step after bind()
  while (1) {
    cl = accept(fd, NULL, NULL);
    while ( (rc=read(cl,buf,sizeof(buf))) > 0) {
      printf("read %u bytes: %.*s\n", rc, rc, buf);
    }
    if (rc == 0) {
      printf("EOF\n");
      close(cl);
    }
  }
  return 0;
}

Unix Domain Socket Clients (Connect to socket):

Clients don't own (bind) the socket. Instead, they connect on-demand.

Shell:

$ nc -U /tmp/sock

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.connect("/tmp/sock")           # Listener must be active before sender
>>> tempsocket.send(data)                     # Try this a few times
13
>>> tempsocket.close()                        # End of stream

C:

#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
int main() {
  struct sockaddr_un addr;
  char buf[100];
  int rc;
  int fd = create_socket(socket path)  // Creating the socket is not done here

  // First step after connect()  
  while( (rc=read(STDIN_FILENO, buf, sizeof(buf))) > 0) write(fd, buf, rc);
  return 0;
}

Quick fun with netcat:

The easiest possible bi-directional socket.

$ nc -lU /tmp/sock   # Enter in Terminal #1
$ nc -U /tmp/sock    # Enter in Terminal #2

Type anything on either terminal.
Look! The two communicate!

Here is an example of using netcat to exchange files this way (from the man page)

$ nc -lU /tmp/sock > filename.out      # Terminal #1 (sender)
$ nc -U /tmp/sock < filename.in        # Terminal #2 (receiver)

---

TCP Sockets (Ports)

TCP ports are very much like Unix sockets. Instead of a file, the data stream is buffered. And Unix sockets are not networkable, but TCP ports are a foundation of the Internet. Ports use client/server, and the server owns (binds) the port that the client connects to. Sockets and Ports use the same shell application (netcat) and Python3 module (socket).

Since there is no file to create or delete, bind() and connect() simply use an IP address and port number.

Processes using a TCP port usually work in the following order (just like a Unix Domain Socket):

• Create a file descriptor (fd) that defines the port
• Server binds to the port
• Server listens on the port
• Client connects to the port
• Server accepts the connection
• Both sides read/write
• Client closes port
• Server closes port (usually during service shutdown)
• Delete/unlink the file descriptor and port (if it won't be reused)

TCP Port Services (Bind to port):

The server owns (binds) and monitors the port.

Shell:

$ nc -l 45678      # Port 45678

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.bind(("127.0.0.1", 45678))   # Tuple: (IP Addr, Port)
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept()   # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8')    # Listener blocks here awaiting send()
'Hello, World'                                  # Try it a few times to catch each send()
>>> tempsocket.close()

C:

#include <stdio.h>
#include <string.h>
#include <netinet/in.h>
int main()
{
     int sock_fd, conn_fd;   // Socket file descriptor and Connection file descriptor
     int port_num = 45678;
     socklen_t clilen;
     char buffer[256];
     struct sockaddr_in serv_addr, cli_addr;
     sockfd = socket(AF_INET, SOCK_STREAM, 0);        // Define socket type
     memset(&serv_addr, 0, sizeof(serv_addr));    // Set array to zeroes 
     serv_addr.sin_family = AF_INET;
     serv_addr.sin_addr.s_addr = INADDR_ANY;
     serv_addr.sin_port = htons(port_num);
     bind(sock_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr));   // Bind
     listen(sock_fd,5);                               // Listen
     clilen = sizeof(cli_addr);
     conn_fd = accept(sock_fd, 
                 (struct sockaddr *) &cli_addr, 
                 &clilen);                        // Accept
     memset(&buffer, 0, sizeof(buffer));          // Fill buffer with zeroes - prevent garbage
     read(conn_fd,buffer,255);                        // Read
     printf("Here is the message: %s\n",buffer);
     write(conn_fd,"I got your message",18);          // Write
     close(conn_fd);
     close(sock_fd);                                  // Close
     return 0; 
}

TCP Port Clients (Connect to port):

The client connects to the port on-demand.

Shell:

$ nc localhost 45678      # IP=loopback, Port #45678

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.connect(("127.0.0.1", 45678))   # Tuple: (IP Addr, Port)
>>> data = "Hello, World".encode("utf-8")
>>> tempsocket.send(data)          # Try this a few times
12
>>> tempsocket.close()

C:

#include <stdio.h>
#include <string.h>
#include <netdb.h>
int main(int argc, char *argv[])
{
    int sock_fd;
    int port_num = 45678;
    struct sockaddr_in serv_addr;
    struct hostent *server;
    char buffer[256];
    sock_fd = socket(AF_INET, SOCK_STREAM, 0);       // Define socket type
    server = gethostbyname(port_num);                // Lookup port owner
    memset(&serv_addr, 0, sizeof(serv_addr));    // Set array to zeroes
    serv_addr.sin_family = AF_INET;
    bcopy((char *)server->h_addr,
         (char *)&serv_addr.sin_addr.s_addr,
         server->h_length);
    serv_addr.sin_port = htons(port_num);
    connect(sock_fd,(struct sockaddr *) &serv_addr,sizeof(serv_addr)); // Connect
    printf("Please enter the message: ");
    memset(&buffer, 0, sizeof(buffer));
    fgets(buffer,255,stdin);                // Keyboard input
    write(sock_fd,buffer,strlen(buffer));   // Write to server
    memset(&buffer, 0, sizeof(buffer)); // Reuse buffer array - clear to prevent garbage
    read(sock_fd,buffer,255);               // Read response
    printf("%s\n",buffer);                  // Print response
    close(sock_fd);                         // Close
    return 0;
}

More fun with netcat

This is the basic way that webservers work.

$ echo "Hello, World" | nc -l 45678   # Enter in Terminal #1
$ nc localhost 45678                  # Enter in Terminal #2
                                      # OR open http://localhost:45678
                                      # in your favorite web browser

Pre-filling an Outlook Express mail in Windows XP

I want a Python3 script to open a pre-filled Outlook Express (OE) new-mail window. A human should read, edit, and approve the e-mail before sending. If I wanted a completely automatic e-mail, I would use Python's smtp module.

The problem is that OE has no COM server, and the Simple-MAPI support is in C++.

---

Command Line

There is an easy way from the Windows command line:

C:>"C:\Program Files\Outlook Express\msimn" /mailurl:mailto:"name%40example.com?cc=cc1%40example.com&bcc=bcc1@example.com&subject=The%20Subject%20Line&body=The%20first%20paragraph.%0A%0AThe%20second%20paragraph.%0A%0AThanks%2C%0AName

This launches OE with a single argument. That argument is a mailto: URL. The URL includes to:, cc:, bcc:, subject:, and the body.

• "C:\Program Files\Outlook Express\msimn" launches OE
• /mailurl:mailto:" defines the mailto: URL. The quote(") is required on the command line, though not in the python version.
• name%40example.com? is the To: line. The To: line ends with the question-mark (?)
• cc=cc1%40example.com& the rest of the fields are defined (cc=) and separated with the ampersand (&).
• Don't use spaces or commas or returns - just text.
  

  space	%20
  ,	%2C
  return	%0A
  @	%40

• I have not figured out a way to include more than one e-mail address on each line.
• Files cannot be attached using this method.

---

Python

Since OE does not have any COM or other programmatic way to do it, we will fall back on subprocess, and have Python simply create a mailto: URL and use the command line.

This simple Python3 script creates a pre-filled Outlook Express window using subprocess:

import subprocess

# The header and body
to      = "name1%40example.com"
cc      = "cc1%40example.com"     # or leave blank: cc = ""
bcc     = "bcc1%40example.com"    # or leave blank: bcc = ""
subject = "The%20Subject"
body    = "The%20first%20paragraph.%0A%0AThe%20second%20paragraph.%0A%0AThanks%2C%0AName"

oe = "C:\Program Files\Outlook Express\msimn"
mailto_url = ('/mailurl:mailto:' + to      + '?' +   # No lone quote (") after colon (:)
              'cc='              + cc      + '&' +
              'bcc='             + bcc     + '&' +
              'subject='         + subject + '&' +
              'body='            + body )            # No ending ampersand (&) 

subprocess.call([oe, mailto_url])

Six ways to make a notification pop-up

UPDATED MAY 2013 to add Go and GObject methods to the existing Command Line and Python methods.
Originally published Feb 27, 2009.

There are several ways to create notifications in Ubuntu. The notify-OSD daemon draws the boxes and creates the notifications, so all these ways simply contact it and tell it what to do.

1. On the command line, use the notify-send command. This command is part of the libnotify-bin package. It's easy:

   $ notify-send -i /usr/share/icons/Tango/32x32/status/sunny.png \
                      "Notification Title" \
                      "This is the body"



2. More command line: dbus-send command does NOT work for notification. It does not support nested message variables. If it did, it should be something really close to this:

   dbus-send --session --dest=org.freedesktop.Notifications \
                            --type=method_call --reply-timeout=10000 \
                            /org/freedesktop/Notifications org.freedesktop.Notifications \
                            string:'Test Application' uint32:0 \
                            string: string:'Notification Title' \
                            string:'This is the body' \
                            array:string: dict:string: int32:10000



3. Using the Python2 or Python3 pynotify module:
   

   #!/usr/bin/env python
   """Python 2.5 script. Creates a Notification pop-up bubble"""
   import pynotify
   title = "Notification Title"
   text  = "This is the body"
   icon  = "/usr/share/icons/Tango/32x32/status/sunny.png"
   pynotify.init("Test Application")
   notification = pynotify.Notification(title, text, icon) 
   notification.set_urgency(pynotify.URGENCY_NORMAL)
   notification.show() 



4. Using the Python2 or Python3 dbus module:
   

   #!/usr/bin/env python
   """Python 2.5 script. Creates a Notification pop-up bubble"""
   import dbus
   item              = "org.freedesktop.Notifications"
   path              = "/org/freedesktop/Notifications"
   interface         = "org.freedesktop.Notifications"
   app_name          = "Test Application"
   id_num_to_replace = 0
   icon              = "/usr/share/icons/Tango/32x32/status/sunny.png"
   title             = "Notification Title"
   text              = "This is the body"
   actions_list      = ''
   hint              = ''
   time              = 5000   # Use seconds x 1000
   
   bus = dbus.SessionBus()
   notif = bus.get_object(item, path)
   notify = dbus.Interface(notif, interface)
   notify.Notify(app_name, id_num_to_replace, icon, title, text, actions_list, hint, time)



5. Using the Python3 GObject Introspection (gi) module:
   

   #!/usr/bin/env python3
   import gi.repository
   from gi.repository import Gio, GLib
   
   session_bus                = Gio.BusType.SESSION
   cancellable                = None
   proxy_property             = 0
   interface_properties_array = None
   destination                = "org.freedesktop.Notifications"
   path                       = "/org/freedesktop/Notifications"
   interface                  = destination
   method                     = "Notify"
   application_name           = "Test Application"
   title                      = "Notification Title"
   text                       = "This is the body"
   id_num_to_replace          = 0
   icon                       = "/usr/share/icons/Tango/32x32/status/sunny.png"
   actions_list               = []
   hints_dict                 = {}
   display_milliseconds       = 5000
   timeout                    = -1
   
   connection = Gio.bus_get_sync(session_bus, cancellable)
   notify = Gio.DBusProxy.new_sync( connection, proxy_property, interface_properties_array,
                                    destination, path, interface, cancellable)
   args = GLib.Variant('(susssasa{sv}i)', ( application_name, id_num_to_replace, icon, 
                       title, text, actions_list, hints_dict, display_milliseconds))
   result = notify.call_sync(method, args, proxy_property, timeout, cancellable)
   id = result.unpack()[0]
   print(id)



6. Using the go programming language (golang):

   package main
   
   import "launchpad.net/~jamesh/go-dbus/trunk"
   import "fmt"
   
   func main() {
       // Set the variables
       var (
           err              error
           conn             *dbus.Connection
           destination      string = "org.freedesktop.Notifications"
           path             string = "/org/freedesktop/Notifications"
           dbus_interface   string = "org.freedesktop.Notifications"
           dbus_method      string = "Notify"
           application_name string = "dbus-test"
           id_num_to_repl   uint32
           icon_path        string = "/usr/share/icons/Tango/32x32/status/sunny.png"
           title            string = "Notification Title"
           text             string = "This is the body"
           actions_list     []string
           hint             map[string]dbus.Variant
           timeout          int32  = -1
       )
   
       // Connect to Session or System buses.
       if conn, err = dbus.Connect(dbus.SessionBus); err != nil {
           fmt.Println("Connection error:", err)
       }
   
       // Create an object proxy
       obj := conn.Object(destination, dbus.ObjectPath(path))
   
       // Call object methods.
       reply, err := obj.Call(
           dbus_interface,             dbus_method,
           application_name,           id_num_to_repl,
           icon_path,                  notif_box_title,
           notif_box_body,             actions_list,
           hint,                       timeout)
       if err != nil {
           fmt.Println("Notification error:", err)
       }
   
       // Parse the reply message
       var notification_id uint32
       if err := reply.Args(&notification_id); err != nil {
           fmt.Println(err)
       }
       fmt.Println("Notification id:", notification_id)
   

Zenity is an application, not a daemon. It creates a regular window that can be abused for notifications.

zenity --info --title "Big Title" --text "This is the\\nbody text"

Logging in Python 3

Here is one way to set up proper logging in Python.

This sample script sends some messages to syslog and others to a different logfile based upon their priority.

#!/usr/bin/python3

# Import the Standard Library modules we need
import logging
import logging.handlers

def set_up_logging():

    # File handler for /var/log/some.log
    serverlog = logging.FileHandler('/var/log/some.log')
    serverlog.setLevel(logging.DEBUG)
    serverlog.setFormatter(logging.Formatter(
        '%(asctime)s %(pathname)s [%(process)d]: %(levelname)s %(message)s'))

    # Syslog handler
    syslog = logging.handlers.SysLogHandler(address='/dev/log')
    syslog.setLevel(logging.WARNING)
    syslog.setFormatter(logging.Formatter(
        '%(pathname)s [%(process)d]: %(levelname)s %(message)s'))

    # Combined logger used elsewhere in the script
    logger = logging.getLogger('wbs-server-log')
    logger.setLevel(logging.DEBUG)
    logger.addHandler(serverlog)
    logger.addHandler(syslog)

    return logger

logger = set_up_logging()
logger.debug('This message should go to the log file')
logger.info('So should this')
logger.warning('And this, too')

From raw idea to useful source code

A couple months ago I had an Idea.

I even blogged about it: A lookup service for US National Weather Service codes. Those codes are necessary to access their machine-readable products.

In this post, I will show how I developed the idea into some code, how I grew the code into a project, added structure and version control, and finally moved the project onto online hosting.

This is not the only way to create a project.
This is probably not the best way for many projects.
It's just the way I did it, so you can avoid the most common mistakes.

You can see my final product hosted online at Launchpad.

---

From Idea to Code:

I know barely enough C to be able to ask there the bathroom is, so it's easier for me to use Python.

Code starts out as a single Python script:
- geolocation.py

As we add more features, a single script gets big and unwieldly, and we break it into smaller pieces.

For example, this structure easily allows more interfaces to be added.
- geolocation_service.py
- command_line_interface.py

Let's add a dbus interface, too. Dbus will semd messages to the interface if it knows about it. Let dbus know about it using a service file.
- dbus_interface.py
- dbus_service_file.service

Let's add an http interface, so not everyone in the world needs to download 5-6MB of mostly-unused lookup databases:
- http_interface.py
- specialized_webserver.py

Let's go back and formalize how we create the databases:
- database_creator.py

We have a lot of hard-coded variables in these scripts. Let's break them out into a config file.
- configfile.conf

There are other possible files, that we're not using. For example:
- Upstart config file (if we want the service to run/stop at boot or upon some system signal, lives in /etc/init)
- Udev rule file (if we want the service to run/stop when a device is plugged in, lives in /etc/udev/rules.d)

But that's a lot of files and scripts! 8 files, plus the databases.
 

---

Adding Version Control:

It's time to get serious about these eight files. We have invested a lot of time creating them, and it's time to start organizing the project so others can contribute, so we can track new bugs and features, and to protect all our invested work.

First, we need to introduce version control. Ideally, we would have done that from the start. But we didn't. So let's fix that.

Version control offers a lot of advantages:
    We can undo mistakes.
    It helps us package the software later.
    It helps us track bugs.
    It helps us apply patches.
    It helps us document changes.

There are plenty of good version control systems available. For this example, I'll use bazaar. The developers have a very good tutorial.

Installing bazaar:

$ sudo apt-get install bzr
$ bzr whoami "My Name "

Since we didn't start with proper version control, we need to create a new directory using version control, move our files into it, and add our files to version control.

$ bzr init-repo My_NEW_project_directory
$ bzr init My_NEW_project_directory
$ mv My_OLD_project_directory/* My_NEW_project_directory/
$ cd My_NEW_project_directory
$ bzr add *

Finally, we need to clean up the old directory, and commit the changes.

$ rm ../My_OLD_project_directory
$ bzr commit -m "Initial setup"

Organizing the code

My project directory is starting to get disorganized, with eight scripts and files, plus six database files, plus version control, plus more to come. I'm going to restructure my project folder like this:

My_project_directory
  +-- data   (all the database files)
  +-- src    (all the python scripts and other non-data files)
  +-- .bzr   (bzr's version control tracking)

Once version control is active, we cannot just move things around. We need to use the version control tools so it can keep tracking the right files.

$ bzr mkdir data src
$ bzr mv *.gz data/
$ bzr mv *.py src/
$ bzr mv dbus_service_file.service src/
$ bzr mv configfile.conf src/

See how bazaar adds the directories and performs the moves?

Now My_project_directory should be empty of files. Once reorganization is complete, remember to commit the change:

$ bzr commit -m "Reorganize the files to a better project structure"

---

Integrating into the system:

We have a problem with our eight files. They run beautifully, but only if they are in our home directory.

That won't work in the long run. A server should not be run as a user with shell access - that's a security hole. Nor should it be run out of a user's /home. Nor should it be run as root. Also, other applications that are looking for our server won't find it - all the files are in the wrong places.

So we need to put our files into the right places. And often that means fixing the scripts to replace hard-coded temporary paths (like '~/server/foo') with the proper locations ('/usr/lib/foo-server/foo').

Where are the right places?

The Linux Filesystem Hierarchy Standard (FHS) is used by Debian to define the right places.

Two files are directly user-launched in regular use:
- specialized_webserver.py: /usr/bin
- command_line_interface.py: /usr/bin

The database files are read-only and available to any application:
- database files: /usr/shared

Three files are launched or imported by other applications or scripts:
- geolocation_service.py: /usr/lib
- dbus_interface.py: /usr/lib
- http_interface.py: /usr/lib

One file is very rarely user-launched under unusual circumstances:
- database_creator.py

The  dbus service file will be looked for by dbus in a specific location:
- geolocation_dbus.service: /usr/share/dbus-1/services

Config files belong in /etc
- geolocation.conf: /etc

Makefiles make organization easier:

Now that we know where the right places are, let's create a Makefile that will install and uninstall the files to the right place. Our originals stay where they are - the makefile copies them during the install, and deletes the copies during uninstall.

Makefiles are really config files for the make application (included in the build-essential metapackage). Makefiles tell make which files depend upon which, which files to compile (we won't be compiling), and where the installed application files should be located, and how to remove the application.

Here is a sample makefile for my project (wbs-server):

DATADIR = $(DESTDIR)/usr/share/weather-location
LIBDIR  = $(DESTDIR)/usr/lib/wbs-server
BINDIR  = $(DESTDIR)/usr/bin
DBUSDIR = $(DESTDIR)/usr/share/dbus-1/services
CONFDIR = $(DESTDIR)/etc
CACHDIR = $(DESTDIR)/var/cache/wbs-webserver

install: 
 # Indents use TABS, not SPACES! Space indents will cause make to fail
 mkdir -p $(DATADIR)
 cp data/*.gz $(DATADIR)/

 mkdir -p $(LIBDIR)
 cp src/geolocation.py $(LIBDIR)/
 cp src/wbs_dbus_api.py $(LIBDIR)/
 cp src/wbs_http_api.py $(LIBDIR)/
 cp src/wbs_database_creator.py $(LIBDIR)/

 cp src/wbs_cli_api.py $(BINDIR)/
 cp src/wbs_webserver.py $(BINDIR)/
 cp src/wbs-server.service $(DBUSDIR)/
 cp src/confile.conf $(CONFDIR)/wbs-server.conf
 mkdir -p $(CACHDIR)

uninstall:
 rm -rf $(DATADIR)
 rm -rf $(LIBDIR)

 rm -f $(BINDIR)/wbs_cli_api.py
 rm -f $(BINDIR)/wbs_webserver.py
 rm -f $(DBUSDIR)/wbs-server.service
 rm -f $(CONFDIR)/wbs-server.conf
 rm -rf $(CACHDIR)

Let's save the makefile as Makefile, and run it using sudo make install and sudo make uninstall.

We run a test:

$ sudo make install
$ /usr/bin/wbs_cli_api.py zipcode 43210
bash: /usr/lib/wbs-server/wbs_cli_api.py: Permission denied

Uh-oh. Let's investigate:

$ ls -l /usr/lib/wbs-server/wbs_cli_api.py 
-rw-r--r-- 1 root root 3287 Dec 23 20:46 /usr/lib/wbs-server/wbs_cli_api.py

Aha. Permissions are correct, but the executable flag is not set. Let's uninstall the application so we can fix the makefile.

$ sudo make uninstall

In the makefile, we can make a few changes if we wish. We can set the executable flag. We can also create links or symlinks, or rename the copy.

For example, wbs_cli_api.py is a rather obtuse name for a command-line executable. Instead of copying it to /usr/bin, let's copy it to /usr/lib with its fellow scripts, make it executable, and create a symlink to /usr/bin with a better name like 'weather-lookup'

install:
        ...
 cp src/wbs_cli_api.py $(LIBDIR)/
 chmod +x $(LIBDIR)/wbs_cli_api.py
 ln -s $(LIBDIR)/wbs_cli_api.py $(BINDIR)/weather-lookup
        ...

uninstall:
        ...
 rm -f $(BINDIR)/weather-lookup
        ...

Another example: It's a bad idea to run a webserver as root. So let's add a couple lines to the makefile to create (and delete) a separate system user to run the webserver.

USERNAM = wbserver
        ...
install:
        ...
 adduser --system --group --no-create-home --shell /bin/false $(USERNAM)
 cp chgrp $(USERNAM) $(LIBDIR)/*
 cp chgrp $(USERNAM) $(CACHDIR)
 # Launch the webserver using the command 'sudo -u wbserver wbs-server'
        ...

uninstall:
        ...
 deluser --system --quiet $(USERNAM)
        ...

---

Sharing the code

Now we have a complete application, ready to distribute.

Thanks to the Makefile, we include a way to install and uninstall.

It's not a package yet. It's not even a source package yet. It's just source code and an install/uninstall script.

We can add a README file, a brief description of how to install and use the application.

We can also add an INSTALL file, detailed instructions on how to unpack (if necessary) and install the application.

It would be very very wise to add a copyright and/or license file, so other people know how they can distribute the code.

After all that, remember to add those files to version control! And to finally commit the changes:

bzr commit -m "Initial code upload. Add README, INSTALL, copyright, and license files."

Finally, we need a place to host the code online. Since I already have a Launchpad account and use bzr, I can easily create a new project on bzr.

And then uploading the version-controlled files is as simple as:

bzr launchpad-login my-launchpad-name
bzr push lp:~my-launchpad-name/my-project/trunk 

You can see my online code hosted at Launchpad.

---

Next time, we'll get into how to package this source code.

Very Simple Database in Python

Experimenting with big lookup tables for my weather code lookup server. Instead of using a big configparse file, I want to try a small database.

Python's dbm bindings are included in the default install of Ubuntu. It's light and easy to use.

#!/usr/bin/env python3
import dbm.gnu                # python3-gdbm package
zipcodes = '/tmp/testdb'

# Create a new database with one entry
# Schema: Key is Zipcode
# Value is Observation_Station_Code, Radar_Station_Code, Forecast_Zone
zipc = dbm.gnu.open(zipcodes, 'c')
zipc['53207'] = b'kmke,mkx,wiz066'

# Close and reopen the database
zipc.close()
zipd = dbm.gnu.open(zipcodes, 'r')

# List of database keys
keys = zipd.keys()

# Retrieve and print one entry
print(zipd[keys[0]].decode('utf-8').split(','))

zipd.close()

It works very well and is very fast. It's not easy to view or edit the database with other applications, since it is binary (not text).

Dbus Tutorial - GObject Introspection instead of python-dbus

Introduction
Introspection
Network Manager
Create a Service
GObject Introspection


In previous posts, I have looked at using the python-dbus to communicate with other processes, essentially using it the same way we use the dbus-send command.

There is another way to create DBus messages. It's a bit more complicated than python-dbus, and it depends upon Gnome, but it's also more robust and perhaps better maintained.

Using Gobject Introspection replacement for python-dbus is described several places, but the best example is here. Python-dbus as a separate bindings project has also suffered with complaints of "lightly maintained," and an awkward method of exposing properties that has been unfixed for years.


These examples only work for clients.  Gnome Bug #656330 shows that services cannot yet use PyGI.

---

Here's an example notification using Pygi instead of Python-DBus. It's based on this blog post by Martin Pitt, but expanded a bit to show all the variables I can figure out....


1) Header and load gi

#!/usr/bin/env python3

import gi.repository
from gi.repository import Gio, GLib

2) Connect to the DBus Session Bus
Documentation: http://developer.gnome.org/gio/2.29/GDBusConnection.html

session_bus = Gio.BusType.SESSION
cancellable = None
connection = Gio.bus_get_sync(session_bus, cancellable)

3) Create (but don't send) the DBus message header
Documentation: http://developer.gnome.org/gio/2.29/GDBusProxy.html

proxy_property = 0
interface_properties_array = None
destination = 'org.freedesktop.Notifications'
path = '/org/freedesktop/Notifications'
interface = destination
notify = Gio.DBusProxy.new_sync(
     connection,
     proxy_property,
     interface_properties_array,
     destination,
     path,
     interface,
     cancellable)

4) Create (but don't send) the DBus message data
The order is determined by arg order of the Notification system
Documentation: http://developer.gnome.org/notification-spec/#protocol

application_name = 'test'
title = 'Hello World!'
body_text = 'Subtext'
id_num_to_replace = 0
actions_list = []
hints_dict = {}
display_milliseconds = 5000
icon = 'gtk-ok'  # Can use full path, too '/usr/share/icons/Humanity/actions/'
args = GLib.Variant('(susssasa{sv}i)', (
                    application_name, 
                    id_num_to_replace,
                    icon, 
                    title,
                    body_text,
                    actions_list,
                    hints_dict,
                    display_milliseconds))

5) Send the DBus message header and data to the notification service
Documentation: http://developer.gnome.org/gio/2.29/GDBusProxy.html

method = 'Notify'
timeout = -1
result = notify.call_sync(method, args, proxy_property, timeout, cancellable)

6) (Optional) Convert the result value from a Uint32 to a python integer

id = result.unpack()[0]
print(id)

Play with it a bit, and you will quickly see how the pieces work together.

---

Here is a different, original example DBus client using introspection and this askubuntu question. You can see this is a modified and simplified version of the above example:

#!/usr/bin/env python3
import gi.repository
from gi.repository import Gio, GLib

# Create the DBus message
destination = 'org.freedesktop.NetworkManager'
path        = '/org/freedesktop/NetworkManager'
interface   = 'org.freedesktop.DBus.Introspectable'
method      = 'Introspect'
args        = None
answer_fmt  = GLib.VariantType.new ('(v)')
proxy_prpty = Gio.DBusCallFlags.NONE
timeout     = -1
cancellable = None

# Connect to DBus, send the DBus message, and receive the reply
bus   = Gio.bus_get_sync(Gio.BusType.SYSTEM, None)
reply = bus.call_sync(destination, path, interface,
                      method, args, answer_fmt,
                      proxy_prpty, timeout, cancellable)

# Convert the result value to a formatted python element
print(reply.unpack()[0])

---

Here is a final DBus client example, getting the properties of the current Network Manager connection

#!/usr/bin/env python3
import gi.repository
from gi.repository import Gio, GLib

# Create the DBus message
destination = 'org.freedesktop.NetworkManager'
path        = '/org/freedesktop/NetworkManager/ActiveConnection/19'
interface   = 'org.freedesktop.DBus.Properties'
method      = 'GetAll'
args        = GLib.Variant('(ss)', 
              ('org.freedesktop.NetworkManager.Connection.Active', 'None'))
answer_fmt  = GLib.VariantType.new ('(v)')
proxy_prpty = Gio.DBusCallFlags.NONE
timeout     = -1
cancellable = None

# Connect to DBus, send the DBus message, and receive the reply
bus   = Gio.bus_get_sync(Gio.BusType.SYSTEM, None)
reply = bus.call_sync(destination, path, interface,
                      method, args, answer_fmt,
                      proxy_prpty, timeout, cancellable)

# Convert the result value to a useful python object and print
[print(item[0], item[1]) for item in result.unpack()[0].items()]

As you can see from this example, dbus communication is actually pretty easy using GLib: Assign the nine variables, turn the crank, and unpack the result.