Saturday, November 24, 2012

Dbus Tutorial - GObject Introspection instead of python-dbus

Introduction
Introspection
Network Manager
Create a Service
GObject Introspection


In previous posts, I have looked at using the python-dbus to communicate with other processes, essentially using it the same way we use the dbus-send command.

There is another way to create DBus messages. It's a bit more complicated than python-dbus, and it depends upon Gnome, but it's also more robust and perhaps better maintained.

Using Gobject Introspection replacement for python-dbus is described several places, but the best example is here. Python-dbus as a separate bindings project has also suffered with complaints of "lightly maintained," and an awkward method of exposing properties that has been unfixed for years.


These examples only work for clients.  Gnome Bug #656330 shows that services cannot yet use PyGI.




Here's an example notification using Pygi instead of Python-DBus. It's based on this blog post by Martin Pitt, but expanded a bit to show all the variables I can figure out....


1) Header and load gi

#!/usr/bin/env python3

import gi.repository
from gi.repository import Gio, GLib


2) Connect to the DBus Session Bus
Documentation: http://developer.gnome.org/gio/2.29/GDBusConnection.html

session_bus = Gio.BusType.SESSION
cancellable = None
connection = Gio.bus_get_sync(session_bus, cancellable)


3) Create (but don't send) the DBus message header
Documentation: http://developer.gnome.org/gio/2.29/GDBusProxy.html

proxy_property = 0
interface_properties_array = None
destination = 'org.freedesktop.Notifications'
path = '/org/freedesktop/Notifications'
interface = destination
notify = Gio.DBusProxy.new_sync(
     connection,
     proxy_property,
     interface_properties_array,
     destination,
     path,
     interface,
     cancellable)


4) Create (but don't send) the DBus message data
The order is determined by arg order of the Notification system
Documentation: http://developer.gnome.org/notification-spec/#protocol

application_name = 'test'
title = 'Hello World!'
body_text = 'Subtext'
id_num_to_replace = 0
actions_list = []
hints_dict = {}
display_milliseconds = 5000
icon = 'gtk-ok'  # Can use full path, too '/usr/share/icons/Humanity/actions/'
args = GLib.Variant('(susssasa{sv}i)', (
                    application_name, 
                    id_num_to_replace,
                    icon, 
                    title,
                    body_text,
                    actions_list,
                    hints_dict,
                    display_milliseconds))


5) Send the DBus message header and data to the notification service
Documentation: http://developer.gnome.org/gio/2.29/GDBusProxy.html

method = 'Notify'
timeout = -1
result = notify.call_sync(method, args, proxy_property, timeout, cancellable)


6) (Optional) Convert the result value from a Uint32 to a python integer

id = result.unpack()[0]
print(id)

Play with it a bit, and you will quickly see how the pieces work together.



Here is a different, original example DBus client using introspection and this askubuntu question. You can see this is a modified and simplified version of the above example:

#!/usr/bin/env python3
import gi.repository
from gi.repository import Gio, GLib

# Create the DBus message
destination = 'org.freedesktop.NetworkManager'
path        = '/org/freedesktop/NetworkManager'
interface   = 'org.freedesktop.DBus.Introspectable'
method      = 'Introspect'
args        = None
answer_fmt  = GLib.VariantType.new ('(v)')
proxy_prpty = Gio.DBusCallFlags.NONE
timeout     = -1
cancellable = None

# Connect to DBus, send the DBus message, and receive the reply
bus   = Gio.bus_get_sync(Gio.BusType.SYSTEM, None)
reply = bus.call_sync(destination, path, interface,
                      method, args, answer_fmt,
                      proxy_prpty, timeout, cancellable)

# Convert the result value to a formatted python element
print(reply.unpack()[0])




Here is a final DBus client example, getting the properties of the current Network Manager connection

#!/usr/bin/env python3
import gi.repository
from gi.repository import Gio, GLib

# Create the DBus message
destination = 'org.freedesktop.NetworkManager'
path        = '/org/freedesktop/NetworkManager/ActiveConnection/19'
interface   = 'org.freedesktop.DBus.Properties'
method      = 'GetAll'
args        = GLib.Variant('(ss)', 
              ('org.freedesktop.NetworkManager.Connection.Active', 'None'))
answer_fmt  = GLib.VariantType.new ('(v)')
proxy_prpty = Gio.DBusCallFlags.NONE
timeout     = -1
cancellable = None

# Connect to DBus, send the DBus message, and receive the reply
bus   = Gio.bus_get_sync(Gio.BusType.SYSTEM, None)
reply = bus.call_sync(destination, path, interface,
                      method, args, answer_fmt,
                      proxy_prpty, timeout, cancellable)

# Convert the result value to a useful python object and print
[print(item[0], item[1]) for item in result.unpack()[0].items()]

As you can see from this example, dbus communication is actually pretty easy using GLib: Assign the nine variables, turn the crank, and unpack the result.

GeoClue vs Geocode-Glib

This post has been superseded by a more recent post with updated information.  

GeoClue, used in Gnome and Unity, is a Dbus service that consolidates location input from multiple sources to estimate a best location. It's lightly maintained, and the most recent maintainer has essentially deprecated it in favor of his newer Geocode-Glib

That announcement is here, plus a bit of reading-between-the-lines and a few subsequent clarifications.

The big advantage if geocode-glib0, is that it uses GObject introspection instead of specialty DBus bindings. The big disadvantage is that it leaves KDE and other non-Gnome environments unsupported...and has less functionality that the older Geoclue.

For now, it looks like I need to stick with GeoClue, and perhaps even help maintain it for my current weather-backend project. geocode-glib simply doesn't do the job I need GeoClue to do.


Here is an example of using python and geocode-glib to get geolocation information. I have not seen any examples of geocode-glib anywhere else, so I may be first here to use the lib with python:

#!/usr/bin/env python3
import gi.repository
from gi.repository import GeocodeGlib

# Create an object and add location information to it 
location = GeocodeGlib.Object()
location.add('city','Milwaukee')

# Do the gelocation and print the response dict
result = location.resolve()
[print(item[0], item[1]) for item in result.items()]

Give it a try...
The relevant packages are libgeocode-glib0 and gir1.2-geocodeglib-1.0
The Geocode-Glib source code and API reference are hosted at Gnome.

Saturday, November 3, 2012

NEW - US National Weather Service geolocation lookup server

Just completed my first hack of a US National Weather Service (NWS) geolocation lookup server.

It doesn't tell you weather...instead it tells you the best Observation Station code and Forecast Zone code so your application can pull the right data directly from the NWS servers.

NWS lookup is important to US users for obvious reasons. Internationally, it's important, too - NWS is one of the easiest and most open sources of international METAR current-condition reports.

Proprietary services like wunderground, weather.com, and Yahoo! will feed weather data based on raw locations...but as the recent and sudden Google Weather shutdown showed us, it's good to keep more than one arrow in your quiver. And this doesn't require registration nor an API key...it's totally free.

This lookup feature has been sorely lacking. In the past, you needed to scrape a NWS web page to figure out those codes...or ask the user to go find them somehow.

Example

Here's an example of how to use the lookup: I have a zipcode (53207), and I need to know the appropriate Observation Station to get current conditions from:

$ wget -q -O - http://kiwkak.dyndns.org:8000/geolookup/zipcode/53207 \
               | sed 's/.*<station_code>\(.*\)<\/station_code>.*/\1/'

kmke

The local station is 'kmke'. Plug that into the NWS current conditions URL:

$ wget -q -O - http://weather.noaa.gov/pub/data/observations/metar/decoded/KMKE.TXT

GEN MITCHELL INTERNATIONAL AIRPORT, WI, United States (KMKE) 42-57N 87-54W 206M
Nov 03, 2012 - 06:52 PM EDT / 2012.11.03 2252 UTC
Wind: from the NE (040 degrees) at 6 MPH (5 KT):0
Visibility: 10 mile(s):0
Sky conditions: mostly cloudy
Temperature: 39.0 F (3.9 C)
Dew Point: 30.0 F (-1.1 C)
Relative Humidity: 69%
Pressure (altimeter): 30.19 in. Hg (1022 hPa)
ob: KMKE 032252Z 04005KT 10SM FEW035 SCT120 BKN250 04/M01 A3019 RMK AO2 SLP231 T00391011
cycle: 23

How to use it

The lookup testing server at http://kiwkak.dyndns.org:8000 accepts 5-digit zipcodes (53207), three-letter airport codes(mke), place names (Milwaukee,WI), and lat/lon (lat=42.57n,lon=87.54w). Lat/Lon is available worldwide, and will return the closest METAR location, the rest are US-only.

http://kiwkak.dyndns.org:8000/geolookup/zipcode/53207
http://kiwkak.dyndns.org:8000/geolookup/airport/mke
http://kiwkak.dyndns.org:8000/geolookup/place/San+Francisco,CA
http://kiwkak.dyndns.org:8000/geolookup/latlon/lat=0.0451N,lon=32.443E

It returns an XML string. The string includes three sections: A data_source credit, the original parsed request, and the location data with the Observation Station code (kmke), a descriptive location string (General Mitchell International Airport, WI), and a Forecast Zone code (wiz066). Non-US METAR sites lack the Forecast Zone, since NWS creates forecasts for the US only.

<loc>
  <data_source>weather-util data files, Oct 25, 2012, v2.0, http://fungi.yuggoth.org/weather/</data_source>
  <request>
    <datatype>coordinates</datatype>
    <data>lat=0.0451n,lon=32.443e</data>
  </request>
  <location>
    <station_code>huen</station_code>
    <station_location>Entebbe Airport, Uganda</station_location>
    <forecast_zone></forecast_zone>
  </location>
</loc>

Limitations on name searches

Some large city requests (City,St) may return multiple locations, or unexpected results since the city may have multiple observation stations or multiple forecast zones within the boundaries. This is a limitation of the public data sets. For example, a search for /place/New+York+City,NY will return an error, since that's not really the city's name. A search for /place/Brooklyn,NY will return a closest Observation Station at Central Park instead of Kennedy Airport. Zipcode searches will generally return more accurate results.

Usage limits

The testing server is limited to five requests per day, so cache those results! Or just contact me if you want to be on the 'unlimited' list. Throttling is merely to spare my server from abuse - it's not meant to be a tease.

Give it a try, and let me know what you think.
If you find a bug, please e-mail me instead of leaving a comment.

Data sets and Credits

The dataset is merely the weather-util-data package, a set of reformatted and correlated lookup tables of freely-available US Government geospatial information. This USG public data is available for public use without copyright or restriction.

Thanks to the US National Weather Service for public access to so many cool products.

Special thanks to Jeremy Stanley, developer of the weather-util* packages, who created most of the important geolocation code and figured out a way to create the lookup tables. I merely wrapped an XML server around those tables. His code is released under the ISC license, which is GPL-compatible, and authorizes reuse.

Source code for this 0.0.1 (still hacking) version is at http://pastebin.ubuntu.com/1330443/, since I haven't actually organized a project around this script yet.