USBIP into an LXD container

In a previous post, I used USBIP to forawrd GPS data from A to B. 'A' was a USB GPS dongle pluged into a Raspberry Pi (Raspbian). 'B' was my laptop.

Now let's take it another step. Let's move 'B' to an LXD container sitting on a headless Ubuntu 19.04 server. No other changes: Same GPS data, same use of USBIP. 'A' is the same USB GPS dongle, the same Raspberry Pi, and the same Raspbian.

Setting up usbip on the server ('B') is identical to setting it up on my laptop. Recall that this particular dongle creates a /dev/ttyUSB_X device upon insertion, and it's the same on the Pi, the Laptop, and the Server

    me@server:~$ lsusb
        Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
        Bus 003 Device 006: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
        Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
        Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

    me@server:~$ ls -l /dev/ttyUSB0
        crw-rw---- 1 root dialout 188, 0 Aug 17 21:13 /dev/ttyUSB0

LXD has a USB Hotplug feature that works for many, but not all USB devices, connecting USB devices on the host to the container. Devices that create a custom entry in /dev (like /dev/ttyUSB_X) generally cannot use the USB Hotplug...but CAN instead use 'usb-char' forwarding which (seems to be) NOT hotpluggable.

Here's that LXD magic at work. In this case, I'm using a container called 'ha-test2', and let's simply name the dongle 'gps'. Do this while the container is stopped, or restart the container afterward

    me@server:~$ lxc config device add ha-test2 gps unix-char path=/dev/ttyUSB0
        Device gps added to ha-test2

Now we start the container, and then jump into a shell inside. We see that /dev/ttyUSB0 has indeed been forwarded. And we test to ensure data is flowing -- that we can read from /dev/tty/USB0.

    me@server:~$ lxc start ha-test2
    me@server:~$ lxc shell ha-test2
        mesg: ttyname failed: No such device

        root@ha-test2:~# ls -l /dev/ | grep tty
            crw-rw-rw- 1 nobody nogroup   5,   0 Aug 18 02:11 tty
            crw-rw---- 1 root   root    188,   0 Aug 18 02:25 ttyUSB0

        root@ha-test2:~# apt install gpsd-clients   // Get the gpsmon application
        root@ha-test2:~# gpsmon /dev/ttyUSB0

Making it permanent

It is permanent already. The 'lxc config' command will edit the config of the container, which is persistent across a reboot.

Cleaning up

There are two options for cleanup of the container.

• You can simply throw it away (it's a container)
• Alternately,

   root@ha-test2:~# apt autoremove gpsd-clients

On the Server:

    me@server:~$ lxc config device remove ha-test2 gps
    me@server:~$ sudo apt autoremove gpsd-clients    // If you installed gpsmon to test connectivity

Also remember to detach USBIP, and uninstall usbip packages.

Garmin etrex Vista and Linux

I have a new (to me) ancient Garmin etrex Vista handheld GPS ($30, ebay). It has a custom serial connector on the back, takes AA batteries, and happily talks in MGRS in addition to degree-minute-second. MGRS was the key - I need it to be compatible when I go out to the field with the Army Reserve.

Onboard features:
Discovering the Software Version and Unit ID: Main Menu -> Setup -> System -> (Zoom) Software Version.

Problem #1:
The date is showing as 2051. That's a known bug, and this page shows how to fix it by hard-resetting the device.

Gpsbabel:
The gpsbabel package, in the Debian Archives and Ubuntu Repositories, can transfer waypoints, tracks, and routes (but not maps) to/from the device. See the online documentation. It's a command-line application. Here are some example commands:

gpsbabel -D9                                                           # Debug verbose flag
gpsbabel -i garmin,get_posn -f /dev/ttyUSB0 -o kml -F myposition.kml   # Get current position
gpsbabel -o garmin,power_off -F /dev/ttyS0                             # Send power-off command

TomTom Home on Linux

Today I tried (again) to get my TomTom One GPS to connect to my linux system. And failed.

Experiment 1: Can the laptop's gpsd read the device as a dumb GPS?
Well, no. GPSD can't locate the device at all. I suspect that's because the TomTom One isn't GPS dongle, it's a full system on it's own. And it doesn't know to pass-through the raw GPS data to simulate a dongle.

Experiment 2: Can TomTome Home 2.8 (for controlling the device, uploading new maps, etc) be installed on Ubuntu 11.10?
No. The available TTH 2.8 clients are for Win and OSX only.

Experiment 3: Can TTH 2.8 be installed on Ubuntu 11.10 using Wine?
Sadly, no. The install itself went poorly. On several attempts, only one was successful. Due to two bugs (some custom jiggery in the TTH does in the Windows Registry, and a bug with Wine support for USB) the device cannot be recognized, loaded, or interacted with.

Experiment 4: Can TTH 2.8 be installed on Ubuntu 11.10 using a Virtual Machine?
Yes. I use a Virtualbox XP VM. TTH 2.8 installed perfectly first time. The device, like all USB devices in VB, must be manually added in the VB 'Device" menu. After that, control of the device was perfect.

GPS dongle, gpsd, and Ubuntu 11.10

This post is an update to my original experiment #1 and experiment #2 with a USB GPS dongle two years ago.

Install the hardware: Plug it in, of course.

• Check the kernel message log to discover the GPS dongle location in the filesystem:
  

  $ dmesg | grep tty
  [1769190.919411] usb 2-1: pl2303 converter now attached to ttyUSB0

• Test for signal with the following command:

  $ cat /dev/ttyUSB0 
  5�y�220,N,08754.5061,W,000.0,316.5,271111,,,A*71
  $GPVTG,316.5,T,,M,000.0,N,000.0,K,A*0C
  $GPGGA,173905.000,4259.7220,N,08754.5060,W,1,08,1.0,190.0,M,-33.8,M,,0000*63
  $GPGSA,A,3,12,04,10,23,02,05,17,25,,,,,1.7,1.0,1.4*31
  $GPRMC,173905.000,A,4259.7220,N,08754.5060,W,000.0,316.5,271111,,,A*71
  

  Use CTRL+C to end the signal check.
  If you get a bunch of zero-data, then try moving closer to a window, since GPS signals are line-of-sight. Or try a USB hub or extension cable, to reduce interference from your computer.

Install the gps daemon so you can use gps data.

• sudo apt-get install gpsd gpsd-clients

• Normally, udev will start gpsd automatically when a USB GPS device is inserted. If it fails to start after 5-10 seconds, you can start it manually:

  gpsd /dev/ttyUSB0  #You must tell gpsd where to look for the dongle

• Test the GPS and gpsd output using the gpsmon and cgps terminal programs, or the xgps program (all included in the gpsd-clients package)
• Capture data for recycling (optional - handy for development). gpsfake will simulate gpsd output
  

  $cat /dev/ttyUSB0 > path/to/datalog  # Create data file. Use CTRL+C to end the capture.
  $gpsfake path/to/datalog             # Run gpsd simulator (not a daemon - it will occupy the terminal)

Command-line tools to get GPS data

• Use the gpspipe command to get gpsd data:

  $ gpspipe -w -n 5
  netlib_connectsock() returns socket on fd 3
  {"class":"VERSION","release":"2.95","rev":"2011-07-27T11:20:24","proto_major":3,"proto_minor":3}
  {"class":"DEVICES","devices":[{"class":"DEVICE","path":"/dev/ttyUSB0"}]}
  {"class":"WATCH","enable":true,"json":true,"nmea":false,"raw":0,"scaled":false,"timing":false}
  {"class":"TPV","tag":"RMC","device":"/dev/ttyUSB0","time":1322418390.001,"ept":0.005,"lat":42.995410000,"lon":-87.908430000,"alt":195.400,"epx":11.739,"epy":17.294,"epv":36.800,"track":342.2000,"speed":0.000,"climb":0.000,"eps":34.59,"mode":3}
  {"class":"SKY","tag":"GSV","device":"/dev/ttyUSB0","xdop":0.66,"ydop":1.15,"vdop":1.46,"tdop":1.01,"hdop":1.32,"gdop":2.22,"pdop":1.97,"satellites":[{"PRN":2,"el":77,"az":360,"ss":26,"used":true},{"PRN":10,"el":58,"az":72,"ss":32,"used":true},{"PRN":12,"el":51,"az":247,"ss":34,"used":true},{"PRN":5,"el":45,"az":192,"ss":21,"used":true},{"PRN":4,"el":40,"az":65,"ss":20,"used":true},{"PRN":25,"el":39,"az":292,"ss":21,"used":true},{"PRN":13,"el":15,"az":60,"ss":18,"used":true},{"PRN":24,"el":14,"az":258,"ss":0,"used":false},{"PRN":29,"el":12,"az":304,"ss":23,"used":false},{"PRN":17,"el":8,"az":121,"ss":17,"used":true},{"PRN":23,"el":7,"az":35,"ss":10,"used":false},{"PRN":31,"el":0,"az":334,"ss":0,"used":false}]}

• Let's refine the command a bit, limit to the first "TPV" (time, position, velocity) sentence:

  $ gpspipe -w -n 5 | grep -m 1 TPV
  netlib_connectsock() returns socket on fd 3
  {"class":"TPV","tag":"RMC","device":"/dev/ttyUSB0","time":1322418481.001,"ept":0.005,"lat":42.995406667,"lon":-87.908428333,"alt":195.300,"epx":10.003,"epy":17.383,"epv":36.800,"track":342.2000,"speed":0.000,"climb":0.000,"eps":34.77,"mode":3}

• We can use basic shell commands to extract single variables from this sentence:
  

  tpv=$(gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4,6-8,13)
  seconds=$(echo $tpv | cut -d, -f1 | cut -d: -f2)
  latitude=$(echo $tpv | cut -d, -f2 | cut -d: -f2)
  longitude=$(echo $tpv | cut -d, -f3 | cut -d: -f2)
  altitude=$(echo $tpv | cut -d, -f4 | cut -d: -f2)
  speed=$(echo $tpv | cut -d, -f5 | cut -d: -f2)

• For fun, let's compare GPS time with system time using shell commands:
  

  date -u +%s   # System utc time in seconds
  gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4 | cut -d: -f2   # GPS utc time in seconds
  
  # Let's compare them side-by side
  $ echo System: $(date -u +%s) , GPS: $(gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4 | cut -d: -f2) 
  netlib_connectsock() returns socket on fd 3
  System: 1322419689 , GPS: 1322419689.000
  # Our sytem time is within one second of GPS time. That's good!

• For fun, let's use shell script to post the GPS location to Google Maps:
  The format is simple enough: http://maps.google.com/?ll=40.480381,-92.373047&z=16 will return a valid map. So that's three variables: Latitude, longitude, and zoom Level.
  

  tpv=$(gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4,6-8,13)
  latitude=$(echo $tpv | cut -d, -f2 | cut -d: -f2)
  longitude=$(echo $tpv | cut -d, -f3 | cut -d: -f2)
  zoom=20
  map_url="http://maps.google.com/?ll=${latitude},${longitude}&z=${zoom}"
  firefox $map_url

Python tools to get GPS data

• Python bindings to gpsd are provided by the python-gps package.

  sudo apt-get install python-gps

• The python tools have changed a *lot* in two years. The old methods don't work anymore. No Python 3 version is available yet. There's a decent example here, but I couldn't get it to work.

Conversion between Lat/Lon and MGRS/UTM and other systems: gpsd doesn't do this, though apparently the geographiclib-tools package does. And so does this website.

Installing Xubuntu 9.04 on an emachines E625-5192

Received my new laptop today - I need it for the fall, and got it a little early due to a sale.

What went well: I created a set of Restore DVDs (in case I want Windows back), then removed windows and installed Xubuntu 9.04 full-disk. Copied over my old /home directory, and installed all my favorite apps. E-mail, web, games, .bashrc, most dektop settings, etc. transferred without a hiccup. Recreated my crontab. Wireless networking and video work great. Built-in card reader reads all cards from my cameras and phone. Machine is noticeably faster. FN-key brightness control works. FN-key multimedia controls work.

Solved Problems:

Sound was tinny and a bit faint. Headphone jack works, not tinny, but low volume. Solution: Adding the following lines to /etc/modprobe.d/alsa-base.conf
somwhat improved speakers and fixed the headphone jack.

# Added by me on <date> while troubleshooting audio
options snd-hda-intel model=6stack

The Volume-FN keys didn't work. Solution: Mapped FN-audio up, down, and mute keys by mapping them to aumix using these instructions.

The suspend FN-key didn't work. Solution: Enabled it in the gnome-power-manager preferences. It was set up to work out-of-the-box, but disabled by default.

DVD: DVDs play only after changing CDROM permissions after each disc insertion. Workaround: Created an alias in .bashrc as the 'fixcd' command

TomTom GPS automatic update doesn't work on Linux - Win and OSX only. Department of Defense forms and other applications are Windows-specific. Workaround: Installed a Virtual Machine (VM) to host an occasional-use XP instance.

Phone-made .3g2 video files play with weird audio - need a way to convert them. Computer .wma songs don't play on phone - need a lossless storage format, and a way to convert them to .mp3. Workaround: Upload videos to YouTube instead of the local hard drive.

Need password to restore from suspend (Xubuntu issue, not hardware-related). Workaround: Remove the screensaver package.

Unsolved Problems: No built-in webcam. Keyboard is different, and will take time to get used to - many typos in the meantime.

You Are Here on a Google Map

This post is obsolete and has been superseded

The following python 2.x script shows your current (GPS-enabled) location on a Google Map. A useful learning experience:

#!/usr/bin/env python

"""This is a python 2.5 script that plot's a GPS receiver's location on the
Google Maps website.

In order to work, you need a network connection, an attached GPS receiver, and
the GPS daemon (gpsd).
"""
import os
#import subprocess and SU
import gps
import dbus
import sys


def test( ):
    """ Step 1: Test for the existence of a running gpsd, test for the existence of an open network connection, and test for a firefox process.
    If any fail, give an error message, don't try to recover. FUTURE: Could also use DBus to test for firefox and gpsd."""

    process_list = os.popen('ps -e')    # os.popen is deprecated in favort of subprocess.Popen
    #process_list = SU.Popen(['ps','e'], stdout=SU.PIPE).stdout.read()  
    gpsd_existence_flag = 0
    firefox_existence_flag = 0
    for line in process_list.readlines():
        if line.count('gpsd') > 0: gpsd_existence_flag = 1
        if line.count('firefox') > 0: firefox_existence_flag = 1

    if not gpsd_existence_flag:
        print ("gpsd is not running. Use 'gpsd -b /dev/ttyUSB0' to start it, and then try again.")
        sys.exit()       
    else: print ('Checking...found gpsd')

    if not firefox_existence_flag:
        print ("firefox is not running. Please start it and try again.")
        sys.exit()
    else: print ('Checking...found firefox')

    bus = dbus.SystemBus()
    nm_item = ('org.freedesktop.NetworkManager')  # This string gets used a lot
    nm_path = ('/org/freedesktop/NetworkManager')
    nm_device = ('org.freedesktop.NetworkManager.Device')

    list_of_interface_paths = dbus.Interface(bus.get_object(nm_item, nm_path), nm_device).getDevices()

    found_network_flag = 0
    for interface_path in list_of_interface_paths:
        one_interface = dbus.Interface(bus.get_object(nm_item, interface_path), nm_device)
        if one_interface.getLinkActive():   # True if there is an active network on this interface
            if one_interface.getType() == 2: # 0 unknown, 1 wired, 2 wireless
                print('Checking...found the wireless network') 
                found_network_flag = 1
            elif one_interface.getType() == 1: 
                print('Checking...found the wired network')
                found_network_flag = 1
                
    if found_network_flag: return
    else:
        print ("cannot find a network connection. Please connect and try again.")
        sys.exit()    


def get_position_fix( ):
    """Step 2: Get a position fix from gpsd."""
    session = gps.gps('localhost','2947')  # Open a connection to gpsd
    session.query('p')                     # Get the location fix 
    lat = session.fix.latitude
    lon = session.fix.longitude
    print ('Location is ' + str(lat) + ' latitude and ' + str(lon) + ' longitude.')
    return (lat, lon)


def show_map(lat_lon_tuple):
    """Step 3: Submit the position fix to Google Maps. Note that the parentheses '()' in the URL must be escaped '\' to work.
    Sample URL format: http://maps.google.com/maps?q=37.771008,+-122.41175+(You+can+insert+your+text+here)&iwloc=A&hl=en"""
    url_string = ('http://maps.google.com/maps?q=' + str(lat_lon_tuple[0]) + ',+' + str(lat_lon_tuple[1]) + '+\(You+Are+Here\)&iwloc=A&hl=en')
    os.popen('firefox ' + url_string)
    return

# Run this script as a standalone program
if __name__ == "__main__" :
    test()
    location = get_position_fix()
    show_map(location)

GPS and Xubuntu 8.04

This post is obsolete and has been superseded

I'm experimenting with USB GPS receiver (dongle). It's a Canmore GT-730F that I received in January 2009. Here's what I've learned so far.

Manually getting data using the command line (source):

1. Check dmesg, the kernel log, to find out where the device has been mounted. In my case, it mounts reliably to /dev/ttyUSB0. If it doesn't mount, try the command sudo modprobe pl2303 to load the correct USB driver.
2. Set the data output rate to 4800 baud using the stty command: stty 4800 > /dev/ttyUSB0
3. Read the data stream using the cat command: cat /dev/ttyUSB0
4. You should see a set of data scroll down the screen. Use CTRL+C to end the test.

The Linux GPS Daemon (gpsd) is the central clearinghouse for receiving GPS data from the receiver, buffering it, and forwarding it to the applications that want it. gpsd has features to broadcast to dbus (system bus), update ntpd, and respond to a multitude of specific queries from clients. References: Project home page, gpsd man page, and a great example program

$sudo apt-get install gpsd gpsd-clients  # Installs the daemon (gpsd) and test set (gpsd-clients) packages
$gpsd -b /dev/ttyUSB0                    # Start gpsd, telling it where to find the receiver
$cgps                                    # Current satellite data - great way to test that the receiver and gpsd are working

gpsfake is a gpsd simulator. It tricks gpsd into reading from a logfile instead of a real GPS device. Very handy for testing without actually using a GPS dongle. It is included with the gpsd package, and has a man page for additional reference. To make a logfile, and then to use gpsfake:

$cat /dev/ttyUSB0 > path/to/testlog  # Create the log file. Use CTRL+C to end the capture.
$gpsfake path/to/testlog             # Run gpsd simulator (not a daemon - it will occupy the terminal)

Python interface to gpsd (python-gps) is a programming tool to build your own gps-aware application.

>>>import gps                             # Load the module
>>>session = gps.gps('localhost','2947')  # Open a connection to gpsd
>>>session.query('o')                     # See man gpsd(8) for the list of commands
>>>print session.fix.latitude             # Query responses are attributes of the session
>>>dir(session)                           # To see the possible responses
>>>del session                            # Close the connection to gpsd

In this case, it seems that I need a periodic session.query('p'), which just gives lat/lon and timestamp.

Time might be an issue, since the system and the GPS may think the time is different. To see if it's an issue, compare them using the python script below. In my tests, they vary from 0.08 to 1.3 seconds apart, not enough to worry about. GPS timestamps use GMT, not localtime.

#!/usr/bin/env python
import calendar, time, gps
system_time = calendar.timegm(time.gmtime())  # System time (in seconds)
session = gps.gps('localhost','2947')         # Open a connection to gpsd
session.query('p')                            # See man gpsd(8) for the list of commands
gps_time = session.timings.c_recv_time        # GPS time (in seconds)
print ('The time difference is ' + str(system_time - gps_time) + ' seconds.')

MGRS (military map coordinates) conversion to/from latitude and longitude is not currently available in Ubuntu...that I can find. The dongle documentation doesn't mention MGRS at all. An online converter is available. The proj package looks promising, but I haven't figured it out yet. Perhaps Lat/Lon -> UTM -> MGRS?

DBus access appears to be undocumented...but there are tantalizing hints on Google that examples are out there. I can listen to dBus using cgps to make traffic, then dbus-monitor --system to see it.

The best storage format for tracklogs, routes, and waypoints seems to be GPX format, since it's easy to understand and cgpxlogger, included with gpsd, will create an XML track in GPX 1.1 format. Google's KML is more powerful, but also much more complex. GPSbabel universal data translator is a command-line application that translates one file type to another, and does convert GPX <-> KML.

$cgpxlogger -i 30 > path/to/logfile         # Save data every 30 seconds to an XML file
$gpsbabel -i gpx -f path/to/gpx_file -x duplicate -o kml -F path/to/kml_file
$#gpsbabel [options] -i INTYPE -f INFILE -x FILTER -o OUTTYPE -F OUTFILE

GPSdrive navigation system looks cool, but I couldn't get maps to load, so it's utility was limited. However, it seems that online plotting of tracklogs, routes, and waypoints is possible on Google Maps (and Yahoo Maps, and others). One example is the cool GPS Visualizer.

Gypsy is an alternative daemon, but not currently in Debian or Ubuntu, so I haven't tried it. Last release 0.6 in March 2008.

GPSman device manager is an app I know nothing about. I couldn't get it to work, so I removed it. The dongle seems small enough and simple enough that it may not need to be 'managed' at all.