Experimenting with USB devices across the LAN with USBIP

USBIP is a Linux tool for accessing USB devices across a network. I'm trying it out.

Reference 1, Reference 2, Reference 3, Reference 4.

At one end of the room, I have a Raspberry Pi with

• A Philips USB Webcam
• A no-name USB GPS dongle
• A Nortek USB Z-Wave/Zigbee network controller dongle

At the other end of the room is my laptop.

Before starting anything, I plugged all three into another system to ensure that they worked properly.

Raspberry Pi Server Setup

The Pi is running stock Raspbian Buster, with the default "pi" user replaced by a new user ("me") with proper ssh keys.

Before we start, here's what the 'lsusb' looks like on the Pi

    me@pi:~ $ lsusb
        Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
        Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Now we plug in the three USB devices and see what changed

    me@pi:~ $ lsusb
        Bus 001 Device 004: ID 10c4:8a2a Cygnal Integrated Products, Inc. 
        Bus 001 Device 005: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
        Bus 001 Device 006: ID 0471:0329 Philips (or NXP) SPC 900NC PC Camera / ORITE CCD Webcam(PC370R)
        Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
        Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And here are the new devices created or modified

    me@pi:~ $ ls -l /dev | grep 12    // 12 is today's date
        drwxr-xr-x 4 root root          80 Aug 12 00:46 serial
        lrwxrwxrwx 1 root root           7 Aug 12 00:46 serial0 -> ttyAMA0
        drwxr-xr-x 4 root root         220 Aug 12 00:47 snd
        crw--w---- 1 root tty     204,  64 Aug 12 00:46 ttyAMA0
        crw-rw---- 1 root dialout 188,   0 Aug 12 00:46 ttyUSB0
        drwxr-xr-x 4 root root          80 Aug 12 00:47 v4l
        crw-rw---- 1 root video    81,   3 Aug 12 00:47 video0

Looks like...

• /dev/ttyAMA0 is the Nortek Z-Wave controller
• /dev/ttyUSB0 is the GPS stick
• /dev/video0 is the webcam

Installing USBIP onto Raspbian Buster is easy. However, it is DIFFERENT from stock Debian or Ubuntu. This step is Raspbian-only

    me@pi:~$ sudo apt install usbip

Now load the kernel module. The SERVER always uses the module 'usbip_host'.

    me@pi:~$ sudo modprobe usbip_host     // does not persist across reboot

List the devices the usbip can see. Note each Bus ID - we'll need those later

    me@pi:~ $ usbip list --local
 - busid 1-1.1 (0424:ec00)
   Standard Microsystems Corp. : SMSC9512/9514 Fast Ethernet Adapter (0424:ec00)

 - busid 1-1.2 (0471:0329)
   Philips (or NXP) : SPC 900NC PC Camera / ORITE CCD Webcam(PC370R) (0471:0329)

 - busid 1-1.4 (067b:2303)
   Prolific Technology, Inc. : PL2303 Serial Port (067b:2303)

 - busid 1-1.5 (10c4:8a2a)
   Cygnal Integrated Products, Inc. : unknown product (10c4:8a2a)

• We can ignore the Ethernet adapter
• The Webcam is at 1-1.2
• The GPS dongle is at 1-1.4
• The Z-Wave Controller is at 1-1.5

Bind the devices.

    me@pi:~$ sudo usbip bind --busid=1-1.2        // does not persist across reboot
        usbip: info: bind device on busid 1-1.2: complete

    me@pi:~$ sudo usbip bind --busid=1-1.4        // does not persist across reboot
        usbip: info: bind device on busid 1-1.4: complete

    me@pi:~$ sudo usbip bind --busid=1-1.5        // does not persist across reboot
        usbip: info: bind device on busid 1-1.5: complete

The USB dongle will now appear to any client on the network just as though it was plugged in locally.

If you want to STOP serving a USB device:

    me@pi:~$ sudo usbip unbind --busid=1-1.2

The server (usbipd) process may or may not actually be running, serving on port 3240. Let's check:

    me@pi:~ $ ps -e | grep usbipd
        18966 ?        00:00:00 usbipd

    me@:~ $ sudo netstat -tulpn | grep 3240
        tcp        0      0 0.0.0.0:3240            0.0.0.0:*               LISTEN      18966/usbipd        
        tcp6       0      0 :::3240                 :::*                    LISTEN      18966/usbipd

We know that usbipd is active and listening. If not, start usbipd with:

    me@:~ $ sudo usbipd -D

You can run it more than one; only one daemon will start. The usbipd server does NOT need to be running to bind/unbind USB devices - you can start the server and bind/unbind in any order you wish. If you need to debug a connection, omit the -D (daemonize; fork into the background) so you can see the debug messages. See 'man usbipd' for the startup options to change port, IPv4, IPv6, etc.

Laptop Client Setup

Let's look at the USB devices on my laptop before starting:

    me@laptop:~$ lsusb
        Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
        Bus 001 Device 003: ID 04f2:b56c Chicony Electronics Co., Ltd 
        Bus 001 Device 002: ID 05e3:0608 Genesys Logic, Inc. Hub
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

In stock Debian (not Raspbian) and Ubuntu, usbip is NOT a separate package. It's included in the 'linux-tools-generic' package, which many folks already have installed...

    me@laptop:~$ apt list linux-tools-generic
        Listing... Done
        linux-tools-generic/disco-updates 5.0.0.23.24 amd64   // Doesn't say "[installed]"

...but apparently I don't. Let's install it.

    me@laptop:~$ sudo apt install linux-tools-generic

Now load the kernel module. The CLIENT always uses the kernel module 'vhci-hcd'.

    me@laptop:~$ sudo modprobe vhci-hcd     // does not persist across reboot

List the available USB devices on the Pi server (IP addr aa.bb.cc.dd). Those Bus IDs should look familiar.

    me@laptop:~$ usbip list -r aa.bb.cc.dd                        // List available on the IP address
        usbip: error: failed to open /usr/share/hwdata//usb.ids   // Ignore this error
        Exportable USB devices
        ======================
         - aa.bb.cc.dd
              1-1.5: unknown vendor : unknown product (10c4:8a2a)
                   : /sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.5
                   : (Defined at Interface level) (00/00/00)
                   :  0 - unknown class / unknown subclass / unknown protocol (ff/00/00)
                   :  1 - unknown class / unknown subclass / unknown protocol (ff/00/00)


              1-1.4: unknown vendor : unknown product (067b:2303)
                   : /sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4
                   : (Defined at Interface level) (00/00/00)

              1-1.2: unknown vendor : unknown product (0471:0329)
                   : /sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2
                   : (Defined at Interface level) (00/00/00)

Now we attach the three USB devices. This will not persist across a reboot.

    me@laptop:~$ sudo usbip attach --remote=aa.bb.cc.dd --busid=1-1.2
    me@desktop:~$ sudo usbip attach --remote=aa.bb.cc.dd --busid=1-1.4
    me@desktop:~$ sudo usbip attach --remote=aa.bb.cc.dd --busid=1-1.5
    // No feedback upon success

The remote USB devices now show in 'lsusb'

    me@laptop:~$ lsusb
        Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
        Bus 003 Device 004: ID 10c4:8a2a Cygnal Integrated Products, Inc. 
        Bus 003 Device 003: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
        Bus 003 Device 002: ID 0471:0329 Philips (or NXP) SPC 900NC PC Camera / ORITE CCD Webcam(PC370R)
        Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
        Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
        Bus 001 Device 003: ID 04f2:b56c Chicony Electronics Co., Ltd 
        Bus 001 Device 002: ID 05e3:0608 Genesys Logic, Inc. Hub
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And we can see that new devices have appeared in /dev. Based upon the order we attached, it's likely that

• The webcam 1-1.2 is at /dev/video2
• The GPS dongle 1-1.4 is probably at /dev/ttyUSB0
• The Z-Wave controller 1-1.5 is at /dev/ttyUSB1
• The same dongle includes a Zigbee controller, too, at /dev/ttyUSB2

The Z-Wave/Zigbee controller has had it's major number changed from 204 to 188. We don't know if that's important or not yet.

    me@laptop:~$ ls -l /dev | grep 12
        drwxr-xr-x  4 root root            80 Aug 12 00:56 serial
        crw-rw----  1 root dialout 188,     0 Aug 12 00:56 ttyUSB0
        crw-rw----  1 root dialout 188,     1 Aug 12 00:56 ttyUSB1
        crw-rw----  1 root dialout 188,     2 Aug 12 00:56 ttyUSB2
        crw-rw----+ 1 root video    81,     2 Aug 12 00:56 video2

Testing Results

I tested the GPS using the 'gpsmon' application, included with the 'gpsd-clients' package. We don't actually need gpsd, we can connect gpsmon directly to the remote USB device.

    me@laptop:~$ gpsmon /dev/ttyUSB0
        gpsmon:ERROR: SER: device open of /dev/ttyUSB0 failed: Permission denied - retrying read-only
        gpsmon:ERROR: SER: read-only device open of /dev/ttyUSB0 failed: Permission denied

Aha, a permission issue, not a usbip failure!
Add myself to the 'dialout' group, and then it works. A second test across a VPN connection, from a remote location, was also successful.

    me@laptop:~$ ls -la /dev/ttyUSB0
        crw-rw---- 1 root dialout 188, 0 Aug 11 21:41 /dev/ttyUSB0    // 'dialout' group

    me@laptop:~$ sudo adduser me dialout
        Adding user `me' to group `dialout' ...
        Adding user me to group dialout
        Done.

    me@laptop:~$ newgrp dialout    // Prevents need to logout/login for new group to take effect

    me@laptop:~$ gpsmon /dev/ttyUSB0
    // Success!

The webcam is immediately recognized in both Cheese and VLC, and plays across the LAN instantly. There is a noticeable half-second lag. A second test, across a VPN connection from a remote location, had the USB device recognized but not enough signal was arriving in timely order for the applications to show the video.

There were a few hiccups along the way. The --debug flag helps a lot to track down the problems:

• Client failed to connect with "system error" - turns out usbipd was not running on the server.
• Client could see the list, but failed to attach with "attach failed" - needed to reboot the server (not sure why)
• An active usbip connection prevents my laptop from sleeping properly
• The Z-wave controller require HomeAssistant or equivalent to run, a bit more that I want to install onto the testing laptop. Likely to have permission issues, too.

Cleaning up

To tell a CLIENT to cease using a remote USB (virtual unplug), you need to know the usbip port number. Well, not really: We have made only one persistent change; we could simply reboot instead.

    me@laptop:~$ usbip port   // Not using sudo - errors, but still port numbers
        Imported USB devices
        ====================
        libusbip: error: fopen
        libusbip: error: read_record
        Port 00:  at Full Speed(12Mbps)
               Philips (or NXP) : SPC 900NC PC Camera / ORITE CCD Webcam(PC370R) (0471:0329)
               5-1 -> unknown host, remote port and remote busid
                   -> remote bus/dev 001/007
        libusbip: error: fopen
        libusbip: error: read_record
        Port 01:  at Full Speed(12Mbps)
               Prolific Technology, Inc. : PL2303 Serial Port (067b:2303)
               5-2 -> unknown host, remote port and remote busid
                   -> remote bus/dev 001/005
        libusbip: error: fopen
        libusbip: error: read_record
        Port 02:  at Full Speed(12Mbps)
               Cygnal Integrated Products, Inc. : unknown product (10c4:8a2a)
               5-3 -> unknown host, remote port and remote busid
                   -> remote bus/dev 001/006

    me@laptop:~$ sudo usbip port    // Using sudo, no errors and same port numbers
        Imported USB devices
        ====================
        Port 00: <port in use> at Full Speed(12Mbps)
               Philips (or NXP) : SPC 900NC PC Camera / ORITE CCD Webcam(PC370R) (0471:0329)
               5-1 -> usbip://aa.bb.cc.dd:3240/1-1.2
                   -> remote bus/dev 001/007
        Port 01: <port in use> at Full Speed(12Mbps)
               Prolific Technology, Inc. : PL2303 Serial Port (067b:2303)
               5-2 -> usbip://aa.bb.cc.dd:3240/1-1.4
                   -> remote bus/dev 001/005
        Port 02: <port in use> at Full Speed(12Mbps)
               Cygnal Integrated Products, Inc. : unknown product (10c4:8a2a)
               5-3 -> usbip://aa.bb.cc.dd:3240/1-1.5
                   -> remote bus/dev 001/006
 
    me@laptop:~$ sudo usbip detach --port 00
        usbip: info: Port 0 is now detached!

    me@laptop:~$ sudo usbip detach --port 01
        usbip: info: Port 1 is now detached!

    me@laptop:~$ sudo usbip detach --port 02
        usbip: info: Port 2 is now detached!

    me@laptop:~$ lsusb              // The remote USB devices are gone now
        Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
        Bus 001 Device 003: ID 04f2:b56c Chicony Electronics Co., Ltd 
        Bus 001 Device 002: ID 05e3:0608 Genesys Logic, Inc. Hub
        Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

    me@laptop:~$ sudo modprobe -r vhci-hcd    // Remove the kernel module

The only two persistent change we made on the CLIENT were adding myself to the 'dialout' group and installing the 'linux-tools-generic' package, so let's remove that. If you ALREADY were in the 'dialout' group, or had the package installed for other reasons, then obviously don't remove it. It's not the system's responsibility to keep track of why you have certain permissions or packages -- that's the human's job. After this step, my CLIENT is back to stock Ubuntu.

    me@laptop:~$ sudo deluser me dialout                  // Takes effect after logout
    me@laptop:~$ sudo apt autoremove linux-tools-generic  // Immediate

Telling a SERVER to stop sharing a USB device (virtual unplug) and shut down the server is much easier. Of course, this is also a Pi, and we did make any changes permanent, so it might be easier to simply reboot it.

    me@pi:~$ usbip list -l
         - busid 1-1.1 (0424:ec00)
           Standard Microsystems Corp. : SMSC9512/9514 Fast Ethernet Adapter (0424:ec00)

         - busid 1-1.2 (0471:0329)
           Philips (or NXP) : SPC 900NC PC Camera / ORITE CCD Webcam(PC370R) (0471:0329)

         - busid 1-1.4 (067b:2303)
           Prolific Technology, Inc. : PL2303 Serial Port (067b:2303)

         - busid 1-1.5 (10c4:8a2a)
           Cygnal Integrated Products, Inc. : unknown product (10c4:8a2a)

    me@pi:~$ sudo usbip unbind --busid=1-1.2
        usbip: info: unbind device on busid 1-1.2: complete
    me@pi:~$ sudo usbip unbind --busid=1-1.4
        usbip: info: unbind device on busid 1-1.4: complete
    me@pi:~$ sudo usbip unbind --busid=1-1.5
        usbip: info: unbind device on busid 1-1.5: complete

    me@pi:~$ sudo pkill usbipd

The only persistent change we made on the Pi is installing the 'usbip' package. Once removed, we're back to stock Raspbian.

    me@pi:~$ sudo apt autoremove usbip

Making it permanent

There are two additional steps to making a permanent server, and essentially the same two steps to make a permanent client. This means a USBIP server that begins serving automatically upon boot, and a client that automatically connects to the server upon boot.

Add the kernel modules to /etc/modules so that the USBIP kernel modules will be automatically loaded at boot. To remove a client or server, delete the line from /etc/modules. You don't need to use 'nano' - use any text editor you wish, obviously.

    me@pi:~$ sudo nano /etc/modules     // usbipd SERVER

        usbip_host

    me@laptop:~$ sudo nano /etc/modules     // usbip CLIENT

        usbip_vhci-hcd

    // Another way to add the USBIP kernel modules to /etc/modules on the SERVER
    me@pi:~$ sudo -s                            // "sudo echo" won't work
    me@pi:~# echo 'usbip_host' >> /etc/modules
    me@pi:~# exit

    // Another way to add the USBIP kernel modules to /etc/modules on the CLIENT
    me@pi:~$ sudo -s                            // "sudo echo" won't work
    me@pi:~# echo 'vhci-hcd' >> /etc/modules
    me@pi:~# exit

Add a systemd job to the SERVER to automatically bind the USB devices. You can use systemd to start, stop, and restart the server conveniently, and for to start serving at startup automatically.

    me@pi:~$ sudo nano /lib/systemd/system/usbipd.service

        [Unit]
        Description=usbip host daemon
        After=network.target

        [Service]
        Type=forking
        ExecStart=/usr/sbin/usbipd -D
        ExecStartPost=/bin/sh -c "/usr/sbin/usbip bind --$(/usr/sbin/usbip list -p -l | grep '#usbid=10c4:8a2a#' | cut '-d#' -f1)"
        ExecStop=/bin/sh -c "/usr/lib/linux-tools/$(uname -r)/usbip detach --port=$(/usr/lib/linux-tools/$(uname -r)/usbip port | grep '<port in use>' | sed -E 's/^Port ([0-9][0-9]).*/\1/')"

        [Install]
        WantedBy=multi-user.target

To start the new SERVER:

    me@pi:~$ sudo pkill usbipd                          // End the current server daemon (if any)
    me@pi:~$ sudo systemctl --system daemon-reload      // Reload system jobs because one changed
    me@pi:~$ sudo systemctl enable usbipd.service       // Set to run at startup
    me@pi:~$ sudo systemctl start usbipd.service        // Run now

Add a systemd job to the CLIENT to automatically attach the remote USB devices at startup. You can use systemd to unplug conveniently before sleeping, and to reset the connection of needed. Note: On the "ExecStart" line, substitute your server's IP address for aa.bb.cc.dd in two places.

    me@laptop:~$ sudo nano /lib/systemd/system/usbip.service

        [Unit]
        Description=usbip client
        After=network.target

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/bin/sh -c "/usr/bin/usbip attach -r aa.bb.cc.dd -b $(/usr/bin/usbip list -r aa.bb.cc.dd | grep '10c4:8a2a' | cut -d: -f1)"
        ExecStop=/bin/sh -c "/usr/bin/usbip detach --port=$(/usr/bin/usbip port | grep '<port in use>' | sed -E 's/^Port ([0-9][0-9]).*/\1/')"

        [Install]
        WantedBy=multi-user.target

To start the new CLIENT attachment(s):

    me@laptop:~$ sudo systemctl --system daemon-reload      // Reload system jobs because one changed
    me@laptop:~$ sudo systemctl enable usbip.service       // Set to run at startup
    me@laptop:~$ sudo systemctl start usbip.service        // Run now