Tuesday, November 29, 2011

Firewall improvement with fwknop

fwknop is an improvement upon the older technique of port-knocking. Quite simply, it looks like an ordinary failed attempt to connect blocked by a closed firewall...but something about the packet identifies it as trustable, and causes the host system to do an action (like briefly open a firewall port).

There are safeguards to prevent the packet from being spoofed, and until the packet is received my SSH port stays safely blocked by the firewall - preventing the use of clever undiscovered exploits.

This is pretty cool because it means that my ssh port can be safely closed behind a firewall, yet still accessible to me (and only me) on the internet.

There are a whole bunch of moving pieces here (firewall, ports, services, client, keys), so I'll talk a lot of detail about the installation and configuration, the some options, then try an example or two. I haven't even touched the options for using my GPG-key for knocking!


Planning: Ahead of time, fill in this list:
  • I want the knock to occur on UDP port #: (default 62201)
  • I want be able to open ports for the following services: (example: sshd)
  • Those services run on ports: (example: tcp/22)
  • I want my knock password (key) to be: (8 characters min -or- GPG)
  • I want to be notified of a successful knock by: (log/email/none)
  • The interface I want to protect is: (only one, usually the same internet-facing interface described in your firewall)
Install and configure fwknop on the server:
  1. Server Install:
    apt-get install fwknop-server   # Debian
  2. Enable fwknopd to autostart upon boot. Edit /etc/default/fwknop-server
    START_DAEMON="no"  #From
    START_DAEMON="yes" #To
  3. Customize the server setup by editing /etc/fwknop/fwknop.conf:
    EMAIL_ADDRESSES             root@localhost;  # From
    EMAIL_ADDRESSES             MyLoginAccountName@localhost;  #to
    
    PCAP_INTF                   eth0;  # From
    PCAP_INTF                   ppp0;  # To
    
    PCAP_FILTER                 udp port 62201;                         # From
    PCAP_FILTER                 udp port <some other port number>; # To
    PCAP_INTF should be the same internet-facing interface used in your firewall (for example, ppp0). You can only choose ONE interface. In this case, ppp0 protects the internet-facing interface only so it does not protect against, say, hackers on the LAN wifi.
    PCAP_FILTER is the udp port to listen for the knock on. Remember that PCAP_FILTER port number - you'll need that again on the client.
  4. Set the server access controls and keys by editing /etc/fwknop/access.conf:
    # Change all three of the following, if needed.
    OPEN_PORTS: tcp/22,tcp/23,tcp/24,udp/85;
    KEY: <Your-8-digit-key>;
    FW_ACCESS_TIMEOUT: 30;
    OPEN_PORTS is the list of ports fwknopd is permitted to open. The client will request a specific port on the list. Remember that 8-digit key, you'll need it again on the client.
  5. Edit the firewall to close all fwknop-protected ports. The fwknopd daemon will tell iptables to open the port as needed.
    iptables -A INPUT -i lan0 -p tcp -dport 22 -j ACCEPT
    iptables -A INPUT -i ppp0 -p tcp -dport 22 -j REJECT
    The first line opens the firewall for ssh from the LAN (not protected by fwknopd). The second line closes the firewall for ssh from the internet-facing ppp0. If the default policy is already DROP or REJECT, you don't need a separate entry at all and the second line can be deleted. fwknopd can open a port regardless of the default policy or any explicit closure rule.
  6. Restart fwknopd (and optionally the firewall) to load the new config(s):
    fwknopd -Restart
Optional Server Notes:
  1. E-mail spamming of successful knocks: I got tired of that pretty fast, so I moved the notifications to a log:
    • In /etc/fwknop/fwknop.conf, change ALERTING_METHODS from ALL; to noemail;
    • Create a new file /etc/rsyslog.d/fwknop.conf:
      # Log fwknopd actions to syslog, auth.log, and iptables.log
      :msg, contains, "fwkno" -/var/log/auth.log
      :msg, contains, "fwkno" -/var/log/iptables.log #my own custom iptables-related log
    • Restart rsyslogd to reload the new config: service rsyslog restart
  2. Server Testing Tip: To prevent locking yourself out of the server (and feeling like a chump), here's a way to test fwknop without erroneously locking yourself out:
    • Edit /etc/ssh/sshd.conf to add a second ssh port (let's say port 2233)
    • Restart sshd to load the new config: service ssh restart
    • Use netstat -tulpn to ensure sshd is really listening on both ports
    • Edit your firewall so the new port (2233) is either explicitly closed or closed by the default policy (it doesn't matter which as long as it's closed). If necessary, reload your firewall to apply the changes.
    • Configure fwknopd to listen on your LAN interface, and only to control the ssh test port.
    • Do your testing and experimenting. Your original ssh port is unaffected.
    • When testing is complete, edit /etc/ssh/sshd.conf to remove the testing port, remove any testing-port related lines in your firewall, and edit /etc/fwknopd/access.conf to remove the test port.
Install and configure fwknop on the client:
  1. Client Install:
    sudo apt-get install fwknop-client   # Ubuntu
  2. Usage with manually-entered key:
    $fwknop --Server-port 12345 --Access 'tcp/22' -R --Destination myserver.dyndns.org
    Enter your key: <Your 8-digit key>
    $ssh -p 22 myserver.dyndns.org
    Server-port is the PCAP_FILTER, the UDP port to send the knock on.
    Access is the firewall port to open. It must be on the server's OPEN_PORTS list.
    The 8-digit key must be the same, of course, as on the server.
    The followup ssh command to connect on the briefly-open port is just an example.
  3. Create a keyfile for automatic use (optional): Use the format servername: key
    echo "myserver.dyndns.org: <Your 8-digit-key> > .knock-keyfile
  4. Usage with keyfile key (optional - handy for scripting):
    $fwknop --Server-port 12345 --Access 'tcp/22' -R --Destination myserver.dyndns.org --get-key .knock-keyfile
    $ssh -p 22 myserver.dyndns.org

Booting an old VIA EPIA-5000L motherboard

I purchased an old VIA fanless mini-ITX motherboard on eBay, including CPU and RAM and shipping, for a reasonable price ($43). I want to play with it and see what I can get it to do. Maybe build a cool case for it...

MB attached to ordinary ATX power supply, VGA monitor, and PS/2 Mouse.
Tiny black rectangle on the right is the on/off jumper.
Step 1: Hook it up

Happily, I have an old PS/2 keyboard, VGA monitor, and ATX power supply handy.
Hook them all up...and nothing.

Of course, the power switch.
Get the manual, read the manual, find the jumper, and use a spare jumper to connect them for a moment.

Hooray! Fans on the power supply start to spin! Beep! Splash appears on the monitor!

I made the following BIOS changes:
  • Always restart upon power loss (since I don't have a power switch!)
  • Enable fast startup (skip memtest, faster boot)
  • Skip splash (annoying)
  • Skip BIOS summary (annoying)
  • Change the boot device to USB-ZIP (not USB-FDD, essential for USB-boot later)


Step 2: Create a Debian Boot USB drive.

This is a pretty old motherboard, so the standard USB-HDD booting option is missing. That means Live-USB (hybrid USB/CD) .iso files won't work without some tweaking. The available USB options are USB-ZIP and USB-FDD. A quick web search turns up this page of instructions, but it doesn't work. Further digging reveals this page, a version of which does work.

  1. Select a USB Drive (small). Everything on it will be erased, so backup if needed.
  2. On a different (working) Debian or Ubuntu system, as root (use 'sudo su' to get a root prompt on Ubuntu). Not all commands will work with sudo alone.
  3. Plugin the USB drive. If your system automatically mounts it, then unmount (not eject) it.
  4. Use dmesg to find the USB drive's /dev/sd* node. For example, mine was sdz and sdz3.
  5. Format the USB drive to a bootable configuration. This will limit the accessible size to 250MB (the old ZIP-format maximum), so CD-sized live .isos won't work. See this page for mkdiskimage commands for other-size drives.
    mkdiskimage -4 /dev/sdz 0 64 32   # 1GB and below USB drives only
  6. Install the MBR:
    cat /usr/lib/syslinux/mbr.bin > /dev/sdz
  7. Get and install the boot image:
    wget -O /tmp/boot.img.gz http://ftp.debian.org/debian/dists/squeeze/main/installer-i386/current/images/hd-media/boot.img.gz
    zcat /tmp/boot.img.gz > /dev/sdz1
  8. Download the Live Session image. In this case, I used a Debian businesscard installer simply because it was a very small download:
    wget -O /tmp/debian-6.0.3-i386-businesscard.iso http://cdimage.debian.org/debian-cd/6.0.3/i386/iso-cd/debian-6.0.3-i386-businesscard.iso
    md5sums /tmp/debian-6.0.3-i386-businesscard.iso
    Compare the md5sum result to http://cdimage.debian.org/debian-cd/6.0.3/i386/iso-cd/MD5SUMS
  9. Mount the USB drive, and install the Live Session image:
    mkdir /tmp/usbmountpoint
    mount /dev/sdz3 /tmp/usbmountpoint
    cp /tmp/debian-6.0.3-i386-businesscard.iso /tmp/usbmountpoint/
    umount /tmp/usbmountpoint
  10. Check the resulting filesystem, and correct any errors (there may be a couple!)
    fsck.msdos -r /dev/sdz3
    My resulting filesystem is about 214MB, below the 250MB limit.
  11. Clean-up is optional, since all working files are in /tmp and will be deleted upon reboot anyway.
  12. Unplug the USB drive (it should already be unmounted)


Step 3: Boot Debian

The BIOS should be set to a startup device of USB-ZIP (step 1), and you should have a Debian USB drive (step 2). Put them together - try to boot the new motherboard from it. In my case, success!

I don't have a hard drive installed yet, so I'm not ready to actually install. But this was a successful power-on and boot.


UPDATE #1: For Xmas, one of the kids got a toy butterly-in-a-jar. It has a "Try Me" feature for in the store, a button that detached after opening the package. The button's connector looked just right to fit the power jumper, and indeed now I have a real power button for the motherboard. This turns out to be handy, because a stable system needs better power control than flipping the switch on the power supply...and my new silent power supply has no switch!

UPDATE #2: A chainloader can be used to bypass the USB-ZIP 250MB boot limit. I did a successful USB-ZIP boot to a complete Debian 6 system.

Sunday, November 27, 2011

GPS dongle, gpsd, and Ubuntu 11.10

This post is an update to my original experiment #1 and experiment #2 with a USB GPS dongle two years ago.

Install the hardware: Plug it in, of course.
  • Check the kernel message log to discover the GPS dongle location in the filesystem:
    $ dmesg | grep tty
    [1769190.919411] usb 2-1: pl2303 converter now attached to ttyUSB0
  • Test for signal with the following command:
    $ cat /dev/ttyUSB0 
    5�y�220,N,08754.5061,W,000.0,316.5,271111,,,A*71
    $GPVTG,316.5,T,,M,000.0,N,000.0,K,A*0C
    $GPGGA,173905.000,4259.7220,N,08754.5060,W,1,08,1.0,190.0,M,-33.8,M,,0000*63
    $GPGSA,A,3,12,04,10,23,02,05,17,25,,,,,1.7,1.0,1.4*31
    $GPRMC,173905.000,A,4259.7220,N,08754.5060,W,000.0,316.5,271111,,,A*71
    
    Use CTRL+C to end the signal check.
    If you get a bunch of zero-data, then try moving closer to a window, since GPS signals are line-of-sight. Or try a USB hub or extension cable, to reduce interference from your computer.

Install the gps daemon so you can use gps data.
  • sudo apt-get install gpsd gpsd-clients
  • Normally, udev will start gpsd automatically when a USB GPS device is inserted. If it fails to start after 5-10 seconds, you can start it manually:
    gpsd /dev/ttyUSB0  #You must tell gpsd where to look for the dongle
  • Test the GPS and gpsd output using the gpsmon and cgps terminal programs, or the xgps program (all included in the gpsd-clients package)
  • Capture data for recycling (optional - handy for development). gpsfake will simulate gpsd output
    $cat /dev/ttyUSB0 > path/to/datalog  # Create data file. Use CTRL+C to end the capture.
    $gpsfake path/to/datalog             # Run gpsd simulator (not a daemon - it will occupy the terminal)

Command-line tools to get GPS data
  • Use the gpspipe command to get gpsd data:
    $ gpspipe -w -n 5
    netlib_connectsock() returns socket on fd 3
    {"class":"VERSION","release":"2.95","rev":"2011-07-27T11:20:24","proto_major":3,"proto_minor":3}
    {"class":"DEVICES","devices":[{"class":"DEVICE","path":"/dev/ttyUSB0"}]}
    {"class":"WATCH","enable":true,"json":true,"nmea":false,"raw":0,"scaled":false,"timing":false}
    {"class":"TPV","tag":"RMC","device":"/dev/ttyUSB0","time":1322418390.001,"ept":0.005,"lat":42.995410000,"lon":-87.908430000,"alt":195.400,"epx":11.739,"epy":17.294,"epv":36.800,"track":342.2000,"speed":0.000,"climb":0.000,"eps":34.59,"mode":3}
    {"class":"SKY","tag":"GSV","device":"/dev/ttyUSB0","xdop":0.66,"ydop":1.15,"vdop":1.46,"tdop":1.01,"hdop":1.32,"gdop":2.22,"pdop":1.97,"satellites":[{"PRN":2,"el":77,"az":360,"ss":26,"used":true},{"PRN":10,"el":58,"az":72,"ss":32,"used":true},{"PRN":12,"el":51,"az":247,"ss":34,"used":true},{"PRN":5,"el":45,"az":192,"ss":21,"used":true},{"PRN":4,"el":40,"az":65,"ss":20,"used":true},{"PRN":25,"el":39,"az":292,"ss":21,"used":true},{"PRN":13,"el":15,"az":60,"ss":18,"used":true},{"PRN":24,"el":14,"az":258,"ss":0,"used":false},{"PRN":29,"el":12,"az":304,"ss":23,"used":false},{"PRN":17,"el":8,"az":121,"ss":17,"used":true},{"PRN":23,"el":7,"az":35,"ss":10,"used":false},{"PRN":31,"el":0,"az":334,"ss":0,"used":false}]}
  • Let's refine the command a bit, limit to the first "TPV" (time, position, velocity) sentence:
    $ gpspipe -w -n 5 | grep -m 1 TPV
    netlib_connectsock() returns socket on fd 3
    {"class":"TPV","tag":"RMC","device":"/dev/ttyUSB0","time":1322418481.001,"ept":0.005,"lat":42.995406667,"lon":-87.908428333,"alt":195.300,"epx":10.003,"epy":17.383,"epv":36.800,"track":342.2000,"speed":0.000,"climb":0.000,"eps":34.77,"mode":3}
  • We can use basic shell commands to extract single variables from this sentence:
    tpv=$(gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4,6-8,13)
    seconds=$(echo $tpv | cut -d, -f1 | cut -d: -f2)
    latitude=$(echo $tpv | cut -d, -f2 | cut -d: -f2)
    longitude=$(echo $tpv | cut -d, -f3 | cut -d: -f2)
    altitude=$(echo $tpv | cut -d, -f4 | cut -d: -f2)
    speed=$(echo $tpv | cut -d, -f5 | cut -d: -f2)
  • For fun, let's compare GPS time with system time using shell commands:
    date -u +%s   # System utc time in seconds
    gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4 | cut -d: -f2   # GPS utc time in seconds
    
    # Let's compare them side-by side
    $ echo System: $(date -u +%s) , GPS: $(gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4 | cut -d: -f2) 
    netlib_connectsock() returns socket on fd 3
    System: 1322419689 , GPS: 1322419689.000
    # Our sytem time is within one second of GPS time. That's good!
  • For fun, let's use shell script to post the GPS location to Google Maps:
    The format is simple enough: http://maps.google.com/?ll=40.480381,-92.373047&z=16 will return a valid map. So that's three variables: Latitude, longitude, and zoom Level.
    tpv=$(gpspipe -w -n 5 | grep -m 1 TPV | cut -d, -f4,6-8,13)
    latitude=$(echo $tpv | cut -d, -f2 | cut -d: -f2)
    longitude=$(echo $tpv | cut -d, -f3 | cut -d: -f2)
    zoom=20
    map_url="http://maps.google.com/?ll=${latitude},${longitude}&z=${zoom}"
    firefox $map_url

Python tools to get GPS data
  • Python bindings to gpsd are provided by the python-gps package.
    sudo apt-get install python-gps
  • The python tools have changed a *lot* in two years. The old methods don't work anymore. No Python 3 version is available yet. There's a decent example here, but I couldn't get it to work.

Conversion between Lat/Lon and MGRS/UTM and other systems: gpsd doesn't do this, though apparently the geographiclib-tools package does. And so does this website.

Saturday, November 26, 2011

Ubuntu Brainstorm - inside the sausage factory

Brainstorm gets about 2,000 submissions each year.
What do we do with all those ideas?
How can they help you, without bogging you down?

Well, we close about 85% of them immediately. That's good - it keeps the system from being spammed by all those bugs, complaints, and other not-really-ideas.

About 2% get reviewed by the Ubuntu Technical Board. We try to get the most out of that 2%!

Most new Brainstormers are new to the Ubuntu community - they find Brainstorm in their first excursion looking for a way to contribute. A Brainstorm idea is their first attempt to speak up, so we try to handle them carefully.

(If you were, er, untactfully treated by any moderator in the past, then please accept my most humble apologies. If you find a recent idea we handled tactlessly, *please* let me know!)


The Idea Sandbox

Of course, most of those first-ideas aren't very good. My first idea was awful, too.
- Make it more like my old OS
- Fix this bug
- This doesn't work very well
- Hey, do this. It's self-evident
- Substitute application X for application Y in the default install
- Create a whole new distro for a very small audience
- Develop a whole new application
- I don't like this
- Someone should be in charge of this
- Make this incredibly minor change to one application
- If only my perfect idea would reach SABDFL, *he* would appreciate it, though none of you other fools do.

In the Sandbox, we gently address every idea. Is it a bug in the wrong venue? Is it a dispute or complaint in the wrong venue? Does it clearly describe a real problem that has multiple possible solutions? Has the problem already been solved some other way? Is this a duplicate? Was it discussed by the community before (and the result)? Is the submitter still interested in the idea, or was it a fire-and-forget notion? Is the issue handled within an existing project or team, or does it cross boundaries?

Most submitters, of course, abandon their idea when they realize that nobody will implement it for them...or that they need to do some research or file a bug report or otherwise become a contributing community member.

We do a lot of basic instruction on how the Ubuntu software ecosystem works, and try to change many of them from a customer-mindset to a community-member mindset. That's a hard transition, and we're still trying to find better ways to do it.

This is an opportunity to recruit new members for YOUR project/team, and turn a lot of potential complainers into contributors and advocates. We're trying, and good ideas on improving our methods are welcome. So are more drop-ins from other teams - come in, answer a few project-specific questions, and leave with the contacts for a half-dozen new people interested in your subject.


30-day voting

15% of that 2000 pass the Sandbox - the problem statement is real and clear, and it won't waste the community's time. Some of that 15% is marginal - the problem is real enough, but needs community input to refine.

We leave those marginal ones open for only 30 days, to give the submitter the public feedback they want. These have the clear understanding that we're leaving it open for feedback to help refine the issue - it won't be seen by the Ubuntu Technical Board.

30-day wonders are also helpful to developers who want to use Brainstorm for project-related polling or questions. Just ping a moderator in #ubuntu-brainstorm so we can skip it past the Sandbox for you.


6-month voting

Around 10% of the original 2000 are left open for the full 6-month voting period (and, indeed, left open long after that. We have still-open ideas from 2008).

If you find a question that's relevant to your project or team, and have an answer, please ping a moderator in #ubuntu-brainstorm, so we can put your answer in the "Developer Comments" section of the idea, and close it for you.

Similarly, if the issue has been raised and answered somewhere else (Launchpad, AskUbuntu, Blog Post), just post the link to the answer, and flag the idea so we know to close it.


The Ubuntu Technical Board

Every three months, the UTB selects 10 of the highest-voted ideas to review. 40 a year out of 2000 initial submissions, or around 2%. The 'review' isn't a promise to implement - indeed most result in a "hey, we agree this is a great idea, somebody in the community should implement this..." Some get added to UDS discussions, some turn into milestones for future releases. Some simply become bug reports.

Every idea that the UTB reviews that simply becomes a "hey, great" or a bug report is a wasted opportunity. You didn't need to wait three months for that kind of feedback - by being involved with the right project or team, or filing the bug report yourself, you could have done all that yourself!

We incorporate past UTB feedback into our moderation - for example we now routinely close ideas to create a whole new project unless they include a realistic resource plan. We encourage idea submitters to engage the appropriate bug tracker / upstream project / Ubuntu Team directly whenever possible. We want the best 40 new ideas before the UTB, real head-scratchers that will require research and create discussion and show up at UDS...and then as taskings.


We have resources for your project

Brainstorm doesn't need to be a filter for only the UTB - we can be a structured-community-feedback filter for *lots* of projects and teams. We have a lot of very smart and knowledgeable community members that help refine the ideas and solutions. If your project/team reviews Brainstorm input on a regular basis, please let us know. If you're looking for certain types of ideas, or want us to moderate your relevant ideas in a different way, just pop in to #ubuntu-brainstorm and let us know.


Are you ready for the challenge?

If you're an AskUbuntu or Forums or IRC guru looking for the next challenge to your technical and diplomatic skills, looking to help shape the next generation of contributors...well, come on by.

NTP on a Debian 6 server

This post is an update to my original in March 2011.


NTP is useful to set the time of LAN devices. The correct time is essential for several services. I installed ntp on an almost-always-connected server, which will in turn broadcast the correct time to other devices on my LAN.

internet ntp servers  <--->  my net ntp client / LAN ntp server  <--->  my LAN ntp clients


The ntp program is both client and server (those used to be separate packages). The same ntp gets installed on all devices. You change the settings in the /etc/ntp.conf file. The version of ntp on those devices isn't important - different versions of ntp interact well together (unlike, say, different versions of the deluge torrent client/server)


These instructions are for debian 6, running as root. For Ubuntu, you need to prepend most commands with 'sudo'


  1. Install NTP with the following command:
    apt-get install ntp
  2. Edit the server's /etc/ntp.conf file (if needed). If you want your device to get time from the default internet time servers (in most cases, this is exactly what you want), then skip this step and make no changes.

    In my case, however, my server is both a client of those upstream internet time servers AND a time server to my LAN. Your LAN probably does not need a timeserver - I'm doing this for fun.
    ## Around Line 48
    # If you want to provide time to your local subnet, change the next line.
    # (Again, the address is an example only.)
    #broadcast 192.168.123.255
    
    # Since my LAN is 192.168.1.0, the corresponding broadcast address is at 192.168.1.255
    broadcast 192.168.1.255
  3. No changes to /etc/hosts.allow and /etc/hosts.deny are needed to limit access to ntpd. Access controls are set by the /etc/ntp.conf file, they default to nobody has any access, and we didn't change those rules.
  4. xinted and dnsmasq do not interact with ntpd, and no special configuration is required.
  5. Firewall rules to open udp port 123 for ntp sync messages. Normally, this won't be necessary. Since the ntp client sends the first packet to the upstream internet server, and outbound packets usually are not blocked, the firewall should automatically let the server's response through.

    If an ntp client has a really locked-down firewall (blocking outgoing traffic), then open the client's UDP port 23 to outgoing traffic.
    iptables -A OUTPUT -p udp --sport 123 -j ACCEPT

    If you run a LAN-facing NTP server, then of course you must ensure that LAN-interface UDP port 23 is open to that same inbound traffic from your clients. (your -i interface name will be different)
    iptables -A INPUT -i lan0 -p udp --dport 123 -j ACCEPT 

    If you run an exposed internet-facing NTP server, then you should really be working with ntp.org for security, and to help support the ntp infrastructure!

    In my case, no firewall changes were needed at all. My firewall blocks inbound packets from the internet, not LAN-only packets, and not outbound-packets (or responses).
  6. Restart/update the firewall (how you do that is up to you), and then restart ntp to reload the changed config file. Let's test ntp to see if it's working:
    $ ntpq -p
         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    -w1-wdc.ipv4.got 10.0.77.54       4 u  396 1024  377   30.567    1.853   2.901
    *ntp3.junkemailf 209.81.9.7       2 u  465 1024  377   64.788   -0.072   1.269
    +vimo.dorui.net  208.90.144.52    3 u  633 1024  377   32.876   -1.081  30.206
    +aeolus.seobeo.c 193.67.79.202    2 u  184 1024  377  110.840    0.577   1.790
     192.168.1.255   .BCST.          16 u    -   64    0    0.000    0.000   0.000
    
    Looks correct. The first character of each row (*/+/-) is the status of each upstream server. See the explanation. The 'when' column is the number of seconds since the last check of that server. The'poll' column is the number of seconds between checks. The first four rows are upstream internet servers. The last row is the LAN broadcast, which is why it doesn't have an upstream status.
  7. The other ntp clients on the LAN need to be modified to listen for the new ntp server. Left alone, they will pull time from the upstream internet time servers (good). But, of course, I now have a server broadcasting time, so let's set each device on my LAN to listen for it. Plus, I still want them to get time from upstream in case my ntp server goes down or starts sending bad time.

    Log in to each device, and look at their /etc/ntp.conf (though it may be in a different location or names differently).

    Here's an example /etc/ntp.conf change from an Ubuntu system on the LAN, starting at around Line #15. We added the LAN ntp server, and marked it as the preferred source. We added minpoll to the internet servers to increase the time between pollings.
    # Specify one or more NTP servers.
    server kiwkak.dyndns.org prefer
    
    # Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
    # on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
    # more information.
    server 0.ubuntu.pool.ntp.org minpoll 12
    server 1.ubuntu.pool.ntp.org minpoll 12
    server 2.ubuntu.pool.ntp.org minpoll 12
    server 3.ubuntu.pool.ntp.org minpoll 12
    Restart ntp (in debian 'service ntp restart'. For ubuntu 'sudo service ntp restart') to reload the new configuration file.

    Let's also take a loot at their ntp status:
    $ ntpq -p
         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    *adsl-76-229-175 184.105.182.7    3 u   28   64  377    0.274    0.805   0.209
     ntp1.Rescomp.Be 169.229.128.214  3 u  838 1024    3   63.979    3.262   0.424
     cheezum.mattnor 129.7.1.66       2 u  818 1024    3   34.490    0.145  25.810
     host2.kingrst.c 204.9.54.119     2 u  789 1024    3   22.359    0.721  17.671
     vf1.bbnx.net    128.4.1.1        2 u  824 1024    3   68.227   -0.820   1.466
    

    This looks good. See how the local server is preferred, and gets checked more often than the backup upstream servers. And the backup upstreams are reachable, but not currently used.
  8. The server itself may not always be the network router/firewall. I automagically switching between server between router and non-router roles using my kingbaron script. I use a script and two config files to change how ntp runs based on if the system is a router (Network King, runlevel 3) or just a LAN client (Faithful Baron, runlevel 4). The decision logic operates in runlevel 2.

    Create two .conf files, one for each runlevel. Let's call them /etc/ntp-king.conf and /etc/ntp-baron.conf. Look at all the .conf stuff above for the contents of those files.

    Use the following command once to automatically prevent ntp from starting in runlevel 2 (the decision logic level), and to stop ntp if you manually switch to runlevel 2 due to a loss of connectivity or for maintenance:
    update-rc.d ntp stop 2

    Add the following patch to /etc/init.d/ntp to restart ntp with the appropriate config files upon runlevel changes.
    test -x $DAEMON || exit 5    # Insert after this line
    
    current_runlevel=echo $(runlevel | cut -d' ' -f2)
    if [ current_runlevel -eq "3" ]; then
         NTPD_OPTS="$NTPD_OPTS -c /etc/ntp-king.conf"
    elif [ current_runlevel -eq "4" ]; then
         NTPD_OPTS="$NTPD_OPTS -c /etc/ntp-baron.conf"
    
    This covers the common use cases (booting, manual force of decision logic), but doesn't cover everything - if I manually switch to runlevels 3/4/5, ntp won't automatically check the config.

Saturday, November 19, 2011

The Unexpected Success of Ubuntu Brainstorm

Once upon a time, there was a busy, vibrant community...with some communication issues. Yes, I'm talking about the Ubuntu community. Back in 2007.

So Ubuntu created a website to foster community discussion about ideas to improve Ubuntu. http://brainstorm.ubuntu.com

Conventional Wisdom from the old-timers who remember those heady days is that it didn't work out. The site was quickly overwhelmed with bugs (not ideas), complaints (not ideas), and unrealistic expectations. It didn't work.

And Conventional Wisdom had those facts right...but reached entirely the wrong conclusion! Brainstorm has been successful in ways that were unexpected at the time.


1) Brainstorm is indeed lousy at directly turning submissions into code.

Well, we knew this already, didn't we? There is no roomful of monkeys poised over their keyboards eagerly awaiting the next idea to begin cranking out Shakesperian code. There never was. There never will be. Implementation was never really expected to be a metric of success...community action was supposed to be the metric.


2) Brainstorm is a great early-warning system for controversy.

What? You can avoid getting blindsided? Brainstorm is awesome at reflecting the community's (lack of) consensus on a wide range of issues. For example, when 'close window' window button changed sides, the ideas on the topic were plentiful. Similarly, today most ideas are about Unity. Brainstorm is also a pretty good barometer of whether an issue is temporary or chronic, and our moderators are excellent at separating valid issues from mere whining (so you don't have to).


3) Brainstorm is a great place to recruit enthusiastic new members.

The website is a portal-of-entry for users who are just starting to get involved, but don't understand the community's structure yet, or how the various teams and projects interact.


4) Brainstorm keeps project community input manageable.

If you are involved with a project, you know how challenging it is to manage certain types of community feedback. Brainstorm has reviewers and moderators who do nothing but handle those tough cases. Better yet, it's set up to let the community itself moderate feedback for you. And it's set up so that you only answer an issue once...ever.


5) Brainstorm finds your blind spots.

Maybe it's documentation, maybe it's a workflow, maybe it's a use case, maybe it's a legacy, maybe it's something else. The trend of feedback across years helps you improve user experience.


So, in the spirit of Ubuntu Community Appreciation day,

- Thanks to nand for writing and maintaining Ideatorrent.

- Thanks to the best moderators and reviewers in the world: Vahan Harutyunyan, DarwinSurvivor, Komputes, DrG, alourie, andruk, and Thonixx

Your valiant efforts to raise the quality of discussions and ideas, and to uphold the Ubuntu Code of Conduct, challenge me to strive. I'm very proud to work with people of such high caliber.

Tuesday, November 1, 2011

Move IPTables log events to a separate logfile

Today some botnet tried to connect to my server over 26,000 times in five hours. They might still be trying.

I have strong firewall protection, and I log all those dropped packets from the firewall. but the records of more than 26,000 dropped packets is filling my syslog and making it unusable.

I used the instructions here to shift that reporting to a separate iptables log, plus enabled logrotate so it gets changed out daily.