I upgraded my home internet connection to fibre (FTTP) last October. I’m still on an 80M/20M service, so it’s no faster than my old VDSL FTTC connection was, and as a result for a long time I continued to use my HomeHub 5A running OpenWRT. However the FTTP ONT meant I was using up an additional ethernet port on the router, and I was already short, so I ended up with a GigE switch in use as well. Also my wifi is handled by a UniFi, which takes its power via Power-over-Ethernet. That mean I had a router, a switch and a PoE injector all in close proximity. I wanted to reduce the number of devices, and ideally upgrade to something that could scale once I decide to upgrade my FTTP service speed.

Looking around I found the MikroTik RB3011UiAS-RM, which is a rack mountable device with 10 GigE ports (plus an SFP slot) and a dual core Qualcomm IPQ8064 ARM powering it. There’s 1G RAM and 128MB NAND flash, as well as a USB3 port. It also has PoE support. On paper it seemed like an ideal device. I wasn’t particularly interested in running RouterOS on it (the provided software), but that’s based on Linux and there was some work going on within OpenWRT to add support, so it seemed like a worthwhile platform to experiment with (what, you expected this to be about me buying an off the shelf device and using it with only the supplied software?). As an added bonus a friend said he had one he wasn’t using, and was happy to sell it to me for a bargain price.

I did try out RouterOS to start with, but I didn’t find it particularly compelling. I’m comfortable configuring firewalling and routing at a Linux command line, and I run some additional services on the router like my MQTT broker, and mqtt-arp, my wifi device presence monitor. I could move things around such that they ran on the house server, but I consider them core services and as a result am happier with them on the router.

The first step was to get something booting on the router. Luckily it has an RJ45 serial console port on the back, and a reasonably featured bootloader that can manage to boot via tftp over the network. It wants an ELF binary rather than a plain kernel, but Sergey Sergeev had done the hard work of getting u-boot working for the IPQ8064, which mean I could just build normal u-boot images to try out.

Linux upstream already had basic support for a lot of the pieces I was interested in. There’s a slight fudge around AUTO_ZRELADDR because the network coprocessors want a chunk of memory at the start of RAM, but there’s ongoing discussions about how to handle this cleanly that I’m hopeful will eventually mean I can drop that hack. Serial, ethernet, the QCA8337 switches (2 sets of 5 ports, tied to different GigE devices on the processor) and the internal NOR all had drivers, so it was a matter of crafting an appropriate DTB to get them working. That left niggles.

First, the second switch is hooked up via SGMII. It turned out the IPQ806x stmmac driver didn’t initialise the clocks in this mode correctly, and neither did the qca8k switch driver. So I need to fix up both of those (Sergey had handled the stmmac driver, so I just had to clean up and submit his patch). Next it turned out the driver for talking to the Qualcomm firmware (SCM) had been updated in a way that broke the old method needed on the IPQ8064. Some git archaeology figured that one out and provided a solution. Ansuel Smith helpfully provided the DWC3 PHY driver for the USB port. That got me to the point I could put a Debian armhf image onto a USB stick and mount that as root, which made debugging much easier.

At this point I started to play with configuring up the device to actually act as a router. I make use of a number of VLANs on my home network, so I wanted to make sure I could support those. Turned out the stmmac driver wasn’t happy reconfiguring its MTU because the IPQ8064 driver doesn’t configure the FIFO sizes. I found what seem to be the correct values and plumbed them in. Then the qca8k driver only supported port bridging. I wanted the ability to have a trunk port to connect to the upstairs switch, while also having ports that only had a single VLAN for local devices. And I wanted the switch to handle this rather than requiring the CPU to bridge the traffic. Thankfully it’s easy to find a copy of the QCA8337 datasheet and the kernel Distributed Switch Architecture is pretty flexible, so I was able to implement the necessary support.

I stuck with Debian on the USB stick for actually putting the device into production. It makes it easier to fix things up if necessary, and the USB stick allows for a full Debian install which would be tricky on the 128M of internal NAND. That means I can use things like nftables for my firewalling, and use the standard Debian packages for things like collectd and mosquitto. Plus for debug I can fire up things like tcpdump or tshark. Which ended up being useful because when I put the device into production I started having weird IPv6 issues that turned out to be a lack of proper Ethernet multicast filter support in the IPQ806x ethernet device. The driver would try and setup the multicast filter for the IPv6 NDP related packets, but it wouldn’t actually work. The fix was to fall back to just receiving all multicast packets - this is what the vendor driver does.

Most of this work will be present once the 5.9 kernel is released - the basics are already in 5.8. Currently not queued up that I can think of are the following:

• stmmac IPQ806x FIFO sizes. I sent out an RFC patch for these, but didn’t get any replies. I probably just need to submit this.
• NAND. This is missing support for the QCOM ADM DMA engine. I’ve sent out the patch I found to enable this, and have had some feedback, so I’m hopeful it will get in at some point.
• LCD. AFAICT LCD is an ST7735 device, which has kernel support, but I haven’t spent serious effort getting the SPI configuration to work.
• Touchscreen. Again, this seems to be a zt2046q or similar, which has a kernel driver, but the basic attempts I’ve tried don’t get any response.
• Proper SFP functionality. The IPQ806x has a PCS module, but the stmmac driver doesn’t have an easy way to plumb this in. I have ideas about how to get it working properly (and it can be hacked up with a fixed link config) but it’s not been a high priority.
• Device tree additions. Some of the later bits I’ve enabled aren’t yet in the mainline RB3011 DTB. I’ll submit a patch for that at some point.

Overall I consider the device a success, and it’s been entertaining getting it working properly. I’m running a mostly mainline kernel, it’s handling my house traffic without breaking a sweat, and the fact it’s running Debian makes it nice and easy to throw more things on it as I desire. However it turned out the RB3011 isn’t as perfect device as I’d hoped. The PoE support is passive, and the UniFi wants 802.1af. So I was going to end up with 2 devices. As it happened I picked up a cheap D-Link DGS-1210-10P switch, which provides the PoE support as well as some additional switch ports. Plus it runs Linux, so more on that later…

Yesterday I did the first release of my OpenPGP compatible keyserver, onak, in 4 years. Actually, 2 releases because I discovered my detection for various versions of libnettle needed some fixing.

It was largely driven by the need to get an updated package sorted for Debian due to the removal of dh-systemd, but it should have come sooner. This release has a number of clean-ups for dealing with the hostility shown to the keyserver network in recent years. In particular it implements some of dkg’s Abuse-Resistant OpenPGP Keystores, and finally adds support for verifying signatures fully. That opens up the ability to run a keyserver that will only allow verifiable updates to keys. This doesn’t tie in with folk who want to run PGP based systems because of the anonymity, but for those of us who think PGP’s strength is in the web of trust it’s pretty handy. And it’s all configurable to taste; you can turn off all the verification if you want, or verify everything but not require any signatures, or even enable v3 keys if you feel like it.

The main reason this release didn’t come sooner is that I’m painfully aware of the bits that are missing. In particular:

• Documentation. It’s all out of date, it needs a lot of work.
• FastCGI support. Presently you need to run the separate CGI binaries.
• SKS Gossip support. onak only supports the email syncing option. If you run a stand alone server this is fine, but Gossip is the right approach for a proper keyserver network.

Anyway. Available locally or via GitHub.

0.6.0 - 13th September 2020

• Move to CMake over autoconf
• Add support for issuer fingerprint subpackets
• Add experimental support for v5 keys
• Add read-only OpenPGP keyring backed DB backend
• Move various bits into their own subdirectories in the source tree
• Add support for full signature verification
• Drop v3 keys by default when cleaning keys
• Various code cleanups
• Implement pieces of draft-dkg-openpgp-abuse-resistant-keystore-03
• Add support for a fingerprint blacklist (e.g. Evil32)
• Deprecate the .conf configuration file format
• Drop version info from armored output
• Add option to deny new keys and only allow updates to existing keys
• Various pieces of work removing support for 32 bit key IDs and coping with colliding 64 bit key IDs.
• Remove support for libnettle versions that lack the full SHA2 suite

0.6.1 - 13th September 2020

• Fixes for compilation without nettle + with later releases of nettle

One of the things I maintain in Debian is OpenOCD. I say maintain, but it’s so far required very little work, as it’s been 3 years since a release (0.10.0). I’ve talked about doing a git snapshot package for some time (I have an email from last DebConf in my inbox about it, and that wasn’t the first time someone had asked), but never got around to it. Spurred on by some moves towards a 0.11.0 release I’ve built a recent snapshot and uploaded it to the experimental suite in Debian.

Of particular interest is the support for more recent architectures that this brings - ARMv8/aarch64 and RISC-V being the big ones, but also MIPS64 and various other ARM improvements. I no longer have access to Xilinx Zynq or Mellanox Bluefield platforms to test against so I’ve just done some some basic tests with a Sheevaplug and BusPirate/STM32F103, but those worked just fine.

Builds should hopefully happen shortly. Enjoy!

A SYNCNI article passed by on my Twitter feed this morning, talking about balancing work life balance while working from home in these times of COVID-19 inspired lock down. The associated Twitter thread expressed an interest in some words of advice from men to other men (because of course the original article has the woman having to do all the balancing).

This post does not contain the words of advice searched for, but it hopefully at least offers some reassurance that if you’re finding all of this difficult you’re not alone. From talking to others I don’t think there’s anything particularly special we’re doing in this house; a colleague is taking roughly the same approach, and some of the other folk I’ve spoken to in the local tech scene seem to be doing likewise.

First, the situation. Like many households both my wife and I work full time. We have a small child (not even a year and a half old yet). I work for a software startup, my wife is an HR business partner for a large multinational, dealing with employees all over the UK and Ireland. We’re both luckily able to work from home easily - our day to day work machines are laptops, our employers were already setup with the appropriate VPN / video conferencing etc facilities. Neither of us has seen any decrease in workload since lock down; there are always more features and/or bugs to work on when it comes to a software product, and, as I’m sure you can imagine, there has been a lot more activity in the HR sphere over the past 6 weeks as companies try to work out what to do.

On top of this our childcare arrangements, like everyone else’s, are completely gone. Nursery understandably shut down around the same time as the schools (slightly later, but not by much) and contact with grandparents is obviously out (which they’re finding hard). So we’re left with trying to fit 2 full time jobs in with full time childcare, of a child who up until recently tried to go down stairs by continuing to crawl forward.

Make no mistake, this is hard. I know we are exceptionally lucky in our situation, but that doesn’t mean we’re finding it easy. We’ve adopted an approach of splitting the day up. I take the morning slot (previously I would have got up with our son anyway), getting him up and fed while my wife showers. She takes over for a bit while I shower and dress, then I take over again in time for her to take her daily 8am conference call.

My morning is mostly taken up with childcare until nap time; I try to check in first thing to make sure there’s nothing urgent, and get a handle on what I might have to work on later in the day. My local team mates know they’re more likely to get me late morning and it’s better to arrange meetings in the afternoon. Equally I work with a lot of folk on the US West coast, so shifting my hours to be a bit later is not a problem there.

After nap time (which, if we’re lucky, takes us to lunch) my wife takes over. As she deals with UK/Ireland folk she often ends up having to take calls even while looking after our son; generally important meetings can be moved to the morning and meetings with folk who understand there might be a lot of pot banging going on in the background can happen in the afternoon.

Having started late I generally work late - past the point where I’d normally get home; if I’m lucky I pop my head in for bath time, but sometimes it’s only for a couple of minutes. We alternate cooking; usually based on work load + meetings. For example tonight I’m cooking while my wife catches up on some work after having put our son to bed. Last night I had a few meetings so my wife cooked.

So what’s worked for us? Splitting the day means we can plan with our co-workers. We always make sure we eat together in the evening, and that generally is the cut-off point for either of us doing any work. I’m less likely to be online in the evening because my study has become the place I work. That means I’m not really doing any of my personal projects - this definitely isn’t a case of being at home during lock down and having lots of time to achieve new things. It’s much more of case of trying to find a sustainable way to get through the current situation, however long it might last.

I’m working on trying to get a Qualcomm IPQ8064 device working properly. I’d got as far as an extremely hacked up chain of networking booting and the serial console working, so the next stage was to try and get a basic userland in place to try and make poking things a bit easier. While I have a cross-compiling environment all setup (that’s how I’m building my kernels) I didn’t really want to get into something complicated just to be able to poke around sysfs etc. I figured I could use a static BusyBox in an initramfs. I was right, but there were a couple of hiccups along the way so I’m writing it up so I remember next time.

First, I cheated and downloaded a prebuilt static binary from https://busybox.net/downloads/binaries/ (very convenient, thanks) - the IPQ8064 is a dual core ARMv7 so I used busybox-armv7l. There were various sites with init scripts to go along with this; I ended up with a variation of Star Brilliant’s gist:

#!/bin/busybox sh

busybox mkdir -p /dev /etc /proc /root /sbin /sys /usr/bin /usr/sbin
busybox mount -t proc proc /proc
busybox mount -t sysfs sys /sys
busybox mount -t devtmpfs dev /dev
echo Starting up...
echo ttyMSM0::respawn:/sbin/getty -L ttyMSM0 115200 vt100 >> /etc/inittab
echo Login with root and no password. > /etc/issue
echo >> /etc/issue
echo root::0:0:root:/root:/bin/sh > /etc/passwd
busybox mdev -s
busybox --install
hostname localhost
ip link set lo up
echo 5 > /proc/sys/kernel/printk
exec /linuxrc

busybox and the init script aren’t sufficient. If you have any problems then you’re going to want a console device. Otherwise you’ll do what I did, forget to change one of the tty references in the script to the right device, and spend too long trying to figure out why you’re not getting any output as soon as you start running userspace. It turns out that CONFIG_DEVTMPFS_MOUNT isn’t sufficient for the initramfs (the Kconfig help even tells you that). So I need to create a /dev/console device file inside the initramfs too. And I don’t build as root (and you shouldn’t either) which makes creating a device node hard. It’s ok though, because the Linux kernel doesn’t need you to be root to build it, and it has a default initramfs with a console device in it. This is helpfully generated using usr/gen_init_cpio, which takes a file defining what you want to put in the cpio file that becomes your initramfs. I used the following:

# Simple busybox based initramfs

dir /dev 0755 0 0
nod /dev/console 0600 0 0 c 5 1
dir /root 0700 0 0
dir /bin 0755 0 0
dir /sbin 0755 0 0
file /init basic-init 0755 0 0
file /bin/busybox busybox-armv7l 0755 0 0

All that then needs to be done is linux/usr/gen_init_cpio initramfs.list > initramfs.cpio and the resulting initramfs.cpio is suitable for inclusion in your kernel. The precompiled binaries have a lot of stuff enabled, which makes for a pretty useful debugging environment with fairly minimal work.