Category: NetBSD

Adventures in Virtualization

I currently pay ~$150/mo for my ISP. It’s a legacy commercial account through my local cable provider, and is 50mbit down, and 10mbit up, with a static IP. Gigabit fiber is available for < $100/mo, but without a static IP. The cable provider has “helpfully” suggested I update my account to higher speeds at higher prices, but it occurred to me that there’s no reason my home needs a static IP if I can set up a tunnel from home to my VPS, which also has a static IP. I take my public services and instead of reverse-proxying them to my local router, I proxy them over VPN to my VPS.

The first step of course is updating my VPS to the “latest and greatest” (it’s still running a very svelte, firewalled, and outdated NetBSD-7) but also to get some hopefully improved VPN capability, because while it would be neat to get an IPSEC tunnel running, (possibly over IPv6,) I suspect it will be a lot easier in practice to get the newly available wireguard support working.

My VPS is one of the few who doesn’t mind running NetBSD, and started out with 32-bit PV instances running on a NetBSD dom0. They shifted to Debian Linux as the company grew, but the lowest-spec instances were still 32-bit PV when I first signed up. They didn’t directly advertise them; you had to do a “view source” in the HTML to find the obscured link, but the single vCPU, 1.4G memory, and 18G storage is plenty for my needs as a DNS server, and soon as a VPN server. (The disk space and the physical CPU backing the vCPU have grown over the years.)

Linux dropped 32-bit PV support in Linux 5.9, and Xen in general has been moving towards PVH from PV, since it utilizes hardware features which were introduced after PV, like page table updates. I was a bit surprised to discover my home server running Ubuntu 22.04, kernel 5.15.0, with Xen 4.16.0 supports PVH, but no longer supports 32-bit PV. A freshly compiled NetBSD-9 32-bit PV kernel completely crashed on my VPS, NetBSD-10 32-bit PV kernel got a little farther but crashed with MMU errors, and the lack of support for 32-bit PV on my home server prevents me from doing more extensive debugging. The solution seems to be to update my VPS to 64-bit PV, which makes things more consistent between my VPS and home.

PVH seems the future, though. A directly specified NetBSD-10 GENERIC kernel seemed to boot, but the Xen docs and some searching indicated that there is a version of OVMF which can be used as well? As a UEFI developer who has been hacking on the build system to greatly improve build times as a side-project, this is interesting. It looks like I have to build it from scratch, though, which I do frequently at work, but doing it under NetBSD instead of Linux or Windows (gasp) has some challenges.

Instead of booting UEFI, it looks like there is the possibility of pvhgrub. Ubuntu includes a 32-bit PVH version of grub, but I can’t seem to find a 64-bit version? Maybe since it’s a naked HVM it runs grub2 in 32-bit mode and then flips to 64-bit as part of the boot process? I have no idea how to boot a NetBSD kernel from a grub2> interactive prompt — it seems to reject the magic number of my kernel, but at least it can see the GPT partition and knows about the UFS2 filesystem. Maybe a menu.lst or grub.cfg and it will magically start working.

For now I’m booting a kernel directly from the xl.cfg side and get on to bootstrapping everything. Maybe ye olde pygrub bootloader will still work here?

Farewell Skolem, the PIII packet-pusher

Skolem was a Pentium III mobile with 128MiB RAM, a 20GiB travelstar HD, and a dock! The dock was important since the LCD was completely shot, and it had two PCI slots (with Lite-on tulip clones) which enabled routing duties. Skolem was my router for many years, taking over after my Ultra 5 started failing, and even survived a DSL to cable ISP change. Continue reading Farewell Skolem, the PIII packet-pusher

continuity of self-bootstrapping

I’ve been collecting build times for over a decade now, in an effort to grok how much faster newer hardware is, how much larger software is getting, and to normalize expectations between my various pieces of hardware. I use the NetBSD world as a microcosm for this, since it is fairly self-contained, and since NetBSD-2, the build process does a full bootstrap including building a (cross-)compiler. A modern Intel Romley or Grantley platform can build the NetBSD-7 amd64 world in less than 20 minutes, and is completely I/O bound. (Of course, I’m not sure when compilation has ever not been I/O bound…)

Self-hosted builds are in some sense “alive” — they beget the next version, they reproduce, and they propagate changes and grow over time. I don’t believe anybody bootstraps from complete scratch anymore these days, with hand-written hand-assembled machine code toggled directly into CPU memory into an environment that supports a macro assembler, which generates an environment that can host a rudimentary C compiler, etc. While there is a base case, it is an inductive process: developers use OS to create OS+1, or cross-compile from OS/foocpu to OS/barcpu. How far back could I go and walk this path? Could I do it across architectures? (Historically, how did things jump from PDP11 to VAX to i386?)

As I’ve been saying goodbye to my oldest hardware, I’ve been trying to get a sense of continuity from those early machines to my latest ones, and wanted to see if I could bootstrap the world on one of my oldest and slowest systems, and compare it with doing the same thing on one of my more modern systems. Modern is relative, of course. I’ve been pitting a circa 1990 12.5MHz MIPS R2000 DECStation (pmin) with 24MiB of RAM against a VM instance running on a circa 2010 3GHz AMD Phenom xII 545, both building the NetBSD 1.4.3A world. AMD (PVHVM) does a full build in 11 minutes. The same process on the pmin takes almost four days. This isn’t a direct apples-to-apples comparison, since the pmin is building NetBSD/pmax and the AMD is building NetBSD/i386, but it gives a good order-of-magnitude scale. (I should throw a 25MHz 80486 into the mix as a point for architectural normalization…)

Now for the continuity. I started running NetBSD on the pmax with 1.2, but I only ran it on DECStations until 1.4, and new architectures were always installed with binary distributions. Could I do it through source? As far as I can tell, the distributions were all compiled natively for 1.4.3. (The cross-compile setup wasn’t standardized until NetBSD-2.) Even following a native (rather than cross-compiled) source update path, there were some serious hiccups along the way: 1.4.3 (not 1.4.3A) doesn’t even compile natively for pmax, for instance. On i386, the jump from 1.4.3 to 1.5 is fiddly due to the switch from a.out to ELF formats. I spent a few evenings over winter break successfully fiddling this out on my i386 VM, recalling that a couple decades ago I was unsuccessful in making a similar jump from a.out to ELF with my Slackware Linux install. (I eventually capitulated back then and installed RedHat from binary.)

So far, I’ve gotten a 1.4.3 pmax to bootstrap 1.4.3A, and gone through the gyrations to get an 1.4.3 a.out i386 to bootstrap 1.5.3 ELF. Next step is doing 1.4.3A -> 1.5.3 on the pmax. We should then be able to do a direct comparison with 1.5.3 -> 1.6 matrix of native vs cross-compiled on both systems, and that will give me crossover continuity, since I could potentially run an i386 that has been bootstrapped from source on the pmin.

I’m also interested in the compile time scaling from 1.4.3 -> 1.4.3A -> 1.5 -> 1.5.3 -> 1.6 across multiple architectures. Is it the same for both pmin and i386? When does 24MiB start hurting? (the pmin didn’t seem overly swappy when building 1.4.3A.) Can I bring other systems (m68k, vax, alpha, sparc) to the party, too?

Some people walk a labyrinth for solace… I compile the world.