Adventures in Virtualization

I currently pay ~$150/mo for my ISP. It’s a legacy commercial account through my local cable provider, and is 50mbit down, and 10mbit up, with a static IP. Gigabit fiber is available for < $100/mo, but without a static IP. The cable provider has “helpfully” suggested I update my account to higher speeds at higher prices, but it occurred to me that there’s no reason my home needs a static IP if I can set up a tunnel from home to my VPS, which also has a static IP. I take my public services and instead of reverse-proxying them to my local router, I proxy them over VPN to my VPS.

The first step of course is updating my VPS to the “latest and greatest” (it’s still running a very svelte, firewalled, and outdated NetBSD-7) but also to get some hopefully improved VPN capability, because while it would be neat to get an IPSEC tunnel running, (possibly over IPv6,) I suspect it will be a lot easier in practice to get the newly available wireguard support working.

My VPS is one of the few who doesn’t mind running NetBSD, and started out with 32-bit PV instances running on a NetBSD dom0. They shifted to Debian Linux as the company grew, but the lowest-spec instances were still 32-bit PV when I first signed up. They didn’t directly advertise them; you had to do a “view source” in the HTML to find the obscured link, but the single vCPU, 1.4G memory, and 18G storage is plenty for my needs as a DNS server, and soon as a VPN server. (The disk space and the physical CPU backing the vCPU have grown over the years.)

Linux dropped 32-bit PV support in Linux 5.9, and Xen in general has been moving towards PVH from PV, since it utilizes hardware features which were introduced after PV, like page table updates. I was a bit surprised to discover my home server running Ubuntu 22.04, kernel 5.15.0, with Xen 4.16.0 supports PVH, but no longer supports 32-bit PV. A freshly compiled NetBSD-9 32-bit PV kernel completely crashed on my VPS, NetBSD-10 32-bit PV kernel got a little farther but crashed with MMU errors, and the lack of support for 32-bit PV on my home server prevents me from doing more extensive debugging. The solution seems to be to update my VPS to 64-bit PV, which makes things more consistent between my VPS and home.

PVH seems the future, though. A directly specified NetBSD-10 GENERIC kernel seemed to boot, but the Xen docs and some searching indicated that there is a version of OVMF which can be used as well? As a UEFI developer who has been hacking on the build system to greatly improve build times as a side-project, this is interesting. It looks like I have to build it from scratch, though, which I do frequently at work, but doing it under NetBSD instead of Linux or Windows (gasp) has some challenges.

Instead of booting UEFI, it looks like there is the possibility of pvhgrub. Ubuntu includes a 32-bit PVH version of grub, but I can’t seem to find a 64-bit version? Maybe since it’s a naked HVM it runs grub2 in 32-bit mode and then flips to 64-bit as part of the boot process? I have no idea how to boot a NetBSD kernel from a grub2> interactive prompt — it seems to reject the magic number of my kernel, but at least it can see the GPT partition and knows about the UFS2 filesystem. Maybe a menu.lst or grub.cfg and it will magically start working.

For now I’m booting a kernel directly from the xl.cfg side and get on to bootstrapping everything. Maybe ye olde pygrub bootloader will still work here?

Goodbye First-gen Roku

I just factory-reset my first-gen roku N1000 in preparation for junking. A cursory ebay search shows these listed on ebay as “vintage.” While it seems like I was able to successfully stream netflix, amazon, and PBS kids through it recently, it was probably a few years ago. (This is what growing old feels like, I guess.)

I type this post on an AMD phenom-II x2 that is only a year newer than the roku, yet runs win10 just fine, and continues to function as a contemporary to newer hardware systems. (A predecessor system with twice the cores but only 4GiB memory did not fare so well.) I added a GTX1650 to replace the anemic on-board video, and my kids have had no complaints so far about their game performance. (They are big fans of Terraria, Minecraft, and Geometry Dash, but it also runs WoW classic at 60FPS.)

Back to the roku… the reason the N1000 no longer works for streaming is not a technical one. It has its hardware limitations of 720p, and I suspect only a fixed set of supported CODECs, but I have run brand-spanking-new software on hardware from the early 90s, so I posit the decision to discontinue support is economic. I wonder what the real or perceived cost in continuing to support roku channels on the N1000 was when they decided to cut it loose? The projected margin for sales new hardware due to upgrades exceeds the cost of maintaining a CI environment for new software on old hardware?

Hobbyists can afford to have a warped sense of economy. Reduction in cost is an indirect goal, not maximization of revenue. For me, the primary cost has been time, followed by power consumption. I can’t recommend running PHP on a 400MHz alpha if you care about speed, but it was functional. Spamassassin took ~30s per message when I moved mail from my dual-processor SPARC 20 to an x86 VM. The same phenom-II x2 that I’m typing this message on now used to host all my VMs, and was replaced a couple years ago by a Dell Romley system which consumes roughly the same amount of power, but with 8x the hardware threads, 16x the memory, and 1/3 the rack space.

Tonight I packed up my c64 which is still able to read most of its disks from c1990 and isn’t dependent on the whims of a vendor for it to continue working correctly. I have three nexus 6 phones in my household now which had support dropped by google in 2017, but are able to run a frequently updated android 10 thanks to lineageOS. My kids’ chromebooks from ~2015 are both supported until mid 2021, and I expect I could install a third party chromeOS distro to extend the lifetime.

The c64 isn’t dependent on a third party to continue functioning. My nexus 6 and chromebooks will continue to work after the vendor drops support, but won’t receive security updates. In contrast, the roku is effectively dead, even though the hardware continues to work correctly. It can play videos from USB media, but the streaming services it utilized have been discontinued.

I want a long tail to hardware. I want the computing equivalent of replacing a refrigerator from 1978 because it no longer efficiently cooling its contents, not because the vendor decided they didn’t want to support cooling for that model. I want the decision to drop support for a given product to be made over a congregation of known users and code escrow being handed over to a non-profit or placed in public domain.

Sorry to see you go, first generation roku N1000. You seem to be working just fine hardware-wise, but apparently I never really owned you, and your true owners have a different software development optimization point than I do.

Loading the Barge

  • DEC
    • VAX
      • VAXStation 3100 M76
      • VAXStation 3100 M30 (VS42A)
      • uVAX-II (BA23 case, no rails)
      • VAXStation 4000 VLC
    • MIPS
      • DECStation 2100
    • Alpha
      • AlphaServer 1000A 5/400
      • Multia / UDB. Three of these, take ’em all! [only one taken so far]
      • Alpha 3000/400
  • Sun
    • SparcStation 1 (possibly 128M memory)
    • SparcStation 2
  • Terminals
    • ADM3a
    • Ann Arbor Ambassador

Systems marked in bold have found new homes.

The Multias are low-end and flaky; I can understand why they haven’t found new homes. The sparcs I’m a little surprised about… no love there?

goodnight clock radios

My wife has been modernizing her family’s vacation property, and all the clock radios have been replaced with newer ones which include USB charging jacks. A couple of them even have built-in cassette players, I suspect with very low use. I’m putting these cassette models on the street in the hopes a Portland Hipster or two will be willing to take them home and give them some love.

With practically everybody having an accurate network-synced clock on their nightstand which can trivially function as a clock radio and media player, are clock radios a technological anachronism being kept alive by millenials?

Plenty of technical baggage for Xeon

Reading through Xeon Bang Per Buck, Nehalem to Broadwell, and came across this gem:

Nehalem is the touchstone against which all Xeons should be measured because this is when Intel got rid of so much technical baggage that had been holding the Xeons back.

Sure, FSB was ditched for QPI, and the memory controllers were brought on-die, but there’s still a heck of a lot of technical baggage being carried around. I started writing this in 2016. It’s now 2020. Do modern Xeons really need to continue carrying around legacy in hardware from the original 8086 and other x86 “mileposts” (80286, 80386, 80486, pentium, pentium pro, etc…)? I bet there’s a Greenlow or Denlow board somewhere in embedded land with ISA slots plumbed in through eSPI, booting DOS 6.

EFI has been shipping since 2000, and Apple started shipping Intel-based Macs with EFI in 2006. Why are we still booting in 16-bit real mode? Why did it take so long for option ROMs for modern on-board ethernet on Intel reference hardware (and PCSD’s commercial versions) to be re-coded for EFI in the intervening decade? (Whitley finally dropped real-mode opROM support.) Add a software-based 16-bit emulator for legacy option ROMs optionally loaded during DXE phase? (DEC had a MIPS emulator in turbochannel Alphas for running turbochannel option ROMs. Sun and other openfirmware systems went one better and used architecture-independent FORTH bytecode.)

The necessity for direct hardware support of legacy 32-bit support is questionable — BIOS claims it needs 32-bit to meet code-size requirements, but the code is not exactly tidy or concise. There seems to be a vicious circle between IBVs and BIOS developers unwilling to update crufty code, with long-aged code bases suffering from a tragedy of the commons, with no clear stewardship or short-term monetary payout for cleaning things up. I do have a few BIOS buddies who grok this and are trying to do what they can, but there’s a lot of momentum in technical debt that’s coming up on two decades old in a company that has historically viewed software as an enabler for hardware.

Surely 32-bit backwards compatibility could be handled at the OS and software layer, not the hardware? IA64 bungled implementation of IA32 backwards compatibility by not making it performant, but that doesn’t make it an inherently bad idea.

Don’t get me started on SMM… or maybe that’s a rant for another post.

(posted from the depths of my drafts years after it was started…)

DEClaser 1152

Feeling a bit nostalgic right now that I had a DEClaser 1152 postscript laser printer left at the family business. It was liquidated when the business relocated. I was given a terse warning by an ex-co-worker to come collect things before the liquidation, but I was dealing with a newborn at the time, and didn’t give it much thought.

http://www.bitsavers.org/pdf/dec/brochures/DEC-DEClaser-1152-PostScriptLevel-2-LaserPrinter.pdf

Continue reading DEClaser 1152

Farewell Skolem, the PIII packet-pusher

Skolem was a Pentium III mobile with 128MiB RAM, a 20GiB travelstar HD, and a dock! The dock was important since the LCD was completely shot, and it had two PCI slots (with Lite-on tulip clones) which enabled routing duties. Skolem was my router for many years, taking over after my Ultra 5 started failing, and even survived a DSL to cable ISP change. Continue reading Farewell Skolem, the PIII packet-pusher

coalescing the VMs

I got a Romley (dual e5-2670 Jaketowns) last November with the plan to pull in the VMs from the three Xen hosts I currently run. I’ve named it “Luxor.” It idles at around 150W, which should save me some power bill, and even though it only currently has 1TB of mirrored storage, thin LVM provisioning should allow me to stretch that a bit. It’s easily the fastest system in my house now, with the possible exception of my wife’s haswell macbook pro for single-threaded performance.

Luxor has 96GiB [now 128GiB] of memory. I think this may exceed the combined sum of all other systems I have in the house. I figured that the price of the RAM alone justified the purchase. Kismet. Looking at the memory configuration, I have six 8GiB DIMMS per socket, but the uneven DIMMs-per-channel prevents optimal interleaving across the four channels. Adding two identical DIMMs or moving two DIMMs from one socket to another should alleviate this. (I doubt it’s causing performance regressions, but given that the DIMMs are cheap and available and I plan on keeping this machine around until it becomes uneconomical to run (or past that point if history is an indicator), DIMMs to expand it to 128GiB should be arriving soon.

In mid-December, the first olde sun x2200m2 opteron (“Anaximander”) had its two VMs migrated and was shut down. A second x2200m2 (“Anaximenes,” which hosts the bulk of my infrastructure, including this site,) remains. While writing this post, a phenom II x2 545 (“Pythagoras”), had its 2TB NFS/CIFS storage migrated to my FreeBSD storage server (“Memphis”) along with some pkgsrc build VMs and secondary internal services.

Bootloader barf-bag for x86 is still in full effect. I couldn’t figure out how to PXE without booting the system in legacy BIOS mode, and I gave up trying to get the Ubuntu installer to do a GPT layout, let alone boot it. I figure I can migrate LVM volumes to new disk(s) on GPT-backed disks, install EFI grub, switch system to EFI mode, and Bob’s your uncle. (He’s my brother-in-law, but close enough.) At least that’s the plan.

The VMs on Anaximenes have been a little trickier to move, since I need to make sure I’m not creating any circular dependencies between infrastructure VMs and being able to boot Luxor itself. Can we start VMs without DHCP and DNS being up, for instance?

Systemd is a huge PITA, and isn’t able to shut down VMs cleanly, even after fiddling with the unit files to add some dependency ordering. Current theory is that it’s killing off underlying qemu instances so the VMs essentially get stuck. Running the shutdown script manually works fine and the VMs come down cleanly.

Time to get rid of my TiVo series 1

Around the turn of the millennium, my now wife and I plunked down our hard-earned money and purchased a Philips TiVo series 1 with lifetime service. We had a friend with a ReplayTV, but the TiVo interface won us over, and we enjoyed the TiVo life through roughly a decade thanks to a dual 128GB hard drive upgrade and silicon dust cache card. The updated hard drives eventually died, and we reverted back to the original 30GB drive for a few months before our directv receiver was replaced with one that had DVR capability. The directv DVR UI sucked, but it was easier than stringing IR blasters and serial control converters together to keep the TiVo going, so the TiVo ended up being neglected.

I had cozy visions of my TiVo recording kids shows from a DTV converter box as part of my basement retro media station. TiVo dropped support for series 1 guide listings in 2016. This obviously had no practical impact on me, but it still makes me a little sad.

So long, TiVo. You were a great box.