continuity of self-bootstrapping

I’ve been collecting build times for over a decade now, in an effort to grok how much faster newer hardware is, how much larger software is getting, and to normalize expectations between my various pieces of hardware. I use the NetBSD world as a microcosm for this, since it is fairly self-contained, and since NetBSD-2, the build process does a full bootstrap including building a (cross-)compiler. A modern Intel Romley or Grantley platform can build the NetBSD-7 amd64 world in less than 20 minutes, and is completely I/O bound. (Of course, I’m not sure when compilation has ever not been I/O bound…)

Self-hosted builds are in some sense “alive” — they beget the next version, they reproduce, and they propagate changes and grow over time. I don’t believe anybody bootstraps from complete scratch anymore these days, with hand-written hand-assembled machine code toggled directly into CPU memory into an environment that supports a macro assembler, which generates an environment that can host a rudimentary C compiler, etc. While there is a base case, it is an inductive process: developers use OS to create OS+1, or cross-compile from OS/foocpu to OS/barcpu. How far back could I go and walk this path? Could I do it across architectures? (Historically, how did things jump from PDP11 to VAX to i386?)

As I’ve been saying goodbye to my oldest hardware, I’ve been trying to get a sense of continuity from those early machines to my latest ones, and wanted to see if I could bootstrap the world on one of my oldest and slowest systems, and compare it with doing the same thing on one of my more modern systems. Modern is relative, of course. I’ve been pitting a circa 1990 12.5MHz MIPS R2000 DECStation (pmin) with 24MiB of RAM against a VM instance running on a circa 2010 3GHz AMD Phenom xII 545, both building the NetBSD 1.4.3A world. AMD (PVHVM) does a full build in 11 minutes. The same process on the pmin takes almost four days. This isn’t a direct apples-to-apples comparison, since the pmin is building NetBSD/pmax and the AMD is building NetBSD/i386, but it gives a good order-of-magnitude scale. (I should throw a 25MHz 80486 into the mix as a point for architectural normalization…)

Now for the continuity. I started running NetBSD on the pmax with 1.2, but I only ran it on DECStations until 1.4, and new architectures were always installed with binary distributions. Could I do it through source? As far as I can tell, the distributions were all compiled natively for 1.4.3. (The cross-compile setup wasn’t standardized until NetBSD-2.) Even following a native (rather than cross-compiled) source update path, there were some serious hiccups along the way: 1.4.3 (not 1.4.3A) doesn’t even compile natively for pmax, for instance. On i386, the jump from 1.4.3 to 1.5 is fiddly due to the switch from a.out to ELF formats. I spent a few evenings over winter break successfully fiddling this out on my i386 VM, recalling that a couple decades ago I was unsuccessful in making a similar jump from a.out to ELF with my Slackware Linux install. (I eventually capitulated back then and installed RedHat from binary.)

So far, I’ve gotten a 1.4.3 pmax to bootstrap 1.4.3A, and gone through the gyrations to get an 1.4.3 a.out i386 to bootstrap 1.5.3 ELF. Next step is doing 1.4.3A -> 1.5.3 on the pmax. We should then be able to do a direct comparison with 1.5.3 -> 1.6 matrix of native vs cross-compiled on both systems, and that will give me crossover continuity, since I could potentially run an i386 that has been bootstrapped from source on the pmin.

I’m also interested in the compile time scaling from 1.4.3 -> 1.4.3A -> 1.5 -> 1.5.3 -> 1.6 across multiple architectures. Is it the same for both pmin and i386? When does 24MiB start hurting? (the pmin didn’t seem overly swappy when building 1.4.3A.) Can I bring other systems (m68k, vax, alpha, sparc) to the party, too?

Some people walk a labyrinth for solace… I compile the world.

bye SGIs

last weekend I dropped off a stack of SGI challenges and an indy to my local recycler. there’s only so much that can be done with a 200MHz R4400 CPU these days. there’s also still some pmap bugs in NetBSD, so I never could complete a build of the world, although I did get some openssl numbers.

I never had these systems in production use, so there’s not a lot of emotional attachment. just the lost potential.

MIPS seems to be bare-bones these days. not even parity memory. I don’t get it. it was a serious workstation- and server-level CPU, and now it’s used for networking gear and more disturbingly for NAS boxes. (seriously, who would ever trust data to a storage subsystem that isn’t at least performing parity checks? yet most MIPS-based NAS boxes don’t have parity or ECC memory!?) MIPS was server-level in the early 90s. no longer.

here’s some dmesg and openssl benchmarks… photos later after I get them exported from raw.

I still like the SGI speckle finish and design. I’ll remember the indy’s fanfare when it was powered up. the electropaint screensaver that inspired xscreensaver’s stonerview. playing mp3 layer 2 files from an early music site… (can’t recall the name this late.) the lp account with no password. the awkwardness of IRIX. OK stopping before the bad memories surface.

here’s some dmesg.

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 7.0 (GENERIC32_IP2x.201510122130Z)
total memory = 128 MB
(768 KB reserved for ARCS)
avail memory = 119 MB
timecounter: Timecounters tick every 10.000 msec
mainbus0 (root): SGI-IP22 [SGI, 690a313e], 1 processor
cpu0 at mainbus0: MIPS R4400 CPU (0x460) Rev. 6.0 with MIPS R4010 FPC Rev. 0.0
cpu0: 48 TLB entries, 16MB max page size
cpu0: 16KB/16B direct-mapped L1 instruction cache
cpu0: 16KB/16B direct-mapped write-back L1 data cache
cpu0: 1024KB/128B direct-mapped write-back L2 unified cache
int0 at mainbus0 addr 0x1fbd9880
int0: bus 100MHz, CPU 200MHz
imc0 at mainbus0 addr 0x1fa00000: revision 3
gio0 at imc0
newport0 at gio0: SGI NG1 (board revision 4, cmap revision 5, xmap revision 5, vc2 revision 0), depth 8
wsdisplay0 at newport0 kbdmux 1
wsmux1: connecting to wsdisplay0
hpc0 at gio0: SGI HPC3 (onboard)
zsc0 at hpc0 offset 0x59830
zstty0 at zsc0 channel 1 (console i/o)
zstty1 at zsc0 channel 0
pckbc0 at hpc0 offset 0x59840
sq0 at hpc0 offset 0x54000: SGI Seeq 80c03
sq0: Ethernet address 08:00:69:0a:31:3e
wdsc0 at hpc0 offset 0x44000: WD33C93B (20.0 MHz clock, BURST DMA, SCSI ID 0)
wdsc0: microcode revision 0x0d, Fast SCSI
scsibus0 at wdsc0: 8 targets, 8 luns per target
haltwo0 at hpc0 offset 0x58000: HAL2 revision 4.1.0
audio0 at haltwo0: half duplex, playback, capture
pi1ppc0 at hpc0 offset 0x59800
pi1ppc0: capabilities=0x8
ppbus0 at pi1ppc0
ppbus0: No IEEE1284 device found.
lpt0 at ppbus0: port mode = 0x1
panel0 at hpc0 offset 0x59850
dsclock0 at mainbus0 addr 0x1fbe0000
ioc0 at mainbus0 addr 0x1fbd9800: rev 0, machine Indy (Guinness), board rev 0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "mips3_cp0_counter" frequency 100000000 Hz quality 100
scsibus0: waiting 2 seconds for devices to settle...
sd0 at scsibus0 target 1 lun 0:  disk fixed
sd0: 17518 MB, 10042 cyl, 12 head, 297 sec, 512 bytes/sect x 35877972 sectors
sd0: sync (100.00ns offset 12), 8-bit (10.000MB/s) transfers, tagged queueing
boot device: sd0
root on sd0a dumps on sd0b

and some openssl results:

OpenSSL 1.0.1p 9 Jul 2015
NetBSD 7.0
options:bn(32,32) md2(int) rc4(ptr,int) des(idx,cisc,16,long) aes(partial) idea(int) blowfish(ptr)
gcc version 4.8.4 (NetBSD nb2 20150115)
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md2                 78.66k      178.98k      257.45k      290.53k      303.10k
mdc2               145.06k      218.49k      252.84k      263.22k      265.83k
md4                272.68k     1020.27k     3510.04k     8881.49k    15989.37k
md5                221.69k      829.48k     2735.18k     6375.68k    10435.90k
hmac(md5)          432.03k     1521.22k     4359.31k     8265.54k    11164.68k
sha1               226.51k      804.40k     2370.03k     4631.29k     6387.03k
rmd160             157.18k      555.69k     1615.45k     3064.52k     4192.54k
rc4               7713.16k     8239.91k     7964.19k     7774.18k     8311.75k
des cbc           1339.39k     1451.10k     1482.15k     1495.11k     1388.01k
des ede3           442.20k      453.40k      452.86k      452.21k      503.50k
idea cbc          1772.67k     1894.70k     1922.57k     1920.86k     1916.00k
seed cbc          1880.76k     2062.38k     2107.36k     2091.69k     2102.61k
rc2 cbc           1549.19k     1627.91k     1639.00k     1637.72k     1641.15k
rc5-32/12 cbc        0.00         0.00         0.00         0.00         0.00
blowfish cbc      3121.18k     3362.69k     3491.58k     3559.60k     3588.10k
cast cbc          2230.75k     2412.86k     2450.63k     2468.83k     2454.86k
aes-128 cbc       2528.06k     2744.85k     2819.34k     2849.51k     2852.23k
aes-192 cbc       2141.74k     2331.49k     2441.50k     2459.99k     2460.35k
aes-256 cbc       1906.51k     2058.35k     2139.82k     2156.59k     2155.50k
camellia-128 cbc     1649.84k     2178.77k     2367.96k     2438.76k     2427.26k
camellia-192 cbc     1389.08k     1737.42k     1857.25k     1886.21k     1884.98k
camellia-256 cbc     1375.05k     1733.15k     1853.87k     1892.86k     1887.72k
sha256             357.50k      866.70k     1587.03k     2004.45k     2181.78k
sha512             140.38k      561.43k      904.85k     1298.90k     1482.75k
whirlpool          109.76k      224.54k      365.65k      435.29k      459.95k
aes-128 ige       2238.29k     2535.68k     2638.35k     2672.33k     2672.61k
aes-192 ige       1964.70k     2209.11k     2280.95k     2303.49k     2335.83k
aes-256 ige       1848.34k     1985.61k     2037.47k     2040.78k     2056.19k
ghash             2965.22k     3198.88k     3252.14k     3205.57k     3271.32k
                  sign    verify    sign/s verify/s
rsa  512 bits 0.054054s 0.004529s     18.5    220.8
rsa 1024 bits 0.291429s 0.014821s      3.4     67.5
rsa 2048 bits 1.931667s 0.053209s      0.5     18.8
rsa 4096 bits 12.870000s 0.198824s      0.1      5.0
                  sign    verify    sign/s verify/s
dsa  512 bits 0.045571s 0.055414s     21.9     18.0
dsa 1024 bits 0.149552s 0.175088s      6.7      5.7
dsa 2048 bits 0.525263s 0.648750s      1.9      1.5
                              sign    verify    sign/s verify/s
 160 bit ecdsa (secp160r1)   0.0152s   0.0746s     65.8     13.4
 192 bit ecdsa (nistp192)   0.0157s   0.0758s     63.7     13.2
 224 bit ecdsa (nistp224)   0.0216s   0.1059s     46.2      9.4
 256 bit ecdsa (nistp256)   0.0325s   0.1772s     30.7      5.6
 384 bit ecdsa (nistp384)   0.0693s   0.3858s     14.4      2.6
 521 bit ecdsa (nistp521)   0.1998s   1.0970s      5.0      0.9
 163 bit ecdsa (nistk163)   0.0176s   0.0755s     56.9     13.2
 233 bit ecdsa (nistk233)   0.0373s   0.1507s     26.8      6.6
 283 bit ecdsa (nistk283)   0.0567s   0.2668s     17.6      3.7
 409 bit ecdsa (nistk409)   0.2597s   0.6200s      3.8      1.6
 571 bit ecdsa (nistk571)   0.3654s   1.4271s      2.7      0.7
 163 bit ecdsa (nistb163)   0.0172s   0.0820s     58.1     12.2
 233 bit ecdsa (nistb233)   0.0368s   0.1602s     27.2      6.2
 283 bit ecdsa (nistb283)   0.0566s   0.2976s     17.7      3.4
 409 bit ecdsa (nistb409)   0.1548s   0.7013s      6.5      1.4
 571 bit ecdsa (nistb571)   0.4786s   1.6467s      2.1      0.6
                              op      op/s
 160 bit ecdh (secp160r1)   0.0637s     15.7
 192 bit ecdh (nistp192)   0.0640s     15.6
 224 bit ecdh (nistp224)   0.0917s     10.9
 256 bit ecdh (nistp256)   0.1524s      6.6
 384 bit ecdh (nistp384)   0.3209s      3.1
 521 bit ecdh (nistp521)   0.9000s      1.1
 163 bit ecdh (nistk163)   0.0366s     27.3
 233 bit ecdh (nistk233)   0.0709s     14.1
 283 bit ecdh (nistk283)   0.1320s      7.6
 409 bit ecdh (nistk409)   0.3094s      3.2
 571 bit ecdh (nistk571)   0.7113s      1.4
 163 bit ecdh (nistb163)   0.0411s     24.3
 233 bit ecdh (nistb233)   0.0795s     12.6
 283 bit ecdh (nistb283)   0.1469s      6.8
 409 bit ecdh (nistb409)   0.3514s      2.8
 571 bit ecdh (nistb571)   0.8185s      1.2

a pang for a DECstation

Tonight I revisited a longstanding question of mine: what does Ubuntu bring to the table over Debian? This of course leads me to look at hardware support for each, especially non-amd64 support. Mix this with current efforts to get my SGI systems running again, and I wonder what the state of Linux is for such platforms.

Linux doesn’t even seem to try to support older platforms anymore. Debian 8 doesn’t support sparc anymore, and mips is limited to malta, octeon, and loongson. No sgimips, no decstation, no alpha, no vax, and no hp300. ARM is the hot thing, but slapping a raspberry pi in a rack doesn’t make it server-class, and actual server-class ARM hardware hasn’t made its way to my basement datacentre… yet.

NetBSD at least still tries to keep running on older hardware. NetBSD-7 boots on my dusted-off sgimips Indy, although it seems to have some issues with cache management, which might be kernel bugs, or might be bad hardware. I don’t know enough about page table fault handling on MIPS to know for sure.

The MIPS action of course made me think of and miss my DECStations. One DECStation, in particular. It was a DECStation 5000/240, and ran almost nonstop from 1999ish to 2013. It was the main brain of my home network, handling DNS, DHCP, YP, HTTP, SMTP, and NFS on my home network for most of its life. I moved SMTP off the DECStation to a dedicated dual CPU sparc 20 when spam filtering became a pain point. HTTP and home directories were moved to an alpha, although I can’t recall if they moved before or after SMTP.

The 5000/240 never let me down. Drives failed, and I had to stop running local backups onto QIC tape at some point due to lockups, (I suspect to a dodgy power supply in the expansion unit which eventually failed,) but the machine itself kept on working until I was <ahem> prompted to clear some things out of the basement in late 2011. I had been trickling the moving of services over to newer active systems since 2011, but it still made me sad to shut it down. Attempting to build the NetBSD world on a 5000/200 and never being successful due to a failing disk (after a couple days of running) really drove the point home of how far software has outstripped the hardware.

Luckily a (now ex-) co-worker was interested in collecting my 5000/240, and it avoided the recycler. I still miss it.