now | writings | rss | github | twitter | contact

beneath this mask there is an idea, and ideas are bulletproof

posted to writings on nov 23rd, 2007 with tags FUCKFUCKFUCK, nerd, openbsd, and work

friday afternoon i decided to install a package on one of my openbsd servers, but it was from a recent snapshot and the snapshot i was running on the server was too old to run it. no problem, i'll just upgrade the server. a usually quick task; just drop a new kernel into /, reboot, untar the new disk sets over /, run mergemaster and reboot again.

remotely rebooting servers that are 350 miles away is always a nerve racking experience. you reboot it, your ssh connection drops, you start a ping waiting for it to reply as you visualize it booting up and thinking about how long each piece usually takes. occasionally something takes longer than normal and you start to panic, but before you reach whoever you need to reach, it starts responding and suddenly a wave of relief comes over you and you resume your work.

but such was not the case on friday. after 10 minutes, my server had still not come back up.

a frustrating 8 hours later, i finally had remote access to the server's console through the hp lom interface to see why it wasn't coming back up. (why it took 8 hours to get access to a leased server's embedded remote management console in a supposedly manned datacenter on a friday afternoon is another story.) the machine was suddenly stalling booting at an ominous "No O/S" error. i assumed that the partition table or boot loader had somehow been corrupted, but not being able to boot to a cd or floppy on a server 350 miles away, i couldn't do much through the remote console.

luckily a pxe boot server was available on the server's network, so i could at least try to boot to a bsd.rd image and try to fix the disk. unluckily though, the lom console suddenly decided to call it quits at that moment and was no longer responding on the network. great.

another 16 hours later (!) someone finally reconfigured the lom and i had the server booted to a ramdisk image to try to rescue the data on the server. during the downtime i had eventually migrated some down services to other servers from backup data, but i prefer spending the time getting the original system back up than restoring a few-days-old backup on a new machine.

a quick look at the partition table and disklabel showed the problem; a mysteriously re-initialized disklabel:

# fdisk sd0
Disk: sd0       geometry: 9728/255/63 [156280320 Sectors]
Offset: 0       Signature: 0xAA55
         Starting       Ending       LBA Info:
 #: id    C   H  S -    C   H  S [       start:      size   ]
------------------------------------------------------------------------
 0: 00    0   0  0 -    0   0  0 [           0:           0 ] unused
 1: 00    0   0  0 -    0   0  0 [           0:           0 ] unused
 2: 00    0   0  0 -    0   0  0 [           0:           0 ] unused
*3: A6    0   1  1 - 9727 254 63 [          63:   156280257 ] OpenBSD

# disklabel sd0
# Inside MBR partition 3: type A6 start 63 size 156280257
[ ... ]
16 partitions:
#             size        offset  fstype [fsize bsize  cpg]
  a:     156280257            63  unused      0     0      # Cyl     0*- 76308*
  c:     156291072             0  unused      0     0      # Cyl     0 - 76313 

normally i would have just restored the disklabel from an off-site backup of the server's /var/backups/disklabel.sd0.current file, but just to make things more interesting, i had no backup of /var/backups available.

having been in a similar situation with my laptop years back, i remembered using the scan_ffs utility to rebuild the disklabel manually. scan_ffs scans the raw filesystem (/dev/rsd0c) looking for ffs partition info and spits out the block numbers as it progresses. however, scan_ffs isn't on the installation ramdisk and since the system disk is not accessible, it can't just be run from a mounted partition. but alas, it's an installation ramdisk, so it does have ifconfig, route and ftp, and with some manual ethernet interface configuration and deletion of some large files on the ramdisk's in-memory filesystem, scan_ffs joined the party:

# ftp -o - ftp://209.242.32.10/pub/OpenBSD/4.1/amd64/base41.tgz | \
    tar xvzf - ./sbin/scan_ffs
# /sbin/scan_ffs -l /dev/rsd0c
X: 4194240 47 4.2BSD 2048 16384 323 # /
X: 4194240 63 4.2BSD 2048 16384 323 # /
X: 1048576 5242880 4.2BSD 2048 16384 323 # /tmp
X: 4194304 6291456 4.2BSD 2048 16384 323 # /var
X: 8388608 10485760 4.2BSD 2048 16384 323 # /usr
X: 8388608 18874368 4.2BSD 2048 16384 323 # /home
X: 5760 25313664 4.2BSD 512 4096 80 # /mnt
X: 2880 25453408 4.2BSD 512 4096 80 # /mnt
X: 129017344 27262976 4.2BSD 2048 16384 323 # /d

and with those numbers, i recreated the disklabel:

[ ... ]
#             size        offset  fstype [fsize bsize  cpg]
  a:       4194240            63  4.2BSD   2048 16384   16 # Cyl     0*-  2047*
  b:       1048577       4194303    swap                   # Cyl  2047*-  2559 
  c:     156291072             0  unused      0     0      # Cyl     0 - 76313 
  d:       1048576       5242880  4.2BSD   2048 16384   16 # Cyl  2560 -  3071 
  e:       4194304       6291456  4.2BSD   2048 16384   16 # Cyl  3072 -  5119 
  f:       8388608      10485760  4.2BSD   2048 16384   16 # Cyl  5120 -  9215 
  g:       8388608      18874368  4.2BSD   2048 16384   16 # Cyl  9216 - 13311 
  h:     129017344      27262976  4.2BSD   2048 16384   16 # Cyl 13312 - 76308*

i reinstalled the boot blocks for good measure, rebooted, and was back in business after almost 30 hours of downtime. i'm still not sure how the on-disk disklabel vanished, though.

so the morals of the story are:

  1. when leasing a server from or colocating with a 3rd party, make sure that the datacenter the server is in is actually staffed 24/7 even if the 3rd party company claims to be able to get access on-demand. if the 3rd party company can't get to the datacenter quickly, at least they can call in a reboot or remote hands request to the 24/7 datacenter staff.
  2. before remotely rebooting a server, make sure that its remote console is actually reachable and usable. or even check it at random since you might need it after an unexpected crash.
  3. always have a backup of /var/backups available (for openbsd machines, anyway) even if the rest of the system isn't worth backing up. hell, put:

    shar /var/backups/*.current | mail -s backups root

    in /etc/weekly.local and just archive the e-mails.

  4. the man page for scan_ffs does not lie:

    It is not perfect, and could do a lot more things with date/time information in the superblocks it finds, but this program has saved more than one butt, more than once.

Comments? Contact me via Twitter or e-mail.