BenV's notes

Some days after tinkering for a little bit you come to the realization that it might be better to stop doing anything with devices and just wait for the day to pass, because everything you touch breaks in the most spectacular ways. Of course this never stopped me from breaking even more, but I’m stupid like that.
Today is a day like that it seems. First our ADSL line at home received an upgrade to FTTH (aka a fiber connection), boosting our internet speed from a lousy 8Mbit down to 50Mbit down, and from less than 1Mbit upstream to 50Mbit upstream.

New FritzBox

Sounds great! Yeah, except for the swapping of modems. Our previous FritzBox 7340 that we barely had for a month got replaced by a somewhat newer FritzBox 7390. Don’t ask me how that’s better, the only thing I noticed was that my IPv6 dropped dead after exporting and importing the settings and restarting internet with the new modem.
According to the guy who installed the fiber connection (we asked if we could do it ourselves, but there was a “NO!!! You don’t have the … uhh… special black box thingy device!” reply from XS4All) this is default for ADSL connections. He also said for FTTH connections the default was IPv6 on. Weird.
Another thing that was way better in this new FritzBox was that when trying to forward IPv6 ports (you didn’t think it would copy those, did you?) you have to specify a port range. Single ports were no option. So I tried to add ‘From: 22, To: 22’, but FritzBox said “No no, your To port needs to be HIGHER than the From port!”. Fortunately updating the FritzBox to the latest firmware fixed this issue.

Linux v3.5

Because our internet would switch IP addresses I figured this would also be a good time to upgrade our server to the latest kernel and Slackware64-current tree.
So after building linux v3.5 using ‘make oldconfig’ and pretty much the defaults for all the new options (“no”) I installed the new kernel and reboot it. No problems there. Yet.

Slackware64-current with udev v1.82

The Slackware64-current changelog had a huge story about how udev-1.82 had been a problem before because of the persistent network rules and yadiyada. In the end they were dumbasses themselves for starting udev more than once, but they figured it out. Since updating udev often introduces problems I figured this might be a good time to test it. Upgrade, reboot! … reboot? … hello?? (ping timeout)
After reconnecting a keyboard and screen to the server I found it in an endless reboot cycle that went a bit like this:

• INTEL!!!!!! (bios splash, take a good look at the nme so you know where to put the blame)
• Grub!!! Boot the latest kernel? (timeout) Of course you do!
• Linux 3.5!!!! eerh… about that root device? Maybe you want an emergency shell to fix stuff
• Emergency shell … ahahaha, that would be too easy, no shell for you!
• Oh your shell is done, let’s reboot!

Seemed like it could find a disk, probably udev related. However, the failing of an emergency shell kind of annoyed me.
“Oh well, let’s boot pmagic from my USB stick” I thought. HA! Today things are not that easy.
After inserting the USB stick and booting I was greeted by the now familiar Grub splash with the new 3.5 kernel in it.
Hmz, why didn’t it boot from USB disk? Probably a BIOS setting. Note that an “INTEL!!!!!!” splash screen isn’t particularly helpful when you’re trying
to find out “WHERE IS THE GODDAMNED FUCKING BOOT MENU OPTION!?!”. Fuck you Intel board.

When I went to the BIOS and put the USB stick on top of the 4 other disks it booted straight into “Boot error”, followed by the now very familiar Grub menu from above. That grub showed me a (fd0,msdos1) that had my pmagic on it.
After meddling with BIOS boot settings for about half an hour (legacy USB on/off, USB emulation settings and some more crap like that) I finally gave up with this approach. Again: Fuck you Intel. Fortunately I ordered new hardware for this machine including a new motherboard that hopefully has a more reasonable BIOS.

Anyhow, I still want to get this working today before I leave for a couple of days on vacation, so next approach: PXE!
This would normally be my first option, but the machine that’s not booting is my DHCP/PXE server 🙂

Setting up a PXE environment

Quick PXE instructions for those of you who don’t know this wonderful invention.
Setting up PXE is easy enough, simply run a dhcp server with a few extra directives in your dhcpd.conf:

filename "/pxelinux.0";
option root-path "/path/to/pxeroot";
next-server 192.168.1.1; # This is obviously the IP to your machine that you host the pxe files on.

And in /path/to/pxeroot you put pxelinux.0 (comes from syslinux), a file called “pxelinux.cfg” that has something like:

DEFAULT pmagic
# Boot the kernel and initramfs over PXE.
LABEL pmagic
LINUX pmagic/bzImage
INITRD pmagic/initramfs
APPEND edd=off load_ramdisk=1 prompt_ramdisk=0 rw vga=normal loglevel=9 max_loop=256

and furthermore you need a few files from the pmagic PXE zip file (check the bottom of the page).
Unzip the file and copy/move the pmagic dir so it’ll be /path/to/pxeroot/pmagic.

The final ingredient for PXE booting is a tftp server. Slackware has this by default, you just need to uncomment and edit the line in /etc/inetd.conf so it looks a bit like this:

# Note that you probably need to change /path/to/pxeroot ;)
tftp dgram udp wait root /usr/sbin/in.tftpd in.tftpd -s /path/to/pxeroot -r blksize

Finally (re)start dhcpd, restart inetd, and try booting a machine from PXE 🙂

Fixing the udev issue

Now that I finally was able to boot my machine into a workable environment again the solution was simple.

1. Start raid device(s), since pmagic doesn’t do this by default (which I think is a good thing! Although a GUI might be nice ;))
2. Mount raid device(s) that contain the slackware system
3. Chroot into slackware64 environment
4. Fix the udev issue

For 1 I issued entered:

root@PartedMagic:~# mdadm --assemble --scan
mdadm: /dev/md127 has been started with 2 drives (out of 2).
mdadm: /dev/md126 has been started with 2 drives (out of 2).
mdadm: /dev/md125 has been started with 2 drives (out of 2).

Fuck. This isn’t necessarily a problem yet, but I remember something about mdadm remembering those retarded device numbers.
Interestingly the /dev/md/ directory contained symlinks to the devices with names that did make some sense, like /dev/md/1_0.
Oh well, after mounting /dev/md/127 and chrooting to it I could finally start fixing the issue.

The issue is quite simple (when you finally know it), but I would have appreciated a warning in the Slackware Changelog guys.
With the new udev version it’s required for your kernel to have the CONFIG_DEVTMPFS option enabled, you can find it in Device Drivers -> Generic Driver Options. Since I never needed that before
it had been turned off in my custom kernels. So: enable the option, recompile and install the kernel and reboot.

My fears about the mdadm command were justified as the new kernel paniced on the missing md1 that was now called md125. Thanks mdadm, piece of cancer.
Now I have to piece it together again manually… *siiiigh*

root@PartedMagic:~# mdadm --examine --scan > /etc/mdadm.conf
# this writes the found arrays in the config file so we can start it
root@PartedMagic:~# mdadm --start /dev/md125
mdadm: /dev/md125 has been started with 2 drives.
root@PartedMagic:~# cat /proc/mdstat
# CHECK WHAT PARTITIONS ARE USED
Personalities : [raid1]
md125 : active raid1 sdc1[0] sda1[1]
412365672 blocks [2/2] [UU]
root@PartedMagic:~# mdadm --stop /dev/md125
mdadm : stopped /dev/md125.
# Enter the correct partitions here!
root@PartedMagic:~# mdadm --assemble /dev/md1 --update=super-minor /dev/sdc1 /dev/sda1
mdadm: /dev/md1 has been started with 2 drives.
# You can check if it's ok now, see what's after "preferred minor" in the output of this:
root@PartedMagic:~# mdadm --examine /dev/sdc1

Reboot, and now it finally works and I’ll stop touching devices before the day is over 😉
(obviously I had to repeat the above for my other 2 raid arrays, but it works)