BenV's notes

You read it right, time for fun!

First we build ourselves a nice cute little server in a 2U rack case made by Chenbro (sounds like Xenbro to me ;)).
Inside we stash a quad core Phenom X2 810, 2 western digital 1TB disks from the “green” series and of course 8GB of DDR1333.
Could be faster, but this should do for not too much coin. (about 600 euros).

We (which is me and lotjuh plus some friends) want this machine as a toy machine that we can all play with which we’re going to put in a rack at Coloclue.
Basically everyone pays a tiny amount of money each month and gets himself a domU on the machine that he can do whatever he wants with, like messing around with
vpn tunnels and ipv6 🙂

Anyway, after putting the stuff inside it looked a bit like this:

Power on, it works! (duh)
Work through the BIOS, making sure to disable the halt on errors prompt and such nonsense. 512MB for integrated video? No thanks, too bad the minimum is still 128MB.
And how come every time I run through a BIOS I find new (and undocumented) settings like “Enable C1E support”? Apparently this is some kind of new AMD power saving stuff,
but from the comments I found it seems like a tickless kernel has issues with it, so I’ll leave it off.

Boot from PXE, slackware64-current, go go go. Created a little md0 raid1 boot partition, ran setup using NFS and a few minutes later it boots into slackware 64.
That was easy. (And fast!)

Now for the more interesting stuff. First, let’s get rid of lilo and put the latest grub on the machine. It’s just such a nicer bootloader, once you get it properly installed that is.
It does support autoconf these days, so I figured I could use my slackbuild script for it. Slackbuild immediately cursed at me for not having the packages dir, so that was fixed as well 😉
I also put the site config in place while I was at it.

root@xenbro:/usr/src# svn co svn://svn.sv.gnu.org/grub/trunk/grub2 grub2-svn
[ ... ]
root@xenbro:/usr/src# cd grub2-svn
root@xenbro:/usr/src/grub2-svn# slackbuild.pl
[ ... ]
root@xenbro:/usr/src/grub2-svn# installpkg /usr/src/packages/grub2-1.97-svn-x86_64-1.tgz
[ ... ]
root@xenbro# grub-install /dev/md0
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

(hd0) /dev/sda
(hd1) /dev/sdb


Well, that went smooth. Now for the config file, this is usually more tricky.

root@xenbro# grub-mkconfig > /boot/grub/grub.cfg
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-huge-2.6.29.6
Found linux image: /boot/vmlinuz-generic-2.6.29.6
done


Looking good…. seems like it even added the mdraid module in the config, grub is improving these days 🙂
Time for a reboot to see if it lied to us. *reboot*
Well, guess what…. it actually worked. Amazing.

Time for Xen!
We have 2 HDDs in there which I split up in a 20 GB boot partition and the rest. Both partitions are mirrored using raid-1 so a disk could basically disappear and the system should still be fine.
The bigger partition will be fed into LVM so we can make nice little chunks there to hand out to the domUs.
Other than that everyone can decide for themselves whatever the hell they want to run on it, as long as it runs under Xen.
Here we go:

root@xenbro:~# cd /usr/src
root@xenbro:/usr/src# mkdir xen
root@xenbro:/usr/src# cd xen/
root@xenbro:/usr/src/xen# wget http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz
[ ... ]
root@xenbro:/usr/src/xen# tar zxvf xen-3.4.1.tar.gz
[ ... ]
root@xenbro:/usr/src/xen# cd xen-3.4.1
root@xenbro:/usr/src/xen/xen-3.4.1# make xen
[ ... ]
root@xenbro:/usr/src/xen/xen-3.4.1# make install-xen
[ ... ]

That went smooth over here, and installed the xen hypervisor. (you should now have a /boot/xen.gz, which is a symlink to xen-3.4.1.gz).
However, we also need a dom0 kernel and the xen tools to run and control the installation.
First the tools:

root@xenbro:/usr/src/xen/xen-3.4.1# make tools
[ .. ]
root@xenbro:/usr/src/xen/xen-3.4.1# make install-tools
[ ... ]

Easy again. You should now have stuff like ‘xm‘ installed.
This also created the /etc/xen directory where we can muck around with the configuration later.

Now for the most challenging part: the kernel.
What usually bites me is getting the kernel config worked out to my wishes, xen seems to wreck havoc with mine most of the times.
Of course you could try to go for the default kernel, but I hate messing around with modules and initrds, so I usually try to make a megahuge kernel.
Let’s go!


root@xenbro:/usr/src/xen/xen-3.4.1# make linux-2.6-xen0-prep

This fetches the kernel sources from hg and prepares them with initial configs etc. It might ask you some stuff, answer them to the best of your abilities, or simply accept the default.
It doesn’t matter much if you messed up the questions, you have a chance to fix it. And fix the rest of the kernel as well while you’re at it:

root@xenbro:/usr/src/xen/xen-3.4.1# cd build-linux-2.6.18-xen0_x86_64/
root@xenbro:/usr/src/xen/xen-3.4.1/build-linux-2.6.18-xen0_x86_64# make menuconfig

You are now presented with the usual kernel config dialogs. Have fun running through all of them as I did and halfway noticing the lack of EXT4 support.
RAAAH!
Kernel 2.6.18.8 for you… it’s ancient. But still default for the xen dom0. I wouldn’t mind that much, if it at least included support for my hardware (another machine we have with Xen required a patch to get AHCI up and running for that controller) and supported my filesystems. Meh.
So after the conclusion that 2.6.18 wouldn’t work, I went for something better: 2.6.29.6 🙂


root@xenbro:/usr/src/xen# wget http://gentoo-xen-kernel.googlecode.com/files/xen-patches-2.6.29-6.tar.bz2
[ ... ]
root@xenbro:/usr/src/xen# wget ftp.kernel.org/pub/linux/kernel/v2.6/linux-2.6.29.6.tar.bz2
root@xenbro:/usr/src/xen# tar jxvf linux-2.6.29.6.tar.bz2
[ *KABOOM* ]
root@xenbro:/usr/src/xen# mkdir xen-patches
root@xenbro:/usr/src/xen# cd xen-patches
root@xenbro:/usr/src/xen/xen-patches# tar jxvf ../xen-patches-2.6.29-6.tar.bz2
root@xenbro:/usr/src/xen/xen-patches# cd ../linux-2.6.29.6
root@xenbro:/usr/src/xen/linux-2.6.29.6# for k in ../xen-patches/* ; do echo Patch $k: ; patch -p1 < $k ; done [ *WHOOSH* ]
If all went well you now have a patched 2.6.29.6 kernel source. Make sure it didn't barf though, but if you used the correct kernel version this should apply cleanly.
Retry on the config!

root@xenbro:/usr/src/xen/linux-2.6.29.6# make menuconfig
[ *yadieyada, don't forget to enable Xen dom0 support* ]
root@xenbro:/usr/src/xen/linux-2.6.29.6# time make bzImage modules modules_install
[ *crunch crunch* ]

Time for some coffee.
When it's done (took about 7 minutes on this machine with my huge list of options) we still need to put it in place and add it to grub's menu.


root@xenbro:/usr/src/xen/linux-2.6.29.6# cp vmlinux /boot/vmlinuz-xen-2.6.29.6
root@xenbro:/usr/src/xen/linux-2.6.29.6# vim /boot/grub/grub.cfg

}

If you did everything allright, yay, it boots with Xen! 🙂
Of course over here it died shortly after booting the kernel because it couldn't mount its root filesystem, but that's nothing new with this stuff 😉

Time to debug the problem. After rebuilding my kernel about 4 times and fighting with grub, I started to seriously wonder what the hell its problem was.
Raid (md) support, check.
Ext4 support, check.
Both in grub and the kernel they were present. Maybe xen in the way? Seems unlikely.
The fact that the kernel gets booted proves to me that grub at least "gets" it.
Finally I created an initrd to see if that would help. It didn't, except for the fact that it gave me a nice little shell to dick around with.
This showed me that md0 would indeed not mount, probably because it wasn't started. After starting it manually using some busybox command (raidgetthefuckstarted /dev/md0 or something)
I could mount it and exit the emergency shell, the system booted.
After I recreated the initrd to include raid support (doh) it actually managed to boot without outsider help. Goody.
The command for this was: (note that I gave my kernel the -BenV local version string)

root@xenbro:/# mkinitrd -c -k 2.6.29.6-BenV -m ext4 -f ext4 -R -r /dev/md0

And the grub config entry now looks like:

menuentry "GNU/Linux, Xen 3.4.1 / Linux 2.6.29.6" {
insmod raid
insmod mdraid
insmod ext2
set root=(md0)
search --no-floppy --fs-uuid --set 2f8afc00-ec48-4d1f-b4fa-5787e18f2387
multiboot /boot/xen-3.4.1.gz dom0_mem=512M
module /boot/vmlinuz-xen-2.6.29.6 root=/dev/md0 ro
module /boot/initrd-xen-2.6.29.6.gz
}


Well, good enough for today. Bedtime!