Easy. Make sure ip forwarding is enabled though (check /proc/sys/net/ipv4/ip_forward, it should be 1) , and for IPv6 forwarding you need to enable /proc/sys/net/ipv6/conf/all/forwarding. Note that you must enable it on the -all- directory, only enabling it on a single interface won’t do what you’d expect (this took me a while to figure out ).
After requesting both a set of IPv4 and IPv6 addresses I stumbled upon Xen’s /etc/xen/scripts/vif-route script which doesn’t do anything with IPv6! Since I do want my IPv6 space routed to my domU (and probably to the others as well) I decided to molest the script a little. My modification assumes you provide both the IPv4 address and IPv6 range you want to use on the domU. Here’s the diff:
xen 3.4 vif-route patch to support IPv6 - Version - SHA: e82dd95b8725c326121603fbd6168b3f88f5b881
Note that my patch adds some extra logging as well, you can hack that out yourself if it annoys you.
If patching is too difficult for you, here’s the patched vif-route script:
Patched Xen 3.4.1 vif-route to support IPv6 - Version - SHA: 29a28f4a8cff787233c1163435c86730fd5346fa

Don’t forget to put your IP addresses in your domU’s config file:

After patching vif-route and enabling the script in xend-config.sxp you still have to set routes to the dom0 in your domU. Example for dom0 with 192.168.1.1 and 1234::1

Now it works (unless you did something stupid of course)

On my laptop this is one of the first things I enabled to make my battery last longer.
Now that we’re going to put a server in a rack where they will bill us depending on the amount of power consumed, this sounds like a welcome thing to have enabled.

Our latest Xen install has Xen 3.4, on which they’ve enabled it by default. Isn’t that nice?
So basically, if your processor etc is good enough to support it, you can mess around with it using the xenpm tool.

For instance we can check the parameters for CPU 0 like this:

As you can see, it’s very much like the whole cpufreq ordeal. It has an ondemand governor, a powersave one, etc. Right now this CPU is using the ondemand governor, which will ramp up the clock when demand increases.
However, since this is a quad core and not a real quad processor machine we can only change the governor for all 4 cores at the same time, not for individual ones. (would be cool to have 1 core burning away power while the rest idles by….).

Anyway, the xenpm tool allows us to monitor and change various aspects of this.
For instance this will do a little benchmark and show the processor states during the benchmark.

As you can see my processor is mostly idle.
State C0 is the most active state, C1 is sleeping with use of the HLT instruction.
Your processor can have many more states, like my laptop which has 4.

Anyway, I put the change to ondemand governor in my /etc/rc.d/rc.local, so let’s hope it helps.

To read some more about the details of this stuff you can check out the Xenpm wiki page.

Since we got Xen up and running yesterday, it’s now time for actually having some fun with it.
The goals are:

1. Getting xend started automagically when booting without destroying my network connection
2. Getting a domU up and running with a network connection
3. Getting an internal network between the domUs and dom0, shielded from the big bad internet.

First things first, yesterday we got xen installed and all, but that’s about as far as we got before the bed was required. Xend almost got it right when it put a startup script in /etc/rc.d/init.d called ‘xend’, but alas, slackware will ignore that file. Personally I simply add the thing to /etc/rc.d/rc.local:

This fixes issue #1, but only the first part.

Before you reboot you might want to alter some xend settings. The file to molest for this is /etc/xend/xend-config.sxp. Mine looks a bit like this:

Almost everything is set to the default value. However, you will want to pay attention to the network-script part at the very least. As you can see I changed this to a little wrapper script.
I won’t go into detail about all the possible Xen networking stuff, see the mailing list and the Xen docs for that, but what we want is the external network bridged to all domU’s, so they can have their own network space there and their external IP address and whatever they want to do without having to mess around in the dom0. To get this working you could stick to the default (network-script network-bridge). Back in the Xen 3.0.3 days this molested my default gateway because their bridge script sucked (it failed to parse some stuff if I remember correctly), but when I tested it today it simply worked.

We want to have an internal network though, so we have to create our own bridge wrapper script.The only thing this script wrapper does is
1. call the network bridge script as the default would do.
2. create a second xen bridge.
Here we go:

As you can see I also add an internal IP address to the bridge. You can make this bridge setup script as complex or as neat as you want, but I can’t be bothered to spend more time on it. It works for me ™

Now that all that is in place, let’s try out the creation of a domU. For testing purposes I steal my images from what used to be jailtime.org, these days it’s called Stacklet. Too bad they don’t have a 64 bit slackware domU yet, but I guess I’ll create that myself a little later. Also, to keep stuff in a place where I can track it, I usually create a /xen directory where I store my host configurations and images.

Edit the slackware.12-2.x86.xen3.cfg file and fix the location of the image file and the kernel. I simply used the same kernel that I use to boot my dom0 for now. Now let’s try it! Try

and it should enter the console of the now booting domU. To get out of the console hit CTRL-[.

Over here I got a bunch of kernel warnings like these:

While these are a nuisance in dmesg, they seem to be fairly harmless. Somewhere on the Xen mailing list someone claims that you can disable CONFIG_DMI in the kernel to get rid of these, which I tend to believe but can’t be bothered. If I’m going to build a new kernel for my domU, I will take the latest vanilla kernel and add Xen domU support in that. It’s a mainline kernel option these days, you can see my post about it here: Xen and booting domU using a vanilla kernel
Login on the domU using root/password, note that your networking actually works, but you still have to enter an IP for eth1 if you want the internal network.

Woohoo, it works!

For an overview of the currently running domains you can use

The xm command has a ton of useful things, so check out the manpage/help for it.

For my real domU I will be using LVM partitions and install slackware 64 on it, but that’s fairly trivial to figure out once you’re this far. So have fun with your Xen install ^^

First we build ourselves a nice cute little server in a 2U rack case made by Chenbro (sounds like Xenbro to me ).
Inside we stash a quad core Phenom X2 810, 2 western digital 1TB disks from the “green” series and of course 8GB of DDR1333.
Could be faster, but this should do for not too much coin. (about 600 euros).

We (which is me and lotjuh plus some friends) want this machine as a toy machine that we can all play with which we’re going to put in a rack at Coloclue.
Basically everyone pays a tiny amount of money each month and gets himself a domU on the machine that he can do whatever he wants with, like messing around with
vpn tunnels and ipv6

Anyway, after putting the stuff inside it looked a bit like this:

Power on, it works! (duh)
Work through the BIOS, making sure to disable the halt on errors prompt and such nonsense. 512MB for integrated video? No thanks, too bad the minimum is still 128MB.
And how come every time I run through a BIOS I find new (and undocumented) settings like “Enable C1E support”? Apparently this is some kind of new AMD power saving stuff,
but from the comments I found it seems like a tickless kernel has issues with it, so I’ll leave it off.

Boot from PXE, slackware64-current, go go go. Created a little md0 raid1 boot partition, ran setup using NFS and a few minutes later it boots into slackware 64.
That was easy. (And fast!)

Now for the more interesting stuff. First, let’s get rid of lilo and put the latest grub on the machine. It’s just such a nicer bootloader, once you get it properly installed that is.
It does support autoconf these days, so I figured I could use my slackbuild script for it. Slackbuild immediately cursed at me for not having the packages dir, so that was fixed as well
I also put the site config in place while I was at it.

Well, that went smooth. Now for the config file, this is usually more tricky.

Looking good…. seems like it even added the mdraid module in the config, grub is improving these days
Time for a reboot to see if it lied to us. *reboot*
Well, guess what…. it actually worked. Amazing.

Time for Xen!
We have 2 HDDs in there which I split up in a 20 GB boot partition and the rest. Both partitions are mirrored using raid-1 so a disk could basically disappear and the system should still be fine.
The bigger partition will be fed into LVM so we can make nice little chunks there to hand out to the domUs.
Other than that everyone can decide for themselves whatever the hell they want to run on it, as long as it runs under Xen.
Here we go:

That went smooth over here, and installed the xen hypervisor. (you should now have a /boot/xen.gz, which is a symlink to xen-3.4.1.gz).
However, we also need a dom0 kernel and the xen tools to run and control the installation.
First the tools:

Easy again. You should now have stuff like ‘xm‘ installed.
This also created the /etc/xen directory where we can muck around with the configuration later.

Now for the most challenging part: the kernel.
What usually bites me is getting the kernel config worked out to my wishes, xen seems to wreck havoc with mine most of the times.
Of course you could try to go for the default kernel, but I hate messing around with modules and initrds, so I usually try to make a megahuge kernel.
Let’s go!

This fetches the kernel sources from hg and prepares them with initial configs etc. It might ask you some stuff, answer them to the best of your abilities, or simply accept the default.
It doesn’t matter much if you messed up the questions, you have a chance to fix it. And fix the rest of the kernel as well while you’re at it:

You are now presented with the usual kernel config dialogs. Have fun running through all of them as I did and halfway noticing the lack of EXT4 support.
RAAAH!
Kernel 2.6.18.8 for you… it’s ancient. But still default for the xen dom0. I wouldn’t mind that much, if it at least included support for my hardware (another machine we have with Xen required a patch to get AHCI up and running for that controller) and supported my filesystems. Meh.
So after the conclusion that 2.6.18 wouldn’t work, I went for something better: 2.6.29.6

If all went well you now have a patched 2.6.29.6 kernel source. Make sure it didn’t barf though, but if you used the correct kernel version this should apply cleanly.
Retry on the config!

Time for some coffee.
When it’s done (took about 7 minutes on this machine with my huge list of options) we still need to put it in place and add it to grub’s menu.

If you did everything allright, yay, it boots with Xen!
Of course over here it died shortly after booting the kernel because it couldn’t mount its root filesystem, but that’s nothing new with this stuff

Time to debug the problem. After rebuilding my kernel about 4 times and fighting with grub, I started to seriously wonder what the hell its problem was.
Raid (md) support, check.
Ext4 support, check.
Both in grub and the kernel they were present. Maybe xen in the way? Seems unlikely.
The fact that the kernel gets booted proves to me that grub at least “gets” it.
Finally I created an initrd to see if that would help. It didn’t, except for the fact that it gave me a nice little shell to dick around with.
This showed me that md0 would indeed not mount, probably because it wasn’t started. After starting it manually using some busybox command (raidgetthefuckstarted /dev/md0 or something)
I could mount it and exit the emergency shell, the system booted.
After I recreated the initrd to include raid support (doh) it actually managed to boot without outsider help. Goody.
The command for this was: (note that I gave my kernel the -BenV local version string)

And the grub config entry now looks like:

Well, good enough for today. Bedtime!

So after the latest local root exploit thing, and testing it on some machines during har2009, I figured it might be nice to get the patch into my kernels. Running a webserver usually means you’re at risk after all, especially when it’s running wordpress

So I grabbed the latest kernel that had that patch and went for the usual make menuconfig ordeal. After half an hour of selecting “yes”, “maybe”, “I guess so”, and
some XEN options (CONFIG_PARAVIRT_GUEST=y, CONFIG_XEN=y, CONFIG_HVC_DRIVER=y, CONFIG_XEN_BLKDEV_FRONTEND=y, CONFIG_XEN_NETDEV_FRONTEND=y, CONFIG_XEN_KBDDEV_FRONTEND=y, CONFIG_HVC_XEN=y, CONFIG_XEN_BALLOON=y, CONFIG_XEN_SCRUB_PAGES=y, CONFIG_XENFS=y, CONFIG_XEN_COMPAT_XENFS=y) I built the thing.
As I usually build bzImages I also did so here. make bzImage modules modules_install.
Half an hour later I had a shiny bzImage. Let’s try it on a test domain!

I copied the bzImage to /boot, changed the kernel = "/boot/vmlinuz-xen-old" option to the new place, and GO!
This is what it told me:

Huh, where’s my console?
Checking xm list it seemed to be there… but paused… and with constantly increasing ids. Waaait a minute!
Checking the log files (xend.log in this case) revealed a secret:

Note the ‘WARNING’ line. Crashed?!
Gee, that’s … interesting. Why? Took me 5 minutes to find the other log file, but xend-debug.log had a magic line:
ERROR Invalid kernel: xc_dom_find_loader: no loader found
… great. Another why.
In case you’re wondering, the increasing status in ‘xm list’ was caused by the on_crash = 'reboot' line in the xen host config.
I quickly destroyed the doman and changed it to a oneshot try: on_crash = 'destroy'

After some searching another hint presented itself. Aren’t they nice?
This hint was: “Xen is a retarded piece of cancer and can’t decypher the bzImage format, try vmlinux instead”. Aha!
Back to the kernel, make vmlinux, copy, and another try to boot it. Obviously this was way too simple. It still crashed.
Checking my old kernel that works for both dom0 and domU with file I noticed that gz should at least work.

Some more details about the crashing domains can be found using xm dmesg
It gave me something like:

As you can see, that’s really useful…..

A useful page with some info can be found here. It confirms that a gz kernel should work (but bz probably doesn’t) since I’m still running Xen 3.2. However, I can’t find a good reason for why it won’t work.

Google to the rescue! Hard to find a useful keyword, since most xen kernel issues seem to be similar, but eventually I found this tidbit:
tiny kernel patch.
Weird that this should be needed, one would hope that basic shit like this would work after 8 kernel versions, but obviously they’re all retards when it comes to this.
Thanks Jeremy Fitzhardinge for the patch!
Just for quick reference, this is the actual patch:

So much for ‘vanilla’.

However, it boots now:

and a lot more yadieyada until it hits a new wall: root device.
Well, that sounds solvable.

In fact, here’s a solution. What you say?
“AAAAH, I GET NOTHING, IT WON’T EVEN CRASH ANYMORE, JUST NO OUTPUT???!”
Ah yeah, I forgot to mention: they changed the console device as well as the block device. So here’s a solution for both issues:
Edit your xen host config file and make it so:

(xvda1 is what used to be sda1, you can figure out the rest).

And after all this: HAHA! Success is mine!
Except of course for the undeniable fact that my domU has a fucked up fstab and all, but no issues there
Now go mess up your own system! Oh, I see… you already did. Good luck fixing it

Update

I just tested a testing kernel — 2.6.31-rc8, and it seems like they fixed it. Probably has been fixed in the testing branch for quite a while now, but still not in stable 2.6.30
Let’s hope they release 2.6.31 soon.