The problem descriptions:
* Your system boots fine (maybe because it’s running on software raid), but your /dev/sd* files are gone.
* Your system doesn’t boot anymore, complaining about not finding your boot device when booting your custom kernel, but the stock kernel does work.
* Mounting partitions doesn’t work anymore, saying stuff like mount: special device /dev/sda does not exist

Reason:
* Your custom kernel has CONFIG_SYSFS_DEPRECATED enabled. To find out:

Since udev version 151 (or something close to that) this will sparsely populate /dev. Yay.

If you don’t believe it, check out /usr/share/doc/udev-*/README:

Another problem solved.

… huh? What’s taking so long? Our server with issues? That’s unlikely since I still used it 5 minutes earlier. After checking out the NFS server and deciding nothing was wrong with it (or at least, it didn’t look like it) I got fed up with it and went to bed.
Don’t you hate it when stuff stops working just when you’re tired and want to go to sleep? Fortunately for me this is our own internal problem so I could actually do so

Today I decided to go find out what the hell was wrong with it. So again I booted the machine to find out the above kernel messages.
Time for some tcpdump on the nfs server to see if that can shine some light on this situation. (needless to say there were no useful log entries anywhere).
Tcpdump:

Hmm, interesting. The first request seems to “work” (read: it receives an answer) while the rest of the messages are somehow ignored. Wonder why…
Just to make sure I checked the firewall on the server, which should allow pretty much everything:

Well, that seems to be true. Maybe something weird in the mount options? Let’s see the /etc/fstab for the NFS client:

Nothing weird… or is there? After messing around with it for a bit I decided it really wasn’t anything in there.
Time to do a test on a different machine in the network. On my workstation I tried to mount the root path after giving myself permissions and re-exporting the nfs paths:

And there it was, being stuck for what seems to be forever. So in other words: the mount worked, but then it doesn’t respond for the ls command. What… the… fuck.
Mind boggling, while the logs on the nfs server gave no hints at all. (as usual).
To make this even more interesting, there is another export on that same nfs server that most workstation use, called general_stores. (to access the music collection etc).
Can’t remember that being broken…. let’s test:

How can this be?! This mount works fine while the other doesn’t? What the hell is so special about that NFS client’s export?

And then it hit me.

*WHACK*
(ouch)

The window that I had open with the ls -al tmp suddenly decided to work. It gave this listing:

Doh! /proc … of course that’s the problem! That’s why tcpdump showed the “proc” entries!

But why?

Well, recently I moved some disks around and added a ‘crossmnt‘ option in the /etc/exports file on the NFS server.
This option gives the nfs clients the ability to cross mount borders, so for example if I have this on the nfs server:
* /mnt/disk with device /dev/sda1 mounted there
* /mnt/disk/stuff with device /dev/sdb1 mounted there
And then export /mnt/disk and mount it on my nfs client, I can see /mnt/disk/stuff with its contents because of the crossmnt option. Without that option the directory would be empty on nfs clients.

So how does this relate to proc? Well, since proc is “special” it has a ton of special files that nfs can’t deal with and therefore can not export.
Now normally this wouldn’t be an issue, since it shouldn’t be mounted in the first place (the nfs server has its own /proc, the nfs cilent mounts its own version so it only needs a place to mount it on). However, a little earlier my girly friend mounted it “because she had to build E“. Don’t ask.
E again…. always the same…
After unmounting the proc dir and rebooting the NFS client it had no more problems and booted fine.

Conclusion:
NFS can’t deal with proc. Don’t hide a mounted procfs in your nfs exports.
NFS sucks because of stuff like this, but I haven’t found anything better so far. Guess we’ll stick with it for now.

No, that has nothing to do with Apple/Mac computers or Media Access Control, it’s
part of the SSH protocol (and others) called Messenge Authentication Code. This blog has a nice explanation: Jan Pechanec on SSH messages.

Funny thing, my brother also had this exact issue with the same kind of laptop. Well…. in his case it was putty failing his connection from a windows machine to this laptop.
The reason? Same as ever, checksum offloading.
You can check if your card does this with the ethtool program:

And the fix for this:

Fixed.

” Wait, what does Pokemon OS have to do with this?”
Well, isn’t that obvious? It’s supposed to magically work, and it doesn’t!
Probably more a kernel thing though… oh well.

Try this:

Apparently this is caused by the p4-clockmod module.
You know, this piece of junk:

p4-clockmod

The reason?
Well, the ondemand and the conservative governors want to be able to switch to a different speed before their calculations become invalid.
The p4-clockmod thing apparently makes the latency of switching so BIG that it becomes impossible. Or so they think.
Which leaves only the sucky powersave and performance governors. (and of course userspace).

The kernel menuconfig help states:

This driver should be only used in exceptional
circumstances when very low power is needed because it causes severe
slowdowns and noticeable latencies. Normally Speedstep should be used
instead.

Lovely. I’ll keep that in mind, thanks.
*enables CONFIG_X86_ACPI_CPUFREQ instead*

Note that when they’re both compiled in (as opposed to modules) the retarded thing seems to go for the clockmod. Just great.

Since we got Xen up and running yesterday, it’s now time for actually having some fun with it.
The goals are:

1. Getting xend started automagically when booting without destroying my network connection
2. Getting a domU up and running with a network connection
3. Getting an internal network between the domUs and dom0, shielded from the big bad internet.

First things first, yesterday we got xen installed and all, but that’s about as far as we got before the bed was required. Xend almost got it right when it put a startup script in /etc/rc.d/init.d called ‘xend’, but alas, slackware will ignore that file. Personally I simply add the thing to /etc/rc.d/rc.local:

This fixes issue #1, but only the first part.

Before you reboot you might want to alter some xend settings. The file to molest for this is /etc/xend/xend-config.sxp. Mine looks a bit like this:

Almost everything is set to the default value. However, you will want to pay attention to the network-script part at the very least. As you can see I changed this to a little wrapper script.
I won’t go into detail about all the possible Xen networking stuff, see the mailing list and the Xen docs for that, but what we want is the external network bridged to all domU’s, so they can have their own network space there and their external IP address and whatever they want to do without having to mess around in the dom0. To get this working you could stick to the default (network-script network-bridge). Back in the Xen 3.0.3 days this molested my default gateway because their bridge script sucked (it failed to parse some stuff if I remember correctly), but when I tested it today it simply worked.

We want to have an internal network though, so we have to create our own bridge wrapper script.The only thing this script wrapper does is
1. call the network bridge script as the default would do.
2. create a second xen bridge.
Here we go:

As you can see I also add an internal IP address to the bridge. You can make this bridge setup script as complex or as neat as you want, but I can’t be bothered to spend more time on it. It works for me ™

Now that all that is in place, let’s try out the creation of a domU. For testing purposes I steal my images from what used to be jailtime.org, these days it’s called Stacklet. Too bad they don’t have a 64 bit slackware domU yet, but I guess I’ll create that myself a little later. Also, to keep stuff in a place where I can track it, I usually create a /xen directory where I store my host configurations and images.

Edit the slackware.12-2.x86.xen3.cfg file and fix the location of the image file and the kernel. I simply used the same kernel that I use to boot my dom0 for now. Now let’s try it! Try

and it should enter the console of the now booting domU. To get out of the console hit CTRL-[.

Over here I got a bunch of kernel warnings like these:

While these are a nuisance in dmesg, they seem to be fairly harmless. Somewhere on the Xen mailing list someone claims that you can disable CONFIG_DMI in the kernel to get rid of these, which I tend to believe but can’t be bothered. If I’m going to build a new kernel for my domU, I will take the latest vanilla kernel and add Xen domU support in that. It’s a mainline kernel option these days, you can see my post about it here: Xen and booting domU using a vanilla kernel
Login on the domU using root/password, note that your networking actually works, but you still have to enter an IP for eth1 if you want the internal network.

Woohoo, it works!

For an overview of the currently running domains you can use

The xm command has a ton of useful things, so check out the manpage/help for it.

For my real domU I will be using LVM partitions and install slackware 64 on it, but that’s fairly trivial to figure out once you’re this far. So have fun with your Xen install ^^

So after the latest local root exploit thing, and testing it on some machines during har2009, I figured it might be nice to get the patch into my kernels. Running a webserver usually means you’re at risk after all, especially when it’s running wordpress

So I grabbed the latest kernel that had that patch and went for the usual make menuconfig ordeal. After half an hour of selecting “yes”, “maybe”, “I guess so”, and
some XEN options (CONFIG_PARAVIRT_GUEST=y, CONFIG_XEN=y, CONFIG_HVC_DRIVER=y, CONFIG_XEN_BLKDEV_FRONTEND=y, CONFIG_XEN_NETDEV_FRONTEND=y, CONFIG_XEN_KBDDEV_FRONTEND=y, CONFIG_HVC_XEN=y, CONFIG_XEN_BALLOON=y, CONFIG_XEN_SCRUB_PAGES=y, CONFIG_XENFS=y, CONFIG_XEN_COMPAT_XENFS=y) I built the thing.
As I usually build bzImages I also did so here. make bzImage modules modules_install.
Half an hour later I had a shiny bzImage. Let’s try it on a test domain!

I copied the bzImage to /boot, changed the kernel = "/boot/vmlinuz-xen-old" option to the new place, and GO!
This is what it told me:

Huh, where’s my console?
Checking xm list it seemed to be there… but paused… and with constantly increasing ids. Waaait a minute!
Checking the log files (xend.log in this case) revealed a secret:

Note the ‘WARNING’ line. Crashed?!
Gee, that’s … interesting. Why? Took me 5 minutes to find the other log file, but xend-debug.log had a magic line:
ERROR Invalid kernel: xc_dom_find_loader: no loader found
… great. Another why.
In case you’re wondering, the increasing status in ‘xm list’ was caused by the on_crash = 'reboot' line in the xen host config.
I quickly destroyed the doman and changed it to a oneshot try: on_crash = 'destroy'

After some searching another hint presented itself. Aren’t they nice?
This hint was: “Xen is a retarded piece of cancer and can’t decypher the bzImage format, try vmlinux instead”. Aha!
Back to the kernel, make vmlinux, copy, and another try to boot it. Obviously this was way too simple. It still crashed.
Checking my old kernel that works for both dom0 and domU with file I noticed that gz should at least work.

Some more details about the crashing domains can be found using xm dmesg
It gave me something like:

As you can see, that’s really useful…..

A useful page with some info can be found here. It confirms that a gz kernel should work (but bz probably doesn’t) since I’m still running Xen 3.2. However, I can’t find a good reason for why it won’t work.

Google to the rescue! Hard to find a useful keyword, since most xen kernel issues seem to be similar, but eventually I found this tidbit:
tiny kernel patch.
Weird that this should be needed, one would hope that basic shit like this would work after 8 kernel versions, but obviously they’re all retards when it comes to this.
Thanks Jeremy Fitzhardinge for the patch!
Just for quick reference, this is the actual patch:

So much for ‘vanilla’.

However, it boots now:

and a lot more yadieyada until it hits a new wall: root device.
Well, that sounds solvable.

In fact, here’s a solution. What you say?
“AAAAH, I GET NOTHING, IT WON’T EVEN CRASH ANYMORE, JUST NO OUTPUT???!”
Ah yeah, I forgot to mention: they changed the console device as well as the block device. So here’s a solution for both issues:
Edit your xen host config file and make it so:

(xvda1 is what used to be sda1, you can figure out the rest).

And after all this: HAHA! Success is mine!
Except of course for the undeniable fact that my domU has a fucked up fstab and all, but no issues there
Now go mess up your own system! Oh, I see… you already did. Good luck fixing it

Update

I just tested a testing kernel — 2.6.31-rc8, and it seems like they fixed it. Probably has been fixed in the testing branch for quite a while now, but still not in stable 2.6.30
Let’s hope they release 2.6.31 soon.