BenV's notes

Yesterday when I was messing around with our homebrew IRC bot (written from scratch in perl) it somehow managed to segfault.
Which surprised me, it doesn’t happen very often that I see perl segfault.
It was caused by the fairly new module that handles RSS feeds. Running it through gdb quickly pointed a finger at XML::LibXML, which probably surprises nobody 😉
(at least I suspected the moment it crashed)
It looked like LibXML tried to free up some resource that didn’t exist anymore.

Today I wrote a little test script to figure out the exact cause. Apparently, using Storable to store things that LibXML returned is a recipe for great fun 🙂
Actually, storing it works pretty good. It works so damn good that LibXML actually believes that the objects loaded through Storable are still valid.

Example.
First store a piece of XML::LibXML to disk:

#!/usr/bin/perl
use Data::Dumper;
use Storable qw/lock_nstore/;
use XML::LibXML;

my $xmlp = XML::LibXML->new();
my $xml = $xmlp->parse_string($rssfeed);
$data->{'xml'} = $xml->find('/rss/channel/title/text()');
my $s = lock_nstore($data, $ARGV[0]);
print "Stored.\n";



#!/usr/bin/perl
use Storable qw/retrieve/;
use XML::LibXML;

die "Usage: $0 file.storable" unless(scalar(@ARGV));
my $s = retrieve($ARGV[0]);
my $link = $s->{'xml'};
print "Segfault in progress! $link\n";


The execution looks like this:

benv@janeman:~:0>./test-store.pl asd
Stored.
benv@janeman:~:0>./test-load.pl asd
Segmentation fault (core dumped)

Fun!

Needless to say Adobe had an update today.
Since we’re running a windows 2008 server with users that don’t have administrator rights, Adobe is very annoying to start with.

<Adobe>”Hey I have an update! Oh, I can’t install it because you need administrator rights, but I’ll keep bugging you with it anyway, MWHOAHAHHAHA”

So if that isn’t enough already (and they have an update about every week … if not more often), the piece of cancer can’t even update. Even when running the updater as administrator:

Mysterious fail

Of course it would be terrible to state what the problem is, so they only give you “Error: 1403” to work with.
Thanks Adobe, really useful. It’s unfortunate that some morons here insist on needing it (Because some idiot customers send us PDF files with chinese fonts and other rubbish that isn’t handled well in better pdf readers).

Feh.

If you’re like me you probably already hate the scumm that’s called Sony.
For installing rootkits on your pc. For taking away the ability to run linux on your PS3.
For being a Sony Cancer.
They’ve been assholes for as long as I can remember. Just check out their latest Playstation network agreement if you want to see how they trample your rights.

Anyhow, the guy being sued by them today (geohot – the one giving back linux on the ps3) has now setup a donation place to get some good lawyers.
So if you have a bit of coin to spare for a good cause, this is it. Anti Cancer! Pro Linux! What else could you want to spend your money on? 🙂

Last week I upgraded our fileserver at work from Debian Lenny to Debian Squeeze.
Obviously a ton of stuff got ‘new’ (read: less ancient) versions, including Apache.
Apart from a reboot or two for new kernels and some config fixes everything went pretty smooth.

This week lotjuh ran into the problem that she couldn’t push to the mercurial repository from windows.
Strange, because everything worked fine from linux. Tested from both the windows 2008 server we have here and another windows 7 machine at home, the both broke with the same cryptic message:

c:\tmp> hg clone --insecure https://fileserver/repository
abort: error: _ssl.c:1325: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

Huh. That’s weird.
Obviously google doesn’t help with this, you get some garbage results on how mercurial didn’t do jack with https certificates before version 1.7 and their struggle to implement it.

After some digging I found this in the apache logs:

[Thu Feb 17 12:10:51 2011] [error] [client 192.168.123.321] Re-negotiation request failed
[Thu Feb 17 12:10:51 2011] [error] SSL Library Error: 336068931 error:14080143:SSL routines:SSL3_ACCEPT:unsafe legacy renegotiation disabled

Feh. Somewhere old SSL libraries are being used! Windows… .always the same.

Solution:
In your apache ssl configuration (mods-enabeld/ssl.conf on Debian), add this:

SSLInsecureRenegotiation on

Note that this obviously isn’t a great solution, but it’s the only way to get it to work on windows at the moment.

It just took me about an hour to figure this one out, so here’s the story for the next time I run into it.
Steps taken:
* New machine
* 2 harddisks (Western Digital Greens, so used wdidle3 on them!)
* Boot Slackware64 installer from PXE/NFS
* cfdisk, create 2 identical partitions, make bootable, set type to FD, write, quit
* mdadm –create /dev/md0 –raid-level=1 –raid-devices=2 /dev/sda1 /dev/sdb2
* install slackware64, grub2 and some other junk
* reboot

Sounds good right?
Well, for some reason the array kept booting up with only 1 of 2 disks active.
No errors or warnings, just kept fucking up. /proc/mdstat looked like

# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md0 : active raid1 sdb1[1]
39061952 blocks [1/2] [_U]

Adding /dev/sda1 back (mdadm /dev/md0 –add /dev/sda1) worked fine too, the array resync-ed without problems.

After about an hour of trying to recreate superblocks and that sort of stuff I found it:
The partition type of /dev/sda1 was set to 0x83 instead of 0xFD.
Thanks cfdisk, last time I used that piece of garbage. (I’m 100% certain I set them to 0xFD, but somehow it’s bugging for me lately in cfdisk).

Every time I install a machine with the latest Slackware, I’m amazed again at the installed boot manager – lilo.
Sure, lilo works. Most of the times. Even when you have a raid-1 boot device.
Unless you don’t have the latest version of lilo of course.

Today I tried to continue a Slackware64 (current) install of a machine that I installed a week ago.
It worked fine, was just about to install Xen when one of the disks started acting up.
Obviously SMART didn’t help for a bit
* Report – No errors!
* Short Self test – Your disk is fine!
* You want a long test that takes 4 hours? Your machine locks up before it completes, haha!
But when the disk kept failing every time when the md1 resync hit 36%, I yanked out the disk and sent it RMA.
Dmesg showed error like this:

[ 3362.784129] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[ 3362.784132] ata1.00: failed command: READ DMA EXT
[ 3362.784135] ata1.00: cmd 25/00:00:3f:60:f4/00:04:57:00:00/e0 tag 0 dma 524288 in
[ 3362.784135] res 40/00:00:02:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3362.784136] ata1.00: status: { DRDY }
[ 3362.784139] ata1: hard resetting link
[ 3364.002049] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 3364.009142] ata1.00: configured for UDMA/33
[ 3364.009148] sd 0:0:0:0: [sda] Result: hostbyte=0x00 driverbyte=0x08
[ 3364.009150] sd 0:0:0:0: [sda] Sense Key : 0xb [current] [descriptor]
[ 3364.009152] Descriptor sense data with sense descriptors (in hex):
[ 3364.009153] 72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
[ 3364.009156] 00 00 00 01
[ 3364.009158] sd 0:0:0:0: [sda] ASC=0x0 ASCQ=0x0
[ 3364.009159] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 57 f4 60 3f 00 04 00 00
[ 3364.009162] end_request: I/O error, dev sda, sector 1475633215
[ 3364.009174] ata1: EH complete


So today I figured I could continue installing with only half a raid-1 array.
But it didn’t boot (“Loading operating system…. *halt*).
I figured lilo must have been installed to the MBR of the disk that I yanked, so I booted from LAN and ran lilo.
Obviously lilo complained, because /dev/sda was only half the raid-1 array and disks were missing!
Fine. I changed my boot device to /dev/md0, hoping that lilo would get the hint.


# lilo
Warning: LBA32 addressing assumed
Fatal: Not all RAID-1 disks are active; use '-H' to install to active disks only
# lilo -H
Warning: LBA32 addressing assumed
Warning: Partial RAID-1 install on active disks only: booting is not failsafe

Warning: Faulty disk in RAID-1 array; boot with caution!!
Fatal: Unusual RAID bios device code: 0xFF


*sigh*
This is why I hate lilo. If it doesn’t work, it doesn’t work.
And it never tells you why. Or maybe it does, just like windows always tells you what’s wrong when you get a blue screen.

It’s probably this bug, but I don’t care. Always something.
Time to find the sources to grub.