Brews you can Use

I have tasted the end of the world, and it was good.

I capped off today's workday by writing yet another pop3 daemon. How many more of these do I have to write still? At least this one only took an hour to knock out. I paused a bit, and my screensaver kicked in. It was fontglide, for some reason stuck on "Zippy mode." It began slowly assembling words, and when I saw them start to crystallize into:

Yow! Is this sexual intercourse yet?

I knew it was definitely time to go home...

For all of you combination math and classical poetry geeks out there, a little memory aid.

(Or at least the coronary arteries...)

Via djymm, a blog devoted to my favorite food and the bane/savior of so many vegetarians and Kosher-observers.

Yummy bacon, it should be the fifth food group. I'm having a mild craving for bacon fat popovers after reading...

(A rare techie post...)

I spent some time today racking my brain over possible shared-disk clustering with FreeBSD. Linux has had this for a while, but, er... I like FreeBSD :-) In particular, RedHat has been distributing the U of MN/Sistina Global Filesystem (GFS), a SAN shared filesystem supporting multiple read/write clients, with basically no bottlenecks or single points of failure. Very cool stuff. If it ran on FreeBSD, I'd be using it in a heartbeat.

I was looking at something more primitive though, namely having a single filesystem with one host mounting it read/write, the other mounting it read-only (if at all), with the possibility of "taking over" that filesystem if the other host fails. This is mostly trivial stuff: just make sure that the SCSI chain is properly terminated, and re-mount the volume read/write if the other host fails. The hard problem is reliably determining that the other host has failed, and distinguishing host failures from network partitioning. Both boxes mounting the filesystem read/write would be really, really bad.

The last time I took a serious look at GFS (back in 2000-2001, when it was going to be ported to FreeBSD), it relied on a SCSI locking primitive to synchronize access. Theoretically, I could use this mechanism to lock the other host out from the shared storage device. This mechanism was unfortunately never standardized or widely implemented, so the modern GFS relies on distributed lock manager software on the participating hosts.

This has the same problem: If a host's network fails while it's holding a lock, it's not safe for other hosts to "steal" that lock because the supposedly-failed host may still access the shared disk thinking it still holds the lock. There needs to be some mechanism to forcibly prevent the failed host from accessing the disk.

One possibility, I thought, would be to somehow physically switch off the SCSI connection to the other host. In a SAN environment this could be done by logging into a switch and turning off the failed host's port. Or, it hit me, we have most of our servers on remotely controllable power switches, and the failover system could run a script that logs into the power switch and turns off its peer. Yuck, what a kludge.

So, I looked back through the GFS sources and docs, and noted some references to "fencing off" a failed node from the storage. This is provided by a cluster service called "fenced", which I promptly downloaded and checked out. Fenced supports multiple mechanisms for separating a failed host from storage, implemented by separate scripts. For example, the fence_brocade script works with Brocade SAN switches by, well, logging in to the switch and shutting down the failed host's port.

And then I noticed one of the scripts was named fenced_apc. Our power switches are APCs, hmm... Yep, they're using the mechanism I'd just thought of and nearly rejected as too ugly...

That's what happens with a lot of my ideas. They go through the following stages:

1. "Hey, this could work!"
2. "Hmm, there is this one problem that needs solving."
3. "Well, I guess <X> would solve that, but what a hack. There has to be a better way."
4. "Somebody else must have encountered this." [Googles]
5. [Finds that somebody who does (or should) know better is using <X>] "Oh, bugger."

This happens a lot. I don't come up with too many original ideas, but I can definitely find the kludgy, almost entertainingly cringe-worthy unoriginal ones.

This hit me the other day at work, when I met one of the more established engineers in the company. We dropped by his cube, and I noticed three "Patent awarded to ..." plaques on his desk. Now there's a trophy. I was quite impressed at that point¹, and the actual meeting confirmed my first impression. The conversation was enlightening and fast-paced enough to make my brain hurt, something that doesn't happen nearly often enough.


¹As a kid, I always wanted to have patents when I grew up. I need to get cracking :-)