When Upgrades Go Bad January 27th, 2008

There have been a number of 500 errors on a freelance site I'm on. For the first round of troubleshooting, I'm updating the web server. My favorite web servers are nginx and lighttpd, but this dedicated server has two legacy apps, both built around Apache.

My favorite platform is Debian Linux because updates are fast and painless. This legacy server runs gentoo.

Unfamiliar Operating System + Unfamiliar Software == Disaster

At 11:00 pm I ran the upgrade. When it came back up, I found my Rails app working wonderfully. For safety, I checked the legacy PHP apps, and they were down. Hard.

It's midnight. I estimate I have until 8:00 a.m. before I'm in deep trouble.

After much research, I learned Gentoo decided to upgrade me from Apache 2.0 to 2.2, and 2.2 breaks a lot of things. One of which is fastcgi, now replaced with fcgid. I haven't used fastcgi in a year, since Mongrel makes it irrelevant for Rails. Fcgid will require much hacking.

It's 1:00 a.m. It's clear I'm better off installing mod_php. You'd think it's as simple as "sudo emerge mod_php". Well mod_php has merged with the base php package. you enable mod_php with a flag in some file. I enable the flag.

I start the compile at 2:00 a.m. And wait. And wait.

You see, Gentoo's philosophy that everything should be compiled from source. So for two hours, php compiled, just so I could get mod_php.

It's a little past 4:00, and things are back. What have I learned?