goaccess -f statistics/logs/access_log

That tells goaccess to load with the logfile at the specified location. You can feed it other logfiles, but the default one for a Hippie Hosting account should be at statistics/logs/access_log.

It will prompt you for the type of log file. Select NCSA Combined (arrow-down, hit enter to select, then F10 to continue. yeah. intuitive software…)

It’ll give you something like this, updating live:

Evan performs his first hard reboot of a server. Memory-induced kernel panics keep locking the box up solid, so it needs a kick to get back online. Not impressed with Dell server hardware…

I had to use my ninja-like server management skills this morning. They consist largely of knowing where the power button is, holding it down until the server shuts off, and then tapping it again to light it back up. Solves 99% of problems.

Donncha O Caoimh, the WordPress guru who’s name I’ll never be able to pronounce, released an updated and refined caching plugin called WP-Super-Cache, based on the great start offered by WP-Cache 2.

Traditionally, if someone wanted to run a high-performance weblog, or an enterprise-level weblogging system, they really had only 2 choices.

1. MovableType – because it generates static HTML for every page of the site, and only runs code when posting new content (posts or comments). It can, in theory, scale pretty much infinitely, but can also take for fracking EVER to publish content because it may have to regenerate static files for EVERY page on the site (if you change the theme, you get to create 5,000 new .html files to reflect it).
2. WordPress on some bad-ass hardware. WordPress.com runs on it, but they use 23 bajillion servers, with jaw dropping MySQL clustering and replication. Dynamic content with query caching, etc… Works great, but needs a fair bit of care on the back end to manage the tiers of MySQL (or Oracle?) database servers, and multiple webservers.

Now, there’s a pretty kickass third option.

3. WordPress on commodity hardware, with WP-Super-Cache

WP-Super-Cache does some great caching, storing the output of dynamically generated pages, and returning those rather than hitting the database. It doesn’t even have to fire up PHP to spit out the cached pages – it’s essentially running as a static file server. Kinda like MovableType. But with lazily bound cache that gets regenerated as needed. You don’t have to staticize an entire website, just the pages that are being requested. And they’re updated every hour or so, so the load on the server should be extremely low.

If your server is kosher, it’ll also gzip the cached pages, saving disk space, time and bandwidth (sadly, neither of my servers seem to play nicely with the gzip “super cache” option – I’ll play more later).

Now, my blog likely still won’t survive a full-on Slashdotting because I’m running on an inexpensive shared hosting provider (Dreamhost). But it should survive a pretty respectable amount of traffic (i.e., way more than I currently get) so I should be good to go. And ucalgaryblogs.ca should be able to serve out some pretty decent traffic loads, assuming I manage to actually market it and get folks to use it…

WP-Super-Cache also works out of the box on WordPress MultiUser. It installs as a mu-plugin with minimal effort, once you RTFM. The options, oddly, don’t show up in “Site Admin” but in the “Options” panel on the main blog of a WPMU installation.

I had been running wp-cache to enable file-based caching, thinking that would help optimize performance of the site (fewer database calls should equal better performance) – except that DreamHost apparently uses NFS-mounted storage for accounts. As a result, filesystem access is a bit laggy, so the file-based caching was actually (apparently) slowing the site down (as suggested by 4+1 ways to speed up wordpress). Disabled wp-cache and set define('WP_CACHE', false); in wp-config.php

I also noticed that there were 2 requests for linked files that were returning 404 errors, which in turn trigger my fancy schmancy 404 page and so add significant lag to the page load. Turns out the OpenID Delegation plugin had bad references to openid.js and openid.css so I fixed that, and page loads are at least cosmetically snappier now.

One other modification I made was to (temporarily?) disable the “Similar Posts” plugin and sidebar display. I really like the functionality that provides, but it was adding too much processing time to generating individual post pages. It works by using the MySQL full text index on the blog posts table, which gets a bit slow with lots of posts and MyISAM tables (table-level locking and lots of extra queries means slower site responses). I’ll look into optimizing that a bit and re-enabling in the future.

Also, I had the Mandigo theme set to automatically rotate through a set of banner images, meaning WordPress was having to crunch through the blogbanner/wide directory itself in order to pick up a banner image. Instead, I just set up my .htaccess file to intercept the URL for the default Mandigo banner image, and Redirect it to rotator.php so it should be a bit better now.

It’s still not running as fast as I’d like, but it’s just a blog. I could always trim out more plugins and pick a simpler theme, but I’m pretty happy with the functionality I have at the moment. At least it’s performing better than Twitter…

Update: I turned off the General Stats widget (but left the plugin active) – there’s no need to count every word of every comment and post when displaying every page on the blog. That info is available on the Archives page, where it belongs. That removed several very heavy queries from the typical page generation load.  I also updated Mandigo, which mentioned some optimizations in the changelog. The blog feels snappy enough now for me to stop worrying for awhile…

We hit a minor snag with the configuration – the onboard NICs weren’t properly lighting up. Some quick Googling, and I believe the solution was found in this thread, and involved running this:

chroot /target
# fix initrd
echo megaraid_sas >> /etc/mkinitramfs/modules
cp /boot/initrd.img-2.6.15-26-amd64-server /boot/initrd.img-2.6.15-26-amd64-server.old
mkinitramfs -o /boot/initrd.img-2.6.15-26-amd64-server 2.6.15-26-amd64-server

After that, everything came up roses. Once I had my admin account, it was pretty trivial to get the rest of the bits set up. I had some minor stumbling when trying to build httrack from source, but a tip from this thread led me to the quick command to install the full developer’s toolkit:

sudo apt-get install build-essential

The whole apt-get stuff is pretty sweet. It’s what Fink and darwinports on MacOSX aspire to, but don’t quite reach. Want to install emacs? It’s just a quick sudo apt-get install emacs away. Easy peasy. Databases, ImageMagick, etc… All trivially installed and updated.

So far, setting this server up has been absolutely trivial. And it’s so stinky fast that it should serve the project for quite some time. I might need to set up an Ubuntu client or server locally to play a bit more. It’s not quite MacOSX, but from a server perspective, it’s pretty close. Actually, having spent about 6+ years dabbling in MacOSX’s UNIXy innards, running an Ubuntu Server is not much of a stretch. The biggest adjustment is learning where all of the various bits are installed, but that’s easy. I’ll be spending a fair bit of time over the break, getting my feet wet in Ubuntu. Should be fun!

I did a quick RTFM , but the MySQL manual recommended not jumping right from 4.0 to 5.0 using the normal upgrade process. It said to go to 4.1 first, then to 5.0. So, I did some more Googling, and realized that a full mysqldump and restore would do the trick, without requiring the milk run through 4.1.

I installed a fresh copy of MySQL 5.0.x on my desktop box, did a full mysqldump from the server using:

mysqldump -u root -p --opt -Q --add-drop-table > mysqldump.sql

That gave me a 400+MB SQL file, handy for a backup, or for migrating between copies of MySQL. I copied the file to my desktop, and ran this:

mysql -u root -p < mysqldump.sql

mysqladmin -u root -p --flush-privileges

And, it Just Worked™ – so the data should safely migrate this way. I did some testing with apps – Drupal etc… and thinks behaved as expected. It takes longer to dump/restore than just updating the MySQL server, but it's safe. And all database users and privileges were intact, so everything should hook up just fine.

Then, I did the same on the server. I killed MySQL, which was still running version 4.0. I ran the installer (both MySQL 5.0 and the StartupItem) via SSH (sudo installer…) and it took care of setting stuff up. But, the old MySQL didn't shutdown cleanly, so I had to do the unthinkable – reboot the box. I know there's a cleaner, more "official" way to clean stuff up, but the box had been up for a couple months, so a reboot wasn't the end of the world.

When the server came back up, MySQL was running as expected. Then, it was a simple. First, I told my shell to use the new copy of MySQL, then imported the data:

alias mysql=/usr/local/mysql/bin/mysql

alias mysqladmin=/usr/local/mysql/bin/mysqladmin

mysql -u root -p < mysql.sql

mysqladmin -u root -p --flush-privileges

And all of our apps are up and running on a shiny new copy of MySQL. The previous version of MySQL is still on the server (but not running) with all data still safe in its own directory, so if something catastrophic has happened, reverting will be trivial.

Now, to upgrade my copy of Moodle to 1.6, which is what prompted the whole MySQL upgrade process in the first place…