Or, how I spent about 15 hours debugging our MediaWiki installation at wiki.ucalgary.ca, trying to figure out why file uploads were mysteriously failing.
We’ve got a fair number of active users on the wiki, and a course in our Werklund School of Education’s grad program is using it now for a collaborative project. Which would be awesome, except they were reporting errors when uploading files. I logged in, tried to upload a file, and BOOM, got this:
Could not create directory "mwstore://local-backend/local-public/c/cf"
um. what? smells like a permissions issue. SSH into the server, check the directories, and yup, they’re all owned and writable by apache (this is on RHEL6). Weird. Maybe the drive’s full? df -h. Nope. Uh oh. Maybe PHP or Apache have gone south - better check with another site on the server. Login to ucalgaryblogs.ca and upload a file. Works perfectly. So it’s nothing inherent in the server.
Lots of searching, reading about LocalSettings.php configuration options. Nothing seems to work. I enable Mediawiki logging, check the apache access and error logs, and find nothing. It should be working just fine. The uploaded file shows up in the /tmp directory, then disappears (as expected) but is never written into the images directory. Weird.
So, I try a fresh install of mediawiki elsewhere on the server (in a separate directory, with a new database called ‘mediawikitest’). Works like a charm. Dammit. So it’s really nothing wrong with the server. Or with mediawiki. Maybe there’s some freaky security restriction on the new server1, so I set up a new VirtualHost to spin up the new MediaWiki install in exactly the same way as wiki.ucalgary.ca (using a full hostname running in its own directory, rather than as a subdirectory of the “main” webserver’s public_html directory). And it works like a charm.
Hrm. Searching for the error message turns up mentions of file permission errors, and file repository configs. I mess around with that, but everything looks fine. Except that uploads fail for some reason.
Maybe there’s something funky about the files in the wiki.ucalgary.ca Mediawiki install - it goes back to May 2005, so there’s over 9 years of kruft building up. There’s a chance. So I copy the whole wiki.ucalgary.ca mediawiki directory and use it to host the test instance (still pointing at the mediawikitest database). Works fine. So it’s nothing in the filesystem. It’s not in the Apache or PHP config. Must be in the database.
So, I switch the test instance to use the production mediawiki database (named ‘wiki.ucalgary.ca’). And uploads fail. Dammit. I assume something is out of sync with the latest database schema, so I eyeball the ‘images’ table in both databases. AHAH! Some of the field definitions are out of date - the production database is using int(5) for a few things, while the new test database uses int(11) - maybe the file upload code is trying to insert a value that’s longer than the table is configure to hold. So I manually adjust the field definitions in our production images table. That’ll solve it. Confidence! But no. Uploads still fail. But the problem’s got to be in the database, so I modify my search tactic, and find a blog post from 2013:
Problem solved. Turns out the new database backend thing in mediawiki doesn’t like database names with dots in them, and doesn’t tell you. Thank you Florian Holzhauer for finding it!
Dafuqbrah? Really? That can’t possibly be it… The wiki’s been working fine all along - it’s up and running and people are actively using it. If the database wasn’t working, surely we’d have noticed earlier…
renames database from wiki.ucalgary.ca to wiki_ucalgary
uploads file
Son of a wiki. It works.
So. At least 15 hours of troubleshooting, debugging, trial and error, modifying configurations, installing test instances, and being completely unable to figure it out. And it was a freaking . in the database name that was doing it. With no mention of that in any error message or log file. Awesome. An error message that says “could not create directory” actually means “hey - a portion of my code can't access databases with . characters in the database name - you may want to fix that.”
-
we moved recently from an old decrepit server onto a shiny new VM server hosted in our IT datacentre, which is awesome but I’m rusty on my RHEL stuff, so there’s a chance I’m missing something important in configuring the server… ↩︎