Lair of the Wolf

This Wednesday, the server that hosts firebot will be down for several hours (during the afternoon/evening EDT) for planned maintenance. This, of course, means that firebot will be offline and not available for your bug watching, tree burning, and factoid requesting pleasure. Thanks.

One of the things that has bugged me over time is how annoying Apache's usual file-based passwd files are to manage and deal with. I don't change the configuration very much as a result. adding a new user is tedious, as I never remember the syntax to pass to the utility to create the file, etc. Being on windows, the joys of modules like mod_auth_mysql just didn't work well, (in fact, in my experience, crashed.) and was rather unreliable.
So, when Apache 2.2 was first released, I was excited that one of the new features was SQL Database support. Sweet, I thought, only to quickly find out that, it wasn't compiled by default for Windows yet. No big deal, I'm rather used to bleeding edge features being a compile-it-yourself kind of thing in open source applications.

Last Fall though, I discovered that the module was included in the windows distributions and set about trying to set things up, following the documentation which instructed me to specify a DBDriver directive to select which driver, and even specifies as an example, mysql. Of course, apr_dbd_mysql.so, as referenced on that page wasn't included, So I set out to find a copy, something I was used to doing for mod_ssl.so, only to find little info, and then to discover, that not only was the module not available by default, it wasn't yet even in the apache source tree but rather a 3rd party because of licensing issues, and further, that in the 1.2.x version of apr-util, it didn't even have dynamic module support for the db drivers, so no such apr_dbd_mysql.so existed without patches to the to eventually become 1.3.x apr-util unstable trunk.
Not to be beaten, I decided to give the module a try on my linux apache install, patch source tree here, figure out to reconfigure apr-util to be mysql enabled and rebuidl, then install, then figure out how to convince mysql to get along so authentication actually worked, after a few hours of banging it surprisingly worked, and worked well. Not ready for my other machines though.

So, that's alot of history, and here's the good news. since that time, Apache now bundles and is using what was then only a myth, APR 1.3.0. In fact, by default in the windows distribution, you'll actually now have dbdrivers to pick from, pgsql, sqlite3 and oracle. Still no MySQL. I've seen a couple of possible reasons for this, one from the apr-util readme.mysql which suggests licensing issues still apply, not sure if this applies to the httpd distributions or not though. The second, and more important issue I found in a readme.win32 in the httpd-win32 source distribution, also printed here, that the MySQL driver wouldn't compile on windows. Sure enough, after setting up a MSVC6 build enviroment with the appropriate Platform SDKs to build apache, trying to build apr_dbd_mysql-1.dll resulted in several fatal errors regarding the use of "long long" types which VC6 (which the ASF binaries are built with) doesn't support. This turns out to be a known issue, already fixed in apr-util 1.3.2 which has been released. (I fully expect that future apache versions (such as a 2.2.10 whenever that is supposed to be) will likely use this version, and whether or not the mysql driver is built we'll have to see then, but until then...) So I quickly grabbed the windows source trees and after a little bit of fighting with VC6, which i'm rather new to, (No, I don't want to build APR for x64 with VC6 on a Intel 32-bit CPU, thanks) I successfully compiled apr_dbd_mysql-1.dll. After dropping in the configuration from my tests last year, the new DLL into the bin folder and restarting, things seemed to be ok. (Without the DLL of course, you get this error "DBD: mod_dbd not compatible with APR in get_driver" which is a rather cryptic way of saying that APR doesn't have the driver you're looking for.) Pointing my browser to my test directory though, gave me a prompt for password and then promptly, an Internal Server Error (500). doh (Along with this very helpful error message in the log "[error] [client 127.0.0.1] Failed to acquire database connection to look up user 'wolf'") reminding me that the apache mysql user actually needed privledges to read from the new DB table.. Fixing that up, and boom, it works. I have only tested it on a limited basis and haven't tried it in production yet, but much like my previous efforts at compiling, I wanted to share with the great internet masses who don't have access to a copy of VC6 and just want to enable this functionality into their Apache 2.2.9 (or later, most likely, but not earlier, since it requires APR-1.3 to work) to share in my happiness...

So here you go, there's no warranty here, and if it fails I most likely cannot help you. That disclaimer aside, go ahead eager one, grab it. :-) It drops right into the standard apache distribution for windows.

Configuration Notes:
This is my configuration, scrubbed for passwords and such.. I have it at the end of the general configuration, before my virtual hosts.. Most of it was stolen from the Apache docs, which aside from being a bit vague about DBDriver are pretty good.

This is probably going to be a bit of a rant. After having spent the last few hours playing with trying to get TVersity 1.0 RC1 to properly read and transcode video I captured using a Plextor ConvertX/Intervideo WinDVD Creator 2.. its pretty straightforward video, it captures at 720x480 and compresses using Divx 6 (6.8 is installed on the machine) but annoyingly enough it doesn't use mp3 for the audio, but rather mp2. Which, I suppose is understandable. Unfortunately, its not the easiest format to work with, there's no VfW decoders for mp2 audio that are free, due to licensing restrictions, which makes tools like VirtualDub fail to be able to convert the audio to something more friendly.
Where the fighting comes in, is that the digital media player here, the D-Link DSM-320, which is pretty nice, does not support MP2 audio in an AVI. Even though the profile tversity has for the device, which specifies what needs to be transcoded to be able to play vs. what can be played natively, enabling a much wider range of videos to be played, thinks it can. The result, audio that's nothing but skipping static. and lots of it. Eek. Luckly just a simple comment in the profile.xml file for the device telling the software that an mp2 in an avi doesn't work, is enough and the transcoder now kicks in to handle the conversion to a format the player can actually handle.

The transcoding feature though, while very nice when it works, can be an absolute headache to get working properly with a variety of formats. Anybody who's saved videos from the web or captured stuff over a long enough time has no doubt collected quite a mess of different formats, even if they only appear to be a few types.. (the container types tend to be nice and misleading to make you think there's only a few.) commonly nowadays, flv, wmv, mov, and avi.. with rm (and ogg) in there somewhere.. this of course, left out all the mpeg, mp4, m4v and the mountain of extensions those use. On top of that, if somebody hands you a .mov, is it H264? MJPEG? or something else? If you're just using Quicktime, its not a big deal, since any type the file can be will play with it, but then you end up with a pile of players installed on your system fighting over file types since they all want to be the one true player, but fail in subtle ways abysmally. As you'd expect though, not everything can just go through the one true player for playback and not expect things to be a bit more compatible, TVersity's transcoder is one of those things. On Windows, which is the only platform for now, it supports, it uses DirectShow to read the videos, so any video you can play in Windows Media Player (or any DirectShow capable player) can be read, and played back to your device.

So this brings up the obvious problem, how do you get DS to be able to handle the popular formats? Well, let's start with a (hopefully) simple explaination of how video is played back...

The image above is graph of youtube flash video, as it would be opened by directshow. First thing is the container is opened by the appropriate filter, in this case the "FLV Splitter", flvsplitter.ax, which as its name implies "splits" the video into its component parts, which are then read to determine what compressor was used, (for youtube, this is currently H.263 for video with MP3 audio) and the appropriate decompressors are selected and the decompressed data would then be passed to to your soundcard ("default directsound device") and player screen ("Video Renderer"). In my case, both H.263 and MP3 for video is handled by ffdshow, but a variety of directshow filters exist.

So, if everything goes well, the flv you saved will now play back in all its glory for you. For quicktime, using mp4splitter.ax ("MP4 Splitter") will usually work since the quicktime format is mpeg4 based. Though some quicktime files seem to like having more quicktime components available, so installing Quicktime is sometimes the only solution. The MediaPlayer classic sourceforge page includes most of these filters, which are really handy. After you get and install the filters you'll need, you'll want to install FFDShow, which is a very nice set of decompressors/compressors for a variety of media types (including all the FLV types, H263, VP6 and H264.)

Now for the rant.... Why is all this so hard? Why, to make use of a fairly simple device do I need to install a crapload of filters and decompressors to transcode video that the 'media server' can't even handle... I know I have an above average variety of videos locally to deal with, some compressors like Indeo and cinepak and all the mpeg varients as well as the average quicktime, flash, real and divx videos, and I don't expect the older videos or the more obscure compressors to work, but its not too much to ask for a real error message when they fail, perhaps with some indication whats wrong, Video codecs have long confused users of all skill levels, people just want their computer to work.. not to spend hours trying to figure out which combination of codec and container in their matrix they missed installing software for. If you send Grandma a video of your cat playing with a ball of yarn, having to think about if they have quicktime (which your digital camera was so nice to preselect for you.. ) or not shouldn't be needed. Thats one of the reasons why Youtube and similar sites which, yes, use flash, which most systems have today, its easy and you know on the other end it'll play. And surprise, now that its become easy to share video without the headaches, millions of people have chosen to do so. Its really a great thing. It does come with its own problems though, alot of people have difficulty or don't know how to save or play back web video later, and content authors often probably think their video will only stay on sites like youtube. (thats speculation, but it makes sense to me anyway).
Its already been proven that the various organizations and companies that develop these compressors and decompressors and players have no intentions of making that world any easier on the rest of us. Otherwise various platforms would have standard ways (like directshow) and players would actually, provide the filters to enable their formats to play, and not try to lock-in users to use only their players...

Now, here comes more fun, though only tangently related to the other paragraphs in this lengthy and wordy post.
Now that video is the web's next big thing.. (thanks to youtube and flash player.) the slow, but steady standardization people have decided, that what we really need in the next version of html, is a