You’ve been waiting patiently, expectantly, dying to know what could make the rapture look like a Tupperware party. Well I did some research. On the one hand, I had confused Tupperware for Rubbermaid. So that rather reduced the magnitude of my promise since Rubbermaid is way better than Tupperware. On the other hand, I had confused the rapture for the time Jesus helped a guy out of a ditch. In fact, the rapture is when the most righteous of Christians get to go bodily to heaven, and the rest of us fight over dwindling resources, as foretold in direct to DVD prophecies.

Boy, was I embarrassed to learn that! I mean, I thought I just had to compete with the miracle of Jesus not getting muck on his robes. I was prepared for that. But I have to admit, what I had in mind just does not compete with the rapture. Please give me another week or two to punch it up a bit.

I was left hunting around for something to fill the gap. So, on Monday (yes I know this post is late) I completed a total upgrade of this blog. Please let me know if you spot any problems. There are very few user-visible changes I can show you. Instead, I’ll write about the process of upgrading.

It started out well! It was a joy to use Mercurial (http://www.selenic.com/mercurial/) to prepare the upgrade. Hg (as Mercurial is also called) is a distributed version control package. For the uninitiated, “version control”, “revision control”, “source control” and other terms all refer to software that manages a history of annotated changes to a set of files. It allows you to do things like look back in time and see what was in a file last month, generate reports of changes, and find the reasons the author gave for making the changes. More importantly, it allows the history to “fork” and “merge”. I can revert my copy of the files to an earlier state and start making changes; the version control software will keep track of the two parallel histories. I can also take two of these forked paths and merge them back into a single version. These features allow a developer or multiple developers to work on various parts of the software at the same time, without the half-completed changes interfering with each other.

The “distributed” qualifier refers to hg’s network model. It is (I think) still the new thing in version control. A traditional version control package has a centralized repository. Developers get a “check out” from the repository. The check out contains a single snapshot of the repository’s history. Changes are sent back over the network to the repository. With a distributed system, there is no central repository. Typically, every check out will contain the entire history (ie, the repository itself). Actually, it is misleading to say that there is no central repository. What I mean is the model does not demand a central repository. However, because you end up with all these copies of the same repository spread out all over the Internet, there is usually a managerial need to bless one of them as the official one against which all others are coordinated (perhaps indirectly).

So as I was saying, Mercurial was one of the better points of the upgrade process. The blog software I use — byteflow (http://byteflow.su/) — is distributed via a Mercurial repository, and I prefer to use Mercurial for my own development, so that part went swimmingly. First I used hg to clone my existing deployment, which was itself a fork — a modified version — of the blessed byteflow repository. My local customizations included this great visual theme, some configuration files, and so on. Then within the clone, I “pull” all the changes that the byteflow developers have made since I started this blog into my repository, and then I “merge” those changes into my check out. Great! I have now upgraded my blog, sorta.

Except, oh, I notice there is a file with a conflict. A conflict occurs when two branches (ie, forks) modify the same line of a file in an incompatible way. I started from an old version of byteflow and made some changes to it (thereby creating a fork), then when I tried to catch up to the latest version, the changes I had made correspond to lines that don’t exist anymore. It seems the byteflow developers changed the way the software is configured slightly, and I had to re-write some of the configuration file.

So now I’m mucking about in Python. A strange complaint, coming from me. Let me explain. I consider myself a casual user of byteflow. Casual users suck. We think that applications should be completely self contained and self explanatory. If I have to configure something as a casual user, I expect the application to pop up an alert box saying, “Hey! You have to configure something! Here’s the configuration interface, do it now and we can proceed.” We’re basically morons and we don’t like our noses rubbed in it.

But, in addition to being a casual user of byteflow, I also happen to be a Python developer. So I did not even notice at the time that I had momentarily traded hats, I just put on my coder beanie without thinking and slipped it back into my pocket a moment later. I only mention because it becomes relevant later in a conceptual sort of way.

Okay, so I have a copy of my blog deployment (saving my live copy from the not unlikely completely chance I will ruin everything and need to start over), and it has my old changes in it, along with byteflow’s new changes, plus the changes I needed to marry the two other sets of changes. Phew. Great. Time to test. Start it up, load the page… Hey! It looks just like when we started! There’s my awesome visual theme, there’s my… oh, hey, where are my posts? Oops, I forgot about the database.

Byteflow is built on top of Django, which is a platform for web applications written in Python. One of the services Django provides is access to a database. A Django application like byteflow specifies how it needs the database to be organized, and then Django does the grunt work of setting it up. Django does not itself contain database software; that is yet another component that I, the long suffering user, must provide. But to its credit, Django knows about a lot of different database packages.

However, in the interval between my initial deployment and this upgrade, the database description had changed. Django itself is… less than helpful on this occasion. Quoth http://docs.djangoproject.com/en/dev/faq/models/#if-i-make-changes-to-a-model-how-do-i-update-the-database,

Q: If I make changes to a model, how do I update the database?

A: If you don’t mind clearing data, your project’s manage.py utility has an option to reset the SQL for a particular application:

manage.py reset appname

This drops any tables associated with appname and recreates them.

If you do care about deleting data, you’ll have to execute the ALTER TABLE statements manually in your database. That’s the way we’ve always done it, because dealing with data is a very sensitive operation that we’ve wanted to avoid automating. That said, there’s some work being done to add partially automated database-upgrade functionality.

My heart sinks. Yes. Yes I do care about deleting data (ie, all my awesome posts). And furthermore, as a casual user, I think you suck, disembodied voice of the Django FAQ. It looks like the only way forward is to get into the database up to my elbows and mess with its guts. My hand reaches for my database monkey fez… hesitates… wilts back. For a day I mope around as casual user and pity myself.

It’s not that I don’t like databases, not in the abstract. They are very handy. I don’t even dislike relational databases (which is what your big name big money big big database companies tend to make). I am comfortable with database design. I will even administer one if the money is good. The problem is SQL. I am DEATHLY allergic to SQL. I get the sweats, the vapors, the shakes and the shimmies. I get tremors and phantom perceptions of temblors. My mouth goes dry and I drool down my shirt. I suffer from obscene priapism accompanied by impotence. I become beside myself with incontinence. I smell toast. I try to explain this alarming and completely legitimate medical condition to my employers, and they don’t believe me!

SQL (pronounced by most as sequel, man that bugs me) stands for structured query language. It is a standard language shared by many database packages. It allows you to set up the database organization (known as the schema) and to add, remove and search for data. Oh, wait, did I say “standard”? Ha ha. That is a joke that database people like to play on us. Implementations of SQL are “standard” the way dress sizes are. It doesn’t really mean anything.

Anyway, I’ve managed to insulate myself from SQL quite well. When I need a relational database, I will usually grab Divmod’s excellent Axiom. It is a Python layer over a relational database. The database they happen to use is called sqlite, but I don’t even have to know that to use it. As with Django, the database is specified and accessed through an entirely Pythonic interface. Unlike Django, Axiom only works with one database package.

Axiom has a distinct advantage that left me unprepared for this unpleasant situation. It was designed conscious of the fact that often changes to your application require changes to your database’s organization. When you change the database description contained in your application, you can provide steps for upgrading existing databases, again written in Python. Cracking open the database and squirting a bunch of SQL at it, that is unheard of with Axiom.

At first I was left wondering why Django would make a choice like this. There is no way a casual user can do this. Either you need to hire a database monkey or you just hope that the developers of the application you are using have gone to elaborate measures to handle this situation for you. The reason started to become clear shortly after I found a shred of salvation.

The developers had indeed made some effort to leave me a lifeline. Whenever they’ve changed the database, they’ve included a snippet of SQL code that maybe a slightly less casual user could use to upgrade their database in place without having to think hard. Hurrah! I did a little sleuthing to figure out where my last version left off, and tried to run the snippets on my database.

ERROR! DOES NOT COMPUTE!

My database won’t accept it. Turns out that my database — sqlite again — doesn’t fully implement the part of the SQL specification that deals with modifying schemas. It rejected about half of the snippets. I’m having nightmare visions of trying to migrate to a new database package. Now I understand why Django does not handle upgrades painlessly like Axiom does. Django has to support all these different crazy implementations of SQL, and Axiom has only one to worry about.

But a bunch of the snippets DID work… And when I try to load the front page now, hey! It’s full of posts! Awesome! I think I’m out of the woods. I try to add another post to my blog. The Universe comes to an end. I guess those SQL statements that wouldn’t run are important. More research. There is a workaround. You can’t change the tables of the database in-place, but you can create a temporary table, move the data, create a new table in the old location with the new organization, move the data back, and delete the temporary table. I look at the fez. It laughs. I’m going to have to put it on after all if I’m going to implement this workaround. I balk.

More time passes. I sulk. I glower. Nothing changes. Finally, I blow my nose, hitch up my jeans, and take another stab at it. By which I mean I googled around to try to find some way out of typing a few lousy stanzas of SQL. And, hurrah! I found: SQLite Manager. It is a firefox extension (of all things) that helps you work with sqlite databases. One of its features is that it does the above mentioned workaround dance for you. And just like that, one download and one firefox restart, the last of my problems are gone.

I push my changes from the testing deployment to the live site, copy up the upgraded database, and here we are. A brand new, just like before Experiment ZM. As I gaze lovingly on my imperceptibly better blog, I reflect on the above experiences.

I mean, look at it. This description, while glossing over a lot of details, goes on and on and on for pages. I’ve tried to write it so that someone who is not in the software thing would be able to follow along, and it contains some irrelevant asides, but a thorough technical description would have been even longer! That’s craziness. Why do I do this when I could just get a Blogger account or something? Perhaps a topic for another day.