Let us say that something horrible has happened to every application you have. And let us say that something horrible happened to all of your backups at the same time.
Armegeddon? Yeah, maybe. But what do you do? Well, you have to rebuild from the ground up. But how?
There is one thing, that if you have it, can mean the difference between rebuilding and sitting in the ashes of your company and cursing whatever higher being that happens to be available at the moment. And that is a runbook.
A runbook is very simple and yet you’d be amazed at the number of places that A) don’t have one or B) never keep it up. A runbook is basically instructions for the morons in Operations (like me) to rebuild your application from the ground up. Simple as that. You make a change to your application, the record of the change goes in the runbook. You move to a new server, the notation goes in the runbook. You do anything to your application, it needs to be noted in the runbook. That way, if disaster strikes you have an ace in the whole.
So who creates and updates the runbook? The development team. And who houses the runbook? Operations. Why? Operations houses it because we are going to be the ones who need it should something happen. Because Operations has multiple applications to watch. We do not know the intricacies of your application like the people who created it. So do not ask Operations if what is in the runbook everything we need. Short answer: we don’t know. It’s your application. You hand us the runbook with the understanding that you have put YOUR knowledge into it. Now when a change goes into the runbook, it is operations responsibility to note the change went in and when. But if you decide to move to a new server or make major structural changes, again, it is the development team’s responsibility to update.
And that is where the problems with runbooks lie. Everyone is pressed for time, no one ever has a technical writer when needed and all of us have a tendency to say “We’ll work on that tomorrow”.
And sometimes, tomorrow is not exactly what you expect.