Bill Clementson's Blog

 Bits and pieces (mostly Lisp-related) that I collect from the ether.

May 2007
Sun Mon Tue Wed Thu Fri Sat
1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31
Apr  Jun

Backups and the Cobbler's Children

Thursday, May 3, 2007

There is an old saying that "the cobblers children have no shoes". You often see this with IT companies and disk backups. In fact, Business 2.0 magazine was recently a victim when it failed to heed its own advice and lost the June issue of the magazine. But, it's not just IT companies that don't back up their work - lots of individuals working in computer-related fields are also guilty of poor backup routines. For example, when Ted Leung and his wife both had their laptops stolen last year, he only had an old backup for his machine and there was no backup for his wife's machine.

I can't throw any stones though - my personal backup routine has been a bit (ehem) "sporadic" as well. I "periodically" cut a CD/DVD with stuff that I don't want to lose; however, if the house burned down, all of my backups would probably be ruined too as everything is kept in the house instead of at an off-site location. Sometimes, I send myself a document via gmail so that I have an off-site copy; however, this is not consistent nor is it very effective. So, if something happened to my laptop or its hard drive, it would be difficult to reconstruct my data and work environment. I've been meaning to buy an external hard drive and formalize my backup strategy, but it's always been something that I've put off "till I had more time".

Recently I read a series of articles that Matt Jadud (who I wrote about in my "Parallel Computing in Lisp - Part 4" post - see, there is "some" lisp content in this post after all!! ;-)) had written on the subject of backups (see here, here, here, here) and these (coupled with Matt's recent post about his hard drive failure) inspired me to get a backup plan in place. I decided to follow Matt's own strategy and use a combination of Amazon S3 and Jungle Disk to do automated off-site backups. The Amazon S3 service is really very, very cheap (some price comparisons are here and here), so it is a very cost-effective backup solution (in addition to being a safer solution than local backups). The nice thing about using the combination of Amazon S3 and Jungle Disk is that:

  1. You're using Amazon's own highly scalable, reliable, fast, inexpensive data storage infrastructure to backup your data
  2. You can connect using SSL and automatically encrypt your data
  3. It's fairly fast (for a non-local solution)
  4. Since Jungle Disk runs as a Webdav server, you can treat S3 as just another disk (so you can use other software to write to it as a disk drive and Jungle Disk does the secure-connection/upload/download/encryption/decryption behind the scenes)
  5. Modification stamps are retained on the backed-up objects
  6. The next release of Jungle Disk will also support backup of in-use files and incremental backup of only the changed portion of large files
I initially wavered over using a third-party, closed utility like Jungle Disk or going for an open source or home-grown alternative, so I looked at a number of different options. The alternatives to Jungle Disk that I considered were:
  1. JetS3t: an open source, Java utility.
  2. Use standard utilities and scripts: for example, like Phil Windley did
  3. Use a CL option: both CL-S3 and CL-WEBDAV were potential "starting points"
In the end, I decided for the "quick solution". If I had to spend a few hours (or days) writing something, that would just delay my getting a backup solution in place again. And, there's nothing to prevent my replacing Jungle Disk with a homegrown solution at a later date. So, I signed up for S3, downloaded Jungle Disk, re-arranged my directories a bit, and kicked off the backups.

Basically, my backup strategy is to backup primarily data and some executables. If I had a drive failure, I would reinstall any Mac OS X applications that I use on a regular basis. So, in addition to saving documents, I save configuration data for some programs. Here's what I save (all saved directories include the subdirectories under each directory):
  1. ~/ : all of the "dot files" in my home directory
  2. ~/bin : all of my custom scripts and utilities
  3. ~/Documents/docs : since lots of programs automatically stick stuff in the ~/Documents folder, I didn't want to backup up the entire folder - instead, I keep the documents that I create in a separate "docs" folder and back that up
  4. ~/Documents/Financial : financial/tax data
  5. ~/Library/Application Support/AddressBook : still haven't completely weened myself off of Address Book yet
  6. ~/Library/Application Support/Firefox/Profiles/92ba5kdv.default/bookmarks.html : just backup the Firefox bookmarks
  7. ~/Library/Application Support/iCal : still haven't completely weened myself off of iCal yet either
  8. ~/Library/Application Support/NetNewsWire : the rss feeds that I read
  9. ~/Library/Application Support/Quicksilver : Quicksilver configuration backup
  10. ~/Library/Keychains : geez, I'd hate to have to setup all of those again!
  11. ~/Library/Preferences/Aquamacs Emacs : configuration info for Aquamacs (the version of Emacs that I use)
  12. ~/lisp : lisp code
  13. ~/Music : iTunes backup
  14. ~/Pictures : iPhoto and other pictures
  15. ~/Documents/Quicken Backup Folder and ~/Documents/Quicken Data.qdfm : Quicken data
  16. ~/Sites : backup of my web site and blog
  17. ~/Documents/jde : backup of JDE programming-related info
  18. ~/Documents/downloads : backup of documents and ebooks that I have downloaded
Now, everything of importance is backed up on a periodic basis and I don't have to worry about losing my data. You can't believe how much of a relief that is!

Update-2007-06-22: Updated list of directories/files that I back up.

emacs Copyright © 2007 by Bill Clementson