My Backup Workflow

Today I'm going to talk about perhaps the least sexy topic of all: backups.  Backups are an absolutely essential part of any serious photographic workflow, and as much as it's hard to spend money on boring stuff like hard drives and not on cool things like film and lenses, it's a necessary evil if you want to avoid serious headaches.  In fact, I'm writing this partly because I've just been through some serious headaches with a drive failure, and I'm hoping that this tutorial will help you avoid the headaches; or at least minimize the amount of Tylenol you need to cope. First, the basics.  All of my post-processing work - be it of film scans or digital raw files - is done in Adobe Lightroom CC 2016.  I've messed around with other digital asset management programs (Capture One Pro and others), but Lightroom just feels right to me.

The ins-and-outs of Lightroom are beyond the scope of what I'd like to deal with here today, but suffice it to say that I use a single Lightroom catalog for all images, be they current work or archives.  I know some people use different catalogs for different projects or for different years, and that's perfectly fine.  But using a single catalog allows me instant access to everything I've ever shot.  My catalog resides on an external OWC Elite Pro Dual Mini, which consists of two 256GB OWC Mercury 6G SSD drives in a RAID-0 stripe, giving me 512GB of blazing fast storage for my Lightroom catalog, Lightroom cache, and Photoshop scratch disk.  Frankly, this is overkill, but it does allow me to have very large cache and scratch disk files, which speeds things up enormously. (Ed. - this is the backup component that a lot of people forget about. Your images are important, of course, but backing up the associated metadata - including all develop settings, keywords, etc - is nearly as important.  Imagine having to redo edits for an entire year's worth of images...)

My work computer is a 2014 5k Retina iMac, with a 4.0GHz Core i7 (4790K) processor, 32GB of RAM, 512GB of internal solid-state storage, and a 4GB AMD Radeon R9 M295X GPU.  This machine is incredibly fast, and the 27" retina display is gorgeous.  I've got it calibrated using a ColorMunki XRite, and things look wonderful to my eyes.

I 'scan' my film using a Nikon D800 and a Tokina 100/2.8 macro lens, so all of my scans  are in .NEF raw format.  These are imported into Lightroom, along with raw files from my Fujifilm digital cameras.  All of the image files from the current calendar year live on a drive called 'MASTER'.  For this, I'm using two HGST 2TB 7200RPM drives, in a 4TB RAID-0 configuration, inside an Other World Computing Elite Pro Dual Thunderbolt enclosure.  I elected for Thunderbolt over USB 3.0 because, frankly, it's faster and, as you'll see, I'm a bit starved for USB 3.0 ports on my my 5k iMac. The OWC TB enclosure offers 10Gps bidirectional transfer, which is FAR more than two HDDs can put out, even in a RAID-0 stripe, so I'm basically delivering images to Lightroom as quickly as possible, without going completely mental with an all-SSD approach.

Image files from previous years are stored on another 4TB drive (an OWC Mercury Elite Pro) called 'ARCHIVE'.  Since it isn't accessed as frequently, this drive doesn't need to be as speedy as 'MASTER' does, so I've gone with a single 4TB drive, connected by USB 3.0.

On each drive, files are stored as follows:

YEAR/FILM/ROLL_ID*/

or

YEAR/DIGITAL/MM_DD

* - I name my film rolls sequentially as I scan them, not as I shoot them.  So 2016_0050 may have been shot after but scanned before 2016_0060, for example.

Each of these two drives ('MASTER' and 'ARCHIVE') are automatically cloned (scheduled at 12:30AM every night) using SuperDuper!, to two other 4TB drives (Seagate Expansion 4TB), connected by USB 3.0.  These drives are called 'MASTER CLONE' and 'ARCHIVE CLONE', respectively.  This protects against failure of my main image drives, and the nightly cloning ensures that no backup is more than one day old.

This isn't all, though.  While I'm protected against drive failure, I'm not protected against fire, flood, or theft.  For this, I have a second line of backup: Backblaze.  Backblaze provides 'always-on' cloud backup for both my 'MASTER' and 'ARCHIVE' drives as well as the clones of each.  It also clones the SSD RAID-0 drive which houses my LR catalog and LR/PS cache.  This cloud-based solution has one major downside: the initial backup is SLOW; even over my 10Mbps upstream connection, initial backup took on the order of two weeks.  Fortunately, I have unlimited monthly bandwidth from my ISP, so this was not an issue (Ed. - Backblaze does offer the option of sending in your data on an external HDD, which is a good option if you have a slow connection or limited bandwidth usage).  Once you've done the initial backup, however, the bandwidth requirements are quite small, since you're only uploading new files.

So I've got local backups of my images to protect against disk failure and cloud-based backup of everything to prevent against local catastrophes (floods, fire, locusts, rapture, etc).  Restoration from local backups is straightforward; simply reformat a new main drive, clone the backup to it, and point Lightroom at the newly populated drive.  Restoration from cloud backup is easy if it's just a few files (say, 20GB or less) that have been lost (just download a .zip file from Backblaze), but is a bit more cumbersome if you need to restore an entire drive.  In that case, it's best to have Backblaze send you the files on an external HDD, since downloading 4TB of data takes a very long time, even on a 50Mbps downstream connection.

And if my Lightroom catalog gets corrupted, I just restore from the cloud backup and I'm back in business.

This is what it looks like:

Backup
Backup

So there you have it.  16TB of local storage, all mirrored to the cloud.  Phew.  Now to sort out the cabling...