A few times on Twitter, I’ve mentioned how “easy” it is to move stuff between servers using the rsync shell command. It’s actually an extremely powerful program for synchronizing two directories – even if they’re not on the same volume, or even on the same computer. To do this, you’ll need to login to one of the servers via SSH. Once there, invoke the geeky incantation:
rsync -rtlzv --ignore-errors -e ssh . username@hostname:/path/to/directory
What that basically says is, “run rsync, and tell it to recursively copy all directories, preserving file creation and modification times, maintaining proper symlinks (for aliases and stuff like that), compress the files as they’re being copied in order to save bandwidth, and provide verbose updates as you’re doing it. Use SSH as the protocol, to securely transfer stuff from the current directory to the server ‘hostname‘ using the username ‘username‘. On that destination server, stuff the files and directories into ‘/path/to/directory‘”
What you’ll need to change:
. – if you want to specify a full path to the source directory, put it in place of the . (which means “here” in shellspeak)username – unless your username on the destination server is “username” – you’ll be prompted for the password for that account after hitting return and starting the program working.hostname – the IP address or domain name of the server you want to send the files to./path/to/directory – I’m always a bit fuzzy on this. Can it be a relative path? from where? So I just specify the full path to where I want the files to go. Something like /home/dnorman/
Because it compresses files, it’s actually pretty efficient at moving a metric boatload of stuff between servers. I’ve used this technique to easily migrate from Dreamhost to CanadianWebHosting.com and I use it regularly to move files around on campus. I use a variation of this technique to regularly backup servers as well – the beauty of rsync is that it only copies files that have been added or modified, so backing up a few gigs worth of server really only involves transferring a few megabytes of files, and can be done routinely in a matter of minutes or seconds.
I’ve been using an rsync-based script to backup my iPhoto library to another computer, just in case my Powerbook blows up or something, so I don’t lose every photo I’ve taken for the last 4 years. It’s worked flawlessly, run every now and then from home, to squirt changes in my Pictures directory to my desktop at work, where I further back it up on an external drive using another rsync script.
But, now that I’ve added Aperture to my work desktop, the script I had been using became suddenly quite dangerous. As in, destructively dangerous. And I hadn’t thought to check out the script in awhile.
So, I ran it last night. It faithfully chugged through my ‘book’s ~/Pictures directory, so I walked away as it started working.
Then, late last night I did a quick scroll through the rsync log, and to my horror found a whole bunch of stuff like “Deleting: ~/Pictures/Aperture Library.aplibrary/...”
Holy. Crap.
I’d accidentally told my computer to nuke my Aperture library on the work desktop.
So, I cracked open the “rsyncpics” script, and lo and behold, I’d left in the “–delete” flag. Which wasn’t a problem – it was actually desired – when I was only using iPhoto. But since Aperture isn’t on the Powerbook, the “–delete” flag told rsync to nuke anything on the desktop that isn’t on the powerbook. Like the Aperture library. Doh.
Thankfully, I also periodically backup my work desktop’s home directory to an external drive, so had a slightly out of date version of the Aperture library which I could just copy back into place. But it was missing everything after Ken Ryba’s session from a week ago. I’d copied the best of the “Campus Tour” photos to my home iPhoto library, so it’s not fatal, but a good lesson learned.
The moral of the story is: be careful with rsync, especially when using --delete. I’ve learned my lesson, and have resurrected most of the lost photos. I’ve also added an Aperture Vault on the external drive. They offer these backup tools, so why not use them?
Update: Between the various locations I’d copied files, and a healthy application of the awesome Flickr Backup utility, all photos worth saving have been restored. Whew.
I’ve been using an rsync-based script to backup my iPhoto library to another computer, just in case my Powerbook blows up or something, so I don’t lose every photo I’ve taken for the last 4 years. It’s worked flawlessly, run every now and then from home, to squirt changes in my Pictures directory to my desktop at work, where I further back it up on an external drive using another rsync script.
But, now that I’ve added Aperture to my work desktop, the script I had been using became suddenly quite dangerous. As in, destructively dangerous. And I hadn’t thought to check out the script in awhile.
So, I ran it last night. It faithfully chugged through my ‘book’s ~/Pictures directory, so I walked away as it started working.
Then, late last night I did a quick scroll through the rsync log, and to my horror found a whole bunch of stuff like “Deleting: ~/Pictures/Aperture Library.aplibrary/...”
Holy. Crap.
I’d accidentally told my computer to nuke my Aperture library on the work desktop.
So, I cracked open the “rsyncpics” script, and lo and behold, I’d left in the “–delete” flag. Which wasn’t a problem – it was actually desired – when I was only using iPhoto. But since Aperture isn’t on the Powerbook, the “–delete” flag told rsync to nuke anything on the desktop that isn’t on the powerbook. Like the Aperture library. Doh.
Thankfully, I also periodically backup my work desktop’s home directory to an external drive, so had a slightly out of date version of the Aperture library which I could just copy back into place. But it was missing everything after Ken Ryba’s session from a week ago. I’d copied the best of the “Campus Tour” photos to my home iPhoto library, so it’s not fatal, but a good lesson learned.
The moral of the story is: be careful with rsync, especially when using --delete. I’ve learned my lesson, and have resurrected most of the lost photos. I’ve also added an Aperture Vault on the external drive. They offer these backup tools, so why not use them?
Update: Between the various locations I’d copied files, and a healthy application of the awesome Flickr Backup utility, all photos worth saving have been restored. Whew.