I didn’t think of the obvious, simple solution first. I came to it in a round about way, perhaps because I was stuck in a traditional (read “old-fashioned”) way of thinking, or perhaps because I was initially just too focused on the related problem. Also because I have almost no personal or professional experience with peer-to-peer technologies, though my kids have a lot from back when illegal music sharing was a thing 🙂
♦ ♦ ♦
The simple solution is, if you want to synchronize the contents of a folder between two (or more) computers on your home network, just download and run Resilio Sync (formerly Bittorrent Sync) on each one, share the folder from one computer with the corresponding folder on the other computer, and let the BitTorrent protocols do their stuff. If one folder is empty to begin with, you get a complete copy. If both folders are partially populated, you get the join – all the files on each computer. The folder(s) can be on external drives. It is amazingly fast, works extremely well with large amounts of data, and best of all is free. The paid version adds some great functionality including selective sync, which is very useful for mobile devices, but is not essential. Yes, there are versions for Android and iOS as well as Windows and Mac OS. Note: this is peer-to-peer, so it does not work for syncing files between two drives on the same computer, but is crazy awesome for cloud-less syncing of files between two or more devices.
♦ ♦ ♦
I am paranoid about backups. Because all disks die, and sooner than most humans. So Penny’s laptop, when it is docked, is connected to capacious external storage (several TB). So is mine, but separately. Each computer runs nightly incremental backups and monthly full backups to its own external storage, automatically. That’s in addition to continuous backups of important files to the cloud, and continuous backups to another disk at home using Apple’s Time Machine. On each computer, the external storage is a fast, quiet-running RAID 5 array. RAID 5 means that you do not lose data in the event of a single drive failure.
You might think these are expensive solutions, and they are, but I figured several years ago that with all the computers I was buying every few years for a family of five, it made sense to invest in some reliable storage for backups in case of theft, drops, or computer-loss-by-liquid-spillage (it happened, twice). Also, since the size of a snapshot of a hard drive is typically between a quarter and half a Terabyte (sometimes more), it’s handy to have a few TB of useable space, in duplicate for a paranoiac. Every few months, not more often because it was such a hassle, I would lug Penny’s RAID array into the other room, connect hers and mine, and make duplicate copies of important stuff like photos and music for safety’s sake.
However, both of my first generation of RAID arrays failed after about seven years. On mine, a single drive went bad and I lost no data. But on Penny’s, two drives failed at the same time and so everything was lost (excepting whatever duplicates were stored elsewhere in my redundant and OCD set of systems). I replaced the two RAID 5 arrays with newer technology a couple of years ago, presumable good for another 7-10 years. But what about when they fail?
♦ ♦ ♦
A few months ago I spent more money on a Synology NAS (Network Attached Storage) and several large (8 TB) disks configured in a RAID 6 array. RAID 6 means that two drives can fail before any data is lost. My thinking was that this would be the backup to the backup, with copies of files from both my external storage and Penny’s, and with files (e.g. our photo and music collections) easily available to both of us. But I became hugely disappointed with the poor performance of Synology’s appliance software. My attempt to copy more than a hundred thousand photos to the shared network drive never completed after running for a week (for circumstantial reasons I strongly suspect 100,000 files is a limit in their software). Other attempts to copy a few hundred GB at a time mostly failed, usually after running for days or a week or more.
So the “related problem” that I started with was how to move files from the external storage on my computer to the NAS, for the purposes of 1) duplicate backup and 2) sharing with Penny. And that’s when I discovered Bittorrent Sync, now Resilio Sync. Resilio Sync is awesome! Resilio provides packages that can be installed on all sorts of NAS devices from different manufacturers. (You do have to identify the correct package for your particular NAS). I installed Resilio sync on the Synology NAS and the MAC OS desktop version on my laptop. The results were incredible. I could sync hundreds of GB in hours instead of days, with no errors or hiccups so far.
And then it dawned on me, this is peer to peer. Duh! I could install Resilio Sync on Penny’s computer too, and sync our precious family photos, music, movies etc. between all three peers – my computer, Penny’s computer, and the NAS. If I’d known this earlier, I could have synced photos (and any other files we want to share) between me and Penny even without the NAS! But I’m happy to have the redundancy of three copies of our most precious files, and room to archive and/or share a lot more stuff besides. I’m good now until the house burns down. Hmmm. Should I also copy the photos to the cloud in case the house burns down?