How to do data hoarding.
My personal setup is:
- 5x USB hard drives
- Veracrypt full-disk encryption on each drive
- SnapRAID with 3 data drives and 2 parities.
- Mergerfs on the data drives.
- SFTP to mount or transfer files.
- Due to SnapRAID not being real-time, I don't keep data for programs such as MariaDB or GitLab on the array. They're on a separate drive and rsync'd to the array nightly.
- In general, get WD easystore disks from Best Buy when they're on sale.
- $15 per TB is great pricing for 8TB+ drives. When on sale, 12TB runs $180 and 14TB runs $200. You can also find $250 14TB bare drives on eBay.
- WD owns Hitachi Global Storage Technologies (HGST). All new HGST drives are WD drives.
- Avoid SMR drives which have worse performance and reliability. Specifically, their write performance is incredibly bad so you'll end up taking multiple days to rebuild parity drives in raid arrays.
- Avoid lower-end Seagate drives (e.g. rosewood).
You should test all hard drives, both new and old, before adding them to your array.
I mainly use VeraCrypt for full disk encryption of every individual disk.
- VeraCrypt for full-disk, partition, or container based encryption.
- Rclone for file-based encryption
- Cryptomator is also another good choice for local file-based encryption.
See https://www.privacytools.io/software/encryption-tools/ for more options.
It is inevitable that one of your drives will eventually fail.
In general, I strongly recommend against hardware Raid and intel RST Raid, despite them being the most popular.
When raid cards or motherboards fail, your data becomes difficult to recover.
I personally only use SnapRAID.
- Archwiki: ZFS is a good choice if you're building a server from scratch with uniform disk sizes
- SnapRaid allows you to create parity files out of existing disks.
- You can mix and match disks but the parity drives must be at least as large as the largest drive.
- You can have up to 6 parity drives for each SnapRaid configuration.
- See selfhostedhome.com: Combining Different Sized Drives with mergerfs and SnapRAID
- See also linuxserver.io The Perfect Media Server 2019
- Archwiki: mdadm is the default software raid which comes with most linux distros.
- rsync for local or ssh backups
- See Archwiki:rsync for more sophisticated incremental backups
- rclone for cloud backups
Union File Systems
- Nextcloud allows you to add an SFTP link as an external storage.
- Alternatively, you can also move the nextcloud directory to your array but then it'll be slightly more difficult to access using other methods.
Closed-source Windows Apps
Would not recommend creating a Windows NAS. If you want to though, most of the above also runs on Windows. Note: I have not tried the applications below.
Here is a list of popular closed-source apps:
- Storage Spaces comes with Windows.
- Stablebit DrivePool is like mergerfs but costs $30. They also have an rclone-esque software which costs $40.