Data Hoarding

From David's Wiki
Jump to navigation Jump to search
\( \newcommand{\P}[]{\unicode{xB6}} \newcommand{\AA}[]{\unicode{x212B}} \newcommand{\empty}[]{\emptyset} \newcommand{\O}[]{\emptyset} \newcommand{\Alpha}[]{Α} \newcommand{\Beta}[]{Β} \newcommand{\Epsilon}[]{Ε} \newcommand{\Iota}[]{Ι} \newcommand{\Kappa}[]{Κ} \newcommand{\Rho}[]{Ρ} \newcommand{\Tau}[]{Τ} \newcommand{\Zeta}[]{Ζ} \newcommand{\Mu}[]{\unicode{x039C}} \newcommand{\Chi}[]{Χ} \newcommand{\Eta}[]{\unicode{x0397}} \newcommand{\Nu}[]{\unicode{x039D}} \newcommand{\Omicron}[]{\unicode{x039F}} \DeclareMathOperator{\sgn}{sgn} \def\oiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x222F}\,}{\unicode{x222F}}{\unicode{x222F}}{\unicode{x222F}}}\,}\nolimits} \def\oiiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x2230}\,}{\unicode{x2230}}{\unicode{x2230}}{\unicode{x2230}}}\,}\nolimits} \)

How to do data hoarding.

My personal setup is:

  • 5x USB hard drives
  • Veracrypt full-disk encryption on each drive
  • SnapRAID with 3 data drives and 2 parities.
  • Mergerfs on the data drives.
  • SFTP to mount or transfer files.
Notes
  • Due to SnapRAID not being real-time, I don't keep data for programs such as MariaDB or GitLab on the array. They're on a separate drive and rsync'd to the array nightly.

Disks

  • In general, get WD easystore disks from Best Buy when they're on sale.
    • $15 per TB is great pricing for 8TB+ drives. When on sale, 12TB runs $180 and 14TB runs $200. You can also find $250 14TB bare drives on eBay.
    • WD owns Hitachi Global Storage Technologies (HGST). All new HGST drives are WD drives.
  • Avoid SMR drives which have worse performance and reliability. Specifically, their write performance is incredibly bad so you'll end up taking multiple days to rebuild parity drives in raid arrays.
  • Avoid lower-end Seagate drives (e.g. rosewood).

HDD Testing

You should test all hard drives, both new and old, before adding them to your array.

Encryption

I mainly use VeraCrypt for full disk encryption of every individual disk.

  • VeraCrypt for full-disk, partition, or container based encryption.
  • Rclone for file-based encryption
  • Cryptomator is also another good choice for local file-based encryption.

See https://www.privacytools.io/software/encryption-tools/ for more options.

Parity

It is inevitable that one of your drives will eventually fail.
In general, I strongly recommend against hardware Raid and intel RST Raid, despite them being the most popular.
When raid cards or motherboards fail, your data becomes difficult to recover.
I personally only use SnapRAID.

Backup

Union File Systems

Front-ends

  • Nextcloud allows you to add an SFTP link as an external storage.
    • Alternatively, you can also move the nextcloud directory to your array but then it'll be slightly more difficult to access using other methods.

Closed-source Windows Apps

Would not recommend creating a Windows NAS. If you want to though, most of the above also runs on Windows. Note: I have not tried the applications below.

Here is a list of popular closed-source apps:

More Resources