File Systems

Revision as of 21:50, 9 February 2024 by David (talk | contribs)
\( \newcommand{\P}[]{\unicode{xB6}} \newcommand{\AA}[]{\unicode{x212B}} \newcommand{\empty}[]{\emptyset} \newcommand{\O}[]{\emptyset} \newcommand{\Alpha}[]{Α} \newcommand{\Beta}[]{Β} \newcommand{\Epsilon}[]{Ε} \newcommand{\Iota}[]{Ι} \newcommand{\Kappa}[]{Κ} \newcommand{\Rho}[]{Ρ} \newcommand{\Tau}[]{Τ} \newcommand{\Zeta}[]{Ζ} \newcommand{\Mu}[]{\unicode{x039C}} \newcommand{\Chi}[]{Χ} \newcommand{\Eta}[]{\unicode{x0397}} \newcommand{\Nu}[]{\unicode{x039D}} \newcommand{\Omicron}[]{\unicode{x039F}} \DeclareMathOperator{\sgn}{sgn} \def\oiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x222F}\,}{\unicode{x222F}}{\unicode{x222F}}{\unicode{x222F}}}\,}\nolimits} \def\oiiint{\mathop{\vcenter{\mathchoice{\huge\unicode{x2230}\,}{\unicode{x2230}}{\unicode{x2230}}{\unicode{x2230}}}\,}\nolimits} \)

There are several common ways to store binary information:

  • Database or key-value store (e.g. PostgreSQL, SQLite) - Good for small files or a finite amount of files which fit within the confines of a database.
  • Object store (e.g. S3) - same as a key-value store but typically designed to scale lots of files across multiple HDDs and hosts.
  • File systems (e.g. EXT4) - good for files where certain operations benefit from a hierarchical data structure, e.g. list, delete. File systems typically come with metadata such as permissions and owners.
  • Block storage - you get raw disk access but need to layout your binary data manually and in fixed block sizes.

Standard File Systems

  • BTRFS
  • ZFS
  • EXT4
  • XFS
  • NTFS

Overlay File Systems

  • MergerFS - a union file system to combine multiple folders on a single computer.

Block Overlays

  • LUKS - encrypts a partition
  • LVM - joins multiple blocks into a pool from which to allocate blocks
  • mdraid

Distributed File Systems

  • GlusterFS - joins filesystem directories across multiple computers
  • Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
  • SeaweedFS - joins drives across multiple computers. Has file (with filer extension) and object storage APIs.
  • JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.
  • Rook - deployment of Ceph using Kubernetes

Databases

  • PostgreSQL
  • MySQL
  • SQLite