File Systems: Difference between revisions
| Line 6: | Line 6: | ||
==Standard File Systems== | ==Standard File Systems== | ||
* BTRFS | |||
* | |||
* | {| class="wikitable" style="margin:auto" | ||
* | |+ File Systems | ||
|- | |||
! Name !! Snapshots !! RAID !! Checksumming !! Compression !! CoW !! Erasure coding || Encryption | |||
|- | |||
| BTRFS<ref name="btrfs_documentation>https://btrfs.readthedocs.io/en/latest/Introduction.html</ref> || Yes, Writable || Block-level<ref name="btrfs_volume>https://btrfs.readthedocs.io/en/latest/Volume-management.html</ref> || Yes || Yes || Yes || Unstable || No | |||
|- | |||
| ZFS || Yes<ref name="zfs_snapshots">https://docs.oracle.com/cd/E19253-01/819-5461/gbcya/index.html</ref>, Writable<ref name="zfs_clone">https://docs.oracle.com/cd/E19253-01/819-5461/gbcxz/index.html</ref> || Disk-level || Yes || Yes || Yes || Yes || Yes | |||
|- | |||
| EXT4 || No || No || No || No || No || - || No | |||
|- | |||
| XFS || No || No || No || No || No || - || No | |||
|- | |||
| bcachefs<ref name="bcachefs">https://bcachefs.org/</ref> || Yes || Yes || Yes || Yes || Yes || Unstable || Yes | |||
|} | |||
Notes: | |||
* I prefer BTRFS for single-drive deployment. | |||
* bcachefs is still under development | |||
For multi-drive deployment, my preferences are: | |||
* If you have multiple same-sized disks, maxmimum storage, and no plans for expansion, use ZFS raidz1 or raidz2. | |||
* If you have different sized disks, maxmimum storage, want maximum size, and don't need live parity then use snapraid + btrfs + mergerfs. | |||
* If you need real-time parity and expansion and don't mind slow rebuilds, use mdraid + btrfs. | |||
Windows: | |||
1. NTRFS | |||
2. ReF | |||
==Overlay File Systems== | ==Overlay File Systems== | ||
Revision as of 07:29, 1 December 2024
There are several common ways to store binary information:
- Database or key-value store (e.g. PostgreSQL, SQLite) - Good for small files or a finite amount of files which fit within the confines of a database.
- Object store (e.g. S3) - same as a key-value store but typically designed to scale lots of files across multiple HDDs and hosts.
- File systems (e.g. EXT4) - good for files where certain operations benefit from a hierarchical data structure, e.g. list, delete. File systems typically come with metadata such as permissions and owners.
- Block storage - you get raw disk access but need to layout your binary data manually and in fixed block sizes.
Standard File Systems
| Name | Snapshots | RAID | Checksumming | Compression | CoW | Erasure coding | Encryption |
|---|---|---|---|---|---|---|---|
| BTRFS[1] | Yes, Writable | Block-level[2] | Yes | Yes | Yes | Unstable | No |
| ZFS | Yes[3], Writable[4] | Disk-level | Yes | Yes | Yes | Yes | Yes |
| EXT4 | No | No | No | No | No | - | No |
| XFS | No | No | No | No | No | - | No |
| bcachefs[5] | Yes | Yes | Yes | Yes | Yes | Unstable | Yes |
Notes:
- I prefer BTRFS for single-drive deployment.
- bcachefs is still under development
For multi-drive deployment, my preferences are:
- If you have multiple same-sized disks, maxmimum storage, and no plans for expansion, use ZFS raidz1 or raidz2.
- If you have different sized disks, maxmimum storage, want maximum size, and don't need live parity then use snapraid + btrfs + mergerfs.
- If you need real-time parity and expansion and don't mind slow rebuilds, use mdraid + btrfs.
Windows: 1. NTRFS 2. ReF
Overlay File Systems
- MergerFS - a union file system to combine multiple folders on a single computer.
Block Overlays
The create a view of one or more block storage, typically using one or more block storage.
- LUKS - encrypts a partition
- LVM - joins multiple blocks into a pool from which to allocate blocks
- mdraid
Object Stores
- Minio - S3-compatible object store
- Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
- Rook - deployment of Ceph using Kubernetes
- SeaweedFS - joins drives across multiple computers to object storage APIs (incl. S3). Has file storage when paired with a database using the SeaweedFS Filer.
Distributed File Systems
- GlusterFS - joins filesystem directories across multiple computers
- Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
- JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.
Databases
SQL
- PostgreSQL
- MySQL
- SQLite
NoSQL
- MongoDB
Cloud Providers
See Cloud Providers