File Systems: Difference between revisions
| Line 13: | Line 13: | ||
! Name !! Snapshots !! RAID !! Checksumming !! Compression !! CoW !! Erasure coding || Encryption | ! Name !! Snapshots !! RAID !! Checksumming !! Compression !! CoW !! Erasure coding || Encryption | ||
|- | |- | ||
| BTRFS<ref name="btrfs_documentation>https://btrfs.readthedocs.io/en/latest/Introduction.html</ref> || Yes, Writable || | | BTRFS<ref name="btrfs_documentation>https://btrfs.readthedocs.io/en/latest/Introduction.html</ref> || Yes, Writable || Yes<ref name="btrfs_volume>https://btrfs.readthedocs.io/en/latest/Volume-management.html</ref> || Yes || Yes || Yes || Unstable || No | ||
|- | |- | ||
| ZFS || Yes<ref name="zfs_snapshots">https://docs.oracle.com/cd/E19253-01/819-5461/gbcya/index.html</ref>, Writable<ref name="zfs_clone">https://docs.oracle.com/cd/E19253-01/819-5461/gbcxz/index.html</ref> || | | ZFS || Yes<ref name="zfs_snapshots">https://docs.oracle.com/cd/E19253-01/819-5461/gbcya/index.html</ref>, Writable<ref name="zfs_clone">https://docs.oracle.com/cd/E19253-01/819-5461/gbcxz/index.html</ref> || Drive-level || Yes || Yes || Yes || Yes || Yes | ||
|- | |- | ||
| EXT4 || No || No || No || No || No || - || No | | EXT4 || No || No || No || No || No || - || No | ||
| Line 25: | Line 25: | ||
Notes: | Notes: | ||
* I prefer BTRFS for | * For single drives/blocks, I prefer BTRFS for it's feature set. | ||
* For multiple drives, ZFS is often preferred for it's reliability. | |||
* bcachefs is still under development | * bcachefs is still under development | ||
For multi-drive deployment, my preferences are: | For multi-drive deployment, my preferences are: | ||
* If you have multiple same-sized disks | * If you have multiple same-sized disks and no need for expansion, use ZFS raid. | ||
* If you have different sized disks | * If you have different sized disks, want maximum storage, and don't need live parity then use snapraid + btrfs + mergerfs. | ||
* If you need real-time parity and expansion and don't mind slow rebuilds, use mdraid + btrfs. | * If you need real-time parity and expansion and don't mind slow rebuilds, use mdraid + btrfs. | ||
Revision as of 07:33, 1 December 2024
There are several common ways to store binary information:
- Database or key-value store (e.g. PostgreSQL, SQLite) - Good for small files or a finite amount of files which fit within the confines of a database.
- Object store (e.g. S3) - same as a key-value store but typically designed to scale lots of files across multiple HDDs and hosts.
- File systems (e.g. EXT4) - good for files where certain operations benefit from a hierarchical data structure, e.g. list, delete. File systems typically come with metadata such as permissions and owners.
- Block storage - you get raw disk access but need to layout your binary data manually and in fixed block sizes.
Standard File Systems
| Name | Snapshots | RAID | Checksumming | Compression | CoW | Erasure coding | Encryption |
|---|---|---|---|---|---|---|---|
| BTRFS[1] | Yes, Writable | Yes[2] | Yes | Yes | Yes | Unstable | No |
| ZFS | Yes[3], Writable[4] | Drive-level | Yes | Yes | Yes | Yes | Yes |
| EXT4 | No | No | No | No | No | - | No |
| XFS | No | No | No | No | No | - | No |
| bcachefs[5] | Yes | Yes | Yes | Yes | Yes | Unstable | Yes |
Notes:
- For single drives/blocks, I prefer BTRFS for it's feature set.
- For multiple drives, ZFS is often preferred for it's reliability.
- bcachefs is still under development
For multi-drive deployment, my preferences are:
- If you have multiple same-sized disks and no need for expansion, use ZFS raid.
- If you have different sized disks, want maximum storage, and don't need live parity then use snapraid + btrfs + mergerfs.
- If you need real-time parity and expansion and don't mind slow rebuilds, use mdraid + btrfs.
Windows: 1. NTRFS 2. ReF
Overlay File Systems
- MergerFS - a union file system to combine multiple folders on a single computer.
Block Overlays
The create a view of one or more block storage, typically using one or more block storage.
- LUKS - encrypts a partition
- LVM - joins multiple blocks into a pool from which to allocate blocks
- mdraid
Object Stores
- Minio - S3-compatible object store
- Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
- Rook - deployment of Ceph using Kubernetes
- SeaweedFS - joins drives across multiple computers to object storage APIs (incl. S3). Has file storage when paired with a database using the SeaweedFS Filer.
Distributed File Systems
- GlusterFS - joins filesystem directories across multiple computers
- Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
- JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.
Databases
SQL
- PostgreSQL
- MySQL
- SQLite
NoSQL
- MongoDB
Cloud Providers
See Cloud Providers