File Systems: Difference between revisions

(11 intermediate revisions by the same user not shown)

Line 1:

~~List~~ of ~~file~~ systems

There are several common ways to store binary information:

* Database or key-value store (e.g. PostgreSQL, SQLite) - Good for small files or a finite amount of files which fit within the confines of a database.

* Object store (e.g. S3) - same as a key-value store but typically designed to scale lots of files across multiple HDDs and hosts.

* File systems (e.g. EXT4) - good for files where certain operations benefit from a hierarchical data structure, e.g. list, delete. File systems typically come with metadata such as permissions and owners.

* Block storage - you get raw disk access but need to layout your binary data manually and in fixed block sizes.

==Standard File Systems==

* BTRFS

* [[ZFS]]

* ~~EXT4~~

{| class="wikitable" style="margin:auto"

* ~~XFS~~

|+ File Systems

* ~~NTFS~~

|-

! Name !! Snapshots !! RAID !! Checksumming !! Compression !! CoW !! Erasure coding || Encryption

|-

| BTRFS<ref name="btrfs_documentation>https://btrfs.readthedocs.io/en/latest/Introduction.html</ref> || Yes, Writable || Yes<ref name="btrfs_volume>https://btrfs.readthedocs.io/en/latest/Volume-management.html</ref> || Yes || Yes || Yes || Unstable || No

|-

| ZFS || Yes<ref name="zfs_snapshots">https://docs.oracle.com/cd/E19253-01/819-5461/gbcya/index.html</ref>, Writable<ref name="zfs_clone">https://docs.oracle.com/cd/E19253-01/819-5461/gbcxz/index.html</ref> || Drive-level || Yes || Yes || Yes || Yes || Yes

|-

| EXT4 || No || No || No || No || No || - || No

|-

| XFS || No || No || No || No || No || - || No

|-

| bcachefs<ref name="bcachefs">https://bcachefs.org/</ref> || Yes || Yes || Yes || Yes || Yes || Unstable || Yes

|}

Notes:

* For single drives/blocks, I prefer BTRFS for it's feature set.

* For multiple drives, ZFS is often preferred for it's reliability.

* bcachefs is still under development

For multi-drive deployment, my preferences are:

* If you have multiple same-sized disks and no need for expansion, use ZFS raid.

* If you have different sized disks, want maximum storage, and don't need live parity then use snapraid + btrfs + mergerfs.

* If you need real-time parity and expansion and don't mind slow rebuilds, use mdraid + btrfs.

===Windows===

# NTRFS

# ReFS

===Mac===

# APFS

# Mac OS Extended

==Overlay File Systems==

Line 12:

Line 46:

==Block Overlays==

The create a view of one or more block storage, typically using one or more block storage.

* [[LUKS]] - encrypts a partition

* [[LVM]] - joins multiple blocks into a pool from which to allocate blocks

* [[mdraid]]

==Object Stores==

* Minio - S3-compatible object store

* Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.

* Rook - deployment of Ceph using Kubernetes

* SeaweedFS - joins drives across multiple computers to object storage APIs (incl. S3). Has file storage when paired with a database using the SeaweedFS Filer.

==Distributed File Systems==

* GlusterFS - joins filesystem directories across multiple computers

* Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.

* SeaweedFS - joins drives across multiple computers. Has file (with filer extension) and object storage APIs.

* JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.

* ~~Rook - deployment of Ceph using Kubernetes~~

==Databases==

===SQL===

* PostgreSQL

* MySQL

* SQLite

===NoSQL===

* MongoDB

==Cloud Providers==

See [[Cloud Providers]]

@@ Line 1: / Line 1: @@
-List of file systems
+There are several common ways to store binary information:
+* Database or key-value store (e.g. PostgreSQL, SQLite) - Good for small files or a finite amount of files which fit within the confines of a database.
+* Object store (e.g. S3) - same as a key-value store but typically designed to scale lots of files across multiple HDDs and hosts.
+* File systems (e.g. EXT4) - good for files where certain operations benefit from a hierarchical data structure, e.g. list, delete. File systems typically come with metadata such as permissions and owners.
+* Block storage - you get raw disk access but need to layout your binary data manually and in fixed block sizes.
 ==Standard File Systems==
-* BTRFS
-* [[ZFS]]
-* EXT4
+{| class="wikitable" style="margin:auto"
-* XFS
+|+ File Systems
-* NTFS
+|-
+! Name !! Snapshots !! RAID !! Checksumming !! Compression !! CoW !! Erasure coding || Encryption
+|-
+| BTRFS<ref name="btrfs_documentation>https://btrfs.readthedocs.io/en/latest/Introduction.html</ref> || Yes, Writable || Yes<ref name="btrfs_volume>https://btrfs.readthedocs.io/en/latest/Volume-management.html</ref> || Yes || Yes || Yes || Unstable || No
+|-
+| ZFS || Yes<ref name="zfs_snapshots">https://docs.oracle.com/cd/E19253-01/819-5461/gbcya/index.html</ref>, Writable<ref name="zfs_clone">https://docs.oracle.com/cd/E19253-01/819-5461/gbcxz/index.html</ref> || Drive-level || Yes || Yes || Yes || Yes || Yes
+|-
+| EXT4 || No || No || No || No || No || - || No
+|-
+| XFS || No || No || No || No || No || - || No
+|-
+| bcachefs<ref name="bcachefs">https://bcachefs.org/</ref> || Yes || Yes || Yes || Yes || Yes || Unstable || Yes
+|}
+Notes:
+* For single drives/blocks, I prefer BTRFS for it's feature set.
+* For multiple drives, ZFS is often preferred for it's reliability.
+* bcachefs is still under development
+For multi-drive deployment, my preferences are:
+* If you have multiple same-sized disks and no need for expansion, use ZFS raid.
+* If you have different sized disks, want maximum storage, and don't need live parity then use snapraid + btrfs + mergerfs.
+* If you need real-time parity and expansion and don't mind slow rebuilds, use mdraid + btrfs.
+===Windows===
+# NTRFS
+# ReFS
+===Mac===
+# APFS
+# Mac OS Extended
 ==Overlay File Systems==
@@ Line 12: / Line 46: @@
 ==Block Overlays==
+The create a view of one or more block storage, typically using one or more block storage.
 * [[LUKS]] - encrypts a partition
 * [[LVM]] - joins multiple blocks into a pool from which to allocate blocks
 * [[mdraid]]
+==Object Stores==
+* Minio - S3-compatible object store
+* Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
+* Rook - deployment of Ceph using Kubernetes
+* SeaweedFS - joins drives across multiple computers to object storage APIs (incl. S3). Has file storage when paired with a database using the SeaweedFS Filer.
 ==Distributed File Systems==
 * GlusterFS - joins filesystem directories across multiple computers
 * Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
-* SeaweedFS - joins drives across multiple computers. Has file (with filer extension) and object storage APIs.
 * JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.
-* Rook - deployment of Ceph using Kubernetes
+==Databases==
+===SQL===
+* PostgreSQL
+* MySQL
+* SQLite
+===NoSQL===
+* MongoDB
+==Cloud Providers==
+See [[Cloud Providers]]