File Systems: Difference between revisions

From David's Wiki
 
(7 intermediate revisions by the same user not shown)
Line 1: Line 1:
List of file systems
There are several common ways to store binary information:
* Database or key-value store (e.g. PostgreSQL, SQLite) - Good for small files or a finite amount of files which fit within the confines of a database.
* Object store (e.g. S3) - same as a key-value store but typically designed to scale lots of files across multiple HDDs and hosts.
* File systems (e.g. EXT4) - good for files where certain operations benefit from a hierarchical data structure, e.g. list, delete. File systems typically come with metadata such as permissions and owners.
* Block storage - you get raw disk access but need to layout your binary data manually and in fixed block sizes.


==Standard File Systems==
==Standard File Systems==
Line 12: Line 16:


==Block Overlays==
==Block Overlays==
The create a view of one or more block storage, typically using one or more block storage.
* [[LUKS]] - encrypts a partition
* [[LUKS]] - encrypts a partition
* [[LVM]] - joins multiple blocks into a pool from which to allocate blocks
* [[LVM]] - joins multiple blocks into a pool from which to allocate blocks
* [[mdraid]]
* [[mdraid]]
==Object Stores==
* Minio - S3-compatible object store
* Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
* Rook - deployment of Ceph using Kubernetes
* SeaweedFS - joins drives across multiple computers to object storage APIs (incl. S3). Has file storage when paired with a database using the SeaweedFS Filer.


==Distributed File Systems==
==Distributed File Systems==
* GlusterFS - joins filesystem directories across multiple computers
* GlusterFS - joins filesystem directories across multiple computers
* Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
* Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
* SeaweedFS - joins drives across multiple computers. Has file (with filer extension) and object storage APIs.
* JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.
* JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.
* Rook - deployment of Ceph using Kubernetes
 
==Databases==
===SQL===
* PostgreSQL
* MySQL
* SQLite
===NoSQL===
* MongoDB
 
==Cloud Providers==
See [[Cloud Providers]]

Latest revision as of 18:02, 15 April 2024

There are several common ways to store binary information:

  • Database or key-value store (e.g. PostgreSQL, SQLite) - Good for small files or a finite amount of files which fit within the confines of a database.
  • Object store (e.g. S3) - same as a key-value store but typically designed to scale lots of files across multiple HDDs and hosts.
  • File systems (e.g. EXT4) - good for files where certain operations benefit from a hierarchical data structure, e.g. list, delete. File systems typically come with metadata such as permissions and owners.
  • Block storage - you get raw disk access but need to layout your binary data manually and in fixed block sizes.

Standard File Systems

  • BTRFS
  • ZFS
  • EXT4
  • XFS
  • NTFS

Overlay File Systems

  • MergerFS - a union file system to combine multiple folders on a single computer.

Block Overlays

The create a view of one or more block storage, typically using one or more block storage.

  • LUKS - encrypts a partition
  • LVM - joins multiple blocks into a pool from which to allocate blocks
  • mdraid

Object Stores

  • Minio - S3-compatible object store
  • Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
  • Rook - deployment of Ceph using Kubernetes
  • SeaweedFS - joins drives across multiple computers to object storage APIs (incl. S3). Has file storage when paired with a database using the SeaweedFS Filer.

Distributed File Systems

  • GlusterFS - joins filesystem directories across multiple computers
  • Ceph - joins drives across multiple computers. Has block, file, and object storage APIs.
  • JuiceFS - creates a POSIX-compatable file storage using an S3 object storage and metadata database.

Databases

SQL

  • PostgreSQL
  • MySQL
  • SQLite

NoSQL

  • MongoDB

Cloud Providers

See Cloud Providers