SHShelnutt2·Sep 1, 2020
Disclaimer: I'm a member of the TileDB team.
> embedded, immutable, syncable relational database.
TileDB Embedded (https://github.com/TileDB-Inc/TileDB) checks all four of these boxes. It is a universal storage engine based on dense and sparse multi-dimensional arrays. I explain each point below.
> embedded
TileDB Embedded [1] is an open source (MIT licensed) embeddable C++ library which exposes C and C++ APIs. We also built APIs for Python, R, C++, Java and Go, and integrations into MariaDB, PrestoDB, Spark, GDAL, PDAL and more. Via MariaDB we even have embeddable SQL[2]. TileDB abstracts the storage backends behind an extendible VFS class, and currently supports S3, GCS, Azure, HDFS, and local disk.
> immutable
TileDB is designed around immutable objects. Every write creates a new "fragment" using a MVCC model[3]. No file is ever update
Read the rest
d in place. TileDB is designed to naturally handle the eventual consistency and restraints of cloud object stores. This allows for features like multi-reader/multi-writer support, time traveling, and update support all within the storage engine without any external orchestration.
> syncable
TileDB's MVCC approach to handling the eventual consistency of cloud object stores, yields directly into its sync-ability. Every write operation in TileDB creates a new "fragment" [3]. A fragment is an immutable folder (or prefix on object stores) that contains all the data from a single write session. Fragments are created with a timestamp and a UUID to ensure uniqueness. The fragment is ignored until a special `.ok` file is available in listing (each write is atomic). Incomplete fragments (without the .ok file) are gracefully ignored.
This means that every fragment is self-contained and syncable. The only requirement is that the special `.ok` file must show up only after the complete fragment folder is synced. With cloud object stores this is not a problem with the read-after-write consistency guarantees. For other systems the synchronization can be managed to handle this behavior.
> relational database
Tables can be easily modeled as multi-dimensional sparse arrays[4]. Through TileDB's integrations with MariaDB and PrestoDB (and Spark), you gain a full SQL interface, while being able to query the data directly via the language APIs without using SQL. For more robust features, such as foreign key enforcement, it is easy to use embeddable MariaDB to achieve this, while we work to push such features into the storage engine itself. We are always seeking feedback on which features are most used and requested to pushdown common operations to the storage engine.
[1] https://tiledb.com/embedded
[2] https://docs.tiledb.com/main/api-usage/embedded-sql
[3] https://docs.tiledb.com/main/basic-concepts/physical-storage
[4] https://docs.tiledb.com/main/handling-dataframes