fdb-record-layer

Java source at https://github.com/FoundationDB/fdb-record-layer

Had been pretty quiet for a number of years, but two pretty sizable capabilities have been added in the last 6 months, which looks much more alive: predicate pushdown and roll-up aggregation. https://github.com/FoundationDB/fdb-record-layer/pull/2916 https://github.com/FoundationDB/fdb-record-layer/pull/2746

FoundationDB is really just a set of structures and tools to build any type of database you want on top of a solid foundation. "FoundationDB decouples its data storage technology from its data model. FoundationDB’s core ordered key-value storage technology can be efficiently adapted and remapped to a broad array of rich data models. Using indexing as an example, FoundationDB’s core provides no indexing and never will. Instead, a layer provides indexing by storing two kinds of key-values, one for the data and one for the index." https://apple.github.io/foundationdb/layer-concept.html Then existing standard layers like the Record layer, providing "(very) roughly equivalent to a simple relational database" providing structured types, index, complex types, queries, etc. https://github.com/FoundationDB/fdb-record-layer Or one for documents, wh

> That being said, I wish there were more layers as the functionality out of the box is very very limited. The record layer https://github.com/FoundationDB/fdb-record-layer which allows to store protobuf, and define the primary keys and index directly on those proto fields is truly amazing: https://github.com/FoundationDB/fdb-record-layer/blob/main/d...

To an extent this is what the FoundationDB Record Layer does. https://github.com/FoundationDB/fdb-record-layer

This is exactly what the engineers behind FoundationDB (FDB) wanted when they open sourced. For those who don't know, FDB provides a transactional (and distributed) ordered key-value store with a somewhat simple but very powerful API. Their vision was to build the hardest parts of building a database, such as transactions, fault-tolerance, high-availability, elastic scaling, etc. This would free users to build higher-level (Layers) APIs [1] / libraries [2] on top. The beauty of these layers is that you can basically remove doubt about the correctness of data once it leaves the layer. FoundationDB is one of the most (if not the) most tested [3] databases out there. I used it for over 4 years in high write / read production environments and never once did we second guess our decision. I could see this project renamed to simply "fdb-sqlite-l

https://github.com/FoundationDB/fdb-record-layer is the SQL/orm like interface. This handles things in a DB like way - joins, indexes. I misspoke when I said SQL regarding this - sorry. There's a go and rust library floating around that's not as good. I've tried em :)

FoundationDB is just a key value store, so it's not comparable. There is a relational layer you can run though called Foundation Record Layer. But this doesn't support SQL AFAICT. The examples on Github just show a query builder interface in Java [0]. There does seem to be a Lucene interface but it doesn't seem to be used [1]. [0] https://github.com/FoundationDB/fdb-record-layer/blob/main/e... [1] https://forums.foundationdb.org/t/lucene-layer-on-foundation...

This looks really promising! Could you provide at a high-level what this aims to offer over FoundationDB's RecordLayer[1]? [1] https://github.com/FoundationDB/fdb-record-layer

You can take it all the way into the db with OSS if you like https://github.com/FoundationDB/fdb-record-layer/blob/master...

The Record Layer doesn't currently support foreign key constraints, so foreign keys are more of an “design pattern” than a first-class feature. For example, in a sample schema in the repository, an “Order” message has have a field called “item_id” that points to the primary key of an “Item” message: https://github.com/FoundationDB/fdb-record-layer/blob/792c95... There isn't an automatic check to make sure the item exists, though, nor are there cascading deletes. That being said, I don't think the architecture is incompatible with that feature, so it would be a reasonable feature request. There are some guidelines regarding field type changes in the schema evolution guide: https://foundationdb.github.io/fdb-record-layer/SchemaEvolut... Most data type changes are incompatible with either Protobuf's serialization format or the FDB Tuple laye

You're right the Record Layer doesn't yet have join support, but note PR #306 [1]. We support aggregate indexes but can't run aggregate “reports” that aren't backed by indexes. The rationale for this is covered in the paper (see [2]), but note that doesn't preclude such support being added into or on top of Record Layer (also discussed in the paper). The Record Layer does a whole bunch of work to deal with index maintenance (see our docs [3])). Our “index maintainer” abstraction (discussed in the paper) makes maintaining our indexes (including those that are basically materialized views) completely seamless from the user's perspective, even for updates and deletes. We also have a lot of tooling for making efficient schema migrations. For example, schema migrations are performed lazily (when the data is accessed), so they aren't limited by