sirix

across both axes: - "What did we THINK the price was on Jan 16?" -> $100 (Transaction time) - "What WAS the price on Jan 1?" -> $95 (Valid time) I've worked on this in my spare time since 2013, following its academic precursor (Idefix/Treetank) at the University of Konstanz. The architecture relies on an append-only physical log and a persistent copy-on-write page trie. A high level view of the architecture: Physical Log (append-only, sequential writes) ┌────────────────────────────────────────────────────────────────────────┐ │ [R1:Root] [R1:P1] [R1:P2] [R2:Root] [R2:P1'] [R3:Root] [R3:P2'] ... │ └────────────────────────────────────────────────────────────────────────┘ t=0 t=1 t=2 t=3 t=4 t=5 t=6 → time Each revision is indexed, and unchanged pages are shared: [Rev 1] [Rev 2] [Rev 3] │ │ │ ▼ ▼ ▼ [Root₁] [Root₂] [Root₃] │ │ │ │ │ │ │ └─────────┐ │ └────────┐ │ └─────────┐ ▼ ▼ ▼ ▼ ▼ ▼ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │ P1 │ │ P2 │ │ P1' │ │ P2' │ └──────┘ └──────┘ └──────┘ └──────┘ Rev 1 Rev 1+2 Rev 2+3 Rev 3 (shared) (shared) Beneath the root pages sit node and secondary indexes, using a novel sliding-snapshot algorithm to balance read/write performance. Everything is queryable using JSONiq via the Brackit compiler. Back in 2019, and even in 2023, SirixDB was very slow due to GC pressure. Unlike most other document stores, SirixDB stores fine-grained nodes, and I came to realize that an on-heap (JVM) representation made up of lots of small objects simply didn't make sense. I measured it with async-profiler — with some help from Andrei Pangin himself — and the result was that the poor throughput was due to the sheer amount of allocations which scaled almost linearly with the number of open transactions. Working a full-time software engineering job, I lacked the energy for a massive spare-time rewrite. About a year ago, I started experimenting with AI. It turned out to be ideal for automating the tedious, repetitive parts of migrating the storage layer to Java's Foreign Function & Memory API, storing pages completely off-heap. Looking further ahead, the append-only, immutable-page design maps naturally onto object storage like S3 and distributed logs like Kafka for a cloud version, and initial prototypes already exist. Maybe that becomes a commercial service one day, but for now, I'm just thrilled to see these core design principles finally proven out.There's an interactive demo, documentation, and the code is on GitHub. I'd love feedback and am happy to answer questions! kind regards Johannes [1] https://sirix.io | https://github.com/sirixdb/sirix [2] https://sirix.io/docs/architecture.html [3] https://demo.sirix.io [4] https://sirix.io/docs/ [5] http://brackit.io

ach level in the trie through some bit shifting. [1] https://github.com/sirixdb/sirix

and other query rewrites :-)

irix.io/docs/concepts.html Furthermore I'm working on a tutorial for a local client usage (work in progress): https://sirix.io/docs/jsoniq-tutorial.html Kind regards Johannes

o the structure and size of an object. Objects can be arbitrarily nested, and updates are cheap. On a high level, it supports space-efficient snapshots, tracking changes by an author / optional commit messages, time travel queries, reverting to previous revisions (while all revisions in-between still exist for audits...), or retrieving the changes of whole (sub)trees. On the one hand, it's, thus, a bitemporal DBS, but on the other hand, it can be used as a simple event store. It stores the state after an event or a change occurs and tracks the changes. Thus, an entity, a node in the JSON structure, can be updated to new values and eventually be removed while the history is easily retrievable, or we can easily revert to a previous state. The system assigns a unique ID to each new node, which never changes and is never reused (even after the deletion of the node). Thus, the system stores the state after the change/event and the event itself (the change event). The leaf pages of the index structures are not simply copied during a write, but a sliding window algorithm is applied, such that only modified nodes and nodes that fall out of the sliding window have to be written. A predefined window length is configurable. The system avoids write-peaks, which would occur due to full snapshots and having to read a long chain of incremental changes in between. Thus, it's best suited for fast flash drives with fast random reads and sequential writes. Data is never overwritten thus, audit trails are given for free. Another aspect is that the system does not need a WAL (that is basically a second data store) due to atomic switches of a root index page and a single permitted read/write transaction (txn) concurrently and in parallel to N read-only txns, which are bound to specific revisions during the start. Reads do not involve any locks.[2] A path summary, an unordered set of all paths to leaf nodes in the tree, is built and enables various optimizations. Furthermore, a rolling hash is optionally built, whereas all ancestor node hashes are adapted during inserts. A dated Jupyter notebook with some examples can be found in [3], and overall documentation in [4]. The query engine[5] Brackit is retargetable (a couple of interfaces and rewrite rules have to be implemented for DB systems) and especially finds implicit joins and applies known algorithms from the relational DB systems world to optimize joins and aggregate functions due to set-oriented processing of the operators.[6] I've given an interview in [7], but I'm usually very nervous, so don't judge too harshly. Give it a try, and happy coding! Kind regards Johannes [1] https://sirix.io | https://github.com/sirixdb/sirix [2] https://sirix.io/docs/concepts.html [3] https://colab.research.google.com/drive/1NNn1nwSbK6hAekzo1Yb... [4] https://sirix.io/docs/ [5] http://brackit.io [6] https://colab.research.google.com/drive/19eC-UfJVm_gCjY--koO... [7] https://youtu.be/Ee-5ruydgqo?si=Ift73d49w84RJWb2

ments has to be read in parallel to reconstruct a page in-memory). [1] https://github.com/sirixdb/sirix | https://sirix.io | https://sirix.io/docs/concepts.html

d-only txns, which are bound to specific revisions during the start. The single writer is permitted on a resource (comparable to a table/relation in a relational DB) basis within a database, reads do not involve any locks at all. The idea is, that the system atomically swaps the tree root to the new version (replicated). If something fails the log can simply be truncated to the former tree root. Thus, the system has many similarities with Git (structural sharing of unchanged nodes/pages) and ZFS snapshots (regarding the latter the keyed trie has been inspired by ZFS, as well as that checksums for child pages are stored in parent pages in the references to the child pages)[2]. You can of course simply execute time travel queries on the whole revision history, add commit comments and the author to answer questions such as who committed what at which point in time and why... The system not only copies full data pages, but it applies a sliding snapshot versioning algorithm to keep storage space to a minimum. Thus, it's best suited for fast flash drives with fast random reads and sequential writes. Data is never overwritten, thus audit trails are given for free. The system stores find granular JSON nodes, thus the structure and size of an object has almost no limits. A path summary is built, which is an unordered set of all paths to leaf nodes in the tree and enables various optimizations. Furthermore a rolling hash is optionally built, whereas during inserts all ancestor node hashes are adapted. Furthermore it optionally keeps track of update operations and the ctx nodes involved during txn commits. Thus, you can easily get the changes between revisions, you can check the full history of nodes, as well as navigate in time to the first revision, the last revision, the next and previous revision of a node... You can also open a revision at a specific system time revert to a revision and commit a new version while preserving all revisions in-between. As said one feature is, that the objects can be arbitrarily nested, thus almost no limits in the number and updates are cheap. A dated Jupyter notebook with some examples can be found in [3] and overall documentation in [4]. The query engine[5] Brackit is retargetable (a couple of interfaces and rewrite rules have to be implemented for DB systems) and especially finds implicit joins and applies known algorithms from the relational DB systems world to optimize joins and aggregate functions due to set-oriented processing of the operators.[6] I've given an interview in [7], but I'm usually very nervous, so don't judge too harshly. Give it a try and happy coding! Kind regards Johannes [1] https://sirix.io | https://github.com/sirixdb/sirix [2] https://sirix.io/docs/concepts.html [3] https://colab.research.google.com/drive/1NNn1nwSbK6hAekzo1YbED52RI3NMqqbG#scrollTo… [4] https://sirix.io/docs/ [5] http://brackit.io [6] https://colab.research.google.com/drive/19eC-UfJVm_gCjY--koOWN50sgiFa5hSC [7] https://youtu.be/Ee-5ruydgqo?si=Ift73d49w84RJWb2

d-only txns, which are bound to specific revisions during the start. The single writer is permitted on a resource (comparable to a table/relation in a relational DB) basis within a database, reads do not involve any locks at all. The idea is, that the system atomically swaps the tree root to the new version (replicated). If something fails the log can simply be truncated to the former tree root. Thus, the system has many similarities with Git (structural sharing of unchanged nodes/pages) and ZFS snapshots (regarding the latter the keyed trie has been inspired by ZFS, as well as that checksums for child pages are stored in parent pages in the references to the child pages)[2]. You can of course simply execute time travel queries on the whole revision history, add commit comments and the author to answer questions such as who committed what at which point in time and why... The system not only copies full data pages, but it applies a sliding snapshot versioning algorithm to keep storage space to a minimum. Thus, it's best suited for fast flash drives with fast random reads and sequential writes. Data is never overwritten, thus audit trails are given for free. The system stores find granular JSON nodes, thus the structure and size of an object has almost no limits. A path summary is built, which is an unordered set of all paths to leaf nodes in the tree and enables various optimizations. Furthermore a rolling hash is optionally built, whereas during inserts all ancestor node hashes are adapted. Furthermore it optionally keeps track of update operations and the ctx nodes involved during txn commits. Thus, you can easily get the changes between revisions, you can check the full history of nodes, as well as navigate in time to the first revision, the last revision, the next and previous revision of a node... You can also open a revision at a specific system time revert to a revision and commit a new version while preserving all revisions in-between. As said one feature is, that the objects can be arbitrarily nested, thus almost no limits in the number and updates are cheap. A dated Jupyter notebook with some examples can be found in [3] and overall documentation in [4]. The query engine[5] Brackit is retargetable (a couple of interfaces and rewrite rules have to be implemented for DB systems) and especially finds implicit joins and applies known algorithms from the relational DB systems world to optimize joins and aggregate functions due to set-oriented processing of the operators.[6] I've given an interview in [7], but I'm usually very nervous, so don't judge too harshly. Give it a try and happy coding! Kind regards Johannes [1] https://sirix.io | https://github.com/sirixdb/sirix [2] https://sirix.io/docs/concepts.html [3] https://colab.research.google.com/drive/1NNn1nwSbK6hAekzo1YbED52RI3NMqqbG#scrollTo… [4] https://sirix.io/docs/ [5] http://brackit.io [6] https://colab.research.google.com/drive/19eC-UfJVm_gCjY--koOWN50sgiFa5hSC [7] https://youtu.be/Ee-5ruydgqo?si=Ift73d49w84RJWb2

stored in parent pages[2]. You can of course simply execute time travel queries on the whole revision history, add commit comments and the author... The system not only copies full data pages, but it applies a sliding snapshot versioning algorithm to keep storage space to a minimum. It's best suited for fast flash drives with fast random reads and sequential writes. Data is never overwritten, thus audit trails are given for free. The system stores find granular JSON nodes, thus the structure and size of an object has almost no limits. A path summary is built, which is an unordered set of all paths to leaf nodes in the tree and enables various optimizations. Furthermore a rolling hash is optionally built, whereas during inserts all ancestor node hashes are adapted. Furthermore it optionally keeps track of update operations and the ctx nodes involved during txn commits. Thus, you can easily get the changes between revisions, you can check the full history of nodes, as well as navigate in time to the first revision, the last revision, the next and previous revision of a node... You can also open a revision at a specific system time revert to a revision and commit a new version while preserving all revisions in-between. As said one feature is, that the objects can be arbitrarily nested, thus almost no limits in the number and updates are cheap. A dated Jupyter notebook with some examples can be found in [3] and overall documentation in [4]. The query engine[5] Brackit is retargetable (a couple of interfaces and rewrite rules have to be implemented for DB systems) and especially finds implicit joins and applies known algorithms from the relational DB systems world to optimize joins and aggregate functions due to set-oriented processing of the operators.[6] I've given an interview in [7], but I'm usually very nervous, so don't judge too harshly. Give it a try and happy coding! Kind regards Johannes [1] https://sirix.io | https://github.com/sirixdb/sirix [2] https://sirix.io/docs/concepts.html [3] https://colab.research.google.com/drive/1NNn1nwSbK6hAekzo1YbED52RI3NMqqbG#scrollTo… [4] https://sirix.io/docs/ [5] http://brackit.io [6] https://colab.research.google.com/drive/19eC-UfJVm_gCjY--koOWN50sgiFa5hSC [7] https://youtu.be/Ee-5ruydgqo?si=Ift73d49w84RJWb2