sdsl-lite

My goto library for succinct data structures is SDSL-Lite [0]. [0] https://github.com/simongog/sdsl-lite

Succinct Data Structures [0] [1]. It encompass many different underlying data structure types but the overarching idea is that you want small data size while still keeping "big O" run time. In other words, data structures that effectively reach a 'practical' entropy lower bound while still keeping asymptotic run time. [0] https://en.wikipedia.org/wiki/Succinct_data_structure [1] https://github.com/simongog/sdsl-lite

You may be interested in any or all of these: Minimal Acyclic Finite State Automata: http://stevehanov.ca/blog/?id=115 (related: https://web.archive.org/web/20120302104036/http://siganakis....) Succinct Tries: http://stevehanov.ca/blog/?id=120 (related: https://alexbowe.com/succinct-debruijn-graphs/) Compression and completion using GPT-2: https://bellard.org/textsynth/index.html (related: https://ed-von-schleck.github.io/shoco/) Search and compression with Finite State Transducers: http://blog.burntsushi.net/transducers/ (related: https://swtch.com/~rsc/regexp/) Collection of succinct string representations: https://github.com/simongog/sdsl-lite (related: http://pizzachili.dcc.uchile.cl/) Search with compressed Radix Trie: https://cr.yp.to/critbit.html related: http://reports-archive.adm.cs.cmu.edu/anon/2020/CMU-CS-20-10...

Thanks. > ...adaptive succinct indexing structures We've been optimising for low-memory and compact / succinct data-structures have been super impactful (especially, static succinct Radix Tree and Data Sketches like CountMinSketch). Apart from the excellent sdsl-lite [0], roaring-bitmaps [1], apache data-sketches [2], and pizza-chilli [3], what are other resources that you recommend we take a look? > ...storage ratio was 1:1000, an order of magnitude This is insane: Is it due to compression techniques or... Could you please point a few reasons why the gap is this high? > The other major contributor to performance is scheduler design... Out of curiosity: Any pointers to papers that we can take a look at? --- [0] https://github.com/simongog/sdsl-lite [1] https://roaringbitmap.org/ [2] https://datasketches.apache.org/ [3] http://pizzachili.d

Perhaps it's worth considering succinct data structures (https://en.wikipedia.org/wiki/Succinct_data_structure), e.g.: - BitMagic: http://bitmagic.io/, https://github.com/tlk00/BitMagic - Succinct Data Structure Library 2.0: https://github.com/simongog/sdsl-lite, https://github.com/simongog/sdsl-lite/wiki/Literature

SDSL-lite is a fantastic library, Simon (the main author) is one of my colleagues and spends a lot of time on making it easy to use: https://github.com/simongog/sdsl-lite Without having read the paper in detail, SODA (the conference where it was presented) papers do have a reputation for being very theoretical, though, so it may not be implemented any time soon.

It is very likely that this will eventually end up in sdsl-lite and/or other compressed data structures libraries https://github.com/simongog/sdsl-lite

At first I found intriguing to see no mention of SDSL[1], which probably is one of the best libraries in the field of compressed data structures, including compressed bitmaps. Probably because the index is built to answer a "different kind of queries". There is a blog commenting on those "other" queries[2]. In the same blog the author discusses some other compressed data structures included in the library. 1: https://github.com/simongog/sdsl-lite 2: http://alexbowe.com/rrr/

This scale is pretty small and can be done on my laptop. If you need to do such operations at a truly large scale, this is an interesting article: http://alexbowe.com/rrr/ C++ implementation: https://github.com/simongog/sdsl-lite/blob/master/README.md

I'm one of the authors of a more complete c++11 succinct data structure library [1] which includes several engineered implementations of wavelet trees, compressed suffix arrays and compressed suffix trees. while you are right that random access is one of the main bottlenecks of succinct data structures, we found that sometimes SDS can be faster than their uncompressed counterparts [2] The github URL contains some additional resources such as tutorial slides and cheatsheets if you want to check it out. [1] https://github.com/simongog/sdsl-lite/ [2] http://people.eng.unimelb.edu.au/sgog/optimized.pdf