lz-string

be a novel innovation! Aside from a lack of time and energy, the main thing holding me back has been, and I'm dead serious, off-by-one errors. It's really, really easy to screw up the order of adding new entries to the dictionary on both the compression/decompression side in such a way that you just get garbage out, and it's a pain to debug. If my algo turns out to be original, I'm tempted to call it LZWAN, or Lempel Ziv With Amnesiac Nodes (referring to the nodes in the trie used to grow the dictionary). [0] https://en.wikipedia.org/wiki/LZ77_and_LZ78 [1] https://pieroxy.net/blog/pages/lz-string/index.html [2] https://ethw.org/History_of_Lossless_Data_Compression_Algori... [3] https://github.com/pieroxy/lz-string/issues/114#issuecomment...

yer-moore) [2], or use a json-map at a cost of higher RAM usage but blazing-fast lookups (~25 MiB), or use succinct radix-trees for optimal RAM usage (~1.5 MiB) but relatively slower lookups, or use bloom-filters with low false-positive but fast membership queries at extremely frugal RAM usage (~200 KiB) [3]. That said, the simplest way to ad-block would be to point the workers instance to adguard-doh. [0] https://news.ycombinator.com/item?id=21598413 [1] https://github.com/pieroxy/lz-string/ [2] https://stackoverflow.com/questions/5562297/fast-search-in-c... [3] https://news.ycombinator.com/item?id=22017868

orm they can validate against.

k's use case look-up performance matters more than insertion performance, which makes it very different from LZString's). But it should be obvious that HNTrie's implementation[2] is something I'm very interested in ;). [0] https://github.com/pieroxy/lz-string/pull/98 [1] https://github.com/pieroxy/lz-string/blob/ba8988028d78962eba... [2] https://github.com/gorhill/uBlock/blob/2a91a685ce3d2dae5d3c2...

, and because JS strings are UTF16, the keys are SMI. It also removes string creation overhead). - for a long time, I used one object for each node, with 0 as the key to look up its value, and charCode+1 to look up next nodes. In recent benchmarks it turned out that { v: val, d: {} } is faster[4][5][6]. This makes sense because: "we don't store information about present indexed properties on the HiddenClass" [0]. In other words: the new options creates a distinct hidden class compared to a plain object, which might be optimised. At the same time adding integer keys to a separate d object will not mess with this hidden class, preventing de-opts. - for really fast performance except in a few degenerate cases, instead of key look-up, use an array with linear search (yes, really! Turns out compact arrays without holes are faster to the point where even linear search makes it worth it - up to a certain length). So this explains a lot of why certain approaches work better than others. Some of the explanations here might let me optimise it even further, but at this point it would become hyper-optimized for V8. Are there some take-aways that I can generalise? And is there a similar blog for the other big javascript engines? I expect that "type stability" - that is, maintaining identical hidden classes - is fairly universal for example. [0] https://v8project.blogspot.se/2017/08/fast-properties.html [1] https://github.com/pieroxy/lz-string/pull/98 [2] https://en.wikipedia.org/wiki/Trie [3] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... [4] https://run.perf.zone/view/LZ-String-2018-01-18-v2-151619850... [5] https://run.perf.zone/view/LZ-String-2018-01-18-v1-151619752... [6] https://github.com/pieroxy/lz-string/pull/98/commits/1894f7c...

.html [1] https://github.com/pieroxy/lz-string/pull/98 [2] https://idyll-lang.github.io/ [3] https://p5js.org/ [4] https://github.com/idyll-lang/idyll/issues/117 [5] https://jobleonard.github.io/idyll-p5/

y more complicated data structure where you replace the object with an array and do a linear search: https://jsfiddle.net/vanderZwan/ccfxv4sg/1/ For most data this is the fastest, because the array will only be a handful of characters (except at the root). However, in certain special cases (specifically, if you have a lot of unique characters at the root) it becomes prohibitively slow.

set as a look-up to reverse the operation. The latter is already a lot shorter when stringified, but also more compressible: the character set is limited to ten digits, commas and square brackets, and different values may be transformed into the same numbers, being distinguished by their position in the array tree. To make this operation easier, I created a few helper functions to declare a "schema" that creates a recursive function for transforming the original JS object to said array of arrays, and vice-versa: https://gist.github.com/JobLeonard/a47692a1f77bebc06c2518f32... As you can see in the gist, that transformation shrunk a URL of 2466 characters down to a (admittedly still crazy) 638 characters. I was thinking on putting it on NPM but it looked so specific in its applicability that I haven't bothered turning it into a proper package yet. [0] https://github.com/ReactTraining/react-router [1] https://github.com/pieroxy/lz-string