t-digest

The state of the art has moved well beyond these algorithms. See these https://github.com/tdunning/t-digest https://www.sciencedirect.com/science/article/pii/S266596382... https://arxiv.org/pdf/2102.09299 And, as I mentioned else-thread, exponential histograms are the best choice in almost all practical situations.

Most quantile sketches (and t-digest in particular) do not assume stationarity. Note also that there are other bounds of importance and each has trade-offs. T-digest gives you a strict bound on memory use and no dynamic allocation. But it does not have guaranteed accuracy bounds. It gives very good accuracy in practice and is very good at relative errors (i.e. 99.999th percentile estimate is between the 99.9985%-ile and 99.9995%-ile) KL-sketch gives you a strict bound on memory use, but is limited to absolute quantile error. (i.e. 99.99%-ile is between 99.9%-ile and 100%-ile. This is useless for extrema, but fine for medians) Cormode's extension to KL-sketch gives you strict bound on relative accuracy, but n log n memory use. Exponential histograms give you strict bounds on memory use, no allocation and strict bounds on relative error in

Sounds like you’re looking for T-digests[1] - most production systems I’ve worked on that do this are using them under the hood. 1: https://github.com/tdunning/t-digest

> I get a lot of benefit from this information but somehow it feels shallow. I take a longer view to this. For example, a few years ago I read about an algorithm to calculate percentiles in real time. [0] It literally just came up at work today. I haven't used that information but maybe two times since I read it, but it was super relevant today and saved my team potential weeks of development. So maybe it's not so shallow. But to your actual question, I have a similar problem. The best I can say is that deadlines help. I usually put down the HN and Youtube when I have a deadline coming up. And not just at work. I make sure my hobbies have deadlines too. I tell people when I think something will be done, so they start bugging me about it when it doesn't get done, so that I have a "deadline". Also one of my hobbies is pixel light sho

I am enamored by data structures in the sketch/summary/probabilistic family: t-digest[1], q-digest[2], count-min sketch[3], matrix-sketch[4], graph-sketch[5][6], Misra-Gries sketch[7], top-k/spacesaving sketch[8], &c. What I like about them is that they give me a set of engineering tradeoffs that I typically don't have access to: accuracy-speed[9] or accuracy-space. There have been too many times that I've had to say, "I wish I could do this, but it would take too much time/space to compute." Most of these problems still work even if the accuracy is not 100%. And furthermore, many (if not all of these) can tune accuracy to by parameter adjustment anyways. They tend to have favorable combinatorial properties ie: they form monoids or semigroups under merge operations. In short, a property of data structures that gave me the ability to solve

On sketches, there is a genre of structure for estimating histogram-like statistics (median, 99th centile, etc) in fixed space, which i really like. Two examples: t-digest https://github.com/tdunning/t-digest DDSketch https://github.com/DataDog/sketches-java

Ah, I misunderstood what you meant. If you are reporting static buckets I get how that is better than what folks typically do but how do you know the buckets a priori? Others back their histograms with things like https://github.com/tdunning/t-digest. It is pretty powerful as the buckets are dynamic based on the data and histograms can be added together.

There are some newer data structures that take this to the next level such as T-Digest[1], which remains extremely accurate even when determining percentiles at the very tail end (like 99.999%) [1]: https://arxiv.org/pdf/1902.04023.pdf / https://github.com/tdunning/t-digest

This is pretty cool. The title would be a bit more descriptive if it were “Fast Rolling Quantile Filters for Python”, since the high-pass/low-pass filter functionality seems to be the focus. The README mentions it uses binary heaps - if you’re willing to accept some (bounded) approximation, then it should be possible to reduce memory usage and somewhat reduce runtime by using a sketching data structure like Dunning’s t-digest: https://github.com/tdunning/t-digest/blob/main/docs/t-digest.... There is an open source Python implementation, although I haven’t used it and can’t vouch for its quality: https://github.com/CamDavidsonPilon/tdigest

Great article! One of my favorite data-structures lately is the https://github.com/tdunning/t-digest. Would be great to see how it compares and what additional accuracy we get but storing some of the data.

Yes, exactly this. It's the fact that you're doing aggregation in two places. Since you're always going to be aggregating on the backend, aggregating in the app is bad news. It may be interesting to think about the class of aggregate metrics that you can safely aggregate. Totals can be summed. Counts can be summed. Maxima can be maxed. Minima can be minned. Histograms can be summed (but histograms are lossy). A pair of aggregatable metrics can be aggregated pairwise; a pair of a total and a count lets you find an average. Medians and quantiles, though, can't be combined, and those are what we want most of the time. Someone who loves functional programming can tell us if metrics in this class are monoids or what. There is an unjustly obscure beast called a t-digest which is a bit like an adaptive histogram; it provides a way to aggregate n

> which was too expensive for Uber to run (since the dataset is in petabytes). Ok, but I am fairly confident Netflix also is at that kind of scale. Netflix has a section on Atlas's documentation about how they get around this: https://github.com/Netflix/atlas/wiki/Overview#cost They also did this nice video that outlines their entire operation including how they do rollups: https://www.youtube.com/watch?v=4RG2DUK03_0 This is how they do the rollup but keep their tails accurate to parts per million and the middle to be parts per hundred: https://github.com/tdunning/t-digest

How does this compare to t-digest? https://github.com/tdunning/t-digest/blob/master/docs/t-dige...

I agree with the premise but it seems that there are more solutions out there. As other commenters noted, you can collect histograms or hdrhistograms. Those have the problem of needing to be precofiguring and of not being able to be merged unless they are configured the same way. Instead you can use the t-digest (https://github.com/tdunning/t-digest), a very cool online quantile estimation data structure from Ted Dunning (which he has recently improved with the Merging approach). There are a number of implementations out there. It is not unreasonable to serialize them and merge them. Unfortunately there’s no easy way to set this up in Prometheus but making that easy could be a fun project