HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
fast-pack
FastPFOR
C++2012–2024
fast-pack / FastPFOR

FastPFOR

The FastPFOR C++ library: Fast integer compression

Data & DatabasesSystems & InfrastructureLibraries & SDKs
Explore on GitHubRead Hacker News discussions
8Distinct people
10Mentions
0Explicit recommenders
978GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

The FastPFOR C++ library: Fast integer compression

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year10 dated mentions

Observed mentions: 2012-09-12 16:34:13 UTC to 2024-10-24 19:25:18 UTC.

01232012-01-01 00:00:00 UTC to 2013-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments20122013-01-01 00:00:00 UTC to 2014-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2014-01-01 00:00:00 UTC to 2015-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20182019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2024

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20122012-01-01 00:00:00 UTC2013-01-01 00:00:00 UTC303
20132013-01-01 00:00:00 UTC2014-01-01 00:00:00 UTC000
20142014-01-01 00:00:00 UTC2015-01-01 00:00:00 UTC000
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC000
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC000
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC000
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC101
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC211
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC101
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC101
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC202

What Hacker News says

10 sources
jcgrillo·Oct 24, 2024
This is looking better! L49 is giving me a little anxiety though, I'd really love to see some kind of justification for that decision. Compare e.g. to Lemire's[1]. EDIT: To be clear, I do understand that I'm making an unreasonable demand. I know the process that came up with this program has no ability to justify this or that "decision" (deliberately scare quoted because it doesn't actually have agency and can't decide at all). And that's the problem. That's why I find it very difficult to trust it. [1] https://github.com/lemire/FastPFor/blob/52e45deeab9c3a481daa...
In Everything I built with Claude Artifacts this week
jcgrillo·Oct 24, 2024
No, that's an interesting idea though. Hard to imagine how that would help with the code correctness issues, though. I haven't even dug into algorithmic correctness yet so I have no real idea whether there's room for improvement there--although I sure do suspect there is! EDIT: Oh my. After digging into the code I found this gem: fn encode_rle(&self, value: u64, count: usize) -> u64 { let selector = Simple8bSelector::RLE as u64; (selector << 60) | ((count as u64) << 30) | (value & 0x3FFFFFFF) } And this one: fn try_rle(&self, input: &[u64]) -> Option<(u64, usize)> { if input.is_empty() { return None; } let value = input[0]; let mut count = 1; for &x in input.iter().skip(1) { if x != value || count >= 0x3FFFFFFF { // Max 30-bit run length break;
Read the rest
} count += 1; } Some((value, count)) } What even is going on here? Compare to an actually sane implementation like[1] or[2]. [1]https://github.com/lemire/FastPFor/blob/master/headers/simpl... [2]https://github.com/timescale/timescaledb/blob/403782a5899c75...
In Everything I built with Claude Artifacts this week
insanitybit·Oct 17, 2023
Well if your integers are sequential you can encode huge numbers of them using diff + RLE in just a few bytes, likely far fewer than 1/2 a byte on average, for the right dataset (in theory you can store 1,2,3,4,5...10_000 in 2 bytes). But for other integer datasets there's FastPFOR https://github.com/lemire/FastPFor The linked papers there will talk about techniques that can be used to store multiple 32bit integers into a single byte, etc. Integer compression is pretty powerful if your data isn't random. The thing with UUIDs is that your data is pretty random - even a UUIDv7 contains a significant amount of random data.
In Making CRDTs More Efficient
naturalplasmoid·May 14, 2022
One notable omission from this piece is a technique to compress integer time series with both positive and negative values. If you naively apply bit-packing using the Simple8b algorithm, you'll find that negative integers are not compressed. This is due to how signed integers are represented in modern computers: negative integers will have their most significant bit set [1]. Zigzag encoding is a neat transform that circumvents this issue. It works by mapping signed integers to unsigned integers so that numbers with a small absolute value can be encoded using a small number of bits. Put another way, it encodes negative numbers using the least significant bit for sign. [2] If you're looking for a quick way to experiment with various time series compression algorithm I highly recommend Daniel Lemire's FastPFor repository [3] (as linked in th
Read the rest
e article). I've used the Python bindings [4] to quickly evaluate various compression algorithms with great success. Finally I'd like to humbly mention my own tiny contribution [5], an adaptation of Lemire's C++ Simple8b implementation (including basic methods for delta & zigzag encoding/decoding). I used C++ templates to make the encoding and decoding routines generic over integer bit-width, which expands support up to 64 bit integers, and offers efficient usage with smaller integers (eg 16 bit). I made a couple other minor tweaks including support for arrays up to 2^64 in length, and tweaking the API/method signatures so they can be used in a more functional style. This implementation is slightly simpler to invoke via FFI, and I intend to add examples showing how to compile for usage via JS (WebAssembly), Python, and C#. I threw my code up quickly in order to share with you all, hopefully someone finds it useful. I intend to expand on usage examples/test cases/etc, and am looking forward to any comments or contributions. [1] https://en.wikipedia.org/wiki/Signed_number_representation [2] https://en.wikipedia.org/wiki/Variable-length_quantity#Zigza... [3] https://github.com/lemire/FastPFor [4] https://github.com/searchivarius/PyFastPFor [5] https://github.com/naturalplasmoid/simple8b-timeseries-compr...
In Time-Series Compression Algorithms
criticaltinker·Aug 21, 2021
> Simply put, it is nicer to build your systems so that, as much as possible, they use a constant amount of memory irrespective of the input size Really good advice - this is a hard earned lesson for many folks. I've worked with quite a few data scientists who were brilliant at experimental design but not necessarily experts in the field of comp sci. Their relatively simple python scripts would run nice and fast initially. As time passed and the organization grew, their scripts would start to run slower and slower as the datasets scaled and swapping to disk started occurring, etc. In some cases they would completely lock up shared machines, taking a good chunk of the team offline for a bit. Anyway, Daniel Lemire's blog is a fantastic resource. I highly recommend taking a look through his publications and open source contributions. I was
Read the rest
able to save my employer a lot of money by building on time series compression algorithms [1] and vectorized implementations [2][3] that he has provided. [1] Decoding billions of integers per second through vectorization https://onlinelibrary.wiley.com/doi/full/10.1002/spe.2203 [2] https://github.com/lemire/FastPFor [3] https://github.com/searchivarius/PyFastPFor
In The big-load anti-pattern

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.