CRcriticaltinker·Aug 21, 2021
> Simply put, it is nicer to build your systems so that, as much as possible, they use a constant amount of memory irrespective of the input size
Really good advice - this is a hard earned lesson for many folks. I've worked with quite a few data scientists who were brilliant at experimental design but not necessarily experts in the field of comp sci. Their relatively simple python scripts would run nice and fast initially. As time passed and the organization grew, their scripts would start to run slower and slower as the datasets scaled and swapping to disk started occurring, etc. In some cases they would completely lock up shared machines, taking a good chunk of the team offline for a bit.
Anyway, Daniel Lemire's blog is a fantastic resource. I highly recommend taking a look through his publications and open source contributions. I was
Read the rest
able to save my employer a lot of money by building on time series compression algorithms [1] and vectorized implementations [2][3] that he has provided.
[1] Decoding billions of integers per second through vectorization
https://onlinelibrary.wiley.com/doi/full/10.1002/spe.2203
[2] https://github.com/lemire/FastPFor
[3] https://github.com/searchivarius/PyFastPFor