pym-particles

1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now? 2. Have added the link for downloading both the enwik9 slice and the nyc dataset. Apologies I forgot to add it. You can get it from here - https://github.com/samyak112/pym-particles/blob/main/README.... 3. Other than zip i tested it with zstd19, and now that you mentioned LZMA2 and BZIP2 I got results on enwik9 100mb slice as zstd - 28mb bzip2 - 30mb lzma2 - 26mb I will mention these and results from ZPAQ in the readme for both files, thanks for pointing them out!!! But the thing is this neural compression approach cant be used right now, as it takes hours to compress and de compress a 100mb file so not really usable and more of a fun project.

Lo and behold, a nice arithmetic coding implementation that wasn't written by an LLM! A sight for sore eyes – a treat, even. Looks like it was written by someone else though. Check it out: https://github.com/samyak112/pym-particles/blob/main/arithme...

I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my AMD 7800XT). Checkout the repo - https://github.com/samyak112/pym-particles