arctic_shift

Path forward is to build as close of a replacement as possible on https://atproto.com/ otherwise the enshittfication dance always continues. Doesn't have to be perfect, it just as to be able to avoid corporate capture, like water running through clasped hands. https://atprotoalternatives.com/ Historical Reddit data remains available via torrent: https://github.com/ArthurHeitmann/arctic_shift/blob/master/d...

I cannot answer this question, but you can always run your own instance scraping X, serving it as a pull through cache, while creating torrents to serve of the archive, all from outside a jurisdiction Musk can get to. If you're paying, you're contributing. https://news.ycombinator.com/item?id=49481557 (potential durable architecture subthread discussion) https://github.com/ArthurHeitmann/arctic_shift | https://github.com/ArthurHeitmann/arctic_shift/blob/master/d... (Reddit example, with the archived datasets available on AcademicTorrents)

the data from 2025-12 has been released already, it is usually released every month, it just needs to be split and reprocessed for 2025 by watchful1. i will probably eventually add support for importing data from the monthly arctic shift dumps so that archives can be updated monthly. https://github.com/ArthurHeitmann/arctic_shift/releases Arctic Shift https://academictorrents.com/browse.php?search=RaiderBDev Watchful1 https://academictorrents.com/browse.php?search=Watchful1

This is a neat project, nice work. You've probably come across this already but there are alternative archives to PushShift that may have differing sets of posts and comments (perhaps depending on removal request coverage?) One is Arctic Shift: https://github.com/ArthurHeitmann/arctic_shift/releases Another is PullPush: https://pullpush.io/

That's pretty neat! I did something similar. I build a tool[1] to import the Project Arctic Shift dumps[2] of reddit into sqlite. It was mostly an exercise to experiment with Rust and SQLite (HN's two favorite topics). If you don't build a FTS5 index and import without WAL (--unsafe-mode), import of every reddit comment and submission takes a bit over 24 hours and produces a ~10TB DB. SQLite offers a lot of cool json features that would let you store the raw json and operate on that, but I eschewed them in favor of parsing only once at load time. THat also lets me normalize the data a bit. I find that building the DB is pretty "fast", but queries run much faster if I immediately vacuum the DB after building it. The vacuum operation is actually slower than the original import, taking a few days to finish. [1] https://github.com/Paul-E/Push

ArcticShift is a project with that goal. It picks up where PushShift left off when the API changes killed that project. https://github.com/ArthurHeitmann/arctic_shift

Reddit's caches are set up to only ever return the last 1,000 of anything. So for example - you can't scroll past 1k items on /new, and if you save more than 1k posts then you'll have to unsave some to retrieve the others. If this extension only edits comments, it'll only touch the most recent 1k. You would need to retrieve the older ones with a Pushshift replacement like this: https://pullpush.io/. But that also shows how ineffective this is. We still have public reddit archives (like Pullpush and https://github.com/ArthurHeitmann/arctic_shift) which contain comments as they were originally posted. This isn't gonna be a problem for Google.