ALAlifatisk·Oct 6, 2024
There's even more alternatives at the bottom https://github.com/dgraham/json-stream?tab=readme-ov-file#al...
In Wikidata as a Giant Crosswalk Filedgraham
json-stream
Ruby2015–2024
dgraham / json-stream
json-stream
A streaming JSON parser that generates SAX-like events.
5Distinct people
5Mentions
0Explicit recommenders
202GitHub stars
What to know before exploring
What it is
A streaming JSON parser that generates SAX-like events.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 2:04:58 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year5 dated mentions
Observed mentions: 2015-12-11 13:17:44 UTC to 2024-10-06 09:39:31 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 2 | 0 | 2 |
What Hacker News says
5 sourcesIt's too bad there aren't more streaming JSON parsers like oboe.js[1]. It would be nice if parsing libraries always supplied an event-based approach like this in addition to parsers that build up the entire data structure in memory.
[1]: https://github.com/jimhigson/oboe.js
EDIT: looking around a bit, I found json-stream ( https://github.com/dgraham/json-stream ) for Ruby.
In Wikidata as a Giant Crosswalk FileThere is no reason you can’t do a SAX-style parser for JSON. A quick bit of searchengineering will find dozens, eg. https://github.com/dgraham/json-stream
In XML is almost always misused(1) Try a language with fast allocations (C, C++, Rust, maybe Go or Java) -- anything except Python or Ruby
or
(2) Try using streaming API (I don't know Ruby, but quick google found https://github.com/dgraham/json-stream ).
Note that this method will require you to massively restructure your program -- you want to avoid having all of the data in memory at once.
The streaming API might work better with jq-based preprocessing -- for example, if you want to compare two unsorted sets, it may be faster to sort them using jq, then compare line-by-line using streaming API.
In Faster and simpler with the command line: deep-comparing JSON files with jqI had a very similar problem with Ruby and some of our workloads. I ended up utilizing this evented parser [1] to process smaller subsets of the JSON document and then clear them from memory. The result is a memory usage that is at most the size of one of these subsets (plus some overhead).
I have been wanting to generalize this approach with some open source software for some time, but I haven't gotten around to it. I think an attractive solution is for a user to specify a JSONPath [2] upfront about the types of subsets they are interested in. In the case of Ruby these chunks could then be yielded by an iterator and cleared, but I wonder if a more general-purpose CLI program which prints the subsets to stdout would also work if the subsets are small enough. But this adds another serialization/deserialization step.
[1] https://github.com/