jiter

Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032. Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse. This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.

A trick I use a lot these days is to take the unit tests from an under-documented library, dump them into an LLM and ask it to write me detailed usage documentation. This works REALLY well. I've even occasionally done some of my own reviewing and editing of those docs and submitted them back to the project. Here's an example: https://github.com/pydantic/jiter/pull/143 - Claude transcript here: https://gist.github.com/simonw/264d487db1a18f8585c2ca0c68e50...

That's neat, I hadn't seen that. Docs were lacking so I submitted a PR: https://github.com/pydantic/jiter/pull/143

Pydantic also have support for parsing partial JSON. https://docs.pydantic.dev/latest/concepts/json/#partial-json... from pydantic_core import from_json partial_json_data = '["aa", "bb", "c' result = from_json(partial_json_data, allow_partial=True) print(result) #> ['aa', 'bb'] You can also use their `jiter` package directly if you don't otherwise use pydantic. https://github.com/pydantic/jiter/tree/main/crates/jiter-pyt...

There is basic GraalPy support in Maturin[0] and PyO3[1], the problem is often that packages require older Maturin/PyO3 versions and/or they use CPython-isms, semi-public APIs, etc., but it is getting better, for example [2]. It is fair to say that large projects with a huge set of dependencies will likely face some compatibility issues, but we're working on ironing this out. There is GraalPy support in setup-python GitHub action. GraalPy is supported in the manylinux image [3]. Hopefully soon also in cibuildwheel [4]. [0] https://github.com/PyO3/maturin/pull/1645 (merged) [1] https://github.com/PyO3/pyo3/pull/3247 (merged) [2] https://github.com/pydantic/jiter/pull/135 (merged) [3] https://github.com/pypa/manylinux/pull/1520 (merged) [4] https://github.com/pypa/cibuildwheel/pull/1538