libpg_query

Hey HN! We built a wrapper on top of the Postgres native SQL parser to allow us to parse and manipulate SQL in Rust. Our library forks libpg_query [1] and its original Rust bindings, pg_query.rs [2]. By not using Protobuf and mapping our structs directly on top of the Postgres arena allocator, we are able to achieve a pretty massive speedup: - 20-60x parse/deparse, depending on query size - 90% memory utilization decrease vs pg_query.rs This library is used in production in PgDog [3], but has no dependencies to it, so you can use it in your Rust apps without issues. It also provides access to Postgres internals used to walk and transform query ASTs. Those are actually really cool. Check out the README for details, but we basically use a bunch of Rust macros to generate 100% compatible Rust structs directly from C headers. This makes it re

Since there seems to be some confusion in the comments about why pg_query chose Protobufs in the first place, let me add some context as the original author of pg_query (but not involved with PgDog, though Lev has shared this work by email beforehand). The initial motivation for developing pg_query was for pganalyze, where we use it to parse queries extracted from Postgres, to find the referenced tables, and these days also rewrite and format queries. That use case runs in the background, and as such is much less performance critical. pg_query actually initially used a JSON format for the parse output (AST), but we changed that to Protobuf a few major releases ago, because Protobuf makes it easy to have typed bindings in the different languages we support (Ruby, Go, Rust, Python, etc). Alternatives (e.g. using FFI directly) make sense for