libpg_query

ally easy to upgrade major versions, i.e., we just bump the submodules and re-compile. Let us know what you think! - PgDog team [1]: https://github.com/pganalyze/libpg_query [2]: https://github.com/pganalyze/pg_query.rs [3]: https://github.com/pgdogdev/pgdog

Rust, but would require a lot of maintained glue code for other languages. All that said, I'm supportive of Lev's effort here, and we'll add some additional functions (see [0]) in the libpg_query library to make using it directly (i.e. via FFI) easier. But I don't see Protobuf going away, because in non-performance critical cases, it is more ergonomic across the different bindings. [0]: https://github.com/pganalyze/libpg_query/pull/321

ps://github.com/pganalyze/libpg_query/tree/17-latest/patc... [1]: https://github.com/pganalyze/libpg_query/blob/17-latest/scri...

can do this — see https://www.pgpool.net/docs/latest/en/html/runtime-config-lo.... > Company policy puts all kinds of restrictions on desktop connections to DBs Presuming these restrictions exist in your company only to ensure that developers' more casually-guarded security credentials aren't exfiltrated from their machines by a virus or a phishing attack and used to attach to your servers from some external IP: consider putting your developer workstations and your DB servers together onto an enterprise-managed Tailscale overlay-network, and then putting a line in pg_hba.conf such that DB users in the "developers" DB group can only connect from the overlay-network's IP range. (AFAIK it's very hard to exfiltrate a logged-in Tailscale session and reuse it elsewhere.) However, presuming you care more about this: > The objective is mostly logging DDL actions and limiting certain DDL operations. One "hack" approach that will get you just these two properties, but otherwise keep things to what you're already doing, is a pretty old-school one: command-line-specific /etc/sudoers lines. Specifically, you would: 1. Make a DB user called ddl_maintenance; make this DB user, and only this DB user, a pseudo-superuser capable of executing DDL statements. Your developer DB users should not have this privilege. 2. On the jump box, create a ddl_maintenance OS user, with /sbin/nologin as their login shell. 3. Also give the ddl_maintenance user a ~/.pgpass file, containing their DB credentials (so that psql can connect automatically when run as this user). chmod it 400, so that other users won't learn these credentials. 4. Write a policy-enforcer program, that parses a .sql file, and exits 0 if every statement in that file is safe to run per your company policy — or with some other exit code if it's not. (Use https://github.com/pganalyze/libpg_query, or one of its HLL bindings. I'd probably recommend the Golang one for simplest deployment.) 5. Write a glue script that receives a path to a .sql file as $1 (and/or a stream of SQL on stdin); buffers it to a spool dir, so that it can be streamed more than once; tests it for conformance with your policy-checker program from #4; and if it passes, executes it with psql. (Don't pass on arbitrary flags to psql — you want to force your devs to use psql a specific, auditable way.) You want something that looks like this: if ddl-looks-good "${sql_in_spool}"; then psql -a -f "${sql_in_spool}" | tee -a "/var/log/ddl_maintenance.log" fi 6. Add a line to /etc/sudoers, that allows users to sudo as ddl_maintenance without a password, but only to execute this glue script using sudo -c. 7. (optional) wrap the required sudo incantation in a friendly wrapper script, so the devs don't have to learn Weird Sudo Tricks to get their job done. Document this final wrapper script; treat everything else as implementation details. --- All that being said, if you have a whitelist of DDL commands you want your devs to run... just make some SECURITY DEFINER stored procedures that do those things (and which audit-log into one or more tables or THROW NOTICEs into the PG log), and grant your DB "developers" group EXECUTE privilege on those sprocs. Problem solved :)

le incomplete inputs, and outputs just the ast, not the cst. To use it for a language server we had to work around these limitations as good as possible. While we leverage procedural macros in rust to generate a lot of the repetitive parser code, there remains a portion that requires a bit of manual work. But the groundwork is completed, and we can finally start working on the data model and the actual server next. Our aim is to bring this to a usable state as swiftly as possible. Huge shout-out to pg_analyze for creating and maintaining libpg_query[1], without which this project would not be possible! [0] https://news.ycombinator.com/item?id=37020610 [1] https://github.com/pganalyze/libpg_query

.e. raw parser + AST structs). Hopefully something we can spend a bit of time on next year. [0]: https://github.com/pganalyze/libpg_query

parser (in a non-C language) myself which was faster." It's true that the core PG code isn't written in a modular way that's friendly to integration piecemeal in other projects (outside of libpq). For THIS PARTICULAR case, the pganalyze team has actually extracted out the parser of PG for including in your own projects: https://github.com/pganalyze/libpg_query

olt can be set up as a replica of your existing MySQL or MariaDB database using standard MySQL binlog replication. Every write becomes a Dolt commit. This is a great way to get the version control benefits of Dolt and keep an existing MySQL or MariaDB database. #. github/gh-ost: https://github.com/github/gh-ost : > Instead, gh-ost uses the binary log stream to capture table changes, and asynchronously applies them onto the ghost table. gh-ost takes upon itself some tasks that other tools leave for the database to perform. As result, gh-ost has greater control over the migration process; can truly suspend it; can truly decouple the migration's write load from the master's workload. #. vlcn-io/cr-sqlite: https://github.com/vlcn-io/cr-sqlite : > Convergent, Replicated SQLite. Multi-writer and CRDT support for SQLite > CR-SQLite is a run-time loadable extension for SQLite and libSQL. It allows merging different SQLite databases together that have taken independent writes. > In other words, you can write to your SQLite database while offline. I can write to mine while offline. We can then both come online and merge our databases together, without conflict. > In technical terms: cr-sqlite adds multi-master replication and partition tolerance to SQLite via conflict free replicated data types (CRDTs) and/or causally ordered event logs. yjs also does CRDTs (Jupyter RTC,) #. pganalyze/libpg_query: https://github.com/pganalyze/libpg_query : > C library for accessing the PostgreSQL parser outside of the server environment #. Ibis + Substrait [ + DuckDB ] https://ibis-project.org/blog/ibis_substrait_to_duckdb/ : > ibis strives to provide a consistent interface for interacting with a multitude of different analytical execution engines, most of which (but not all) speak some dialect of SQL. > Today, Ibis accomplishes this with a lot of help from `sqlalchemy` and `sqlglot` to handle differences in dialect, or we interact directly with available Python bindings (for instance with the pandas, datafusion, and polars backends). > [...] `Substrait` is a new cross-language serialization format for communicating (among other things) query plans. It's still in its early days, but there is already nascent support for Substrait in Apache Arrow, DuckDB, and Velox. #. ibis-project/ibis-substrait: https://github.com/ibis-project/ibis-substrait #. tobymao/sqlglot: https://github.com/tobymao/sqlglot : > SQLGlot is a no-dependency SQL parser, transpiler, optimizer, and engine. It can be used to format SQL or translate between 19 different dialects like DuckDB, Presto, Spark, Snowflake, and BigQuery. It aims to read a wide variety of SQL inputs and output syntactically and semantically correct SQL in the targeted dialects. > It is a very comprehensive generic SQL parser with a robust test suite. It is also quite performant, while being written purely in Python. > You can easily customize the parser, analyze queries, traverse expression trees, and programmatically build SQL. > Syntax errors are highlighted and dialect incompatibilities can warn or raise depending on configurations. However, it should be noted that SQL validation is not SQLGlot’s goal, so some syntax errors may go unnoticed. #. benbjohnson/postlite: https://github.com/benbjohnson/postlite : > postlite is a network proxy to allow access to remote SQLite databases over the Postgres wire protocol. This allows GUI tools to be used on remote SQLite databases which can make administration easier. > The proxy works by translating Postgres frontend wire messages into SQLite transactions and converting results back into Postgres response wire messages. Many Postgres clients also inspect the pg_catalog to determine system information so Postlite mirrors this catalog by using an attached in-memory database with virtual tables. The proxy also performs minor rewriting on these system queries to convert them to usable SQLite syntax. > Note: This software is in alpha. Please report bugs. Postlite doesn't alter your database unless you issue INSERT, UPDATE, DELETE commands so it's probably safe. If anything, the Postlite process may die but it shouldn't affect your database. #. > "Hosting SQLite Databases on GitHub Pages" (2021) re: sql.js-httpvfs, DuckDB https://news.ycombinator.com/item?id=28021766 #. >> - bittorrent/sqltorrent https://github.com/bittorrent/sqltorrent >> Sqltorrent is a custom VFS for sqlite which allows applications to query an sqlite database contained within a torrent. Queries can be processed immediately after the database has been opened, even though the database file is still being downloaded. Pieces of the file which are required to complete a query are prioritized so that queries complete reasonably quickly even if only a small fraction of the whole database has been downloaded. #. simonw/datasette-lite: https://github.com/simonw/datasette-lite datasette, *-to-sqlite, dogsheep "Loading SQLite databases" [w/ datasette] https://github.com/simonw/datasette-lite#loading-sqlite-data... #. awesome-db-tools: https://github.com/mgramin/awesome-db-tools Lots of neat SQLite/vtable/pg/replication things

://github.com/pganalyze/libpg_query/tree/15-latest/src/... Further, there isn't a canonical way to output node trees into a text format today in core, besides the rather hard to work with output of debug_print_parse - there have been discussions on -hackers to potentially utilize JSON here, which may make this a bit easier. Note that in libpg_query we currently use Protobuf (but used to use JSON), which does have the benefit of getting auto-generated structs in the language bindings - but Protobuf is not used in core Postgres at all today. All in all, I think there is some upstream interest, but its not clear that this is a good idea from a maintainability perspective.