trustfall

SQL really sucks. In general I'm not a fan of SQL-like syntax since it makes queries with lots of joins really difficult to read -- it's hard to keep track which value came from where. GraphQL has some neat ideas and I really love the clarity the nesting provides. But GraphQL is also too limiting: it can't express arbitrary filters, recursion, optional edges (i.e. left joins), etc. My own take on this is an embeddable GraphQL-like language with all those features and with strong guarantees about execution performance. I've been calling it Trustfall, and you can play around with it (or check out example queries over HN data) in your browser here: https://play.predr.ag/ For example, here's the query "Find the GitHub or Twitter accounts listed in the bios of HN users that comment on HN stories pointing to Open AI's website": https://play.pre

Joint work with https://twitter.com/PaavanBhavsar and https://eyurtsev.github.io/ Execution is 100% client-side in the browser. There's no backend server, predr.ag is hosting only static files. Open the dev tools network pane and run a query -- you'll see that the only requests going out are to HN APIs. Still a bit rough around the edges: - error cases need polish - the UI is not mobile-friendly at the moment - some open Safari bugs prevent it from working on Safari or on iOS devices, etc. That said, you can query for things like: - patio11 comments on HN stories about his own blog posts: https://play.predr.ag/hackernews#?f=1&q=IyBDb21tZW50cyBieSBw... - story comments that link to other HN stories: https://play.predr.ag/hackernews#?f=1&q=IyBDb21tZW50cyB0aGF0... - discussions where the top-level comment author replied 2+ more times in the

There's an interesting tension here. On one hand, GraphQL is versatile and can simplify frontend data handling code. On the other, as schema and data sizes grow, query expressiveness is limited and performance can be quite difficult to maintain. I've put GraphQL in front of TB-scale data sources with X00,000 tables, and it was clear that (a) GraphQL wasn't designed for anything that big, and (b) if we could get it there, it would be transformational. This is why I built Trustfall[0]: a system that keeps the elegant GraphQL syntax, and tweaks the semantics to ensure great performance and better expressiveness: left joins / recursion / aggregations / arbitrary filters etc. Trustfall flips the proposition around: acknowledge we're wrapping a database around every data source (API server / database / filesystem / ML etc.) and make it the best

Trustfall (see sibling comment: https://news.ycombinator.com/item?id=32120007 ) is my attempt at what you're describing: GraphQL syntax with custom directives like @recurse / @optional / aggregations / arbitrary filters, executed client-side over whatever remote APIs are available (REST or otherwise). It's by no means fully polished yet and lacks many features that'd be useful to have, but I'd love your thoughts if you decide to take a peek! It supports Rust, Python, and (extremely soon!) WASM in the browser. https://github.com/obi1kenobi/trustfall

I designed Trustfall[0] out of frustration with existing systems that either: - didn't allow optimizations like batch get and predicate pushdown even when I knew they'd help, or - made it really difficult to add them piecemeal after-the-fact on an existing schema, by either demanding everything be batched and optimized, or that nothing be batched and optimized. In my experience, it's usually a small component of the schema that is causing most of the runtime cost, and it's not always predictable ahead of time where it's going to be. It's dependent on the queries and the data size and shape, so it changes over time. So I wanted to be able to write a working schema and adapter quickly, and then optimize the parts that were slow in practice for my workload and not just ones that "hypothetically might be slow," and this is what Trustfall lets

I'm building out Trustfall[0] with most of the same goals in mind: query everything with a single composable syntax and fast performance, incrementally evolve the schema, and build up the types of objects it can understand and query into your own personal (or shared!) graph. I recently gave a conference talk on how we use queries over Dockerfiles + project configuration + deployment info to triage bugs and prevent them from coming back at my workplace. An example query is "find Python projects whose pyproject.toml specifies Python 3.9 but which get deployed with a Dockerfile that installs Python 3.7 instead." Here's the talk link: https://www.hytradboi.com/2022/how-to-query-almost-everythin... If you'd like to see Trustfall in action, I recently used it to build a Rust semver-checking tool[1] that's just a bunch of Trustfall queries like

I made a query language that parses as legal GraphQL using a vanilla parser, but has directives like `@filter, @recurse, @optional` etc. that make it a real query language. Also, instead of returning a giant fully-nested result, it flattens results and emits them row-by-row like a SQL database. This means the query evaluation can be lazy and incremental — if you write a query that has a billion results but only load 20 of them, then only 20 rows' worth of work happens. My company has been using this in production for 6 years now across everything from TB-scale SQL clusters with X00,000 tables/views, to querying our own codebase, configuration, and deployment information to find and prevent bugs. I gave a 10min talk at a conference about this recently, if you'd like to learn more: https://www.hytradboi.com/2022/how-to-query-almost-everythi

Yesterday, I found trustfall[1], which tries to query almost everything in GraphQL'ish format. This[2] talk by author explains it in few minutes. [1]: https://github.com/obi1kenobi/trustfall [2]: https://www.hytradboi.com/2022/how-to-query-almost-everythin...

100% agree that pagination is extremely awkward, especially with nesting. Between the pagination problem and the "oops I asked for too much data and blew up the server" problem, I think it's more work than one might think to run a GraphQL API. For my own work, I took things in a different direction: I made a query language that parses as legal GraphQL using a vanilla parser, but has directives like `@filter, @recurse, @optional` etc. Instead of returning a giant fully-nested result, it flattens results and emits them row-by-row like a SQL database. This means the query evaluation can be lazy and incremental — if you write a query that has a billion results but only load 20 of them, then only 20 rows' worth of work happens. My company has been using this in production for 6 years now across everything from TB-scale SQL clusters with X00,00

Why hand-roll one when you can use one that's already available and thoroughly tested :) https://github.com/obi1kenobi/trustfall (Hi Dustin!)

If you want to see what a GraphQL with an algebra could look like, I built one! The query language is parsed with a vanilla GraphQL parser, but has directives like `@filter, @recurse, @optional` etc. 10min talk video: https://www.hytradboi.com/2022/how-to-query-almost-everythin... GitHub: https://github.com/obi1kenobi/trustfall