go-mysql-server

We implemented a query optimizer with a flexible intermediate representation in pure Go: https://github.com/dolthub/go-mysql-server Getting the IR correct so that it's both easy to use and flexible enough to be useful is a really interesting design challenge. Our primary abstraction in the query plan is called a Node, and is way more general than the IR type described in the article from OP. This has probably hurt us: we only recently separated the responsibility to fetch rows into its own part of the runtime, out of the IR -- originally row fetching was coupled to the Node type directly. This is also the query engine that Dolt uses: https://github.com/dolthub/dolt But it has a plug-in architecture, so you can use the engine on any data source that implements a handful of Go interface.

https://github.com/dolthub/go-mysql-server One item under "Scope of this project": Provide a runnable server speaking the MySQL wire protocol, connected to data sources of your choice.

https://github.com/dolthub/go-mysql-server/blob/main/SUPPORT... is popular but not 100%

I just wanted to say thanks for https://github.com/dolthub/go-mysql-server This is incredibly useful for anyone who wants to build their own DB or wrap another datasource so it's queryable via MySQL protocol.

a very cool project they also maintain is a MySQL server framework for arbitrary backends (in Go): https://github.com/dolthub/go-mysql-server You can define a "virtual" table (schema, how to retrieve rows/columns) and then a MySQL client can connect and execute arbitrary queries on your table (which could just be an API or other source)

https://github.com/dolthub/go-mysql-server 170kloc of pure Go

Thanks for this write up! I've been really interested in postgres compatibility in the context of a tool I maintain (https://github.com/mergestat/mergestat) that uses SQLite. I've been looking for a way to expose the SQLite capabilities over a more commonly used wire-protocol like postgres (or mysql) so that existing BI and visualization tools can access the data. This project is an interesting one: https://github.com/dolthub/go-mysql-server that provides a MySQL interface (wire and SQL) to arbitrary "backends" implemented in go. It's really interesting how compatibility with existing protocols has become an important feature of new databases - there's so much existing tooling that already speaks postgres (or mysql), being able to leverage that is a huge advantage IMO

In Go you've got: go-mysql-server [0], vitesse's parser [1], and one I wrote [2]. In JavaScript you've got: Alasql's [3] (this is a large file) and here's another SQLite parser [4]. In Java there's JSQLParser [5] and I think you can access Presto's parser too [6]. [0] https://github.com/dolthub/go-mysql-server [1] https://github.com/blastrain/vitess-sqlparser [2] https://github.com/eatonphil/gosql [3] https://github.com/agershun/alasql/blob/develop/src/alasqlpa... [4] https://github.com/codeschool/sqlite-parser [5] https://github.com/JSQLParser/JSqlParser [6] https://github.com/prestodb/presto

Interesting, hadn't seen the virtual table feature of SQLite. I'd have to do some benchmarking but that could be worth switching to in future for perhaps getting fresher data. Yes, at the moment there's a batch job to download your calendars and convert them into a SQLite DB. Then, all queries are done directly on that concrete DB (read only). Originally I was using https://github.com/dolthub/go-mysql-server which is quite similar to the virtual table feature (you just provide a struct which implements some getter methods). Unfortunately I found it a bit slow though (it had to call back into my table many times for even simple queries). Might just be a problem with that implementation and not a limitation of virtual tables themselves so thanks for pointing it out!

Nice article. Btw go-mysql-server[1] is a great project! [1]: https://github.com/dolthub/go-mysql-server