Multicorn

> I wonder why this wasn't built on top of, or an enhancement to, the existing (excellent) multicorn[1] project Have to agree with you there, multicorn is extremely cool. I'm a big sqlalchemy fan so their default SQLA wrapper was a killer feature to give up (although maybe we could do something similar with launchbadge/sqlx[1]). We investigated using multicorn early this year and had a few hiccups. Activity on the original repo[2] quieted way down ~3 years ago. For example, pg14 support hasn't landed and the newest supported python version is EOL in 2022. There is new fork[3] with pg14 support (15 in the pipe) that might pick up in adoption but thats still TBD. Supabase aims to support new major Postgres versions within 2-3 months so we have to be very careful taking on dependencies that might slow that process. > I'd love to understand m

I think the easiest is python https://github.com/Segfault-Inc/Multicorn

I think FDWs are not more used because they're not easy to get into. The best link/example I found was https://github.com/beargiles/passwd-fdw and it's quite easy to follow the code and understand all the moving parts. Once you've written a FDW you'll see them everywhere. In fact the same author wrote https://github.com/beargiles/zip file-fdw and https://github.com/beargiles/tarfile-fdw If you (still?) need inspiration and want to see what already exists: https://wiki.postgresql.org/wiki/Foreign_data_wrappers Simpler, using a 'generic' file-based FDW shipping with pg's sources: https://aaronparecki.com/2015/02/19/8/monitoring-cpu-memory-... There's a python wrapper to get your feet wet (or prototype an idea) : https://github.com/Segfault-Inc/Multicorn (though I'm not sure how maintained this is). The only annoying part is that you're pl

Postgres would probably be even easier, since it already has the framework to do this (foreign data wrappers) and a nice API to write said wrappers in python[1] - 200 lines to use your SQL client as an email reader[2], I imagine that wrapping /proc would be considerably less :) [1] http://multicorn.org/foreign-data-wrappers/ [2] https://github.com/Kozea/Multicorn/blob/master/python/multic...

I was at OSCON Data this year that held a talk about the new foreign data connectors piece, it's great that Postgres has them now. Want twitter data? Make an interface to translate twitter data to rows, it's that easy! LucidDB has had foreign data connectors for a while but you have to use Java at least for part of it. I like the generator-style of yielding list-rows shown here: https://github.com/Kozea/Multicorn/blob/master/python/multic...

Thank you for your interest! The postgresql plan is parsed, and passed as a list of "quals", objects representing simple filters. As an implementer, you don't HAVE to enforce those, since postgresql will recheck them for you anyway, but they can be quite handy. For example, you can look at the imap foreign data wrapper (https://github.com/Kozea/Multicorn/blob/master/python/multic...) to see how the conditions from postgresql are converted to an IMAP filter, allowing for server side filtering. The required columns are also provided, so if you don't need the email payload, the foreign data wrapper will not fetch it. For joins, it will depend on the execution plan. There is still plenty of work on parsing the postgresql execution plan into something more useful, but the current set of "optimizations" is sufficient for our main use cases.