splitgraph

sgr

Python2020–2023

splitgraph / sgr

sgr

sgr (command line client for Splitgraph) and the splitgraph Python library

Explore on GitHub Read Hacker News discussions

2Distinct people

11Mentions

0Explicit recommenders

325GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

sgr (command line client for Splitgraph) and the splitgraph Python library

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 1:03:20 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year11 dated mentions

Observed mentions: 2020-06-24 20:17:49 UTC to 2023-02-01 21:47:46 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	5	5
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	3	3
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	2	2
2023	2023-01-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	1	1

What Hacker News says

3 sources

chatmasta·Nov 12, 2021

We are building a solution for this problem at Splitgraph [0] – it sounds like we could probably help with your use case, assuming this is for analytical (OLAP) data. You can get it to work yourself with our open source code [1], but our (private beta, upcoming public) SaaS will put all your schemas on a more scalable “data delivery network,” which incidentally, happens to be implemented with PgBouncer + rewriting + ephemeral instances. We also have private repositories / fine-grained ACL shipping shortly. In a local engine (just a Postgres DB managed by Splitgraph client to add extra stuff), there is no PgBouncer or batteries-included authN/Z, but we use Foreign Data Wrappers to accomplish the same query resolving. Our goal is for you to have the ability to do everything locally as an individual, with the SaaS becoming useful for “multi

Read the rest

player” (teams and orgs). On Splitgraph, every dataset – and every version of every dataset – has an address. Think of it like tagged Docker images. The address either points to an immutable “data image” (in which case we can optionally download objects required to resolve a query on-the-fly, although loading up-front is possible too) or to a live data source (in which case we proxy directly to it via FDW translation). This simple idea of _addressable data products_ goes a long way – for example, it means that computing a diff is now as simple as joining across two tables (one with the previous version, one with the new). Please excuse the Frankenstein marketing site – we’re in the midst of redesign / rework of info architecture while we build out our SaaS product. Feel free to reach out if you’ve got questions. And if you have a business case, we have spots available in our private pilot. My email is in my profile – mention HN :) [0] https://www.splitgraph.com/connect [1] examples: https://github.com/splitgraph/splitgraph/tree/master/example...

In Scalable PostgreSQL Connection Pooler

chatmasta·Jun 30, 2021

Hey! Socrata is awesome :) The basic idea here is that you can write a Splitgraph plugin for any type of data source [0], so we like to use Socrata as a convenient example. Each plugin includes a Postgres Foreign Data Wrapper (FDW) to translate from SQL to whatever upstream query language (e.g. Socrata FDW [1]). By default we don't copy any data into Splitgraph (although we do cache query results keyed by AST). When a query arrives, we parse it for any table identifiers, and "mount" a temporary foreign table for each, using the appropriate FDW. Foreign tables are mostly transparent to the Postgres query planner, so a JOIN will work (example [2]), and an FDW can even optionally implement aggregate pushdown. But yes, especially in the case of cross-dataset JOINS, there are some pathological cases. When your use case outgrows live querying (

Read the rest

e.g. due to pathological JOIN queries), you can ingest any data source into a Splitgraph image, a versioned snapshot of data similar to a Docker image. [3] Our focus right now is on the internal enterprise use case. We use the Socrata plugin as a demo, but our customers are more interested in plugins for data sources they use internally, e.g. Postgres, Snowflake, Google Search Console, CSV in S3, etc. We support all those too; you can mount them all locally, and you can also mount them on Splitgraph.com (or an internal deployment), but still thin docs / no web UI yet. [0] https://www.splitgraph.com/blog/foreign-data-wrappers [1] https://github.com/splitgraph/splitgraph/blob/master/splitgr... [2] https://www.splitgraph.com/workspace/ddn?layout=hsplit&query... [3] https://www.splitgraph.com/docs/concepts/images

In Globally Distributed Postgres

chatmasta·May 7, 2021

On the public DDN (data.splitgraph.com:5432), we enforce a (currently arbitrary) 10k row limit on responses. You can construct multiple queries using LIMIT and OFFSET, or you can run a local Splitgraph engine without a limit. We also have a private beta program if you want a managed or self-hosted cloud deployment with the full catalog and DDN features. And we are planning to ship some "export to..." type workflows for exporting to CSV and potentially other formats. For live/external data, we proxy the query to the data source, so there is no theoretical data size limit except for any defined by the upstream. For snapshotted data, we store the data as fragments in object storage. Any size limit depends on the machine where Splitgraph's Postgres engine is running, and how you choose to materialize the data when downloading it from object s

Read the rest

torage. You can "check out" an entire image to materialize it locally, at which point it will be like any other Postgres schema. Or you can use "layered querying" which will return a result set while only materializing the fragments necessary to answer the query. Regarding ClickHouse, you could watch this presentation [0] my co-founder Artjoms gave at a recent ClickHouse meet-up on the topic of your question. We also have specific documentation for using the ClickHouse ODBC client with the DDN [1], as well as an example reference implementation. [2] [0] https://www.youtube.com/watch?v=44CDs7hJTho [1] https://www.splitgraph.com/connect [2] https://github.com/splitgraph/splitgraph/tree/master/example...

In Google Dataset Search