ingestr

n. I understand it is sometimes required due to organizational complexity or legacy database reasons, mine is just a personal preference. If anyone is looking for an open-source CDC tool that runs as a standalone Go CLI, check out ingestr: https://github.com/bruin-data/ingestr

lso due to legacy reasons. Think usecases where the columns are updated without the timestamp being updated. - For hard deletes. Even though I do believe there are ways to solve each of these, it ended up putting us in a disadvantage, and we decided to build the CDC connectors instead. ingestr CDC works in two modes now: batch (bad name, I know) and stream. The batch mode is reading the changelog entries from the last load until the starting timestamp, and the streaming mode keeps reading and landing them to the destination databases. ingestr has a few advantages compared to a more traditional debezium + kafka setup: - it's a standalone Go binary and does not require any additional infra. - it has very low resource consumption, ~100MB baseline. - it can run on your own computer during development, and can be converted into a streaming prod deployment when it is ready. - supports 20+ destinations already, primarily analytical platforms like snowflake, databricks, generic iceberg destinations, etc. i would love to hear any feedback on what we could do to make it easier for cdc workloads! https://github.com/bruin-data/ingestr

s of a CLI application written in Golang, and a VS Code extension that supports it with a local UI. Bruin supports quite a few stuff: - Data ingestion using ingestr (https://github.com/bruin-data/ingestr) - Data transformation in SQL & Python, similar to dbt - Python env management using uv - Built-in data quality checks - Secrets management - Query validation & SQL parsing - Built-in templates for common scenarios, e.g. Shopify, Notion, Gorgias, BigQuery, etc This means that you can write end-to-end pipelines within the same framework and get it running with a single command. You can run it on your own computer, on GitHub Actions, or in an EC2 instance somewhere. Using the templates, you can also have ready-to-go pipelines with modeled data for your data warehouse in seconds. It includes an open-source VS Code extension as well, which allows working with the data pipelines locally, in a more visual way. The resulting changes are all in code, which means everything is version-controlled regardless, it just adds a nice layer. Bruin can run SQL, Python, and data ingestion workflows, as well as quality checks. For Python stuff, we use the awesome (and it really is awesome!) uv under the hood, install dependencies in an isolated environment, and install and manage the Python versions locally, all in a cross-platform way. Then in order to manage data uploads to the data warehouse, it uses dlt under the hood to upload the data to the destination. It also uses Arrow’s memory-mapped files to easily access the data between the processes before uploading them to the destination. We went with Golang because of its speed and strong concurrency primitives, but more importantly, I knew Go better than the other languages available to me and I enjoy writing Go, so there’s also that. We had a small pool of beta testers for quite some time and I am really excited to launch Bruin CLI to the rest of the world and get feedback from you all. I know it is not often to build data tooling in Go but I believe we found ourselves in a nice spot in terms of features, speed, and stability. https://github.com/bruin-data/bruin I’d love to hear your feedback and learn more about how we can make data pipelines easier and better to work with, looking forward to your thoughts! Best, Burak

to monitor, how to schedule, etc. I ended up figuring out solutions for each of these matters, but the process always felt suboptimal. I like coding but for more novel stuff rather than trying to copy a table from Postgres to BigQuery. There are libraries like dlt (awesome lib btw, and awesome folks!) but that still required me to write, deploy, and maintain the code. Then there are solutions like Fivetran or Airbyte, where there's a UI and everything is managed through there. While it is nice that I didn't have to write code for copying the data, I still had to either pay some unknown/hard-to-predict amount of money to these vendors or host Airbyte myself which is roughly back to square zero (for me, since I want to maintain the least amount of tech myself). Nothing was versioned, people were changing things in the UI and breaking the connectors, and what worked yesterday didn't work today. I had a bit of spare time a couple of weeks ago and I wanted to take a stab at the problem. I have been thinking of standardizing the process for quite some time already, and dlt had some abstractions that allowed me to quickly prototype a CLI that copies data from one place to another. I made a few decisions (that I hope I won't regret in the future): - everything is a URI: every source and every destination is represented as a URI - there can be only one thing copied at a time: it'll copy only a single table within a single command, not a full database with an unknown amount of tables - incremental loading is a must, but doesn't have to be super flexible: I decided to support full-refresh, append-only, merge, and delete+insert incremental strategies, because I believe this covers 90% of the use-cases out there. - it is CLI-only, and can be configured with flags & env variables so that it can be automated quickly, e.g. drop it into GitHub Actions and run it daily. The result ended up being `ingestr` (https://github.com/bruin-data/ingestr). I am pretty happy with how the first version turned out, and I plan to add support for more sources & destinations. ingestr is built to be flexible with various source and destination combinations, and I plan to introduce more non-DB sources such as Notion, GSheets, and custom APIs that return JSON (which I am not sure how exactly I'll do but open to suggestions!). To be perfectly clear: I don't think ingestr covers 100% of data ingestion/copying needs out there, and it doesn't aim that. My goal with it is to cover most scenarios with a decent set of trade-offs so that common scenarios can be solved easily without having to write code or manage infra. There will be more complex needs that require engineering effort by others, and that's fine. I'd love to hear your feedback on how can ingestr help data copying needs better, looking forward to hearing your thoughts! Best, Burak