kiba

Hell yes :-) For the record, I am a deep Sequel fan, and wrote ETL connectors that use it: https://www.kiba-etl.org/kiba-pro https://github.com/thbar/kiba/wiki#kiba-pro (full doc with snippets) It is one of the most underrated gems of the Ruby universe IMO!

I started https://github.com/thbar/kiba#kiba-etl to scratch my own itch & be able to write properly structured ETL jobs in Ruby. It was a blank-slate rewrite of something larger (activewarehouse-etl) which I could not maintain anymore. This landed me not strictly a job, but long term consulting gigs with a number of companies in EU, UK & US. The job was directly related to the project: companies wanted the expertise of data engineering & ETL, often with Kiba directly, but also in general. This "side project" was totally worth it :-)

Definitely! As a matter of fact, this is the default data structure I use when writing Ruby ETL code (e.g. https://github.com/thbar/kiba/wiki). Methods like "except" (https://docs.ruby-lang.org/en/3.2/Hash.html#method-i-except) or "fetch" (raising an error on missing key) are very convenient to write defensive data processing code! Similarly, in Elixir, I use Maps a lot for the same type of jobs (https://hexdocs.pm/elixir/1.15.4/Map.html), with similar properties.

in ruby you have the async gems from https://github.com/socketry/async i dont know for data manipulation but i guess also ruby has it. maybe this https://github.com/thbar/kiba

Happy to officially release Kiba Pro v2 (commercial extensions to Kiba ETL, a popular open-source Ruby framework to build ETL & data pipelines). This is my effort to build a sustainable (open core) "data pipeline" solution for Ruby. Feedback is most welcome! Direct links: - https://www.kiba-etl.org -> main OSS home - https://github.com/thbar/kiba/wiki -> documentation (OSS + Pro) - https://www.kiba-etl.org/kiba-pro -> Pro short description - https://thibautbarrere.com -> my blog (with articles on data processing with Kiba)

(I’m the author of a popular Ruby ETL framework named Kiba ETL) An « ETL » framework is for me something that will allow you to express and run maintainable data pipelines, as a coder. Kiba is exactly that, in the sense that it does not include built-in sources/transforms/destinations, but rather defines a set of guidelines you can follow to easily implement reusable ETL components with high quality & unit tests. More information in the README: https://github.com/thbar/kiba/blob/master/README.md

https://github.com/thbar/kiba This has been the standard for years

https://github.com/thbar/kiba This is still my workhorse and has never let me down. I do a ton of ETL. https://github.com/thbar/kiba-ex This looks interesting though.

Responding with http://www.kiba-etl.org (a ruby data processing framework): 1) Why did I start it: I started it because Kiba ancestor https://github.com/activewarehouse/activewarehouse-etl was crawling under its own technical debt, and I still wanted a way to express data processing jobs with Ruby, without having too much maintenance work on the ETL framework itself. 2) How did people find it: I wrote a number of blog posts at http://thibautbarrere.com (including one video), and also spent quite a bit of time working on documentation at https://github.com/thbar/kiba/blob/master/README.md. 3) If I recall well, I posted on the ruby subreddit, twitter, but also gave talks at conferences. 4) It's a matter of personal preference IMO! To summarise, as marketing for an OSS project, I'm writing blog posts, detailed documentation, videos, speaking