connect

Check out redpanda connect / warpstream bento (depending on your license needs). Both came out of what was benthos. https://github.com/redpanda-data/connect https://github.com/warpstreamlabs/bento

Its a rust port of Redpanda Connect (benthos), but with less connectors https://github.com/redpanda-data/connect

lots of Go based CDC stuff going on these days. Redpanda Connect (formerly benthos) recently added support for Postgres CDC [1] and MySQL is coming soon too [2]. 1: https://github.com/redpanda-data/connect/pull/2917 2: https://github.com/redpanda-data/connect/pull/3014

Hi, Burak, it looks interesting. I was wondering, do you know about connect? Maybe you can take advantage of some of its ready-made components. In addition, it is also developed using Go - https://docs.redpanda.com/redpanda-connect/home/ - https://github.com/redpanda-data/connect

The text "Licensed as a Redpanda Enterprise file under the Redpanda Community" appears in the two RCL-licensed connectors as listed on the web site: github.com/redpanda-data/connect/v4@v4.28.0/internal/impl/snowflake/output_snowflake_put.go github.com/redpanda-data/connect/v4@v4.28.0/internal/impl/splunk/template_output.yaml But also in an apparently unrelated file (Kafka seems to fall under Apache 2 from the website): github.com/redpanda-data/connect/v4@v4.28.0/internal/impl/kafka/topic_logger.go Now I am a bit puzzled. What's up with this? I am furiously rewriting my way out of Benthos but I would like to keep the FreeBSD port in shape :D

It feels pretty uncharitable for Redpanda to enforce their terms when they haven't done anything of value with it yet. They made a bold claim that you'll have to pay them to use these features, but you certainly don't as they're still available under MIT licensing. One does not simply buy Open Source Software. Until Redpanda actually makes any code changes, the ~three now-proprietary plugins are still available as Open Source Software: just browse to the commit before they slapped their license at the top. These are all MIT and bit-for-bit identical to the now-proprietary plugins: - Splunk HEC: https://github.com/redpanda-data/connect/blob/e653dc3f8a6eee... - Snowflake: https://github.com/redpanda-data/connect/blob/e653dc3f8a6eee... - Kafka topic logger: https://github.com/redpanda-data/connect/blob/e653dc3f8a6eee...

As a freelance engineer who's a long time Benthos contributor and who volunteered a lot of community support for this project in the past several years, I don't think it makes sense to fork it and I'm perfectly happy with the current approach where the core engine (https://github.com/redpanda-data/benthos) is MIT licensed as @jeffail mentioned and 3rd party plugins can live in other people's repos and have various licenses, one example being https://github.com/redpanda-data/connect. I'm 100% committed to keep contributing to Benthos as long as it remains free and open source and I'm also happy to continue offering community support to whomever requests it on the official channels on Discord, Slack, GitHub etc.

Hey, the HTTP output has a fixed number of retries, after which you could either have some mechanism in place to fall back on or by default it will simply continue the retries again whilst blocking upstream. You might also be interested in running Benthos in streams mode: https://github.com/Jeffail/benthos/tree/master/docs/streams Streams mode lets you run as many isolated stream pipelines as you want in the same process, which in your case could be a simple queue -> webhook bridge. You can manage these pipelines either statically in config files, or dynamically through a REST API.