redpanda

Porting LevelDB[1] to Seastar[2], for internal metadata storage in Redpanda[3]. Before you ask why can’t something off the shelf be used, seastar has unique constraints around its runtime and its memory allocator that means we can’t reuse an existing library. 1: https://github.com/google/leveldb 2: https://github.com/scylladb/seastar 3: https://github.com/redpanda-data/redpanda/pull/28351

This sounded interesting to me, and it looks like the plan is to make Redpanda open-source at some point in the future, but there's no timeline: https://github.com/redpanda-data/redpanda/tree/dev/licenses

Yeah I built a custom sysroot for Redpanda (Bazel/C++/Distributed Kafka) using a really simple docker image: https://github.com/redpanda-data/redpanda/blob/dev/bazel/too... We also have an dockerfile for clang/LLVM in that repo so the whole thing is hermetic. It’s a bit of shame Bazel doesn’t come with stronger options/defaults here, because I feel like I want to reproduce this same toolchain on every C++ project with Bazel

Yeah it’s just the stack switching itself that is a handful of cycles, but there is not much more overhead for the full VM switch if you structure your embedding the right way. Code the code is source available if you want to peek at it! https://github.com/redpanda-data/redpanda/blob/dev/src/v/was...

that's a littlebit of a stretch. when you say "no shortage" - outside of redpanda what product exists that actually compete in all deployment modes? it's a misconception that redpanda is simply a better kafka. the way to think about it is that is a new storage engine, from scratch, that speaks the kafka protocol. similar to all of the pgsql companies in a different space, i.e.: big table pgsql support is not a better postgres, fundamentally different tech. you can read the src and design here: https://github.com/redpanda-data/redpanda. or an electric car is not the same as a combustion engine, but only similar in that they are cars that take you from point a to point b.

Hmm? I'm pretty sure it's written in C++. See also, their install dependencies script. https://github.com/redpanda-data/redpanda/blob/dev/install-d...

we use it for redpanda https://github.com/redpanda-data/redpanda - other langs are go for k8s integration, python (dev prod tooling), ... js for ui that's mostly it.

While still meeting most of the OSI criteria for being open source. It basically says "you can use the features covered by this license for most uses, except creating your own SaaS offering of this product", plus some of the typical boilerplate about ensuring the license travels with the code. It's basically a license that was created in response to how Amazon has been operating, co-opting open source technologies (whose business models focused on creating SaaS offerings of their open source tech) and creating their own managed services, directly competing with the creators (Redis even called this out a few years ago when they changed their license, see https://techcrunch.com/2019/02/21/redis-labs-changes-its-ope...). They also indicate that at some point the license will change to Apache 2 ( https://github.com/redpanda-data/redpanda/tree

https://github.com/redpanda-data/redpanda You're welcome to use it yourself and not pay us a dime. We're a business so we need to monetize but our core is to build a better Kafka. The nice thing is that if you've already invested in building against Kafka but you're exhausted by running Kafka, ours works better and faster. (Again, I'm biased so don't trust me, verify.)

a small note on redpanda. Is not a `rewrite`. It is an entire new storage engine, different consensus protocol, different on-disk format, different internal RPC, no external dependencies, ...., in fact when i started it it was using flatbuffers as the API. What we did later to make money is add a kafka-API translation layer, but in terms of code size, is a very small part. The way i'd recommend thinking about it is a new storage engine that is compatible w/ kafka clients. https://github.com/redpanda-data/redpanda/tree/dev/src/v/kaf... - this is basically all the kafka-related code, the rest is net-new design.

this is cool. i wanted to try this w/ redpanda (i'm a maintainer https://github.com/redpanda-data/redpanda), but I didn't see it using testcontainers or docker or anything like that.

Your dream has arrived! Redpanda [1] is a non-Java, ZooKeeper-free streaming platform. It's Kafka compatible, so you can use any Kafka client, of which there are many. Finally, and perhaps most importantly, it is substantially faster than Kafka. Disclosure: I work there. --- [1] https://github.com/vectorizedio/redpanda

for the record, you can do compaction with redpanda - all the same compaction strategies you know w/ kafka work w/ redpanda. https://github.com/vectorizedio/redpanda/tree/dev/src/v/storage - see all 'compaction_*' for deets