kafka

I have been doing some reading about Kafka lately so decided to check the repo and it looks like a great reference: https://codewiki.google/github.com/apache/kafka

I still use it. They finally fixed my biggest complaint about it a year ago, which is that you couldn't use vanilla Clojure lambdas for the Java functional interface, and so you'd have to reify that interface and it was bulky and ugly. Now it works fine so long as the interfaces actually have the @FunctionalInterface attribute. Not every project uses @FunctionalInterface, but I've been trying to add it to places [1] [2] [3], and now I'm able to use Clojure in a lot more places. [1] https://github.com/LMAX-Exchange/disruptor/pull/492 [2] https://github.com/apache/kafka/pull/19234 [3] https://github.com/apache/kafka/pull/19366

Grok says you are right (https://grok.com/share/c2hhcmQtMg%3D%3D_ddbace62-c299-4b7b-9...) however... https://github.com/iluwatar/java-design-patterns https://github.com/spring-projects/spring-framework https://github.com/apache/kafka https://github.com/neo4j https://github.com/inforkgodara/store-poshttps://github.com/...

I’ve open-sourced Timestone, a small Java library that lets you abstract time sources for testable, deterministic time-based logic. It provides an interfaces `Time` with two implementations `SystemTime` for real system clock, and `MutableTime` that can be advanced programmatically. It lets you control time precisely, without relying on mocking frameworks. What prompted me to build this is that many large projects (like Apache Kafka and Apache Flink) implement their own internal solutions for time control: - in Kafka: [org.apache.kafka.common.utils.Time](https://github.com/apache/kafka/blob/trunk/clients/src/main/...) - in Flink: [org.apache.flink.util.clock.Clock](https://github.com/apache/flink/blob/master/flink-core/src/m...) I saw a pattern of duplicated fake clock logic across open source and internal codebases, and decided to make a

Another fun fact: this isn't the only software named after Trogdor: Apache Kafka's test framework is also called Trogdor[0]. -- [0] https://github.com/apache/kafka/tree/trunk/trogdor

One clarification: > A data streaming platform (of which Kafka is one example) will push the data The Kafka broker keeps hot data in RAM. The clients pulls: https://github.com/apache/kafka/blob/3.3.2/clients/src/main/.... Most often it’s a frequent pull with a short timeout. This is one of the major differences from MQTT. MQTT broker pushes to the subscriber when the latter is connected, or stores for redelivery if subscriber is offline (qos 1 and 2) but the redelivery will also be a real push on reconnect.

Thanks for putting things in perspective, EdwardDiego. > The fact that MM2 happened, and Confluent didn't try to stop it, despite it being awfully similar to Replicator, makes me think that Confluent are acting in good faith. Let me share an anecdote related to this example. We (Confluent) were actually the ones who contributed the documentation for MirrorMaker v2 to the Apache Kafka docs (https://kafka.apache.org/documentation/#georeplication). The development lead on MM2 was (an engineer at) Cloudera, yet they never spent the time to provide user-facing documentation to the Kafka project. I don't want to speculate about reasons, yet I noticed that MM2 was documented in the Cloudera docs. If we didn't care for the Kafka community at Confluent, we would not have spent our own resources and time to fill that gap, given that we have a propr

Is there a compiled package for 2.8.0-rc0 available? I only see sources on Github https://github.com/apache/kafka/releases/tag/2.8.0-rc0 and couldn't find anything regarding 2.8.0-rc0 on the Kafka downloads page https://kafka.apache.org/downloads.

True, but for all intents and purposes, it's basically public domain with some attribution clauses and litigation indemnity: "Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form." https://github.com/apache/kafka/blob/trunk/LICENSE#L67-L72

Well parts of it yes but not 100%: https://github.com/apache/kafka/tree/trunk/clients/src/main/...

I should also mention that system/integration tests of the new functionality was a requirement for releasing it. For example, here's one of the tests https://github.com/apache/kafka/blob/trunk/tests/kafkatest/t...

It's not Jepsen, but we actually do a fair amount of system and integration testing, some of which does things like kill nodes (randomly, the controller, etc) and validates data is delivered correctly. There is some ongoing work to add other fault injection tests: https://issues.apache.org/jira/browse/KAFKA-5476 This has already caught quite a few bugs, including one that led to a change to the replication protocol which is included in this 0.11.0.0 release. See the community KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+... One cool thing that happened recently with these tests is that they were modified to make the client implementation pluggable: https://github.com/apache/kafka/pull/2048 Confluent uses this functionality to test all of its clients (librdkafka, confluent-kafka-python, confluent-kafka-go, confluen

It is a mixture of Java and Scala, though mostly Scala. GitHub mirror of the source: https://github.com/apache/kafka

This is becoming more and more common for Scala projects that want to build libs in Scala and only export a Java interop API (as opposed to pure Java at the bottom and building Scala on top). E.g. https://github.com/apache/kafka/tree/0.8/core/src/main/scala...