HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
redpanda-data
redpanda
C++2020–2025
redpanda-data / redpanda · Archived

redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

Systems & InfrastructureDeveloper ToolsData & DatabasesLanguages & CompilersDevOps & Cloud
Explore on GitHubRead Hacker News discussions
14Distinct people
26Mentions
0Explicit recommenders
12.7KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 2:04:58 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year26 dated mentions

Observed mentions: 2020-11-12 22:41:15 UTC to 2025-11-09 22:11:38 UTC.

024682020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 6 mentions; 0 submissions, 6 comments20202021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 7 mentions; 0 submissions, 7 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 8 mentions; 2 submissions, 6 comments20222023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC606
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC707
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC826
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC202
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC303

What Hacker News says

6 sources
agallego·Dec 13, 2020
The write is a 2 stage function which works quite well for the kafka layer since it's all batch. Let me explain. 1. First, we have a core-local shared chunk cache w/ DMA aligned buffers. https://github.com/vectorizedio/redpanda/blob/dev/src/v/stor... 2. Second, we eagerly dispatch IO blocks w/ manual accounting of which offset on the DMA section it is at https://github.com/vectorizedio/redpanda/blob/dev/src/v/stor... 3. We adaptively fallocate data to prevent metadata contention on the file handle itself. 4. We issue an fdatasync() - since we are interested in the data being safe (data corruption w/ checksums, etc it's too long to type, but i can expand on a blog post if interested) 5. so imagine a big write (for simplicity) say 1MB. This will get broken up into 16-128KB DMA writes. The last step is an fdatasync for acks=-1 (raft needs t
Read the rest
his) There are nuances between each line, but hopefully this gives you a hint on how it's done.
In Thread-Per-Core Buffer Management for a modern storage system
agallego·Dec 12, 2020
Oh shoot! thank you... fixing the link give me 5 mins. So currently the partition allocator - https://github.com/vectorizedio/redpanda/blob/dev/src/v/clus... - is primitive. But we have a working-not-yet-exposed HTTP admin api on the controller that allows for Out Of Band placement. so the mechanics are there, but not yet integrated w/ the partition allocator. Thinking that we integrate w/ k8s more deeply next year. The thinking at least is that at install we generate some machine labels say in /etc/redpanda/labels.json or smth like that and then the partition allocator can take simple constraints. I worked on a few schedulers for www.concord.io with Fenzo on top of mesos 6 years ago and this worked nicely for both 'affinity', 'soft affinity' and anti-affinity constraints. Do you have any thoughts on how you'd like this exposed?
In Thread-Per-Core Buffer Management for a modern storage system
agallego·Dec 12, 2020
Indeed. I think you have different saturation points the wider the use cases you hit. One example w/ a single-core (which btw, agreed whole heartedly for io) is checksumming + decoding. For kafka, we have multiple indexes - a time index and an offset index which are simple metadata. the trouble becomes on how you handle decompression+checksumming+compression for supporting compacted topics. ( https://github.com/vectorizedio/redpanda/blob/dev/src/v/stor... ) So single core starts to get saturated while doing both fore-ground and background requests. ..... Now assume that you handle that with correct priorities for IO and CPU scheduling.... the next bottleneck will be keeping up w/ background tasks. So then you start to add more threads. but as you mentioned and what I tried to highligiht in that article was that the cost of implicit or si
Read the rest
mple synchronization is very expensive (as noted by you intuition) The thread-per-core buffer management with defer destructors is really handy at doing 3 things explicitly 1. your cross core communication is explicit - that is you give it shares as part of a quota so that you understand how your system priorities are working across the system for any kind of workload. This is helpful to prioritize foreground and background work. 2. there is effectively a const memory addresses once you parse it - so you treat it is largely immutable and you can add hooks (say crash if modified on a remote core) 3. makes memory accounting fast. i.e.: instead of pushing a global barrier for the allocator you simply send a message back to the originating core for allocator accounting. This becomes hugely important as you start to increase the number of cores.
In Thread-Per-Core Buffer Management for a modern storage system

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.