JGjgraettinger1·Nov 11, 2019Submission
Show HN: Gazette: towards unification of batch and stream processing
12 points · Open discussiongazette
core
Go2019–2024
gazette / core
core
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
2Distinct people
6Mentions
0Explicit recommenders
799GitHub stars
What to know before exploring
What it is
Build platforms that flexibly mix SQL, batch, and stream processing paradigms
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 3:38:16 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year6 dated mentions
Observed mentions: 2019-10-02 00:58:44 UTC to 2024-12-22 19:18:52 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 3 | 2 | 1 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
3 sourcesAsk HN: Infinite Byte Streams AaS?
I’m looking for signal from fellow HN’ers on whether this is appealing as a managed service:
Append to a stream:
$ curl -X PUT --data-binary @- https://service/my/stream/name << EOF
{
"some": “arbitrary”,
“data”: [1,2,3]
}
{"payload”: 4}
EOF
Appends are byte-oriented (here, they only happen to be JSON). The API promises that appends don’t interleave (streams are linearizable), and are durable upon a 200 OK response. PUT from cloud functions, lambdas, app engine, etc, or integrate streams as third-party webhook callbacks.
Read from a stream:
$ curl "https://service/my/stream/name?offset=24&block=true"
“data”: [1,2,3]
}
{"payload”: 4}
{“next”: “append”}
Blocking reads receive new content as it commits to the stream. You can think of it as a restart-able firehose (offset=-1 reads from the current stream end).
Read the rest
Permission a stream to partners, customers, or the world.
Content of the stream is batched into regular compressed files that persist to a s3://, etc bucket you own and permission to the service. Historical reads serve directly from your bucket, and the service doesn’t retain PUT content for any longer than is required to durably persist it. Files themselves hold only raw content — if you PUT JSONL, you’ll get JSONL files you can feed to your processing pipelines.
You may want to have multiple streams: annotate & organize them into topics and partitions using Kubernetes-syntax labels and selectors (--selector message-type=MyMessage,environment=qa).
Backstory: I’m describing a managed offering of Gazette’s broker (github.com/gazette/core). The project has a bunch of other neat stuff, but for now we’re exploring what might resonate as a distilled MVP. Thanks!
You might be interested in Gazette (https://github.com/gazette/core). It presents durable, append-only, transactional byte streams ("journals") while delegating all actual storage to S3/GCS/Azure etc. Brokers themselves are fully ephemeral. Written journal content ends up in predictably-named "fragments" in a configured bucket path, where each fragment holds only raw written bytes (eg, if you write JSONL messages, your fragments are also just JSONL).
The broker is pure Go, and Etcd is used for distributed consensus. The project has been around for years (as closed source), but still lacks a bunch of polish as an open-source offering. Working on that...
In Kafka Spawns Open-Source KarelDB