HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
gazette
core
Go2019–2024
gazette / core

core

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

Data & Databases
Explore on GitHubRead Hacker News discussions
2Distinct people
6Mentions
0Explicit recommenders
799GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Build platforms that flexibly mix SQL, batch, and stream processing paradigms

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year6 dated mentions

Observed mentions: 2019-10-02 00:58:44 UTC to 2024-12-22 19:18:52 UTC.

01232019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 3 mentions; 2 submissions, 1 comments20192020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC321
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC000
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC101
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC110
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC101

What Hacker News says

6 sources
jgraettinger1·Dec 22, 2024
> To me the best solution seem like combining storing writes on EBS (or even NVMe) initially to minimize the time until writes can be acknowledged, and creating a chunk on S3 standard every second or so. Yep, this is approximately Gazette's architecture (https://github.com/gazette/core). It buys the latency profile of flash storage, with the unbounded storage and durability of S3. An addendum is there's no need to flush to S3 quite that frequently, if readers instead tail ACK'd content from local disk. Another neat thing you can do is hand bulk historical readers pre-signed URLs to files in cloud storage, so those bytes don't need to proxy through brokers.
In Introducing S2
jgraettinger1·Oct 18, 2021
If you're in the Go ecosystem, Gazette [0] offers transactional integrations [1] with remote DB's for stateful processing pipelines, as well as local stores for embedded in-process state management. It also natively stores data as files in cloud storage. Brokers are ephemeral, you don't need to migrate data between them, and you're not constrained by their disk size. Gazette defaults to exactly-once semantics, and has stronger replication guarantees (your R factor is your R factor, period -- no "in sync replicas"). Estuary Flow [2] is building on Gazette as an implementation detail to offer end-to-end integrations with external SaaS & DB's for building real-time dataflows, as a managed service. [0]: https://github.com/gazette/core [1]: https://gazette.readthedocs.io/en/latest/consumers-concepts.... [2]: https://github.com/estuary/flow
In Issues we've encountered while building a Kafka based data processing pipeline
jgraettinger1·Nov 6, 2019Submission

Ask HN: Infinite Byte Streams AaS?

I’m looking for signal from fellow HN’ers on whether this is appealing as a managed service: Append to a stream: $ curl -X PUT --data-binary @- https://service/my/stream/name << EOF { "some": “arbitrary”, “data”: [1,2,3] } {"payload”: 4} EOF Appends are byte-oriented (here, they only happen to be JSON). The API promises that appends don’t interleave (streams are linearizable), and are durable upon a 200 OK response. PUT from cloud functions, lambdas, app engine, etc, or integrate streams as third-party webhook callbacks. Read from a stream: $ curl "https://service/my/stream/name?offset=24&block=true" “data”: [1,2,3] } {"payload”: 4} {“next”: “append”} Blocking reads receive new content as it commits to the stream. You can think of it as a restart-able firehose (offset=-1 reads from the current stream end).
Read the rest
Permission a stream to partners, customers, or the world. Content of the stream is batched into regular compressed files that persist to a s3://, etc bucket you own and permission to the service. Historical reads serve directly from your bucket, and the service doesn’t retain PUT content for any longer than is required to durably persist it. Files themselves hold only raw content — if you PUT JSONL, you’ll get JSONL files you can feed to your processing pipelines. You may want to have multiple streams: annotate & organize them into topics and partitions using Kubernetes-syntax labels and selectors (--selector message-type=MyMessage,environment=qa). Backstory: I’m describing a managed offering of Gazette’s broker (github.com/gazette/core). The project has a bunch of other neat stuff, but for now we’re exploring what might resonate as a distilled MVP. Thanks!
8 points · Open discussion
jgraettinger1·Oct 2, 2019
You might be interested in Gazette (https://github.com/gazette/core). It presents durable, append-only, transactional byte streams ("journals") while delegating all actual storage to S3/GCS/Azure etc. Brokers themselves are fully ephemeral. Written journal content ends up in predictably-named "fragments" in a configured bucket path, where each fragment holds only raw written bytes (eg, if you write JSONL messages, your fragments are also just JSONL). The broker is pure Go, and Etcd is used for distributed consensus. The project has been around for years (as closed source), but still lacks a bunch of polish as an open-source offering. Working on that...
In Kafka Spawns Open-Source KarelDB

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.