HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
confluentinc
bottledwater-pg
HISTORICAL RECORD2015–2020
confluentinc / bottledwater-pg · Not publicly available

bottledwater-pg

Discover this repository through its Hacker News conversations.

Data & Databases
Explore on GitHubRead Hacker News discussions
12Distinct people
15Mentions
0Explicit recommenders
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A public description has not been verified.

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year15 dated mentions

Observed mentions: 2015-04-27 14:50:22 UTC to 2020-11-16 17:53:45 UTC.

024682015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20152016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 8 mentions; 1 submissions, 7 comments20172018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2020

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC110
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC303
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC817
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC000
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC202
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC101

What Hacker News says

15 sources
marco_craveiro·Dec 21, 2019
Before I proceed, let me just state that I have no particular knowledge of either RDBMS' internals or Kafka, so this is nothing but my own amateur musings on the subject. Now, I wholeheartedly agree that Kafka is more scalable, but I think the key point here is that there is no particular law of nature as to why that is the case. It may just be an historical accident of how RDBMS - and PosgreSQL in particular - have evolved. Further: many of the properties of Kafka are in fact also desirable properties for the PostgreSQL transaction log. My take on the inopinatus observation, together with the Samza article [1] mentioned on this discussion, are as follows. You can think of Postgres as two "products" (bounded contexts, if you like): - a stream-based, possibly replicated, transaction log; - a projection of that transaction log into relation
Read the rest
al calculus, plus all of the associated machinery. Thus far we never had the need to think of these as clearly separate "products", but Kafka makes it obvious that they are. In truth, the amount of tools processing WAL outside of PostgreSQL were already hinting in this direction; Kafka just made it obvious. From this perspective, it seems a tad expensive to take the original transaction log, convert it to a RDBMS representation, then convert it to events and, in some cases, then store it as an event stream in Kafka. It would be much more efficient to simply use the original transaction log directly - and this is why, to me, even Debezium [2] / Bottled Water [3] appear to be one layer too many. To the best of my understanding, this line of reasoning is also line with the observations in the Samza article [1]. Where I believe I differ from the article is in thinking that the RDBMS representation also adds a lot of value to applications - I see both having a role (e.g. streaming vs batch processing sort of thing). I think this would derail the present discussion too much, so I won't go in to it. In conclusion: to the untrained eye, it seems that the right thing to do is to extract the transaction log out of PostgreSQL and make it as scalable as Kafka. Then, allow for it to log "things" which are not necessarily "projectable" into the relational plane. PostgreSQL then becomes just a client of the transaction log, together with other "kinds" of clients. I suspect that this is what will ultimately happen, but the engineering work required will probably span a decade or more. My 2 Angolan Kwanzas, at any rate. [1] https://www.confluent.io/blog/turning-the-database-inside-ou... [2] https://debezium.io/ [3] https://github.com/confluentinc/bottledwater-pg
In When an SQL Database Makes a Great Pub/Sub
marco_craveiro·Dec 20, 2019
Indeed, I've also puzzled over this a fair bit. Its almost as if we are lacking a lower-level interface to the transaction log that enables one to push events without going via the higher-level representation of tables etc. However, the implementation details are somewhat beyond me :-) Postgres' Bottled Water [1] was what made me think about this. I mean, why bother with exporting to Kafka at all, and instead just use the Postgres transaction log directly? [1] https://github.com/confluentinc/bottledwater-pg
In When an SQL Database Makes a Great Pub/Sub
ruslan_talpa·Jul 1, 2017
Short answer no, it doesn't (at least now, release 1 week ago :) ), but dismissing it only for that would be a superficial look. Most cases don't need those guarantees, i.e. you can tolerate a few lost messages. For example, you are implementing real time updates in your app using this. What's the probability of a network glitch happening at the same time as two users being logged in at the same time in one system where an event produced by one needs to be synced to the other, even more, say he lost that event, is it that critical considering he will soon move to another screen and reload the data entirely? From rabbitmq point of view, db&bridge are producers. You are really asking here, does the "producer" guarantee delivery? To do that, it means the producer needs to become himself a "queue" system in case he fails to communicate with
Read the rest
rabbit. Considering we are talking web here, the producers are usually scripts invoked by a http call so there is no such guarantee in any system (when communication to rabbitmq fails). However i think network (in the datacenter) is quite reliable so there is no point in overengineering for that case. If the system can tolerate a few seconds of downtime, it's easy enough to implement a heartbeat system which would restart this tool in case it's needed, also, you can run 2-3 of them to make it redundant then use corelationId to dedup the messages. A more robust tool would be https://github.com/confluentinc/bottledwater-pg but it's for kafka and the major downside for me is that it cant be used with RDS since it's installed as a plugin to postgresql
In What is ‘skip locked’ for in PostgreSQL 9.5? (2016)
steve-chavez·Jun 30, 2017
This tool enables a decoupled architecture. A simple example is sending emails when a user signs up. Instead of having explicit code in your signup function that does the work (and slows down your response), you just have to worry about inserting the row into the database. After this, a database trigger will generate an event which gets sent to RabbitMQ. From there, you can have multiple consumers reacting to that event (email, sms, log, cache). Those consumers tend to be very short, self contained scripts. A more complex example is if you pair pg-amqp-bridge and the Web STOMP plugin for RabbitMQ, you can enable flexible and robust real time updates with almost zero code for your frontend. The larger goal is to enable the development of backends around PostgREST [1] / subZero [2] philosophy. Check out the PostgREST Starter Kit [3] to see
Read the rest
how pg-amqp-bridge fits in a larger project. Similar work (for kafka) is bottledwater-pg [4] which has a nice blog post [5] explaining the benefits of the architecture [1] https://github.com/begriffs/postgrest [2] https://subzero.cloud [3] https://github.com/subzerocloud/postgrest-starter-kit [4] https://github.com/confluentinc/bottledwater-pg [5] https://www.confluent.io/blog/bottled-water-real-time-integr...
In Show HN: pg-amqp-bridge – Send Messages from PostgreSQL to RabbitMQ (rust)
bpicolo·Jan 11, 2017
There are actually a few strong solutions out there for Mysql, most starting with change data capture like: https://github.com/shyiko/mysql-binlog-connector-java (I link that one in particular because he links to alternatives right in his readme!) Pgsql is a bit harder, but if I needed to start somewhere it would be with: https://github.com/debezium/debezium or https://github.com/confluentinc/bottledwater-pg These are the start of pretty sophisticated solutions where you need super real-time elasticsearch indexes and can bring up infra like Kafka. For many applications, queueing an update when something hits your ORM to update, with the hourly/daily refresh is pretty satisfactory.
In Moving persistent data out of Redis
SEJeff·Nov 1, 2016
FYI I've used bottledwater and it can cause segfaults that take down the postgres database hard. I ran into both of these: https://github.com/confluentinc/bottledwater-pg/issues/53 https://github.com/confluentinc/bottledwater-pg/issues/61 They're fixed now, but this is very much beta stuff. It wasn't as magical as we'd hoped so we ended up ripping it out and switching the application to use SSE[1] which streamed into a kafka producer. [1] https://en.wikipedia.org/wiki/Server-sent_events
In Bottled Water – Real-Time Integration of PostgreSQL and Kafka
bladecatcher·Jun 12, 2016
Postgres 9.4+ lets you use the write-ahead log and "replication slots" using which you can create streams of database changes that can be replayed to a client in the order they were made on the origin server [1] EDIT: In case the question was which library did I use, I used BottledWater and it's worked great for me so far [2] [1] https://www.postgresql.org/docs/9.4/static/logicaldecoding-e... [2] https://github.com/confluentinc/bottledwater-pg
In Thoughts on Algolia vs. Solr and Elasticsearch
Cieplak·Feb 15, 2016
What is the best way to ETL data from one Postgres cluster to another (by ETL I mean things like data sanitization and denormalization)? a) I think most people end up doing some kind of batch synchronization, but I'm interested in streaming solutions. b) A lot of folks use trigger based replication, but triggers have to be on the primary/master node, and not just on the replicas. c) Another common solution is to force the database client to write to a message broker, but that opens the door to data discrepancies and synchronization issues. d) In theory I think the best way is to do something like bottledwater-pg or pg_kafka [1] [2] [3], but I'm not sure how battle hardened these are. I think logical replication of the WAL is the right approach, but there is still not much tooling around this. [1] https://github.com/confluentinc/bottledwa
Read the rest
ter-pg [2] https://github.com/xstevens/pg_kafka [3] https://github.com/xstevens/decoderbufs PS: There are a bunch of interesting MySQL solutions out there, such as Zendesk's Maxwell: https://github.com/zendesk/maxwell
In A list of PostgreSQL libraries, tools and resources

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.