kine

It's possible they are using kine: https://github.com/k3s-io/kine

For what it's worth, Kine (software that k3s uses to replace etcd with SQL databases) implements etcd watches on SQLite through polling[1]. The reason being that SQLite does not offer NOTIFY/LISTEN like MySQL and Postgres do. Ironically, Honkey attempts implementing NOTIFY/LISTEN through polling. k3s has been running on my home server for about three years now (using the default SQLite backend), and there doesn't seem to be excessive CPU usage despite dozens of watches existing in the simulated etcd. Of course, this doesn't say much about Honker, but it's nonetheless worth pointing out that sometimes the choice of database forces one towards a certain design. [1] https://github.com/k3s-io/kine/blob/648a2daa/pkg/logstructur...

I am extremely Not A Database Person but I understand that the rationale for Kubernetes adopting etcd as its preferred data store was more about its distributed consistency features and less about query throughput. etcd is slower cause it's doing RAFT things and flushing stuff to disk. Projects like kine allow K8s users to swap sqlite or postgres in place of etcd which (I assume, please correct me otherwise) would deliver better throughput since those backends don't need to perform consenus operations. https://github.com/k3s-io/kine

See https://github.com/k3s-io/kine, k3s uses this to shim etcd to MySQL, Postgres and sqlite

The API server is the thing. It so happens that the API server can mostly be a thin shell over etcd. But etcd itself while so common is not sacrosanct. https://github.com/k3s-io/kine is a reasonably adequate substitute for etcd. sqlite, MySQL, PostgreSQL can also be substituted in. Etcd is from the ground up built to be more scale-out reliable, and that rocks to have baked in. But given how easy it is to substitute etcd out, I feel like we are at least a little off if we're trying to say "etcd is also the entire point of k8s" (the APIserver is)

1. The built-in types are already protos. Imo gRPC wouldn't be a good fit - actually will make the system harder to use. 2. Already can be achieved today via kine[0] 3. Couldn't you build this today via regular CNI? Cilium NetworkPolicies and others basically do this already 4,5 probably don't require 2.0 - can be easily added within existing API via KEP (cri-o already does userns configuration based on annotations) [0] - https://github.com/k3s-io/kine

Kine (https://github.com/k3s-io/kine) is a shim or an external process (when not k3s) that translates the etcd api to enable compatibility with a database or alternative data store. Kubernetes natively talks etcd, so this translation is what enables its usage with sqlite or another database, but it incurs an overhead. I don't have specific numbers unfortunately since it was years ago I benchmarked Kine against etcd. But I had a better results with etcd both in cluster and single node. I happened to stumble upon this paper that echos my experience. https://programming-group.com/assets/pdf/papers/2023_Lightwe... Particularly, the high controller cpu usage (even for an empty cluster), and higher latencies.

Looking at [1], is there any reason why this couldn’t run in something like DynamoDB? I feel like the operations required would fit https://github.com/k3s-io/kine/blob/master/pkg/drivers/gener...

Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0]. > I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far. As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer). One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from

I could spent >0s looking into this, but I've been curious how kline implements Etcd's Watch API[1]. For the matter, I don't have much idea in general how kline handles data. Kline would be a great candidate for a blog post or introductory technical article on. I do quickly see one issue- evidently watches presently use polling. The issue requests use of postgres triggers[2]. [1] https://etcd.io/docs/v3.2/learning/api/#watch-api [2] https://github.com/k3s-io/kine/issues/20

https://github.com/k3s-io/kine#readme but it's "caveat emptor" since to the best of my knowledge no one runs conformance tests on top of it

from this https://github.com/k3s-io/kine it seems like k3s was born out of a rancher labs project called 'kine' that does what you're describing?

yes looks to be polling https://github.com/k3s-io/kine/blob/master/pkg/server/watch....

Not to be mistaken for high-availability Dqlite[1], which is one of the options one can run the k3s kubernetes distribution on (instead of etcd), via the Kine etcd shim[2]. Ultimately though the K3s team replaced Dqlite with an embedded etcd to get high-availability[3]. [1] https://dqlite.io/ [2] https://github.com/k3s-io/kine [3] https://rancher.com/docs/k3s/latest/en/installation/ha-embed...