collector

/discovery/usability a nightmare, and require an agent in the loop to drive/integrate the tools when you really just need to give them the actual APIs and documentation). That ends up getting rather hairy, we ended up actually building a declarative meta-lexer/parser/transpiler (meta basically just meaning it’s generalized across languages and self-hosting/bootstrapped) recently (https://github.com/accretional/gluon) because it turns out building a cross-language distributed type system is rather difficult. But reflection alone gets you halfway there as far as benefits.

ay to scale it. Basically it's a proto ORM based on sqlite that auto configures a bunch of CRUD/search and management endpoints for each "Collection", and via a CollectionRepo additional Collections can be created at runtime. The main thing I invested time into was backups/cloning these sqlite dbs since that's what allows you to scale up and down more flexibly than a typical db and not worry as much about availability or misconfiguration.

ta in the proto wire format. Two major problems: nothing in proto itself enforces that field numbers are assigned sequentially, because there is no single source of truth for the proto schema (you can still have one of your own, but it’s not a “thing” in proto). Also, the whole point of field numbers is that they can be selectively missing/reserved/ignored and allow you to deserialize messages without special handling for version changes in your code at runtime. So, field numbers aren’t a dense, easily enumerable range of numbers, they’re basically tags that can be any number between 1 and 536,870,911 except for the reserved 19,000-19,999. This smells like serious tech debt/ a design flaw that completely closes the door for even fixing this at Google or anywhere else, because it’s arbitrarily in the middle of the range of field numbers and a leaked implementation detail from the internals. You couldn’t build your own dense field number management/sequential enforcement system on top of proto without ripping that part out, but your existing proto usage relies on that part and changing it would break existing clients because you’re removing field numbers, which is the whole fucking point of proto, and makes it difficult to roll out even if you did fix it yourself. So, representing union/enumerable types in proto is impossible. For proto enums to have forward compatibility, they have to handle adding new enum values over time, or need to remove and reserve old ones. So, proto enums end up being basically just field numbers. That’s exactly what you see in Golang enums and I don’t think it’s a coincidence: Google has no good way to serialize/deserialize/operate on enumerable enums or union types anywhere they use proto/grpc. Golang inherits this “enums” implementation from protobuf because it’s the context in which it was created.

tually, just a bit tricky because most LLMs aren't trained to operate this way without very explicit instructions for how to do so, and for collector specifically the API surface is probably too big. But I am pretty sure neither would take long to fix if enough people were adopting this kind of pattern.

ntuition for avoiding this situation going forward. Because I do understand the mental model behind what this is supposed to look like at its core, and I need to maintain some kind of human-friendly guard rails, I'm adding integration tests in a different repo and a README/project "constitution" that claude can't change but is accountable for maintaining, and configuring it to keep them in context while working on my project. Kind of a microcosm of startups' reluctance to institute employee handbook/kpis/PRDs followed by resignation that they might truly be useful coordination tools.

nd of expertise in actually properly providing a simple interface to a complex solution. So why not just create a compact universal representation of that? Because it's not so big a leap to go beyond eating SaaS to eating integrations, migration costs/bad moats, and the marketing/documentation/wrapper.

ncy. For what is worth, here is a benchmark comparing IO performance on a few linux filesystem. Including some sqlite tests. https://www.phoronix.com/review/linux-615-filesystems

s are spending 10-40% of their budget on compute, payroll, and SaaS to host CRUD applications that could probably run on a small number of servers without a "platform" team behind it, just a couple of developers who know what they're doing. Unless they're paying $$$ each of these deployments is running their own control plane and dealing with all the operational and cognitive overhead that entails. Most of those are running in a small number of datacenters alongside a bunch of other people running/managing/operating kubernetes clusters of their own. It's insanely wasteful because if there were a proper multitenant service mesh implementation (what I'm working on) that was easy to use, everybody could share the same control plane ~per datacenter and literally just consume the Kubernetes APIs they actually need, the ones that let them run and orchestrate/provision their application, and forget about all the fucking configuration of their cluster. BTW, that is how Borg works, which Kubernetes was hastily cobbled-together to mimic in order to capitalize on Containers Being So Hot Right Now. The vast majority of these Kubernetes users just want to run their applications, their customers don't know or care that Kubernetes is in the picture at all, and the people writing the checks would LOVE to not be spending so much and money on the same platform engineering problems as every other midsize company on the Internet. > what is the use case or benefit of not having a control plane? All that is to say, it's not having to pay for a bunch of control plane nodes and SaaS and a Kubernetes guy/platform team. At small and medium scales, it's running a bunch of container instances as long as possible without embarking on a 6-24mo, $100k-$10m+ expedition to Do Kubernetes. It's not having to secure some fricking VPC with a million internal components and plugins/SaaS, it's not letting some cloud provider own your soul, and not locking you in to something so expensive you have to hire an entire internal team of Kubernetes-guys to set it up. All the value in the software industry comes from the actual applications people are paying for. So the better you can let people do that without infrastructure getting in the way, the better. Making developers deal with this bullshit (or deciding to have 10-30% of your developers deal with it fulltime) is what gets in the way: https://kubernetes.io/docs/concepts/overview/components/

olutely love sqlite so far - would still use Spanner for some tasks with an infinite budget but once we get Collector to implement partitions I don't think I would ever use Postgres again.