flatbuffers

irst parse that array (usually accumulating a copy of the elements in a Vec<Object> or similar, which in turn means that the whole array was effectively copied), rather than providing a "cursor" into the original buffer. Parsing JSON requires traversing the entire buffer and copying everything you come across. Protocol buffers works much the same way: because structures are variable length (and in fact, scalars are too -- integers are stored as base 128 varints), if you want the Nth element of an array, you have no choice but to parse all the proceeding elements (rather than nudging a cursor's offset by N×sizeof(Elem) -- you can't do that because the size of any element is unknown until after you've parsed it). Because you want O(1) indexing after parsing, the logical implication is that whenever you parse a protobuf message, the protobuf library parses (and copies) the entire thing, and any array/repeated field ends up as a new array allocation in your language (e.g. Vec<Elem>). Contrast with something like flatbuffers or Cap'n Proto: the code generated from your schema file gives you structures that (more or less) have two fields: a reference to a buffer, and an offset into that buffer. When you do something like person.age(), the offset of the age field (which is constant) is added to the offset of this person record, and that combined offset if used to index into the buffer (something like buffer.read_u32(offset)). Using a library like this gives you performance similar to what you'd have dereferencing array indices and struct fields in plain old data types in your language of choice. You don't pay in memory and processor time parsing everything up front, you only pay for the scalar memory reads you actually use (and a little bit of quasi-pointer chasing, not unlike what would happen with native structs). Put another way: a zero copy (de)serialization protocol is one where the on-disk format is the same as the (readily usable) in-memory format. This rules out things like string escaping (just store the original bytes), variable sized records/integers, variable length arrays stored inline with records (otherwise that would make records themselves variable length), storing pointers in records (because those pointers will be meaningless when read from disk; instead: store offsets), etc. You can read more about zero copy as it relates to Rust here: https://manishearth.github.io/blog/2022/08/03/zero-copy-1-no... Here's the Wikipedia article: https://en.wikipedia.org/wiki/Zero-copy Examples of zero copy (de)serialization libraries: https://github.com/rkyv/rkyv https://github.com/google/flatbuffers https://github.com/capnproto/capnproto

r consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

at all - so maybe you'd either skip this step or put in the effort to parse the schema yourself? [0]: https://github.com/bufbuild/connect-go/blob/main/codec.go#L5... [1]: https://github.com/bufbuild/connect-go/blob/main/internal/ge... Also, thank you for making me stop and think about this in detail. I just pulled Flatbuffers out of a hat when I was writing that part of the docs and assumed that `flatc` supported plugins like `protoc`. Turns out that when you assume... I'm updating that portion of the documentation now :)

(Java, V8, and C# are particular offenders), so oftentimes the speed benefits are lost in translation. But people have certainly been thinking about this issue. [1] https://capnproto.org/ [2] https://github.com/google/flatbuffers