As a maintainer of a Python module that uses abi3 now, thank you!
Just one question: any plans to promote APIs like PyUnstable_EnableTryIncRef() and PyUnstable_TryIncRef() into the limited API? So far we have found that these APIs are necessary to implement the weak-valued caches we've always used in the past: https://github.com/protocolbuffers/protobuf/blob/6559a9f9622...
In Python 3.15protocolbuffers
protobuf
C++2014–2026
protocolbuffers / protobuf
protobuf
Protocol Buffers - Google's data interchange format
89Distinct people
124Mentions
0Explicit recommenders
72.2KGitHub stars
What to know before exploring
What it is
Protocol Buffers - Google's data interchange format
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year124 dated mentions
Observed mentions: 2014-12-15 12:38:35 UTC to 2026-10-09 18:07:50 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2014 | 2014-01-01 00:00:00 UTC | 2015-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2015 | 2015-01-01 00:00:00 UTC | 2016-01-01 00:00:00 UTC | 11 | 2 | 9 |
| 2016 | 2016-01-01 00:00:00 UTC | 2017-01-01 00:00:00 UTC | 11 | 3 | 8 |
| 2017 | 2017-01-01 00:00:00 UTC | 2018-01-01 00:00:00 UTC | 5 | 0 | 5 |
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 12 | 0 | 12 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 13 | 0 | 13 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 14 | 1 | 13 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 7 | 0 | 7 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 6 | 0 | 6 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 8 | 1 | 7 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 14 | 2 | 12 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 15 | 1 | 14 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 7 | 0 | 7 |
From the project README
An excerpt from the repository, fetched Nov 13, 2026.
Read the project’s introduction
Protocol Buffers - Google's data interchange format
Copyright 2008 Google LLC
Overview
Protocol Buffers (a.k.a., protobuf) are Google's language-neutral,
platform-neutral, extensible mechanism for serializing structured data. You
can learn more about it in protobuf's documentation.
This README file contains protobuf installation instructions. To install
protobuf, you need to install the protocol compiler (used to compile .proto
files) and the protobuf runtime for your chosen programming language.
Working With Protobuf Source Code
Most users will find working from
supported releases to be
the easiest path.
If you choose to work from the head revision of the main branch your build will
occasionally be broken by source-incompatible changes and insufficiently-tested
(and therefore broken) behavior.
If you are using C++ or otherwise need to build protobuf from source as a part
of your project, you should pin to a release commit on a release branch.
This is because even release branches can experience some instability in between
release commits.
Bazel with Bzlmod
Protobuf supports
Bzlmod with Bazel 8 +.
Users should specify a dependency on protobuf in their MODULE.bazel file as
follows.
bazel_dep(name = "protobuf", version = <VERSION>)
Users can optionally override the repo name, such as for compatibility with
WORKSPACE.
bazel_dep(name = "protobuf", version = <VERSION>, repo_name = "com_google_protobuf")
Bazel with WORKSPACE
Users can also add the following to their legacy
WORKSPACE file.
Note that with the release of 30.x there are a few more load statements to
properly set up rules_java and rules_python.
http_archive(
name = "com_google_protobuf",
strip_prefix = "protobuf-VERSION",
sha256 = ...,
url = ...,
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")
rules_java_dependencies()
load("@rules_java//java:repositories.bzl", "rules_java_toolchains")
rules_java_toolchains()
load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()
Protobuf Compiler Installation
The protobuf compiler is written in C++. If you are using C++, please follow
the C++ Installation Instructions to install protoc along
with the C++ runtime.
For non-C++ users, the simplest way to install the protocol compiler is to
download a pre-built binary from our GitHub release page.
In the downloads section of each release, you can find pre-built binaries in
zip packages: protoc-$VERSION-$PLATFORM.zip. It contains the protoc binary
as well as a set of standard .proto files distributed along with protobuf.
If you are looking for an old version that is not available in the release
page, check out the Maven repository.
These pre-built binaries are only provided for released versions. If you want
to use the github main version at HEAD, or you need to modify protobuf code,
or you are using C++, it's recommended to build your own protoc binary from
source.
If you would like to build protoc binary from source, see the C++ Installation Instructions.
Protobuf Runtime Installation
Protobuf supports several different programming languages. For each programming
language, you can find instructions in the corresponding source directory about
how to install protobuf runtime for that specific language:
Language
Source
C++ (include C++ runtime and protoc)
src
Java
java
Python
python
Objective-C
objectivec
C#
csharp
Ruby
ruby
Go
protocolbuffers/protobuf-go
PHP
php
Dart
dart-lang/protobuf
JavaScript
protocolbuffers/protobuf-javascript
Quick Start
The best way to learn how to use protobuf is to follow the tutorials in our
developer guide.
If you want to learn from code examples, take a look at the examples in the
examples directory.
Documentation
The complete documentation is available at the Protocol Buffers doc site.
Support Policy
Read about our version support policy
to stay current on support timeframes for the language libraries.
Developer Community
To be alerted to upcoming changes in Protocol Buffers and connect with protobuf developers and users,
join the Google Group.
Read the full README on GitHubWhat Hacker News says
124 sourceshttps://github.com/protocolbuffers/protobuf/commit/50f9ac3ba...
On a sufficiently anemic CPU doing enough fast allocation you can get measurable improvements from that kind of thing, but for a superscalar CPU and a general purpose allocator I agree you'd be hard pressed.
In Size-Specialized Memory Allocation> This is a bit in the weeds but inside Google we static link everything which makes is far easier to do it, and static linking is how we recommend you use C++Proto in general.
Statically linking everything doesn't work in this case because each C extension module (i.e. each Python package with some native code in it) is, in effect, a shared library.
If you statically linked the C++ protobuf library into both the Python protobuf library and the Python tensorflow library then you'd end up with two copies of the library in the process (with two copies of globals/statics such as the protobuf global descriptor pool). That's not a problem in itself, but it wouldn't be possible to pass protobuf objects between them.
(In principle, you could perhaps statically link protobuf into the CPython executable, but I really don't think Google is doing th
Read the rest
at.)
The usual solution is the exact opposite to static linking: you have to build the C++ protobuf library as a shared library (libprotobuf.so) and then link to it from both of your C extension modules (i.e. from the other two shared libraries).
Following the tiny crumbs of evidence of how to do this for protobuf, it seems like that was previously how you would do it. But it's possibly not now? It seems like proto_api.h [1] has a way to get the global descriptor pool, possibly to work around needing a common shared library? But also now that header is deprecated anyway [2]? (I think this is roughly where I got to before when I gave up.)
[1] https://github.com/protocolbuffers/protobuf/blob/main/python...
[2] https://github.com/protocolbuffers/protobuf/issues/9464
I agree you can't trust your compiler, but you can control its behavior more reliably with __builtin_expect_with_probability
https://github.com/protocolbuffers/protobuf/commit/9f29f02a3...
In Trust your compiler: Modern C++I've spent some brainpower on binary search and have not been able to beat this:
https://github.com/protocolbuffers/protobuf/blob/44025909eb7...
1. Check for dense list O(1)
2. Check upper bound
3. Constant trip count binary search
The constant trip count is great for the branch predictor, and the core loop is pretty tightly optimized for the target hardware, avoiding multiplies. Every attempt to get more clever made the loop worse and did not pay for itself. It's hard because it's an array-of-structs format with a size of 12, and mostly pretty small N.
In You can beat the binary searchYou're saying we choose Protobufs [1] because Google maintains it but not FlatBuffers [2]?
[1] - https://github.com/protocolbuffers/protobuf: Google's data interchange format
[2] - https://github.com/google/flatbuffers: Also maintained by Google
In Replacing Protobuf with RustLook at the long chains of methods in https://github.com/amantinband/error-or?tab=readme-ov-file#n.... The code looks quite unnatural.
In google/c++ you can do much simpler, but with preprocessor magic.
Example:
absl::StatusOr<User> loadUserById(int userId) { ... }
absl::Status populateItems(User user, std::vector<Item>& items) {...}
absl::StatusOr<Item> findItems(int userId) {
ASSIGN_OR_RETURN(auto user, loadUserById(userId));
std::vector<Item> items;
RETURN_IF_ERROR(populateItems(user, items));
for (auto& item: items) {
...
}
}
ASSIGN_OR_RETURN and RETURN_IF_ERROR essentially preprocessor macroses, that expand, more or less into.
absl::StatusOr<Item> findItems(int userId) {
auto userOrStatus = loadUserById(userId);
if (!userOrStatus.ok()) return user
Read the rest
OrStatus.status();
auto user = *userOrStatus;
std::vector<Item> items;
absl::Status st2 = populateItems(user, items));
if (!st2.ok()) return st2;
}
No long and ugly method invocation chains, no weirdly looking code - everything just works. You can see real life example here: https://github.com/protocolbuffers/protobuf/blob/bd7fe97e8c1...
Again, even inside Google there were docs that considered those macroses bad and suggested to write straightforward code, but I'm in the camp who considers them useful and, maybe, sole good use of preprocessor macros that I ever seen, as there are no other way to clearly and concisely express that in majority of languages.
F# has something like that with Computational Expressions, but they are still limited.
Protobuf is the example I had in mind. It uses sun.misc.Unsafe which is being removed in upcoming Java releases, but it has a slow fallback path. It logs a warning when it runs if it can tell it's only using the fallback path but the fast path is still available if the application owner set a flag to turn it back on if they want to:
https://github.com/protocolbuffers/protobuf/issues/20760
Java Protobuf also logs a warning now if you can tell you are using gencode old enough that it's covered by a DoS CVE. They actually did a release that broke compatability of the CVE covered gencode but restored it and print a warning in a newer release.
In Log level 'error' should mean that something needs to be fixedYou can influence the choice of conditional moves (usually inserting them) with
__builtin_expect_with_probability(..., 0.5)
https://github.com/protocolbuffers/protobuf/commit/9f29f02a3...
In When O3 is 2x slower than O2If I have a library that let me curry trailing arguments for a function, then I can see how something like std::flip() could be useful in letting me curry different arguments without costing extra lines of code. The library I had in mind is Google's RPC callbacks:
https://github.com/protocolbuffers/protobuf/blob/main/src/go...
This library was written in the C++98 era. It might seem silly now because with C++11, we could use std::bind or lambda expressions instead.
In `std::flip`> Your guess is as good as mine for why an enum can’t be used as a map key.
I filed an issue requesting this and it was denied with an explanation:
https://github.com/protocolbuffers/protobuf/issues/7791#issu...
In Protobuffers Are Wrong (2018)Out of curiosity, does this idiom work in fil-c?
https://github.com/protocolbuffers/protobuf/blob/cb873c8987d...
// This somewhat silly looking add-and-subtract behavior provides provenance
// from the original input buffer's pointer. After optimization it produces
// the same assembly as just casting `(uintptr_t)ptr+input_delta`
// https://godbolt.org/z/zosG88oPn
size_t position =
(uintptr_t)ptr + e->input_delta - (uintptr_t)e->buffer_start;
return e->buffer_start + position;
It does use the implementation defined behavior that a char pointer + 1 casted to uintptr is the same as casting to uintptr then adding 1.
In Fil's Unbelievable Garbage CollectorI tried implementing an optimized varint encoder that did something similar, by populating an 8 byte value and then storing it to ram, but unaligned overlapping stores caused big regressions. The approach that worked required a different technique. This one is for encoding backwards:
1. One branch for the one-byte case, always inlined, just store the byte
2. Calculate the size size of the unencoded zero prefix with a branch-free construction:
(((uint32_t)clz + 7) * 9) >> 6
3. Hand roll a jump table taking advantage of arm's fixed instruction width to calculate the jump target with a shift.
https://github.com/protocolbuffers/protobuf/commit/b039dfe26...
This results in one branch for 1 byte varints, plus one additional branch for any larger size, and the branch predictor does not have to track a varying trip count through a loop. This ap
Read the rest
proach resulted in a 2.64% speedup for overall encoding (which includes a lot more than just varints) on mid sized arm cores.
I think it's very difficult to beat a single comparison and branch for the 1 byte case for actual encoding forwards or backwards, unless you know there's going to be long runs of one-byte values.
This kind of "expected next field" optimization has a long history in protobuf, but results are mixed.
The generated code in C++ used to check for the expected next field before falling back to the switch() (example here: https://github.com/protocolbuffers/protobuf/blob/460e7dd7c47...) but this was removed in 2016 when load tests found that it hurt more than it helped.
One tricky part of making this optimization work is making a good guess about what the next field should be. Miguel's article alludes to this:
> Each field specifies which fields to try next. This allows the compiler to perform field scheduling, by carefully deciding which order to try fields in based both on their declaration order and a rough estimation of their “hotness”, much like branch scheduling happens in a program compiler. This avoids almost all of the work of lo
Read the rest
oking up the next field in the common case, because we have already pre-loaded the correct guess.
> I haven’t managed to nail down a good algorithm for this yet, but I am working on a system for implementing a type of “branch prediction” for PGO, that tries to provide better predictions for the next fields to try based on what has been seen before.
One delightful thing about the tail-call parser design is that the CPU's branch predictor effectively takes over the job of guessing. With a tail call parser, the dispatch sequence ends up looking like this:
cmp QWORD PTR [rdi+0x8],rsi # Bounds check
jbe .fallback
movzx r10d,WORD PTR [rsi] # Load two bytes of tag
mov eax,r10d
and eax,0xf8
mov r9,QWORD PTR [rcx+rax*2] # Load table data
xor r9,r10
mov rax,QWORD PTR [rcx+rax*2+0x8] # Load field parser function
jmp rax # Tail call to field parser
That "jmp rax" instruction is an indirect branch that can be predicted by the CPU. The CPU effectively guesses for us!
And unlike any kind of guess we might have performed ahead-of-time, the branch predictor will constantly adapt to whatever patterns it is seeing in the data. This is good, because statically guessing ahead-of-time is hard.
I think I can shed some light on this, as the creator and lead of upb.
Calling a Protobuf Parser an "interpreter VM" is a little bit of rhetorical flourish. It comes from the observation that there are some deep structural similarities between the two, which I first observed in an article a few years back: https://blog.reverberate.org/2021/04/21/musttail-efficient-i...
> It may seem odd to compare interpreter loops to protobuf parsers, but the nature of the protobuf wire format makes them more similar than you might expect. The protobuf wire format is a series of tag/value pairs, where the tag contains a field number and wire type. This tag acts similarly to an interpreter opcode: it tells us what operation we need to perform to parse this field’s data. Like interpreter opcodes, protobuf field numbers can come in any order, so we have to
Read the rest
be prepared to dispatch to any part of the code at any time.
This means that the overall structure of a protobuf parser is conceptually a while() loop surrounding a switch() statement, just like a VM interpreter.
The tricky part is that the set of "case" labels for a Protobuf parser is message-specific and defined by the fields in the schema. How do we accommodate that?
The traditional answer was to generate a function per message and use the schema's field numbers as the case labels. You can see an example of that here (in C++): https://github.com/protocolbuffers/protobuf/blob/f763a2a8608...
More recently, we've moved towards making Protobuf parsing more data-driven, where each field's schema is compiled into data that is passed as an argument to a generic Protobuf parser function. We call this "table-driven parsing", and from my read of the blog article, I believe this is what Miguel is doing with hyperpb.
The trick then becomes how to make this table-driven dispatch as fast as possible, to simulate what the switch() statement would have done. That question is what I cover at length in the article mentioned above.
Showing the 15 most recent of 124 sources. Enable JavaScript to search and browse every indexed mention.