HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
protocolbuffers
upb
C2016–2023
protocolbuffers / upb · Archived

upb

a small protobuf implementation in C

4Distinct people
9Mentions
0Explicit recommenders
1.5KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

a small protobuf implementation in C

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year9 dated mentions

Observed mentions: 2016-01-30 16:47:53 UTC to 2023-08-18 18:39:55 UTC.

0122016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20162017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments20192020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC110
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC000
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC202
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC000
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC101
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC202
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC101
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC202

What Hacker News says

9 sources
> There are undeniably power users for whom every last bit of performance translates to very large sums of money, and I don’t claim to know how to satisfy them otherwise. That is the key, right there. In the 1970s, C may have been considered a general-purpose programming langauge. Today, given the landscape of languages currently available, C and C++ have a much more niche role. They are appropriate for the "power users" described above, who need every last bit of performance, at the cost of more development effort. When I'm working in C, I'm frequently watching the assembly language output closely, making sure that I'm getting the optimizations I expect. I frequently find missed optimization bugs in compilers. In these scenarios, undefined behavior is a tool that can actually help achieve my goal. The question I'm always asking myse
Read the rest
lf is: what do I have to write in C to get the assembly language output I expect? Here is an example of such a journey: https://blog.reverberate.org/2021/04/21/musttail-efficient-i... I created the https://github.com/protocolbuffers/upb project a long time ago. It's written in C, and over the years it has gotten to a state where the speed and code size are pretty compelling. Both speed and code size are very important to the use cases where it is being used. It's a relatively small code base also. I think focused, performance-oriented kernels are the area where C makes the most sense.
In C and C++ prioritize performance over correctness
Lua uses "chained scatter" (linked list, but links point to other entries in the same table, to maintain locality): https://github.com/lua/lua/blob/master/ltable.c This is a good visual depiction of chained scatter: https://book.huihoo.com/data-structures-and-algorithms-with-... Inspired by Lua, I did the same for upb (https://github.com/protocolbuffers/upb). I recently benchmarked upb's table vs SwissTable for a string-keyed table and found upb was faster in both insert and lookup (in insert upb is beating SwissTable by 2x). It's true that the links do create more memory overhead though.
In Implementing Hash Tables in C
> But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful. For what it's worth, this sounds somewhat similar to protobuf (which also supports dicts, arrays, etc). After spending many years trying to figure out the smallest, fastest, and simplest way to implement protobuf in https://github.com/protocolbuffers/upb, the single best improvement I found was to make the entire memory management model arena-based. When you parse an incoming request, all the little objects (messages, arrays, maps, etc) are allocated on the arena. When you are done with it, you just free the arena. In my experience this results in code that is both simpler and faster than trying to memory-manage all of the sub-objects independently. It also integrates nicely with existing memory-managem
Read the rest
ent schemes: I've been able to adapt the arena model to both Ruby (tracing GC) and PHP (refcounting) runtimes. You just have to make sure that the arena itself outlives any reference to any of the objects within. (Protobuf C++ also supports arenas, that's actually where the idea of using arenas for protobuf was first introduced. But Protobuf C++ also has to stay compatible with its pre-existing API based on unique ownership, so the whole API and implementation are complicated by the fact that it needs to support both memory management styles).
In Asahi Linux for Apple M1 progress report, August 2021
> Google's implementations, at least C++ and Java, are a bunch of bloated crap (or maybe they're very good, but for a use case that I haven't yet encountered). As someone who has been working on protobuf-related things for >10 years, including creating a size-focused implementation (https://github.com/protocolbuffers/upb), and has been working on the protobuf team for >5 years, I have a few thoughts on this (thoughts are my own, and I don't speak for anybody else). I think it is true that protobuf C++ could be a lot more lean than it currently is (I can't speak to Java as I don't work on it directly). That's why I created upb to begin with. But there's also a bit more to this story. The protobuf core runtime is split into two parts, "lite" and "full". If you don't need reflection for your protos, it's better to use "lite" by using "opti
Read the rest
on optimize_for = LITE_RUNTIME" in your .proto file (https://developers.google.com/protocol-buffers/docs/proto#op...). That will cut out a huge amount of code size from your binary. On the downside, you won't get functionality that requires reflection, such as text format, JSON, and DebugString(). Even the lite runtime can get "lighter" if you compile your binary to statically link the runtime and strip unused symbols with -ffunction-sections/-fdata-sections/--gc-sections flags. Some parts of the lite runtime are only needed in unusual situations, like ExtensionSet which is only used if your protos use proto2 extensions (https://developers.google.com/protocol-buffers/docs/proto#ex...). If you avoid these cases, the lite runtime is quite light. However, there is also the issue of the generated code size. Unlike the runtime, this is not a fixed cost, but is proportional to the number of messages you use. If you have a lot of messages it can quickly dwarf the size of the runtime. For this reason, C++ also supports "option optimize_for = CODE_SIZE" which uses reflection-based algorithms for all parsing/serialization/etc instead of using generated code. This means you pay the fixed size hit from linking in the full runtime, but the generated code size is much smaller. On the downside, "optimize_for = CODE_SIZE" has a severe ~10x speed penalty for parsing and serialization. I have long had the goal of making https://github.com/protocolbuffers/upb competitive with protobuf C++ in speed while achieving much smaller code size. With the benefit of 10 years of hindsight and many wrong turns, upb is beginning to meet and even surpass these goals. It is an order of magnitude smaller than protobuf C++, both in the core runtime and the generated code, and after some recent experiments it is beginning to significantly surpass it in speed also (I want to publish these results soon, but the code was merged in this PR: https://github.com/protocolbuffers/upb/pull/310). upb has downsides that prevent it from being fully "user ready" yet: the API is still not 100% stable, there is no C++ API for the generated code yet (and C APIs for protobuf are relatively verbose and painful), it has a bunch of legacy APIs sitting around that I am just on the verge of being able to finally delete, and it doesn't support proto2 extensions yet. On the upside, upb is 100% conformant on every other protobuf feature, it supports reflection, JSON, and text format, but also lets you omit these if you don't want to pay the code size. I hope 2021 is a year when I'll be able to publish more about these results, and when upb will be a more viable choice for users who want a smaller protobuf implementation.
In Don't Use Protobuf for Telemetry
> FWIW, I have been experimenting with auto-generating CMake from my Bazel BUILD file for my project https://github.com/google/upb. My plan is to use Bazel for development but have the CMake build be a fully supported option also for users who don't want to touch Bazel. Thank you for your answer. I really wish the auto-generation of CMake build scripts will solve the problem(s). But even that I have doubts. The way Bazel works will make very hard to auto-generate "standard modern" CMake that allows external dependencies. This is orthogonal to the way Bazel works.
In Tensorflow 2.0: models migration and new design
> Nix, Guix and Spack packager managers solved the C++ ABI issue a long time ago Yes this is certainly something that is solvable at the package manager level also. And this approach will certainly have shorter compile times. I agree it would be nice if Bazel integrated with package managers like this more easily. I hope Bazel adds support for this. There is a trade-off though: with less control over the specific version of your dependencies, there is a greater risk of build failure or bugs arising from an untested configuration. Basically this approach outsources some of the testing and bugfixing from the authors to the packagers. But it's a trade-off I know many people are willing to make. > Can I use that to specify a flag to some target and not some other ? Without having to build sequentially each of them ? You can put copts=["<
Read the rest
opt>"] in the cc_library() rules in the BUILD file. This will give per-target granularity. You can add a select() based on compilation_mode if you need to define opt-only flags: https://docs.bazel.build/versions/master/configurable-attrib... > Bazel try to build in isolated environment but do half of the job. It still depends on system compiler, and do not chroot nor "compiler-wrap" ( c.f Spack ) making the build still very vulnerable of system side effect and update. Bazel allows you to define your own toolchain. This can support cross-compiling and isolating the toolchain I believe, though I don't have any direct experience with this: https://docs.bazel.build/versions/master/toolchains.html > I can understand that Bazel is very convenient in Google environment with Google resources. But it's a nightmare for everyone I talked to outside of Google. I hear you and I hope that we see some improvements to integrate better with package managers. FWIW, I have been experimenting with auto-generating CMake from my Bazel BUILD file for my project https://github.com/google/upb. My plan is to use Bazel for development but have the CMake build be a fully supported option also for users who don't want to touch Bazel.
In Tensorflow 2.0: models migration and new design

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.