sqlite_protobuf

A SQLite extension for extracting values from serialized Protobuf messages

Data & Databases

Explore on GitHub Read Hacker News discussions

2Distinct people

4Mentions

0Explicit recommenders

92GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

A SQLite extension for extracting values from serialized Protobuf messages

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year4 dated mentions

Observed mentions: 2019-03-01 12:15:44 UTC to 2022-06-21 14:15:04 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	2	2
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	0	0
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	1	1
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	1	1

What Hacker News says

4 sources

rgovostes·Jun 21, 2022

SQLite's virtual table API (https://www.sqlite.org/vtab.html) makes it possible to access other data structures through the query engine. You don't need to know much if anything about how the database engine executes queries, you only need to implement the callbacks it needs to do its job. A few years ago I wrote an extension to let me search through serialized Protobufs which were stored as blobs in a regular database. https://github.com/rgov/sqlite_protobuf

In One-liner for running queries against CSV files with SQLite

rgovostes·Apr 3, 2021

Some time ago I joined a project that was recording data logs in Protobuf format. I demonstrated that we could log the serialized Protobuf structures to a SQLite database and then use a plugin to query across the message fields: https://github.com/rgov/sqlite_protobuf Learning how to use the SQLite extension API was a little mind-expanding but it worked like a charm.

In SQLite Plus: Missing SQLite functions

scottlamb·Mar 1, 2019

Interesting library, thanks! > Any query is going to require deserializing every row in the table (JSON and Protobuf alike) which is basically a non-starter for any project with performance requirements. Apparently you can create indexes on the json functions, so I assume on your proto ones too. > I believe that Protobuf messages are decoded as a whole, rather than just the desired field, which is going to be slower. With the official library, yes, but it's an implementation/API choice. The wire format [1] can be skimmed fairly efficiently. Most significantly, message/byte/string fields all are written as tag number, length, data. So if there are submessages (entire trees) you don't care about, you can just skip over them. I've seen custom proto decoding things do this. I think the most efficient approach would be to parse the path once p

Read the rest

er SQL query into a tag number-based path. And then have the per-row function just follow those with custom decoding logic. Unfortunately from my quick skim, it looks like SQLite's extension API doesn't really support this. You'd want it to build some context object for a given path/proto, then call extract with it a bunch of times, then tear it down. Still, I suppose you could do a LRU cache of these or something. btw, I see your code is compiling a regex [edit: originally wrote proto by mistake] in the per-row path. [2] I haven't profiled, but I'd bet that's slowing you down a fair bit. You could just make it a 'static const std::regex* kPathElementRegexp = new std::regex("...")' to avoid this. (static initialization is thread-safe in C++. The heap allocation is because it's good practice to ensure non-POD globals are never destructed. Alternatively, there's absl::NoDestructor for this.) [1] https://developers.google.com/protocol-buffers/docs/encoding [2] https://github.com/rgov/sqlite_protobuf/blob/0a148ac6a5a2c02...

In JSON with Sqlite

rgovostes·Mar 1, 2019

I recently wrote an SQLite extension that lets me query stored Protobuf messages. I was inspired by the JSON extension. Adding virtual tables to SQLite with an extension is tricky but kind of magical when you finally have the full query language to play with. https://github.com/rgov/sqlite_protobuf

In JSON with Sqlite