kstenerud

concise-encoding

ANTLR2019–2024

kstenerud / concise-encoding

concise-encoding

The secure data format for a modern world

Explore on GitHub Read Hacker News discussions

5Distinct people

63Mentions

0Explicit recommenders

280GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

The secure data format for a modern world

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 8:59:20 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year63 dated mentions

Observed mentions: 2019-05-31 18:05:03 UTC to 2024-08-13 16:16:01 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2019	2019-01-01 00:00:00 UTC	2020-01-01 00:00:00 UTC	9	0	9
2020	2020-01-01 00:00:00 UTC	2021-01-01 00:00:00 UTC	17	1	16
2021	2021-01-01 00:00:00 UTC	2022-01-01 00:00:00 UTC	13	0	13
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	15	2	13
2023	2023-01-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	6	1	5
2024	2024-01-01 00:00:00 UTC	2025-01-01 00:00:00 UTC	3	0	3

What Hacker News says

13 sources

kstenerud·Dec 17, 2021

The problem is that we're still stuck with only binary floating point types in our CPUs and compilers and runtime environments, over a decade after ieee754 released their decimal float spec [1]. Once we finally move away from binary floats, these nasty binary-exponent-decimal-exponent lossy conversions will end (as will the horribly complicated nasty scanf and printf algorithms). This is a solved problem. Our actual problem now is momentum of adoption. I've anticipated that this will eventually happen (hopefully sooner rather than later), and proactively marked binary floats as "legacy formats" [2]. [1] https://en.wikipedia.org/wiki/Decimal64_floating-point_forma... [2] https://github.com/kstenerud/concise-encoding/blob/master/ce...

In Floating point numbers, and why they suck

kstenerud·Nov 20, 2021

I'll be working on the reference implementation [1] of Concise Encoding [2], which is a secure data format for the modern world. My aim is to replace insecure and clunky formats like JSON and XML and the various binary formats that do similar things less conveniently. In a nutshell: - Existing ad-hoc formats are too loosely defined to be secure, and that's becoming a huge problem as the bad guys become more sophisticated. CE is tightly specified and designed to mitigate exploitation of codecs. - CE is a twin text and binary format. Humans view and edit in text, and machines send it in binary, so you get the convenience of text and the efficiency of binary for free. - CE supports the fundamental types natively. Stringifying is buggy, causes incompatibilities, and opens security holes. And it's completely unnecessary with a properly designe

Read the rest

d data format. [1] https://github.com/kstenerud/go-concise-encoding [2] https://concise-encoding.org/ [3] https://github.com/kstenerud/concise-encoding

In Ask HN: What are you working on this weekend?

kstenerud·Nov 2, 2021

I'm continuing to work on https://concise-encoding.org which is a new security-conscious ad-hoc encoding format to replace JSON/XML and friends. I've been at it for 3 years so far and am close to a release. In a nutshell: - Edit in text, transmit in binary. One can be seamlessly converted to the other, but binary is far more efficient for processing, storage and transmission, while text is better for humans to read and edit (which happens far less often than the other things). - Secure by design: Everything is tightly specced and accounted for so that there aren't differences between implementations that can be exploited to compromise your system. https://github.com/kstenerud/concise-encoding/blob/master/ce... - Real type support because coercing everything into strings sucks (and is another security risk and source of incompatibilities).

Read the rest

XML had a good run but was replaced by JSON which was a big improvement. JSON also had a good run but it's time for it to retire now that the landscape has changed even further: Security and efficiency are the desires of today, and JSON provides neither. I've got the spec nailed down and can finally see the light at the end of the tunnel for the reference implementation in golang. I still need to come up with a system for schemas, but I'm hoping that https://cuelang.org will fit the bill.

In Ask HN: What you up to? (Who doesn't want to be hired?)

kstenerud·Oct 24, 2021

This is where security and limits comes in: https://github.com/kstenerud/concise-encoding/blob/master/ce...

In RFC 3339 vs. ISO 8601

kstenerud·Oct 24, 2021

This is the biggest reason why I had to develop a new time format for Concise Encoding :/ https://github.com/kstenerud/concise-encoding/blob/master/ce... https://github.com/kstenerud/concise-encoding/blob/master/ct... ISO 8601 and RFC 3339 are only useful for times in the past (which works fine for most internet protocols because they're only interested in recording events).

In RFC 3339 vs. ISO 8601

kstenerud·Jun 25, 2021

Unfortunately, the QR code "binary" mode specification defaults to ISO 8859-1 for the encoding (because it was not originally intended to store actual binary data), and there's also no way to indicate what format is actually encoded. So all decoders of course just assume ISO 8859-1 because they have no way of knowing otherwise. However, we could in theory get around this by using binary data formats that always begin with an invalid text character (such as 0x80-0x9f). This way, an implementation can know that the data is not ISO 8859-1, and try to decode whatever format it discovers through the beginning byte signature. I've actually put this into Concise Encoding [1] [1] https://github.com/kstenerud/concise-encoding/blob/master/cb...

In The Base45 Data Encoding

kstenerud·Jun 25, 2021

Unfortunately, the QR code "binary" mode specification defaults to ISO 8859-1 for the encoding (because it was not originally intended to store actual binary data), and there's also no way to indicate what format is actually encoded. So all decoders of course just assume ISO 8859-1 because they have no way of knowing otherwise. However, we could in theory get around this by using binary data formats that always begin with an invalid text character (0x80-0x9f). This way, an implementation can know that the data is not ISO 8859-1, and try to decode whatever format it discovers through the beginning byte signature. I've actually put this into Concise Encoding [1] [1] https://github.com/kstenerud/concise-encoding/blob/master/cb...

In The Base45 Data Encoding

kstenerud·Jun 8, 2021

I got so sick of the broken time formats (ALL of them are broken or deficient, including ISO-8601) that I eventually made my own: one in binary [1], and one in text [2]. Time is one of the most difficult things to get right in software engineering. [1] https://github.com/kstenerud/compact-time/blob/master/compac... [2] https://github.com/kstenerud/concise-encoding/blob/master/ct... and https://github.com/kstenerud/concise-encoding/blob/master/ce...

In TLS certificates have at least two internal representations of time

kstenerud·May 8, 2021

You can put emphasis on high order bits, but that makes decoding more complex. With little endian the decoder builds low to high, which is MUCH easier to deal with, especially on spillover. For example, with ULEB128 [1], you just read 7 bits at a time, going higher and higher up the value you're reconstituting. If the value grows too big and you need to spill over to the next (such as with big integer implementations), you just fill the last bits of the old value, then put the remainder bits in the next value and continue on. With a big endian encoding method (i.e. VLQ used in MIDI format), you start from the high bits and work your way down, which is fine until your value spills over. Because you only have the high bits decoded at the time of the spillover, you now have to start shifting bits along each of your already decoded big intege

Read the rest

r portions until you finally decode the lowest bit. This of course gets progressively slower as the bits and your big integer portions pile up. Encoding is easier too, since you don't need to check if for example a uint64 integer value can be encoded in 1, 2, 3, 4, 5, 6, 7 or 8 bits. Just encode the low 8 bits, shift the source right by 8, repeat, until the source value is 0. Then backtrack to the as-yet-blank encoded length field in your message and stuff in how many bytes you encoded. You just got the length calculation for free. Use a scheme where you only encode up to 60 bit values, place the length field in the low 4 bits, and Robert's your father's brother! For data that is right-heavy (i.e. the fully formed data always has real data on the right side and blank filler on the left - such as uint32 value 8 is actually 0x00000008), you want a little endian scheme. For data that is left-heavy, you want a big endian scheme. Since most of the data we deal with is right-heavy, little endian is the way to go. You can see how this has influenced my encoding design in [2] [3] [4]. [1] https://en.wikipedia.org/wiki/LEB128 [2] https://github.com/kstenerud/concise-encoding/blob/master/cb... [3] https://github.com/kstenerud/compact-float/blob/master/compa... [4] https://github.com/kstenerud/compact-time/blob/master/compac...

In The Byte Order Fiasco

kstenerud·Apr 3, 2021

> I noticed the spec allows unquoted strings. What's the reasoning for this? In my experience unquoted strings cause nothing but trouble, and are confusing to humans who may interpret them as keywords. Unquoted strings are much nicer for humans to work with. All special keywords and object encodings are prefixed with sigils (@, &, $, #, etc), so any bare text starting with a letter is either a string or an invalid document, and any bare text starting with a numeral is either a number or an invalid document. > Any reason for not using RFC2119 keywords in the spec? Using them should make the spec easier to read. I use a superset of those keywords to give more precision in meaning: https://github.com/kstenerud/concise-encoding/blob/master/ce...

In The Norway Problem

kstenerud·Apr 3, 2021

The worst tragedy of this is the security implications of subtly different parsers. As your application surface increases, you're likely to mix languages (and thus different parsers), which means that the same input data will produce different output data depending on whether your parser replaces, truncates, ignores, or otherwise attempts to automatically "fix up" the data. A carefully crafted document could exploit this to trick your data storage layer into storing truncated data that elevates privileges or sets zero cost, while your access control layer that ignores or replaces the data is perfectly happy to let the bad document pass by. And here's something else to keep you up at night: Just think of how many unintentional land mines lurk in your serialized data, waiting to blow up spectacularly (or even worse, silently) as soon as you

Read the rest

attempt to change implementation technologies! This is why I've been so anal about consistent decoder behavior in Concise Encoding https://github.com/kstenerud/concise-encoding/blob/master/ce... https://concise-encoding.org/

In The Norway Problem

kstenerud·Mar 23, 2021

Slightly more complex. The type code distribution is based on how often the type is likely to occur in your average document. The type field is almost always 1 byte long, except for typed arrays (which would generally in aggregate be long enough to offset the cost) and two uncommon types/variations. These have an effective 2-byte type field (0x94 selects a secondary type plane, and the next byte selects the specific type from there). The most common integer values (-100 to 100) are encoded directly into the types 0x9c to 0x64 (wraparound), such that interpreting them directly as signed 8-bit signed integers yields their actual value (type 0x00 = integer 0, type 0x64 = integer 100, type 0x9c = -100, etc). Strings are also optimized such that types 0x80 - 0x8f are used for the most common string lengths (0 to 15) so as not to require a sepa

Read the rest

rate length field. The rest have a 1-byte type field. There are also 3 reserved type codes left in the first plane in case something big comes up in the future. You can see the chart here: https://github.com/kstenerud/concise-encoding/blob/master/cb...

In FastDoubleParser: Java port of Daniel Lemires fast_double_parser

kstenerud·Feb 26, 2021

ISO-8601 was a good early attempt, but unfortunately it has some negative points: - Only offset-based time zones are allowed (which are next to useless). - Omitting the time zone defaults to local time, which renders the time ambiguous. Defaulting to UTC makes much more sense. - BC dates are all offset by 1 ("-1" = 2 BC). I know the reasons for this, but this could have been moved to the implementation and not exposed to the user. - Lots of silly things like fractional-minutes and multiple ways to represent the same data bloat the spec and implementations. This is why I rejected iso-8601 and developed my own for https://github.com/kstenerud/concise-encoding/blob/master/ct... Examples: 2019-8-5 : August 5, 2019 -300-12-21 : December 21, 300 BC (proleptic Gregorian) 12:05:50.102 : 12:05:50 and 102 milliseconds UTC 4:00:00/Asia/Tokyo : 4:00:

Read the rest

00 Tokyo time 2019-01-23/14:08:51.941245 : January 23, 2019, at 14:08:51 and 941245 microseconds, UTC 1985-10-26/01:20:01.105/America/Los_Angeles : October 26, 1985, at 1:20:01 and 105 milliseconds, Los Angeles time 25192-11-01/03:00:00/48.86/2.36 : November 1st, 25192, at 3:00:00, at whatever is in the place of Paris at that time (global coordinates 48.86, 2.36)

In ISO 8601: a better date format