HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
kstenerud
concise-encoding
ANTLR2019–2024
kstenerud / concise-encoding

concise-encoding

The secure data format for a modern world

5Distinct people
63Mentions
0Explicit recommenders
280GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

The secure data format for a modern world

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year63 dated mentions

Observed mentions: 2019-05-31 18:05:03 UTC to 2024-08-13 16:16:01 UTC.

0510152019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 9 mentions; 0 submissions, 9 comments20192020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 17 mentions; 1 submissions, 16 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 13 mentions; 0 submissions, 13 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 15 mentions; 2 submissions, 13 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 6 mentions; 1 submissions, 5 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2024

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC909
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC17116
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC13013
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC15213
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC615
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC303

What Hacker News says

63 sources
I haven't made a programming language (and never will), but I did build a BNF-inspired metalanguage for describing text and binary formats to scratch the itch of trying to describe a binary data format I was developing: The metalanguage: https://dogma-lang.org/ It's even got a syntax highlighter: https://marketplace.visualstudio.com/items?itemName=ksteneru... The binary format I wanted to describe: https://github.com/kstenerud/concise-encoding/blob/master/cb...
In What you learn by making a new programming language
I built a grammar to tackle these sorts of problems when I had trouble writing formal grammar notations for my binary data format. It's even got a syntax highlighter. https://dogma-lang.org/ So far it's been able to describe 90% of what's out there. Some examples: - 802.3 layer 2 Ethernet: https://github.com/kstenerud/dogma/blob/master/v1/examples/8... - Microsoft ICO format: https://github.com/kstenerud/dogma/blob/master/v1/examples/i... - Android Dex v39: https://github.com/kstenerud/dogma/blob/master/v1/examples/d... - IPv4: https://github.com/kstenerud/dogma/blob/master/v1/examples/i... - DNS query: https://github.com/kstenerud/dogma/blob/master/v1/examples/d... - Microsoft Minidump: https://github.com/kstenerud/dogma/blob/master/v1/examples/m... - Concise Binary Encoding: https://github.com/kstenerud/concise-encoding/blob/master/cb..
Read the rest
In Interval Parsing Grammars for File Format Parsing (2023) [pdf]
kstenerud·Sep 27, 2023Submission

Show HN: Concise Encoding – Last Prerelease

After 5 years of development, I'm about to release v1.0 of Concise Encoding: a general purpose, human and machine friendly, compact representation of semi-structured hierarchical data. I could do with one last round of constructive criticism before I do! It's designed for: * Security when ingesting data * High efficiency * Human readability * Native support for all basic types * Simple to use Its purpose is to give a secure and efficient alternative for whenever you'd normally reach for JSON or similar ad-hoc formats. Example (using the text format): c1 { "boolean" = true "binary int" = -0b10001011 "octal int" = 0o644 "decimal int" = -10000000 "hex int" = 0xfffe0001 "very long int" = 100000000000000000000000000000000000009 "decimal fl
Read the rest
oat" = -14.125 "hex float" = 0x5.1ec4p+20 "very long flt" = 4.957234990634579394723460546348e+100000 "not-a-number" = nan "infinity" = inf "neg infinity" = -inf "string" = "Strings support escape sequences: \n \t \[1f415]" "url" = @"https://example.com/" "email" = @"mailto:me@somewhere.com" "uuid" = f1ce4567-e89b-12d3-a456-426655440000 "date" = 2019-07-01 "time" = 18:04:00.948/Europe/Prague "timestamp" = 2010-07-15/13:28:15.415942344/America/Vancouver "null" = null "media" = @application/x-sh[23 21 2f 62 69 6e 2f 73 68 0a 0a 65 63 68 6f 20 68 65 6c 6c 6f 20 77 6f 72 6c 64 0a] "list" = [1 2.5 "a string"] "map" = {"one"=1 2="two" "today"=2020-09-10} "bytes" = @u8x[01 ff de ad be ef] "int16 array" = @i16[7374 17466 -9957] "uint16 hex" = @u16x[91fe 443a 9c15] "float32 array" = @f32[1.5e10 -8.31e-12] } Website: https://concise-encoding.org/ Structure: https://github.com/kstenerud/concise-encoding/blob/master/ce... Binary Format: https://github.com/kstenerud/concise-encoding/blob/master/cb... Text Format: https://github.com/kstenerud/concise-encoding/blob/master/ct... Reference implementation (go): https://github.com/kstenerud/go-concise-encoding Command line tool: https://github.com/kstenerud/enctool
1 points · Open discussion
This is basically why I ended up rolling my own text date format for Concise Encoding: - https://github.com/kstenerud/concise-encoding/blob/master/ct... - https://github.com/kstenerud/concise-encoding/blob/master/ce... e.g. 2019-8-5 // August 5, 2019 5081-03-30 // March 30, 5081 -300-12-21 // December 21, 300 BC (proleptic Gregorian) 20-01-01 // January 1, 20 (NOT 1920, NOT 2020) 09:04:21 // 9:04:21 UTC 23:59:59.999999999 // 23:59:59 and 999999999 nanoseconds UTC 12:05:50.102/Z // 12:05:50 and 102 milliseconds UTC 4:00:00/Asia/Tokyo // 4:00:00 Tokyo time 17:41:03/-13.54/-172.36 // 17:41:03 Samoa time 9:00:00/L // 9:00:00 local time 2019-01-23/14:08:51.941245 // January 23, 2019, a
Read the rest
t 14:08:51 and 941245 microseconds, UTC 1985-10-26/01:20:01.105/M/Los_Angeles // October 26, 1985, at 1:20:01 and 105 milliseconds, Los Angeles time 5192-11-01/03:00:00/48.86/2.36 // November 1st, 5192, at 3:00:00, at whatever is in the place of Paris at that time 1985-10-26/01:20:01.105+0700 // October 26, 1985, at 1:20:01 and 105 milliseconds, UTC+7:00 2000-01-14/10:22:00-0200 // January 14, 2000, at 10:22, UTC-2:00 ISO 8601 and RFC 3339 are fine for dates in the recent past, but they're not great as a general time format.
In RFC 3339 vs. ISO 8601
Yup, agreed. I've been building similar building blocks to this, and it's VERY hard and long and tedious work to make it actually happen. And yes, doing this kind of thing while employed full time is frustrating to no end, because everything takes 5x longer. After 5 years, I have Dogma [1] (a metalanguage for building text and binary grammars) finished, and Concise Encoding [2] (an ad-hoc hierarchical data format) almost finished (it's been slowed by the fact that I needed to develop Dogma to describe it [3], and also from switching to Antlr for the text parsing). I expect that the bidirectional, multi-stream encrypted communications protocol that uses these technologies will be even more pain once I dig into it beyond the 6 months I've given it so far. And only THEN will I be able to make interesting things with it (such as a re-imagined
Read the rest
file API alternative to POSIX). Building this kind of stuff is a marathon with no end, and you absolutely can't do it unless you either are already independently wealthy, or you're leveraging your daytime job to that end. [1] https://dogma-lang.org/ [2] https://concise-encoding.org/ [3] https://github.com/kstenerud/concise-encoding/blob/master/cb... and https://github.com/kstenerud/concise-encoding/blob/master/ct...
In Facing Imminent Homelessness, Gagoot Developer Seeks Project Funding
This looks similar to https://concise-encoding.org/ Dogma was developed as a consequence of trying to describe Concise Binary Encoding. The CBE spec used to look like the preserves binary spec, full of hex values, tables and various ad-hoc illustrations: https://preserves.dev/preserves-binary.html Now the CBE formal description looks like this: https://github.com/kstenerud/concise-encoding/blob/master/cb... And the regular documentation looks like this: https://github.com/kstenerud/concise-encoding/blob/master/cb... Dogma also does text formats because Concise Encoding has a text and binary format, so I needed a metalanguage that could do both in order to make it less jarring for a reader: Docs: https://github.com/kstenerud/concise-encoding/blob/master/ct... Formal: https://github.com/kstenerud/concise-encoding/blob/master/ct...
In Ask HN: Please critique my metalanguage: “Dogma”
It's to scratch a personal itch of mine: formal, human readable descriptions of binary data formats. I couldn't find anything suitable for describing the binary format for Concise Encoding (https://concise-encoding.org/), so I did the crazy thing, thinking "how hard could it be?" But it worked out and I have a formal description for it now: * https://github.com/kstenerud/concise-encoding/blob/master/cb... * https://github.com/kstenerud/concise-encoding/blob/master/ct... So it's primarily geared towards documentation. That being said, there's no reason why tools couldn't use it as well (I'm trying to keep it consistent enough that tools could be built, or possibly even a specialized parser-generator language that uses Dogma as a base, similar to how Antlr uses a modified EBNF). But I've got too many projects going on as it is, so anyone wh
Read the rest
o wants to build tooling, I'm all for it and will help where I can :) Error handling would be an implementation detail. The spec only describes what counts as a malformed grammar, and what counts as an ambiguous grammar (and a small bit about undefined behaviour, which counts as ambiguous).
In Ask HN: Please critique my metalanguage: “Dogma”
kstenerud·Dec 29, 2022Submission

Ask HN: Please help me by reviewing KBNF

Hi everyone! Merry Christmas and happy new year! I'm making a modernized BNF-style metalanguage that supports grammars for text and binary formats, but unless there's some kind of outside review, I won't find any blind spots I have until long after release. If you have experience in grammars or protocols (text or binary), could you take a quick look? Note: I'm not aiming for perfect expressivity; just 80-90% of use cases without making things overcomplicated so that it can be used descriptively in documentation. https://github.com/kstenerud/kbnf/blob/master/kbnf.md Some examples of it in use: * Internet Prodocol, version 4: https://github.com/kstenerud/kbnf/blob/master/ipv4.kbnf * Concise Text Encoding: https://github.com/kstenerud/concise-encoding/blob/master/cte.kbnf * Concise Binary Encoding: https://github.com/kstenerud/concise-encodi
Read the rest
6 points · Open discussion
Hey thanks for taking the time to critique! I actually do have an ANTLR file that is about 90% of the way there ( https://github.com/kstenerud/concise-encoding/tree/master/an... ), so I could use those as a basis... One thing I'm not sure about is how to define a BNF rule that says for example: "An identifier is a series of characters from unicode categories Cf, L, M, N, and these specific symbol characters". BNF feels very ASCII-centric...
In Keep ’em coming: Why your first ideas aren’t always the best
It's still in the prerelease stage, but v1 will be released later this year. I'm mostly getting hits from China since they tend to be a lot more worried about security. I expect the rest of the world to catch on to the gaping security holes of JSON and friends in the next few years as the more sophisticated actors start taking advantage of them. For example https://github.com/kstenerud/concise-encoding/blob/master/ce... There are still a few things to do: - Update enctool (https://github.com/kstenerud/enctool) to integrate https://cuelang.org so that there's at least a command line schema validator for CE. - Update the grammar file (https://github.com/kstenerud/concise-encoding/tree/master/an...) because it's a bit out of date. - Revamp the compliance tests to be themselves written in Concise Encoding (for example https://github.com/ksten
Read the rest
erud/go-concise-encoding/blob/master... but I'll be simplifying the format some more). That way, we can run the same tests on all CE implementations instead of everyone coming up with their own. I'll move the test definitions to their own repo when they're done and then you can just submodule it. I'm thinking that they should look more like: c1 { "type" = { "identifier" = "ce test" "version" = 1 } "tests" = [ { "name" = "Some kind of test" "success" = [ // These must successfully convert to each other { "cte" = "c1 [1 2 3 4]" "cbe" = |u8x 81 01 7a 01 02 03 04 7b| // Events are a kind of text shorthand for what the parsing of cte or cbe should produce "events" = ["v 1" "l" "n 1" "n 2" "n 3" "n 4" "e"] } { ... } ] "failure" = [ // These must fail to decode { "cte" = "c1 100 ]" } { "cbe" = |u8x 81 01 64 7b| } ] } ] }
In Working in the software industry, circa 1989

Showing the 15 most recent of 63 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.