json

to bootstrap it we of course need to actually have keywords and operators there to build upon.) Since modules can add new things, we would need to update the list of known tokens as we are parsing. Which means that even if Raku did the tokenization thing, it would have to happen at the same time as the other steps. Tokenization seems like an antiquated way to create compilers. It was needed as there wasn't enough memory to have all of the stages loaded at the same time. --- Here is an example parser for JSON files using regexes in a grammar to show the simplicity and power of parsing things this way. grammar JSON::Parser::Example { token TOP { \s* <value> \s* } rule object { '{' ~ '}' <pairlist> } rule pairlist { <pair> * % ',' } rule pair { <string> ':' <value> } rule array { '[' ~ ']' <arraylist> } rule arraylist { <value> * % ',' } proto token value {*} token value:sym<number> { '-'? # optional negation [ 0 | <[1..9]> <[0..9]>* ] # no leading 0 allowed [ '.' <[0..9]>+ ]? # optional decimal point [ <[eE]> <[+-]>? <[0..9]>+ ]? # optional exponent } token value:sym<true> { <sym> } token value:sym<false> { <sym> } token value:sym<null> { <sym> } token value:sym<object> { <object> } token value:sym<array> { <array> } token value:sym<string> { <string> } token string { ｢"｣ ~ ｢"｣ [ <str> | ｢\｣ <str=.str_escape> ]* } token str { <-["\\\t\x[0A]]>+ } token str_escape { <["\\/bfnrt]> } } A `token` is a `regex` with `:ratchet` mode enabled (no backtracking). A `rule` is a `token` with `:sigspace` also enabled (whitespace becomes the same as a call to `<.ws>`). The only one of those that really looks anything like traditional regexes is the `value:sym<number>` token. (Raku replaced non capturing grouping `(?:…)` with `[…]`, and character classes `[eE]` with `<[eE]>`) This code was copied from https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Gra... but some parts were simplified to be slightly easier to understand. Mainly I removed the Unicode handling capabilities. It will generate a tree based structure when you use it. my $json = Q:to/END/; { "foo": ["bar", "baz"], "ultimate-answer": 42 } END my $result = JSON::Parser::Example.parse($json); say $result; The above will display the resultant tree structure like this: ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ value => ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ object => ｢{ "foo": ["bar", "baz"], "ultimate-answer": 42 } ｣ pairlist => ｢"foo": ["bar", "baz"], "ultimate-answer": 42 ｣ pair => ｢"foo": ["bar", "baz"]｣ string => ｢"foo"｣ str => ｢foo｣ value => ｢["bar", "baz"]｣ array => ｢["bar", "baz"]｣ arraylist => ｢"bar", "baz"｣ value => ｢"bar"｣ string => ｢"bar"｣ str => ｢bar｣ value => ｢"baz"｣ string => ｢"baz"｣ str => ｢baz｣ pair => ｢"ultimate-answer": 42 ｣ string => ｢"ultimate-answer"｣ str => ｢ultimate-answer｣ value => ｢42｣ You can access parts of the data using array and hash accesses. my @pairs = $result<value><object><pairlist><pair>; my @keys = @pairs.map( *.<string>.substr(1,*-1) ); my @values = @pairs.map( *.<value> ); say @pairs.first( *.<string><str> eq 'ultimate-answer' )<value>; # ｢42｣ You can also pass in an actions class to do your processing at the same time. It is also a lot less fragile. See https://github.com/moritz/json/blob/master/lib/JSON/Tiny/Act... for an example of an actions class. --- Note that things which you would historically talk about as a token such as `true`, `false`, and `null` are written using `token`. This is a useful association as it will naturally cause you to write shorter, more composable regexes. Since they are composable, we could do things like extend the grammar to add the ability to have non string keys. Or perhaps add `Inf`, `-Inf`, and `NaN` as values. grammar Extended::JSON::Example is JSON::Parser::Example { rule pair { <key=.value> ':' <value> } # replaces existing rule # adds to the existing set of value tokens. token value:sym<Inf> { <sym> } token value:sym<-Inf> { <sym> } token value:sym<NaN> { <sym> } } This is basically how Raku handles adding new keywords and operators under the hood.