lark

Some years ago I used the Python library Lark (https://github.com/lark-parser/lark) to parse my programming language. It lets you write your grammar in basically EBNF, builds parse trees (that you can automatically reduce by removing leaves for parentheses, for example) and generates standalone pure-Python LALR(1) parsers for your grammar. Back then, I had a blast with it, it was fast, super easy to use and just worked. Now it's got 5k stars and seems to be a mature and production-ready library. Thus, I vote Python: it has this amazing parser library, it's garbage-collected (no need to fiddle with memory allocations, like in C), it's dynamically typed (so you basically don't need to think about types at all), it now has a `match` statement (!) that makes it super easy to dissect your parse trees and abstract syntax trees.

To add to your survey, I have been reading the lark documentation https://github.com/lark-parser/lark and like the cut of it's jib, I have not used it yet as I don't really have any projects that need a full parser.

Lark supports, and recommends, writing and storing the grammar in a .lark file. We have syntax highlighting support in all major IDEs, and even in github itself. For example, here is Lark's built-in grammar for Python: https://github.com/lark-parser/lark/blob/master/lark/grammar... The rationale is that it's more terse and has less visual clutter than a DSL over Python, which makes it easier to read and write.

If you are trying out parsing in python, I would recommend giving [Lark](https://github.com/lark-parser/lark) a try. I have used it for smallish projects are its really easy to use especially with the Earley parser.

When I surveyed what was out there last year I wrote down these 3 as the most interesting: https://github.com/lark-parser/lark https://github.com/neogeny/TatSu https://github.com/dabeaz/sly Lark seems to be the one with traction.

I don't know if this is true, but I would assume that the tokenizers they used for Codex use actual language parsers which would drop invalid files like this and make this attack infeasible. When I was playing around a couple years ago with the Fastai courses in language modeling I used the Python tokenize module to feed my model, and with excellent parser libraries like Lark[0] out there it wouldn't take that long to build real quality parsers. Of course I could be totally wrong and they might just be dumping pure text in, shutter. [0]: https://github.com/lark-parser/lark

Lark [1] is a battle-tested pure-Python Earley parser. My company uses it in production. It is by far the easiest parser generator I've ever used. 1. https://github.com/lark-parser/lark

You are one of the "couple of people" I was referring to. :) I know SPARK's docstring use influenced PLY. PLY doesn't use Earley, but "Earley" does come up in the show notes of an interview with Beazley, PLY's author, at https://www.pythonpodcast.com/episode-95-parsing-and-parsers... . No transcript, and I'm not going to listen to it just to figure out the context. https://github.com/lark-parser/lark "implements both Earley(SPPF) and LALR(1)". Kegler, the author of that timeline I linked to, is the author of Marpa. Home page is http://savage.net.au/Marpa.html . The most recent HN comments about it are from a year ago, at https://news.ycombinator.com/item?id=24321395 .

I would recommend most software engineers to avoid inventing a new configuration or other DSL and the parser/lexer for it. This can easily lead to hard-to-debug program and long-term technical debt. Always research for existing and well-tested solutions first (even JSON!). Even if you don't invent your language, you can avoid writing a low-level parser/lexer by using a higher-level format, like context-free grammar (see Lark https://github.com/lark-parser/lark). Define and maintain a grammar is much easier.