super

Maybe integrate with https://github.com/brimdata/zed ?

Your observations are definitely valid. Structured errors were introduced in Zed some time ago, but as you highlight, they're not yet being generated in all the places they could be and with appropriate detail. There were a couple existing to-do items in the Zed backlog to make these enhancements in packages where it would have a high impact, but we've now created an additional Epic https://github.com/brimdata/zed/issues/4608 to gather these up and remind ourselves to address this in a comprehensive way. Thanks for helping to shine a light on this!

Cool, I didn't realize you used sqlite-utils for your performance demo! It's not particularly designed for speed - it should be fast as far as Python code goes (I use some generator tricks to stream data and avoid having to load everything into memory at once) but I wouldn't expect "sqlite-utils insert" to win any performance competitions with tools written in other languages. Those benchmarks against sqlite itself are definitely interesting. I'm looking forward to playing with the "native ZNG support for Python" mentioned on https://github.com/brimdata/zed/blob/main/docs/libraries/pyt... when that becomes available.

Hi, all. Author here. Thanks for all the great feedback. I've learned a lot from your comments and pointers. The Zed project is broader than "a jq alternative" and my bad for trying out this initial positioning. I do know there are a lot of people out there who find jq really confusing, but it's clear if you become an expert, my arguments don't hold water. We've had great feedback from many of our users who are really productive with the blend of search, analytics, and data discovery in the Zed language, and who find manipulating eclectic data in the ZNG format to be really easy. Anyway, we'll write more about these other aspects of the Zed project in the coming weeks and months, and in the meantime, if you find any of this intriguing and want to kick the tires, feel free to hop on our slack with questions/feedback or file GitHub issue

And it's BSD 3 Clause, for those interested: https://github.com/brimdata/zed/blob/v1.0.0/LICENSE.txt

Closed source? https://github.com/brimdata/zed/blob/main/runtime/query.go Yes, it’s an obscure query language. But if you were interested in jq, that clearly wasn’t a barrier to entry. I agree the author is happy to show off their tool, but disagree that that is somehow disqualifying. They made a cool thing, they’re allowed to be proud about it.

There are a few examples in the ZSON spec... https://github.com/brimdata/zed/blob/main/docs/formats/zson.... And you can easily see whatever data you'd like formatted as ZSON using the "zq" CLI tool, but I just made this gist (with some data from the brimdata/zed-sample-data report) so you can have a quick look (the bstring stuff is a little noisy and an artifact of the data source being Zeek)... https://gist.github.com/mccanne/94865d557ca3de8abfd3eb09e8ac...

This is a very real problem being addressed here and I am intrigued by all the great comments in this thread. In the Zed project, we've been thinking about and iterating on a better data model for serialization for a few years, and have concluded that schemas kind of get in the way (e.g., the way Parquet, Avro, and JSON Schema define a schema then have a set of values that adhere to the schema). In Zed, a modern and fine-grained type system allows for a structure that is a superset of both the JSON and the relational models, where a schema is simply a special case of the type system (i.e., a named record type). If you're interested, you can check out the Zed formats here... https://github.com/brimdata/zed/tree/main/docs/formats

We're still early in development, but the ZNG format is one we're working on for schema-less, semi-structured, zero-copy operations: https://github.com/brimsec/zq/blob/master/zng/docs/spec.md#1... We currently use this to store security log data, but think it's an interesting midpoint between having no schema at all vs requiring schema registries to do useful work.