arquero

I think the Vega team would agree with you since it seems to me like they are doing exactly that with Arquero (https://github.com/uwdata/arquero). From my experience it's the right architecture. You cannot have a great data visualization library without a great data transformation library, but the data transformation functions should be at lower level; not provided by the visualization layer.

This looks fantastic. In the past I've often used bits and pieces from lodash and d3 to do these sort of data manipulations. Anyone know if there are any performance comparisons anywhere? I know that in the past Mike Bostock has suggested array processing may be a lot faster[0], and on the Github [1] it mentions it's designed to be fast. I really think this, in combination with other parts of the ecosystem like vega can be a game changers for how people use open data. One obstacle to get government to start publishing high quality machine readable open data has been that relatively few users want it - most seem stuck in the world of Excel. With tools like this and Observable, the task of loading it in and charting it become a few simple commands that work even with large datasets. This in turn makes the case for publishing it much st

The Github repo, for anyone who's interested: https://github.com/uwdata/arquero