arquero

In a similar vein is https://pbeshai.github.io/tidy/ which I've used for 3+ years. It's a really nice lightweight transformer. I've also used https://github.com/uwdata/arquero once (better performance for large datasets).

Somewhat related, who has played with performant JS dataframe libraries? I mostly want to serialize from pandas/polars to JS in a fast way, with some very minor selection and indexing operations JS side. Ideally I could find a library that does dataframe like operations from reading base64 encoded Buffers/Arrays into their JS objects, then doing the right things. I'm investigating https://arrow.apache.org/docs/js/ https://github.com/vega/falcon https://github.com/pola-rs/nodejs-polars https://github.com/uwdata/arquero

You can get a long way nowadays with Arquero[0] and Observable[1]. Arquero allows columnar based data storage and processing, with a grammar of data processing verbs similar to e.g. dplyr. Not as fast as vectorized computations in e.g. Python or R, but faster than has previously been possible. I'm not suggesting these are the first tools you'd reach for for data science in production, but I've found them extremely useful for prototyping, experimenting with algorithms, and visualization. I think it's got to the stage they should be seriously considered for some types of relatively simple data processing work due to their ease of deployment. [0]https://github.com/uwdata/arquero [1]https://observablehq.com/

There is a Typescript arrow library that can operate off the IPC format. As an example use case, take a look at the arquero data processing library from the University of Washington (from the same group that created D3). https://github.com/uwdata/arquero

Arrow is definitely one of the top 10 new things I'm most excited about in the data science space, but not sure I'd call it the most important thing. ;) It is pretty awesome, however, particularly for folks like me that are often hopping between Python/R/Javascript. I've definitely got in on the roadmap for all my data science libraries. Btw, Arquero from that UW lab looks really neat as well, and is supporting Arrow out of the gate (https://github.com/uwdata/arquero).

I think the Vega team would agree with you since it seems to me like they are doing exactly that with Arquero (https://github.com/uwdata/arquero). From my experience it's the right architecture. You cannot have a great data visualization library without a great data transformation library, but the data transformation functions should be at lower level; not provided by the visualization layer.

This looks fantastic. In the past I've often used bits and pieces from lodash and d3 to do these sort of data manipulations. Anyone know if there are any performance comparisons anywhere? I know that in the past Mike Bostock has suggested array processing may be a lot faster[0], and on the Github [1] it mentions it's designed to be fast. I really think this, in combination with other parts of the ecosystem like vega can be a game changers for how people use open data. One obstacle to get government to start publishing high quality machine readable open data has been that relatively few users want it - most seem stuck in the world of Excel. With tools like this and Observable, the task of loading it in and charting it become a few simple commands that work even with large datasets. This in turn makes the case for publishing it much st

The Github repo, for anyone who's interested: https://github.com/uwdata/arquero