siuba

https://github.com/machow/siuba for one

Hey--I maintain a port of dplyr to python, called siuba[1]! Right now it supports a decent number of verbs + SQL generation. I tried to break down why R users find pandas difficult in an RStudioConf talk last year[2]. Between siuba and tools like polars and duckdb, I'm hopeful that someone hits the data analysis sweet spot for python in the next couple years. [1]: http://github.com/machow/siuba [2]: https://youtu.be/w4Mi0u4urbQ

For further inspiration, this is a pretty good-looking "dplyr for Python": https://github.com/machow/siuba There's precedent in Ecto for a "magic" operator (`^` if I'm not mistaken), so it wouldn't be a stretch to implement it here as well.

This is not a new idea. siuba (a dplyr-like Pandas alternative) is one project that does it, and it also has a "pipeline operator": https://github.com/machow/siuba (mtcars >> group_by(_.cyl) >> summarize(avg_hp = _.hp.mean()) ) (Edit: ok, it seems siuba doesn't have the "parameter fixing" thing.) Related: Coconut - a functional superset of Python - https://github.com/evhub/coconut range(10) |> map$(pow$(?, 2)) |> list There were others but I can't remember which now.

See 'https://github.com/machow/siuba'.

Author here--happy to answer questions :) Siuba has come a long way since I wrote this, and now can optimize for fast grouped operations!: * https://github.com/machow/siuba * https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

For what it's worth, I maintain a library called siuba that lets you generate SQL code from pandas methods. It's crazy to me how people use SELECT * -> pandas, but also how people in SQL type a ton of code over and over. https://github.com/machow/siuba

Hey, thanks for pointing out Self--I definitely need to dig into fastcore more! One motivation for developing siuba is that the grouped agg you show requires users specify only one operation on one column. E.g. 1. Calculate mean of x However, common operations like demeaning a column are multiple operations: 1. Calculate mean of x 2. Subtract result of (1) from x In siuba you can just write mutate(res = _.x -_.x.mean()). This isn't possible from something like gdf.x.agg("mean"), and from what I can tell deeply confusing to analysts :/. In vanilla pandas I really like to use the chaining method you laid out, and siuba to me is mostly a utility library for making the approach a little more succinct / performant[1]. siuba has experimental autocompletion (thanks to Tim Mastny!), and there's a pretty hefty technical write up on how it uses IPy

Interesting approach! I've been working on it from the other angle: having pandas code generate SQL. If you're interested in checking it out, happy to try and show how it would generate the query in your readme! https://github.com/machow/siuba One thing I was wondering about the gnarly ast stuff you mention, what about operator overloading? E.g. Q("Select a from" + subquery + "where a < 1")

Working on siuba, a data analysis tool for python. It's a port of the R library dplyr, and can produces SQL queries! I've programmed in python for much longer than R, and really want to be able to move at the same speed when using python for data analysis :o. It's a weird problem though because the two languages have basically opposite approaches to DataFrames. pandas has a very fat DataFrame implementation, R an extremely minimal one. (Pros and cons to both approaches). https://github.com/machow/siuba

Articles like these are interesting, but what surprises me is that they rarely set up a holistic use case, so most debates imagine how long it would take an expert user to use each tool. But time constraints (eg spent coding) separates expert from novice performance in many domains. FWIW I have 2020 set aside to implement siuba, a python port of the popular R library dplyr (siuba runs on top of pandas, but also can generate SQL). A huge source of inspiration has been screencasts by Dave Robinson using R to analyze data he's never seen before lightning fast. Has anyone seen similar screencasts with pandas? I suspect it's not possible (given some constraints on its interface), but would love to be wrong here, because I'd like to keep all my work in python :o. Expert R screencasts: https://youtu.be/NY0-IFet5AM Siuba: https://github.com/macho

porting the R library dplyr to python. :) (I set aside 2020 to work on this, but quarantine definitely created much more time for it) https://github.com/machow/siuba

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4edcc5541ff3f6470f... And then, for the ubiquitous Object dtype, often figure out which of the many possible more specific types to cast it to. If you think that is easy, ask yourself what this outputs: import numpy as np np.array([np.nan, 'a']) Lo and behold--it produces an array where the np.nan has been converted to the string "nan". And yet import pandas as pd pd.Series([np.nan, "a"]) Knows this, has your b

I've been working on a library over the past year that does exactly that, including generating dbplyr style SQL queries! Would love your feedback :) https://github.com/machow/siuba