fg-data-profiling

WhatTheDuck does SQL with duckdb-wasm Pygwalker does open-source descriptive statistics and charts from pandas dataframes: https://github.com/Kanaries/pygwalker ydata-profiling does open-source Exploratory Data Analysis (EDA) with Pandas and Spark DataFrames and integrates with various apps: https://github.com/ydataai/ydata-profiling #integrations, #use-cases

Hey guys, We've been recently at the Dask Demo Day and we're hoping to launch a new feature on ydata-profiling, with the support for Dask dataframes! We're looking for Dask Wizards to start collaborating on this feature, so if you're interested, please join us to define the roadmap of the project and start making it real Current GitHub branch is here: https://github.com/ydataai/ydata-profiling/tree/feat/dask Dedicated dask channel here: https://discord.gg/EHDBuSSDuy

For those who are going through the thread finding new tools: pandas-profiling[0] is a library for automatic EDA (which bamboolib[1], mentioned elsewhere, also does). [0]: https://github.com/pandas-profiling/pandas-profiling [1]: https://bamboolib.com/

Visions is a python library for working with user defined data type systems. Out of the box, it provides type inference and automated data cleaning of sequence data with backend specific implementations for pandas, spark, python, and numpy. We often use it as a first pass cleaning step when working with tabular data and to simplify the backend logic of both pandas-profiling[1] and our tabular data compression library compressio[2]. Because data types are user defined, we can build user customizable libraries based around types without adding code complexity. In the case of compressio that means offering users the ability to use any compression algorithm they want by simply passing a dictionary mapping `{type: compression algorithm}` or defining new compression algorithms for otherwise unsupported data types like shapely geometries, images

This is really cool! I'm glad there's more work going into the area - visions[1] is a similar tool which embeds the type system (user defined schema) into a traversable graph rather than encoding all type information in the trained network. We originally wrote it as the backend for pandas-profiling[2]. I'm not sure if any of the authors are on here but y'all might look into Sherlock[3] as well. They've got pre-trained models for many other semantic types than you've currently implemented. 1. https://github.com/dylan-profiler/visions 2. https://github.com/pandas-profiling/pandas-profiling 3. https://sherlock.media.mit.edu/

Great concept, although worried about uploading datasets with sensitive data to your cloud. I usually use this for Pandas Profiling to accomplish EDA tasks https://github.com/pandas-profiling/pandas-profiling

Hi, I'm working on this if anyone is interested (work in progress): https://github.com/dylan-profiler/visions The key idea is to allow abstraction of physical types (boolean, integers) by defining custom semantic types (URLs, paths, probabilities). The idea originated while working on pandas-profiling [0] and running into similar problems. We found this abstraction to be effective for many other downstream tasks, too, including compression and AutoML. More coming soon... [0]: https://github.com/pandas-profiling/pandas-profiling

In the past, I've looked at OpenRefine and Jupyter integration. Once I've learned to do data transformation with pandas and sklearn with code, I'll report back to you. Pandas-profiling has a number of cool descriptive statistics features as well. https://github.com/pandas-profiling/pandas-profiling There's a new IterativeImputer in Scikit-learn 0.22 that it'd be cool to see visualizations of. https://twitter.com/TedPetrou/status/1197150813707108352 https://scikit-learn.org/stable/modules/impute.html A plugin model would be cool; though configuring the container every time wouldn't be fun. Some ideas about how we could create a desktop version of binderhub in order to launch REES-compatible environments on our own resources: https://github.com/westurner/nbhandler/issues/1

For nice looking basic statistics of your dataset the pandas-profiling library will do the trick https://github.com/JosPolfliet/pandas-profiling

For nice looking basic statistics of your dataset the pandas-profiling library will do the trick https://github.com/JosPolfliet/pandas-profiling