visions

Visions is a python library for working with user defined data type systems. Out of the box, it provides type inference and automated data cleaning of sequence data with backend specific implementations for pandas, spark, python, and numpy. We often use it as a first pass cleaning step when working with tabular data and to simplify the backend logic of both pandas-profiling[1] and our tabular data compression library compressio[2]. Because data types are user defined, we can build user customizable libraries based around types without adding code complexity. In the case of compressio that means offering users the ability to use any compression algorithm they want by simply passing a dictionary mapping `{type: compression algorithm}` or defining new compression algorithms for otherwise unsupported data types like shapely geometries, images

This is really cool! I'm glad there's more work going into the area - visions[1] is a similar tool which embeds the type system (user defined schema) into a traversable graph rather than encoding all type information in the trained network. We originally wrote it as the backend for pandas-profiling[2]. I'm not sure if any of the authors are on here but y'all might look into Sherlock[3] as well. They've got pre-trained models for many other semantic types than you've currently implemented. 1. https://github.com/dylan-profiler/visions 2. https://github.com/pandas-profiling/pandas-profiling 3. https://sherlock.media.mit.edu/

Hi, I'm working on this if anyone is interested (work in progress): https://github.com/dylan-profiler/visions The key idea is to allow abstraction of physical types (boolean, integers) by defining custom semantic types (URLs, paths, probabilities). The idea originated while working on pandas-profiling [0] and running into similar problems. We found this abstraction to be effective for many other downstream tasks, too, including compression and AutoML. More coming soon... [0]: https://github.com/pandas-profiling/pandas-profiling