DataProfiler

My team built a similar tool in Python to load any delimited file, json, parquet and Avro with one command: https://github.com/capitalone/DataProfiler Effectively loads anything into a dataframe. After that you can profile the data, merge, save, load, and take differences between profiles before generating a report.

The thing is, tensorflow has more ability to run cross platform. I help maintain https://github.com/capitalone/DataProfiler Our sensitive data detection library is exported to iOS, android, and Java; in addition to Python. We also run distributed and federated use cases with custom layers. All of which are improved in tensorflow. That said, I’d use pytorch if I could. Simply put, it has a better user experience.

I really should write up how we did delimiter and quote detection in this library: https://github.com/capitalone/DataProfiler It turns out delimited files IMO are much harder to parse than say, JSON. Largely because they have so many different permutations. The article covers CSVs, but many files are tab or null separated. We’ve even seen @ separated with ‘ for quotes. Given the above, it should still be possible to use the method described. I’m guessing you’d have to detect the separators and quote chars first, however. You’d have to also handle empty rows and corrupted rows (which happen often enough).

Having used both ray, dask, and writing custom threads, my personal view is that while there are advantages I wouldn’t want to use any of these unless absolutely necessary. My personal approach for most of these tasks are to try to break down the problem to be as asynchronous as possible. Then you can create threads. The nice thing about dask is really the way you can effectively use it on pandas dataframe; Without any large overhead. Having said that, we opted to write our own parallelization for this library: https://github.com/capitalone/DataProfiler As opposed to using the dask frame. Effectively, the dataframe processing wasn’t the bottleneck and easier to maintain the threading ourselves given the particular approaches taken. That said, if I was working with large pandas dataframes, id likely use dask. For large datasets which coul

My team has worked on a library for a similar purpose: https://github.com/capitalone/DataProfiler Load any document, profile and monitor the profiles for changes that would impact downstream applications. Very common problem, you all are in a great space! Very interested and will check out!

It would be possible to build a similar system via a library my team has built: https://github.com/capitalone/dataprofiler Effectively, you can monitor changes between profiles: data1 = dp.Data("file_a.csv") # Load a CSV file profile1 = dp.Profiler(data1) # Generate a profile data2 = dp.Data("file_b.csv") # Load another CSV file profile2 = dp.Profiler(data2) # Generate another profile diff_report = profile1.diff(profile2) print(json.dumps(diff_report, indent=4)) The system we have generates reports, it might be worth adding it OP.

Not exactly the same, but we wrote a library to easily load any delimited type of file and finds header (even if not first row). It also works to load JSON, Parquet, AVRO and loads it into a dataframe. Not CLI exactly, but pretty easy: https://github.com/capitalone/dataprofiler Can Install: pip install dataprofiler[ml] --user How it works: csv_data = Data('your_file.csv') # Load: delimited, JSON, Parquet, Avro csv_data.data.head(10) # Get head csv_data.data.sort_values(by='name', inplace=True) # Sort Anyway, pretty interesting Miller CLI. I'm not sure how the header detection works, especially if the header isn't the first row (which is often the case)

Having built a model to identify sensitive data having a solid data labeling solution would be awesome. I can attest this is a real problem. Here's the library we built: https://github.com/capitalone/DataProfiler In this space, Prodigy really dominates: https://prodi.gy/ We actually built our own internal system which integrates and can export the labels (does predictive labeling, etc). Of course, we only focused on text data at the moment. All that being said, this is going to become a crowded space and highly competitive. Plus, once the data is labeled companies often drop their labelers. I would recommend ensuring some consistent use. Potentially, hosting their models off-prem or something to lock companies in.

Telm.ai (YC S21) - Real-time data quality monitoring Looks interesting! I worked on https://github.com/capitalone/DataProfiler We are looking to monitor correlation changes over time, see if sensitive data gets entered, track schema changes, etc and see the impact of down stream modeling, etc I'm curious how heavy the input is? because usually these systems take a lot of effort to setup. Any idea?

My team and I wrote an NLP application to detect sensitive data and detect / validate schemas, etc as well as the other items provided by pandas-profiling. https://github.com/capitalone/DataProfiler That being said, we noted the same thing. It shouldn't matter what modeling you use. It's the data pipelining where 99% of the work typically is. Modeling itself always needs the same basic input -- matrix of data and outputs a matrix of data. Some libraries are good at specific components. Others have improved speeds ups, etc. But it's all so new it's effectively going to change month-to-month. So I always tell the team to build what you can as fast as you can, with the tools you have. We can always update it later, once the pipeline is in place.

We built a similar tool, utilizing a CNN. It works on structured (and unstructured) data and provides additional info. https://github.com/capitalone/DataProfiler Cool part, is you can “extend” the intern name-entity recognition model by refitting with the new data. Out if the box, the DataProfiler does something like 18 entities including most of the PII dada.

This looks pretty awesome! Something that might be cool is also identifying the schemas and the ranges. You would probably be able to automatically generate some of these blocks. You could use something like this: https://github.com/capitalone/DataProfiler With enough data, you could probably further automate this process with AI. Good luck! Feel free to reach out for ideas or anything.

Hi HN, I am looking for the most accurate tool which I can use to infer Data Types from Tabular data (csv,tsv,excel) I need to be able to perform some small customization, if possible, to the detection algorithm. For example if I have a 9 digit number, starting with 0, then treat it as a String. So far - I have found Frictionless Framework [0] which seems good, but I can't see any way of specifying customizations to the profiling algorithm, and Data Profiler [1] which uses ML for type detection, and it seems I should be able to train some new rules but I need a CUDA capable machine, which at the moment I do not have. Hoping the collective HN brain can point me to something better if it exists. [0] - https://framework.frictionlessdata.io/ [1] - https://github.com/capitalone/DataProfiler

Hope this helps someone: https://github.com/capitalone/DataProfiler We’re working to automate much of that as well in a Python library. The end goal is to: 1. point at any dataset and load it with one command 2. Calculate statistics and identify entities with one command 3. Generate robust reports with one command. Regarding the data wrangling... Believe it or not, even automatically detecting a delimited file with a header is hard work. Imagine a header can be on the 3rd row and has a title and author ship one rows 1 and 2 respectively. Further, the delimiter might be the “@“ symbol! The linked library wrote handles that scenario. But that’s just CSVs, there’s also Json, parquet, Avro, etc etc... This is an extraordinarily deep and complex field.