DataProfiler

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

bug fixes and adding static typing to the library too

newest release of the dataprofiler, which includes a big addition for profiling graph structured data

I spent some time working on something called DataProfiler python library https://github.com/capitalone/DataProfiler The gist is that you can point to any common dataset and load it directly into pandas. from dataprofiler import Data data = Data("your_file.csv") # Auto-Detect & Load: CSV, AVRO, Parquet, JSON, Text, URL I simply hate dealing with loading data, so it's my go-to.

My team built a similar tool in Python to load any delimited file, json, parquet and Avro with one command: https://github.com/capitalone/DataProfiler Effectively loads anything into a dataframe. After that you can profile the data, merge, save, load, and take differences between profiles before generating a report.

The thing is, tensorflow has more ability to run cross platform. I help maintain https://github.com/capitalone/DataProfiler Our sensitive data detection library is exported to iOS, android, and Java; in addition to Python. We also run distributed and federated use cases with custom layers. All of which are improved in tensorflow. That said, I’d use pytorch if I could. Simply put, it has a better user experience.

I really should write up how we did delimiter and quote detection in this library: https://github.com/capitalone/DataProfiler It turns out delimited files IMO are much harder to parse than say, JSON. Largely because they have so many different permutations. The article covers CSVs, but many files are tab or null separated. We’ve even seen @ separated with ‘ for quotes. Given the above, it should still be possible to use the method described. I’m guessing you’d have to detect the separators and quote chars first, however. You’d have to also handle empty rows and corrupted rows (which happen often enough).

Having used both ray, dask, and writing custom threads, my personal view is that while there are advantages I wouldn’t want to use any of these unless absolutely necessary. My personal approach for most of these tasks are to try to break down the problem to be as asynchronous as possible. Then you can create threads. The nice thing about dask is really the way you can effectively use it on pandas dataframe; Without any large overhead. Having said that, we opted to write our own parallelization for this library: https://github.com/capitalone/DataProfiler As opposed to using the dask frame. Effectively, the dataframe processing wasn’t the bottleneck and easier to maintain the threading ourselves given the particular approaches taken. That said, if I was working with large pandas dataframes, id likely use dask. For large datasets which coul

My team has worked on a library for a similar purpose: https://github.com/capitalone/DataProfiler Load any document, profile and monitor the profiles for changes that would impact downstream applications. Very common problem, you all are in a great space! Very interested and will check out!

It would be possible to build a similar system via a library my team has built: https://github.com/capitalone/dataprofiler Effectively, you can monitor changes between profiles: data1 = dp.Data("file_a.csv") # Load a CSV file profile1 = dp.Profiler(data1) # Generate a profile data2 = dp.Data("file_b.csv") # Load another CSV file profile2 = dp.Profiler(data2) # Generate another profile diff_report = profile1.diff(profile2) print(json.dumps(diff_report, indent=4)) The system we have generates reports, it might be worth adding it OP.

Not exactly the same, but we wrote a library to easily load any delimited type of file and finds header (even if not first row). It also works to load JSON, Parquet, AVRO and loads it into a dataframe. Not CLI exactly, but pretty easy: https://github.com/capitalone/dataprofiler Can Install: pip install dataprofiler[ml] --user How it works: csv_data = Data('your_file.csv') # Load: delimited, JSON, Parquet, Avro csv_data.data.head(10) # Get head csv_data.data.sort_values(by='name', inplace=True) # Sort Anyway, pretty interesting Miller CLI. I'm not sure how the header detection works, especially if the header isn't the first row (which is often the case)

Having built a model to identify sensitive data having a solid data labeling solution would be awesome. I can attest this is a real problem. Here's the library we built: https://github.com/capitalone/DataProfiler In this space, Prodigy really dominates: https://prodi.gy/ We actually built our own internal system which integrates and can export the labels (does predictive labeling, etc). Of course, we only focused on text data at the moment. All that being said, this is going to become a crowded space and highly competitive. Plus, once the data is labeled companies often drop their labelers. I would recommend ensuring some consistent use. Potentially, hosting their models off-prem or something to lock companies in.