presidio

Our solve is to allow it to work with a local dev database and it's output is a script. Then that script gets checked into version control (auditable and reviewed). Then that script can be run against production. Slower iteration but worth the tradeoff for us. Giving LLM even read access to PII is a big "no" in my book. On PII, if you need LLMs to work on production extracted data then https://github.com/microsoft/presidio is a pretty good tool to redact PII. Still needs a bit of an audit but as a first pass does a terrific job.

I wonder if this would have been useful https://github.com/microsoft/presidio - its heavy but looks really good. There is a lite version..

Like many of you, we've spent the last two years dealing with requests to sprinkle LLM-powered features everywhere. One recurring problem we faced in almost every project was related to data governance. In most cases, it was extremely hard to implement the needed granular control over the retrieved data. We developed custom solutions each time, but when we realized most of the solutions could be reused in subsequent projects, we started thinking about creating a modular framework. Today we're releasing that framework! We've already built some modules using open-source solutions such as Microsoft Presidio (1) or custom fine-tuned models like our own Financial BERT (2), but the goal is to let you play with it and add the modules you need for your own use case. Leave a star on GitHub if you find this problem interesting, join our community i

I wonder what SOTA open-source PII stripping libraries there are? Something like https://github.com/microsoft/presidio for stripping out PII might fill the role I expected https://github.com/capeprivacy/private-ai to do.

Hey, I'm one of the maintainers of Presidio OSS. We built this tool mainly for Microsoft's strategic customers and decided to open-source it for others to use. No hidden agenda here. The engine has two main advantages: It's easily expandable and customizable, and it works well at scale. We do know of organizations who are very close to production with it. Every organization has their own requirements for PII entities, many of them specific to the org itself, so the engine allows a developer to easily add support for new PII entities using code, regex or black-lists. See hree: https://github.com/microsoft/presidio/blob/master/docs/custo... As for the productization aspect, we did some performance tests and are confident with taking it to production. Single instance cluster with a medium size machine has a ~65ms response time for a 100 word

So you might have more fun with this email one https://github.com/microsoft/presidio/blob/master/presidio-a... :) ... seriously, they should have broken that down, or used the multi-line with comments pattern, if they really want people to contribute to this.

They also have a NLP engine, the regexes are just predefined recognizers for known formats. https://github.com/microsoft/presidio/tree/master/presidio-a...

answering myself, yes. It is a pile of regex https://github.com/microsoft/presidio/blob/master/presidio-a... and i can't find the body of test data easily... that's not good.

To be fair, there are different levels of detection [1], the demo is probably using the weakest one. [1] https://github.com/microsoft/presidio/blob/74ea983cc50ff76d7...

License: MIT Demo: https://presidio-demo.westeurope.cloudapp.azure.com/ Documentation: https://github.com/microsoft/presidio/blob/master/docs/index...