Replibyte

Great to see such a project. We are using datanymizer [1] right now but it has gone unmaintained and we are using my patched version [2] and it is working pretty well for us. I saw a new project that is getting close in terms of having the feature set I need and has them on their roadmap [3]. To ensure that we are marking columns as PII, we run a job that compares the anonymization configuration to a comment on the column- we have a comment on every column to mark it as PII (or not). [1] https://github.com/datanymizer/datanymizer [2] https://github.com/digitalmint/datanymizer/tree/digitalmint [3] https://github.com/GreenmaskIO/greenmask Other tools I found that do some anonymization but didn't meet my needs: * https://github.com/DivanteLtd/anonymizer * https://postgresql-anonymizer.readthedocs.io/en/stable * https://nitzano.github.i

I like rust binaries because they are cross-platform. I've found cool ones like - https://github.com/static-web-server/static-web-server/ - https://github.com/Qovery/Replibyte If you know any repos where I may find compilation of tools I'll be thankful!

Maybe you're looking for this: https://github.com/Qovery/Replibyte.

Fun to see that we share some very similar ideas with Replibyte - an open source tool to seed a dev database with real data. https://github.com/Qovery/replibyte

Thanks: 1. What do you mean it is not safe? 2. Telemetry can be removed with the option --no-telemetry and you can inspect the code > https://github.com/Qovery/replibyte/blob/main/replibyte/src/telemetry.rs

That was the first thing I checked whether it supports data masking. And it does, via transforms, as evoxmusic already pointed out. However, there may be times when data masking must be nuanced. Suppose some random email/domain pair is bad and you would rather replace all "example.com" domain instances with "fake.com", and not "random1.com", "random2.com", etc (for ML, 3rd party random analysis). Out of the box I don't see it is provided, HOWEVER I see that you can write custom transformer: https://github.com/Qovery/replibyte/tree/main/examples/wasm and fulfill your needs. Excellent :)

That's why RepliByte uses S3 to store transformed data. Then the real production data never leave the production environment. https://github.com/Qovery/replibyte/blob/main/docs/DESIGN.md

You can use env vars in the conf yaml file -> https://github.com/Qovery/replibyte/blob/main/examples/sourc...

I was thinking "oh! this is awesome!", and then noticed it didn't support MSSQL. Not to worry, I'll just contribute a connector. Let's take a look at their existing connector code... https://github.com/Qovery/replibyte/blob/main/replibyte/src/... Not a single comment to say what anything does. Sigh. It's the same for the other drivers too.

- Works on large database (> 10GB) (read Design) Can anyone explain to me how this works in RepliByte? The design document only talked about Postgres. For example let's say I have a MySQL database, how does RepliByte copy that database into S3? Does it use mysqldump or are they coping the database index files? We have a script that automatically backs up our production database at intervals to S3 and then a program to download the latest backup and scrub the data. It takes a heck of a long time to download and impacts the server when it happens... it's been on my todo list to replace with Percona's Xtrabackup [1] but doesn't look that's what these guys are doing? - Database Subsetting: Scale down a production database to a more reasonable size What about this? Does the database need foreign keys to prevent related rows in tables being

Please don't require static AWS credentials: https://github.com/Qovery/replibyte/blob/v0.4.4/replibyte/sr... or at least either include "AWS_SESSION_TOKEN" in that setup (if it is present) in order to allow "aws sts assume-role" to work, or allow `AWS_PROFILE`, or just use the aws-sdk's normal credential discovery mechanism which at least on their "main" SDKs is a fallback list of them, but I couldn't follow the docs.rs soup in order to know if their rust sdk is up to speed or what

A bunch of developers and myself have created RepliByte - an open-source tool to seed a development database from a production database. Features: - Support data backup and restore for PostgreSQL, MySQL and MongoDB - Replace sensitive data with fake data - Works on large database (> 10GB) (read Design) - Database Subsetting: Scale down a production database to a more reasonable size - Start a local database with the prod data in a single command - On-the-fly data (de)compression (Zlib) - On-the-fly data de/encryption (AES-256) - Fully stateless (no server, no daemon) and lightweight binary - Use custom transformers My motivation: As a developer, creating a fake dataset for running tests is tedious. Plus, it does not reflect the real-world data and painful to keep updated. If you prefer to run your app tests with product