greenmask

ation software. Having worked as a database engineer for 8 years, I frequently struggled with routine tasks like setting up development environments—this was a common request. Initially, I used custom scripts to handle this, but things became increasingly complex as the number of services grew, especially with the rise of microservices architecture. When I began exploring tools to solve this issue, I listed my key expectations for such software: documentation; type safety (the tool should validate any changes to the data); streaming (I want the ability to stream the data while transformations are being applied); consistency (transformations must maintain constraints, functional dependencies, and more); reliability; customizability; interactivity and usability; simplicity. I found a few options, but none fully met my expectations. Two interesting tools I discovered were pganonymizer and replibyte. I liked the architecture of Replibyte, but when I tried it, it failed due to architectural limitations. With these thoughts in mind, I began developing Greenmask in mid-2023. My goal was to create a tool that meets all of these requirements, based on the design principles I laid out. Here are some key highlights: * It is a single utility - Greenmask delegates the schema dump to vendor utilities and takes responsibility only for data dumps and transformations. * Database Subset (https://docs.greenmask.io/latest/database_subset) - specify the subset condition and scale down size. We did a deep research in graph algorithms and now we can subset almost any complexity of database. * Database type safety - it uses the DBMS driver to decode and encode data into real types (such as int, float, etc.) in the stream. This guarantees consistency and almost eliminates the chance of corrupted dumps. * Deterministic engine (https://docs.greenmask.io/latest/built_in_transformers/trans...) - generate data using the hash engine that produces consistent output for the same input. * Dynamic parameters for transformers (https://docs.greenmask.io/latest/built_in_transformers/dynam...) - imagine having created_at and updated_at dates with functional dependencies. Dynamic parameters ensure these dates are generated correctly. We are actively maintaining the current project and continuously improving it—our public roadmap at https://github.com/orgs/GreenmaskIO/projects/6. Soon, we will release a Python library along with transformation collections to help users develop their own complex transformations and integrate them with any service. We have plans to support more database engines, with MySQL being the next one, and we are working on tools which will integrate seamlessly with your CI/CD systems. To get started, we’ve prepared a playground for you that can be easily set up using Docker Compose: https://docs.greenmask.io/latest/playground/ I’d love to hear any questions or feedback from you!

es the following features: * Database Subset (https://docs.greenmask.io/latest/database_subset) * Validate transformation and DB schema changes (https://docs.greenmask.io/latest/commands/validate) * Variety of transformers (https://docs.greenmask.io/latest/built_in_transformers/stand...) * Deterministic engine (https://docs.greenmask.io/latest/built_in_transformers/trans...) * Dynamic parameters for transformers (https://docs.greenmask.io/latest/built_in_transformers/dynam...) * Extensible (https://docs.greenmask.io/latest/built_in_transformers/advan...) * Backups retention management (https://docs.greenmask.io/latest/commands/delete) * PostgreSQL 17 compatibility * Database type safe (it controls type encode-decode ops) * Parallel execution (reduces dump/restore duration) You can find the full documentation here: https://docs.greenmask.io If you're wondering how to get started, we’ve prepared a playground for you that can be easily set up using Docker Compose: https://docs.greenmask.io/latest/playground/ Our public roadmap is available here: https://github.com/orgs/GreenmaskIO/projects/6 Join our Discord community to discuss and help shape the future of the Greenmask project: https://discord.com/invite/rKBKvDECfd I’d love to hear any questions or feedback from you!

* Validation: Powerful validation commands allow you to check transformation differences, constraint violations, and schema changes. * Highly Customizable Transformation: For instance, customizable transformations such as random email generation (https://greenmask.io/v0.2beta1/built_in_transformers/standar...). * Database Type Safety: Integrated with the pgx driver, ensuring accurate value encoding/decoding. * Dynamic Parameters: Allows resolution of functional dependencies between columns (https://greenmask.io/v0.2beta1/built_in_transformers/dynamic...). * Deterministic and Random Engines: Offers both deterministic and random transformation engines (https://greenmask.io/v0.2beta1/built_in_transformers/transfo...). * Single Utility: Easy to set up and run. * Schema Dumping and Restoration: Delegates these tasks to PostgreSQL's native utilities, as they are the most reliable. * Backward Compatibility: Dumps created by Greenmask can be successfully restored by pg_restore. * Large Objects Support: Handles the dumping and restoration of LargeObjects (https://www.postgresql.org/docs/current/largeobjects.html). * Control Schema Changes: Checks your schema changes against the previous snapshot, prints the differences, and returns a non-zero exit code if changes are detected. This feature can be helpful in your CI/CD pipelines. There is definitely a lot of work ahead. Have a look at the planned features in our Roadmap (https://github.com/orgs/GreenmaskIO/projects/6). We have prepared a playground sandbox for experiments and for evaluating the Greenmask utility. You can access it here: (https://greenmask.io/v0.2beta1/). If you want to run a Greenmask playground for the beta version execute: git checkout tags/v0.2.0b1 -b v0.2.0b1 docker-compose run greenmask-from-source I would appreciate feedback.

o/dbzar/ * https://github.com/Qovery/Replibyte