neosync

aset. At least the way we were thinking about it. [1]: https://github.com/nucleuscloud/neosync

added new integrations. In addition to supporting Postgres and Mysql, we’re introducing first class support for DynamoDB, MongoDB and SQL Server. You can also sync to object storage like S3 and GCP Cloud storage. Next, we’ve completely revamped our transformers. Transformers are how you anonymize sensitive data and generate new data. We’ve added new Transformers that you can use out of the box or you can write your own custom one in javascript. We’ve added real time validation and the ability to combine transformers together to create your own anonymization scheme. We’ve also added in new features to make Neosync easier to use. For example, the ability to automatically map transformers to your schema. The ability to only append new records instead of a full refresh. And to stop jobs from running when the schema changes. We've also upgrade our AI Synthetic Data features. You can use any LLM to generate synthetic data and Neosync will handle the orchestration between your database and the LLM. Lastly, we’re also announcing Neosync Cloud. Our hosted platform that allows you to use Neosync without having to run any of the infrastructure yourself. All you have to do is connect your source and destination databases(s), configure your schema and you’re done. Of course, you can use Neosync Open Source on-prem and hundreds of companies do. Neosync is written in Go and Typescript and can be started locally with a single make command. We'd love any feedback you have and contributions are always welcome.

ction errors would take a long time and would often resurface. When we see a bug in production, the first thing we want to do is reproduce it locally, but if we can’t reproduce the state of the data locally, then we’re kind of flying blind. Working directly with production data would solve both of these problems but most teams can’t because of: (1) privacy/security issues and (2) scale. So we set out to solve these two problems with Neosync. We solve the privacy and security problem using anonymization and synthetic data. We have 40+ pre-built transformers (or you can write your own in code) that can anonymize PII or sensitive data so that it’s safe to use locally. Additionally, you can generate synthetic data from scratch that fits your existing schema across your database. The second problem is scale. Some production databases are too big to fit locally or just have more data than you need. Also, in some cases, you may want to debug a certain customer’s data and you only want their data. We solve this with subsetting. You can pass in a SQL query to filter your table(s) and Neosync will handle all of the heavy lifting including referential integrity. At the core of Neosync does three things: (1) It streams data from a source to one or multiple destination databases. We never store your sensitive data. (2) While that data is being streamed, we transform it. You define which schemas and tables you want to sync and at the column level, select a transformer that defines how you want to anonymize the data or generate synthetic data. (3) We subset your data based on your filters. We do all of this while handling referential integrity. Whether you have primary keys, foreign keys, unique constraints, circular dependencies (within a table and across tables), sequences and more, Neosync preserves those references. We also ship with APIs, a Terraform provider, a CLI and Github action that you can use to hydrate a CI database. Neosync is an open source project written in Go and Typescript and can be run on Docker Compose, Bare Metal, or Kubernetes via Helm. You can also use our hosted platform or managed platform that you can deploy in your VPC. We also have a hosted platform with a generous free tier - https://neosync.dev Here's a brief loom demo: https://www.loom.com/share/ac21378d01cd4d848cf723e4960e8338?... We'd love any feedback you have!

ndom float64, etc). You can create your own custom transformations as well. We've designed our transformers to be as flexible as possible so you can use them across almost any data type. You could also use Neosync in passthrough mode which means that none of the data will be transformed and you can use it for data replication. 4. Lastly, you can defined subsets. This is a way to filter the data that gets sent to the destination. You can provide a custom SQL query or filters to do this. For example, you can filter the data by an id, customerType, column, date, etc. This is very flexible. And that's it! The job will run on the schedule you determine. We handle things like retries and backoffs and referential integrity between tables. We also ship with APIs, a CLI and Github action so that you can use Neosync to hydrate a CI database in your CI pipeline. As a developer, you can use the CLI to hydrate a local database from S3 or another source. We're working on releasing a Terraform provider shortly. Deployment is pretty straightforward. You can deploy Neosync using Docker Compose (we provide a script) or on Kubernetes using our helm chart. So what's next? Here's a brief overview: Real time mode (hook up Neosync to Kafka/SQS and anonymize and send the data to destinations in real time) and more connections (MongoDB, Snowflake, CSV). On the ML side, supporting use-cases like consistent data generation (providing a seed value), statistically consistent data and more. You can check out our roadmap in our Github project. Here's a brief loom demo: https://www.loom.com/share/66224a81074a464b92808ecdc82b7ddb?... We'd love your feedback and contributions. We strongly believe that your data should be yours and it should stay on your infrastructure and open source is the best way to bring that vision to life.

ndom float64, etc). You can create your own custom transformations as well. We've designed our transformers to be as flexible as possible so you can use them across almost any data type. You could also use Neosync in passthrough mode which means that none of the data will be transformed and you can use it for data replication. 4. Lastly, you can defined subsets. This is a way to filter the data that gets sent to the destination. You can provide a custom SQL query or filters to do this. For example, you can filter the data by an id, customerType, column, date, etc. This is very flexible. And that's it! The job will run on the schedule you determine. We handle things like retries and backoffs and referential integrity between tables. We also ship with APIs, a CLI and Github action so that you can use Neosync to hydrate a CI database in your CI pipeline. We're working on releasing a Terraform provider shortly. Deployment is pretty straightforward. You can deploy Neosync using Docker Compose (we provide a script) or on Kubernetes using our helm chart. So what's next? Here's a brief overview: Real time mode (hook up Neosync to Kafka/SQS and anonymize and send the data to destinations in real time) and more connections (MongoDB, Snowflake, CSV). On the ML side, supporting use-cases like consistent data generation (providing a seed value), statistically consistent data and more. You can check out our roadmap in our Github project. Here's a brief loom demo: https://www.loom.com/share/66224a81074a464b92808ecdc82b7ddb?... We'd love your feedback and contributions. We strongly believe that your data should be yours and it should stay on your infrastructure and open source is the best way to bring that vision to life.