Pontoon

offer data syncs?”. It turns out, building reliable cross-cloud data syncs is difficult. That’s why we built Pontoon. We designed Pontoon to be: - Easily deployed: we provide a single, self-contained Docker image for easy deployment and Docker Compose for larger workloads (https://pontoon-data.github.io/Pontoon/getting-started/quick...) - Support modern data warehouses: we support syncing to/from Snowflake, BigQuery, Redshift, and Postgres. - Sync cross cloud: sync from BigQuery to Redshift, Snowflake to BigQuery, Postgres to Redshift, etc. - Developer friendly: data syncs can also be built via the API - Open source: Pontoon is free to use by anyone Under the hood, we use Apache Arrow (https://arrow.apache.org/) to move data between sources and destinations. Arrow is very performant - we wanted to use a library that could handle the scale of moving millions of records per minute. In the shorter-term, there are several improvements we want to make, like: - Adding support for DBT models to make adding data models easier - UX improvements like better error messaging and monitoring of data syncs - More sources and destinations (S3, GCS, Databricks, etc.) - Improve the API for a more developer friendly experience (it’s currently tied pretty closely to the front end) In the longer-term, we want to make data sharing as easy as possible. As data engineers, we sometimes felt like second class citizens with how we were told to get the data we needed - “just loop through this api 1000 times”, “you probably won’t get rate limited” (we did), “we can schedule an email to send you a csv every day”. We want to change how modern data sharing is done and make it simple for everyone. Give it a try https://github.com/pontoon-data/Pontoon. Cheers!

offer data syncs?”. It turns out, building reliable cross-cloud data syncs is difficult. That’s why we built Pontoon. We designed Pontoon to be: - Easily deployed: we provide a single, self-contained Docker image for easy deployment and Docker Compose for larger workloads (https://pontoon-data.github.io/Pontoon/getting-started/quick...) - Support modern data warehouses: we support syncing to/from Snowflake, BigQuery, Redshift, and Postgres. - Sync cross cloud: sync from BigQuery to Redshift, Snowflake to BigQuery, Postgres to Redshift, etc. - Developer friendly: data syncs can also be built via the API - Open source: Pontoon is free to use by anyone Under the hood, we use Apache Arrow (https://arrow.apache.org/) to move data between sources and destinations. Arrow is very performant - we wanted to use a library that could handle the scale of moving millions of records per minute. In the shorter-term, there are several improvements we want to make, like: - Adding support for DBT models to make adding data models easier - UX improvements like better error messaging and monitoring of data syncs - More sources and destinations (S3, GCS, Databricks, etc.) - Improve the API for a more developer friendly experience (it’s currently tied pretty closely to the front end) In the longer-term, we want to make data sharing as easy as possible. As data engineers, we sometimes felt like second class citizens with how we were told to get the data we needed - “just loop through this api 1000 times”, “you probably won’t get rate limited” (we did), “we can schedule an email to send you a csv every day”. We want to change how modern data sharing is done and make it simple for everyone. Give it a try: https://github.com/pontoon-data/Pontoon. Cheers!