dbt

Hey all, as a potential competitor of Looker, I'm not sure how I should feel about this news. :) Here are some of the facts: 1. When Google acquired Alooma, they slowed down the development and dropped the support for other destinations such as Redshift and Hive. Even though Alooma is a data pipeline tool which makes it similar to Looker's case, the deal was $150M (compared to $2.6B) so I'm not sure the comparison makes sense. 2. Looker's sale team is so aggressive and their support team is great. In fact, that's why Looker became so big in the last few years. Google is not famous in terms of support. 3. Google is serious on BigQuery and I'm almost sure it will make Looker part of the Google Cloud. Since most of Looker customers are enterprise companies, Google will probably chase them to switch to BigQuery. On the other hand, Google has

Here's one approach: 1. Document all datasets at their sources of record 2. Setup jobs to dump their data into an S3 bucket (daily, hourly, whatever makes sense). Use IAM to lock down access if necessary. 3. Setup AWS Athena to give you some analysis capability on those files, while you: 4. Setup jobs to [denormalise/cleanup and] load those files from S3 into postgres RDS/citus data/redshift (alternatively you could denormalise after the load ELT-style using materialised views or [dbt](https://github.com/fishtown-analytics/dbt)) You'll still need a tool to orchestrate it all. I'm excited about trying [pachyderm](https://www.pachyderm.io/) for my next project.