MRmrshu·May 29, 2020
1. Managed, on AWS ECS -- it was not that difficult to set up using AWS services (Aurora for the DB, ElastiCache for Redis)
2. Freedom. We generally use PythonOperators but it is not uncommon to run containers as well. I agree with pyrophane, setting up containerized operators really is a non-trivial and far from straightforward to test locally. Still, it seems worth doing, particularly if you do not want various execution to affect one another.
3. Again, echoing what pyrophane said, a custom solution is needed for anything that's more than a couple hundred bytes in size. There even exists (now mostly abandoned) plugin that allows you to streamline the whole process: https://github.com/industrydive/fileflow Writing directly to something like S3 is almost always sufficient in combination with passing the path for the file from one task to
Read the rest
another.
Having said that, I would encourage you to try it out, even if the setup may sound daunting at first. If you can model your tasks as DAGs whose graphs are known in advance, I would argue it almost always makes sense because you get many things "for free" that were not mentioned before: logging, backfill, standard handling of connections/secrets and a ton of metrics in a nice UI that helps a lot with the visibility part.