HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
apache
airflow
Python2016–2026
apache / airflow

airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Science & EngineeringAI & Machine LearningData & DatabasesLearning & ReferenceDevOps & CloudAutomation & Workflows
Explore on GitHubRead Hacker News discussions
25Distinct people
32Mentions
0Explicit recommenders
47.5KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year32 dated mentions

Observed mentions: 2016-07-21 12:54:52 UTC to 2026-04-20 13:07:56 UTC.

02468102016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments20162017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 9 mentions; 0 submissions, 9 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 5 mentions; 1 submissions, 4 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC312
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC202
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC909
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC202
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC312
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC202
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC211
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC303
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC514
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC101

What Hacker News says

32 sources
art_mach·Apr 20, 2026
Yeah, I was pondering a few months away when I checked Pathway, an ETL solution. I've never heard about it but I saw some news that they have created a better model than transformer. So stats: - link: https://github.com/pathwaycom/pathway - watch: 115, fork: 1.6k, star: 63.5k - issues: 32, PR-s: 3 And compare to other ETL tool, like Apache Airflow - used by me and many machine learning folks: - link https://github.com/apache/airflow - watch: 777, forks 16.9k!!!!!, Stars: (only!) 45.1k - issues: 1200 (!!!), PR-s (501!!!).
In GitHub's fake star economy
jlaneve·Mar 26, 2025
Hi HN, We've built an SDK for building DAGs / data pipelines with LLMs in Apache Airflow [1] using Pydantic AI [2] under the hood. I've seen success across the board with Airflow users building simple LLM workflows before moving on to "AI agents". In my experience, the noise around building agents means that people forget that there are other ways to get more immediate value out of LLMs. Coupling Airflow for orchestration and Pydantic AI for LLM interactions has turned out to be a very pragmatic approach to building these workflows (and agents). Neither tool "gets in the way" of what you're trying to do. Airflow's been around for 10+ years and has a very well-built orchestration engine rich with everything you need to write production grade data pipelines, and Pydantic AI's been a refreshing take on working with LLMs. Would love some feed
Read the rest
back from this community! [1] https://github.com/apache/airflow [2] https://ai.pydantic.dev
In Show HN: Airflow AI SDK to build simple LLM workflows
annexrichmond·Aug 12, 2024
We only use KubernetesOperators, but this has many downsides, and it's very clearly a 2nd thought of the Airflow project. It creates confusion because users of Airflow expect features A, B, and C, and when using KubernetesOperators they aren't functional because your biz logic is separated. Eg., if your biz logic knows what S3 it talks to in an external task, how can Airflow? So now its Dataset feature is useless. There are a number of blog posts echoing a similar critique[1]. Using KubernetesOperators creates a lot of wrong abstractions, impedes testability, and makes Airflow as a whole a pretty overkill system just to monitor external tasks. At that point, you should have just had your orchestration in client code to begin with, and many other frameworks made this correct division between client and server. That would also make it e
Read the rest
asier to support multiple languages. According to their README: https://github.com/apache/airflow#approach-to-dependencies-o... > Airflow has a lot of dependencies - direct and transitive > The important dependencies are: SQLAlchemy, Alembic, Flask, werkzeug, celery, kubernetes Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server? [1] Anecdotes from various posts - https://medium.com/bluecore-engineering/were-all-using-airfl... - https://eng.lyft.com/orchestrating-data-pipelines-at-lyft-co... - https://dagster.io/blog/dagster-airflow > Airflow, in its design, made the incorrect abstraction by having Operators actually implement functional work instead of spinning up developer work. > By simply moving to using a Kubernetes Operator, Airflow developers can develop more quickly, debug more confidently, and not worry about conflicting package requirements. > Airflow lacks proper library isolation. It becomes hard or impossible to do if any team requires a specific library version for a given workflow > There is no way to separate DAGs to development, staging, and production using out-of-the-box Airflow features. That makes Airflow harder to use for mission-critical applications that require proper testing and the ability to roll back > Data pipelines written for Airflow are typically bound to a particular environment. To avoid dependency hell, most guides recommend defining Airflow tasks with operators like the KubernetesPodOperator, which dictates that the task gets executed in Kubernetes. When a DAG is written in this way, it’s nigh-impossible to run it locally or as part of CI. And it requires opting out of all of the integrations that come out-of-the-box with Airflow.
In New Apache Airflow Operators for Google Generative AI
caprock·Jul 21, 2024
From what I've seen, there are sort of two paths if you're running on your own hardware or VMs. I'll provide a well known example from each. 1. lang specific distributed task library For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config. * https://github.com/celery/celery Or lower level: * https://github.com/dask/dask 2. DAG Workflow systems There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow: * https://github.com/apache/airflow
In Ask HN: What's the right tool for this job?
KptMarchewa·May 23, 2022
>If I need a for-loop, I'll write my own script to generate the JSON. That's how you end with extreme mess in logs, UI and metrics. > But with Airflow, you need to figure out what _Airflow_ operator and method wraps the Google SDK method you're trying to call. Or you can use PythonOperator with hooks, that generally integrate external APIs with Airflow connection system. https://github.com/apache/airflow/blob/main/airflow/provider... I think the bigger problem with Airflow's Operator concept is N*M problem of integrating multiple systems. That's how you end with GoogleCloudStorageToS3Operator and stuff like that.
In Lessons learned from running Apache Airflow at scale
jamestimmins·Aug 26, 2021
Come help out with Apache Airflow! It's a great project to get involved with because it has a ton of users and real world problems, but it's still early enough that there's low hanging fruit in terms of adding functionality and improving existing code. A helpful place to start is the provider packages, since Airflow has integrations with so many 3rd party providers, and if you have knowledge in any of them it can be a good jumping off point. https://github.com/apache/airflow/
In Ask HN: What is a good Python project for a mid lv engineer to contribute to?
JAlexoid·Apr 19, 2021
That's really not true. I like to point out Airflow as an example. It's still a complete nightmare to deal with and it has many steps for just packaging the application. https://github.com/apache/airflow I have less trouble packaging and running a 3 year unmaintained Maven Java application, than actively maintained Airflow. And let's not forget that I can target Java 8, without having to install Java 8 on my machine. (Which only adds to Python's complexity)
In Java Is Underhyped
rywalker·May 29, 2020
1) An official Helm chart is coming very soon. We (Astronomer) have a commercial platform that aims to solve this completely including observability, configuration and deployment. Happy to team up to improve Airflow's open-source k8s story if you have some ideas. 2) Yes the UI is outdated, and not responsive. We're going to kick off a process to build a new modern UI in Q3 (a full-featured Swagger API is being built now, which the new UI will rely upon.) 3) I personally think catchup true is a fine default, but whatever. Generally when I launch a new DAG I want to generate some historical data using the DAG. 4) Airflow has tags now since 1.10.8 https://airflow.readthedocs.io/en/latest/howto/add-dag-tags.... - we decided not to do folders. 5) That's true, but it also provides a low bar to entry. There are some guides written on unit testin
Read the rest
g DAGs, but I agree we should be a test-first community. On my roadmap. 6) Celery w/ KEDA is pretty nice - check out https://www.astronomer.io/blog/the-keda-autoscaler/ 7) I personally love the PythonOperator for simple DAGs. Agree that DockerOperator (KubernetesPodOperator if you're running Airflow in K8s) 8) Yes, logging UI will be improved in the UI rewrite. What's your favorite UI for this? 9) That feature is in the queue https://github.com/apache/airflow/issues/7911
In Introduction to Apache Airflow

Showing the 15 most recent of 32 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.