HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
Dao-AILab
flash-attention
Python2022–2026
Dao-AILab / flash-attention

flash-attention

Fast and memory-efficient exact attention

Explore on GitHubRead Hacker News discussions
14Distinct people
15Mentions
0Explicit recommenders
25.3KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Fast and memory-efficient exact attention

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:49:33 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year15 dated mentions

Observed mentions: 2022-05-31 10:56:20 UTC to 2026-03-05 19:03:48 UTC.

02462022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments20222023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 6 mentions; 0 submissions, 6 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20242025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC312
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC606
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC202
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC303
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC110

What Hacker News says

6 sources
cs702·May 5, 2023
After a very quick read, that's my understanding too: It's just KNN search with some bells and whistles. So I agree on points 1-3. When something works well, I don't care much about point 4. Personally, I've had only mixed success with KNN search on long sequences. Maybe I haven't done it right? I don't know. In my experience, nothing seems to work quite as well as explicit token-token interactions by some form of attention, which as we all know is too costly for long sequences (O(n²)). Lately I've been playing with https://github.com/hazyresearch/safari , which uses a lot less compute and seems promising, though it reminds me of things like FNet. Otherwise, for long sequences I've yet to find something better than https://github.com/HazyResearch/flash-attention for n×n interactions and https://github.com/glassroom/heinsen_routing for n×m
Read the rest
interactions. If anyone has other suggestions, I'd love to hear about them.
In Unlimiformer: Long-Range Transformers with Unlimited Length Input
cs702·Apr 24, 2023
Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e., mapping n tokens to m tokens. It's like a souped-up version of attention. I've used it to pull up more than a million tokens as context. Memory use and compute are O(nm). It works, but in my (limited) experience, it doesn't work out-of-the-box as well as FlashAttention for n² attention. Code: https://github.com/glassroom/heinsen_routing * RWKV: A sort-of-recurrent model which claims to have performance comparable to n² attenti
Read the rest
on in transformers. In my (limited) experience, it doesn't. Others seem to agree: https://twitter.com/arankomatsuzaki/status/16390003799784038... . Code: https://github.com/BlinkDL/RWKV-LM * RMT (this method): I'm skeptical that the recurrent connections will work as well as n² attention or n×m routing in practice, but I'm going to give it a try. Code: https://github.com/booydar/t5-experiments/tree/scaling-repor... In addition, the group that developed FlashAttention is working on state-space models (SSMs) that look promising to me. The idea is to approximate n² attention dynamically using only O(n log n) compute. There's no code available, but here's a blog post about it: https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn... [CORRECTION: Code is available. See comment by lucidrains below. I'm hopeful this will go to the top of my list.] If anyone here has other suggestions for working with long sequences (hundreds of thousands to millions of tokens), I'd love to learn about them.
In Scaling Transformer to 1M tokens and beyond with RMT

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.