FZfzliu·Apr 1, 2024Submission
1 points · Open discussionlucidrains
x-transformers
Python2021–2024
lucidrains / x-transformers
x-transformers
A concise but complete full-attention transformer with a set of promising experimental features from various papers
3Distinct people
3Mentions
0Explicit recommenders
6KGitHub stars
What to know before exploring
What it is
A concise but complete full-attention transformer with a set of promising experimental features from various papers
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year3 dated mentions
Observed mentions: 2021-05-08 17:38:27 UTC to 2024-04-01 05:19:36 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 0 | 0 | 0 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 1 | 0 | 1 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
3 sourcesIt could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better if your attention is subquadratic - again there is a dozen of varieties to pick from), see:
"A Length-Extrapolatable Transformer"
https://arxiv.org/abs/2212.10554
"Language Is Not All You Need: Aligning Perception with Language Models"
https://arxiv.org/abs/2302.14045
Notably, this positional embedding has been implemented by lucidrains in his x-transformers package: https://github.com/lucidrains/x-transformers/blob/main/x_tra...
In GPT-4 architecture: what we can deduce from research literatureX-Transformers: A fully-featured transformer with experimental features
96 points · Open discussion