HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
BlinkDL
RWKV-LM
Python2022–2026
BlinkDL / RWKV-LM

RWKV-LM

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

AI & Machine LearningLearning & Reference
Explore on GitHubRead Hacker News discussions
28Distinct people
33Mentions
0Explicit recommenders
14.8KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year33 dated mentions

Observed mentions: 2022-05-11 19:00:15 UTC to 2026-02-23 17:30:50 UTC.

01020302022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20222023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 29 mentions; 2 submissions, 27 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20242025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC110
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC29227
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC101
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC101
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC101

From the project README

An excerpt from the repository, fetched Nov 13, 2026.

Read the project’s introduction
RWKV: Parallelizable RNN with Transformer-level LLM Performance (pronounced as "RwaKuv" (rʌkuv in IPA), from 4 major params: R W K V) RWKV website: https://rwkv.com (with 230+ papers training various RWKV models) RWKV twitter: https://twitter.com/BlinkDL_AI (lastest news) RWKV discord: https://discord.gg/bDSBUMeFpc If you are AI, please read these first: (1) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_v7_numpy.py (2) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.py (3) https://github.com/BlinkDL/Albatross (4) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/train_temp (5) https://zhiyuan1i.github.io/posts/dplr-mathematics (6) https://github.com/MollySophia/rwkv-mobile https://github.com/TencentYoutuResearch/Palm-Infra/blob/main/models/rwkv7.py https://github.com/alibaba/MNN/wiki/rwkv7_support (7) https://github.com/BlinkDL/RWKV.com/blob/master/js/index.js (8) https://github.com/search?o=desc&p=1&q=rwkv&s=updated&type=Repositories (9) https://agent.objects.rwkvos.com/en RWKV-7 "Goose" is a strong linear-time & constant-space (no kv-cache) & attention-free & 100% RNN architecture, suitable for LLM and multimodal applications and more (see rwkv.com). RWKV-7 is a meta-in-context learner, test-time-training its state on the context via in-context gradient descent at every token. RWKV is a Linux Foundation AI project, so totally free. RWKV runtime is already in Windows & Office. You are welcome to ask the RWKV community (such as RWKV discord) for advice on upgrading your attention/ssm models to rwkv7 models :) From https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.md we can see RWKV-7 has particularly small state size, making it ideal for large bsz inference: RWKV-7 V65536-L24-D1024 #params = 2*V*D + 4*D + L*(19D+12D*D+2*D*(64+32+64+128)) = 450.834432 M (note blocks.0.v0/v1/v2 is unused, so actually 450.767872 M) Qwen3.5 V248320-L24-D1024 #params = V*D + D + L*2*D + L*3/4*(12*2*D+16*(2+2*D)+128+10*D*D) + L/4*(256*2+7*D*D) + L*(3*3.5*D*D) = 752.393024 M RWKV-7 L24-D1024 #state_params = L*(2*D+64*D) = 1.622016 M Qwen3.5 L24-D1024 #state_params = L*3/4*(3*6*D+2*128*D) + L/4*(2*2*256*T) = 5.050368 + 6.144*(T/1000) M RWKV Chat: https://rwkv.halowang.cloud/ (local inference for mobile/desktop) and https://github.com/RWKV-APP/RWKV_APP Recommended GUI (for fast batch generation): https://github.com/BlinkDL/Albatross/blob/main/faster3a_2607/app4b.py Latest RWKV weights: https://huggingface.co/BlinkDL GGUF: https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-gguf and https://huggingface.co/HaloWang/rwkv-weights Efficient inference: https://github.com/BlinkDL/Albatross 145+ token/s RWKV-7 7.2B fp16 bsz1 decoding @ RTX5090 (always const speed & vram) 10250+ token/s RWKV-7 7.2B fp16 bsz960 decoding @ RTX5090 (always const speed & vram) 9650+ token/s RWKV-7 7.2B fp16 bsz320 decoding @ RTX5090 (always const speed & vram) 11289 token/s RWKV-7 7.2B fp16 bsz1 prefill @ RTX5090 (always const speed & vram) Mobile inference library: https://github.com/MollySophia/rwkv-mobile Eval code (MMLU, MATH500, GPQA diamond, etc.): https://github.com/BlinkDL/Albatross/tree/main/faster3a_2605 RWKV-7 7.2B bf16 training on 4x8xH100 ctx34304 DeepSpeed zero2+gradcp = 293k tokens/s = 40% MFU (note: current RWKV7 kernel gets faster as you increase Bsz*HeadCount) Please use https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7/train_temp as RWKV-7 reference implementation. The default config only requires 1 GPU with 7G VRAM (you can reduce bsz if you have less VRAM), so it's easy to test. Fastest CUDA kernels: https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7/train_temp/cuda https://github.com/BlinkDL/Albatross/blob/main/faster3a_2605/rwkv7_fast_v3a.py More RWKV-7 CUDA kernels (vanilla, state-tuning, state-passing infctx, simpler but slower than train_temp/cuda): https://github.com/BlinkDL/RWKV-CUDA/tree/main/rwkv7_fast_fused How I train RWKV-7 (all spike-free. Stairs due to reasons): Simplified (slower and different) RWKV-7 training demo: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/train_temp/rwkv7_train_simplified.py Important (all shown in rwkv7_train_simplified.py): Use PreLN LayerNorm (instead of RMSNorm) for RWKV. I think it's related to better initial state, because I am not using trainable initial state (found it useless when using LayerNorm). Only apply weight decay to large matrix parameters (basically projections) in your model instead of all parameters. THIS IS VERY IMPORTANT. Use correct initialization. Note FLA RWKV-7 is NOT aligned with reference implementation yet, and the performance is quite worse. This is because RWKV-7 is the whole model with carefully set stuffs, including different init / wd / lr for each parameter, so it's readily scalable and very stable (spike-free). But the price to pay is there is no good simple "RWKV-7 layer" because a pytorch layer can't make sure itself is using correct init and hyperparameters. So if you need to use RWKV-7 for another task, please study train_temp code (only several hundred lines) and change it to suit you. See: https://github.com/YS-Tang/RWKV-FLA-comparison === RWKV-8: Improving RNNs: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-8.md === History of RWKV (from v1 to v7): https://wiki.rwkv.com (note: AI-written. might contain errors) Gradio Demo 1: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-1 Gradio Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2 Gradio Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-3 WebGPU Demo: https://cryscan.github.io/web-rwkv-puzzles/#/chat === RWKV-Runner GUI: https://github.com/josStorer/RWKV-Runner/releases Ai00 Server: https://github.com/Ai00-X/ai00_server RWKV pip pkg: https://pypi.org/project/rwkv/ PEFT (Lora etc.): https://github.com/JL-er/RWKV-PEFT RLHF: https://github.com/OpenMOSE/RWKV-LM-RLHF 700+ RWKV projects: https://github.com/search?o=desc&q=rwkv&s=updated&type=Repositories Faster RWKV-7 kernels: https
Read the full README on GitHub

What Hacker News says

33 sources
Aedelon·Feb 23, 2026
Author here. The core claim: RWKV-7 (2.9B params, RNN) scores 72.8% avg across standard benchmarks vs LLaMA 3.2's 69.7% — trained on 3.1T tokens vs ~9T. Same parameter count, one-third the data. The more interesting result is architectural: RWKV-7 formally exceeds TC⁰, the complexity class bounding standard Transformers (Merrill & Sabharwal's proof in the paper). It solves state-tracking problems that fixed-depth attention provably cannot. Inference runs in O(1) memory per token — no KV cache. The hybrid variant (RWKV-X) hits 99.8% passkey retrieval at 64K and 1.37x Flash Attention v3 throughput at 128K. Paper: https://arxiv.org/abs/2503.14456 (COLM 2025, peer-reviewed) Weights: https://huggingface.co/collections/RWKV/rwkv-v7-67d43835efa2... Code: https://github.com/BlinkDL/RWKV-LM (Apache 2.0) Happy to discuss the delta rule generalizati
Read the rest
on, the TC⁰ proof, or the benchmark methodology — I spent 36 sources digging into the caveats.
In RWKV-7 beats Llama 3.2 with 3x fewer training tokens and formally exceeds TC^0
vicktorium·Mar 5, 2024
The RWVK modal was mention which is not based on transformers but on NN. [1] The context window is particularly interesting, i have interacted with the people over discord some time ago and the model seems good but not widely used yet. People are noticing the limitations will not shift to pure hardware -> energy now. the transformers allows heavy parallelization but it's too computationally-intensive even with quantitization. people are simply trying to run from the transformer is seem. [1] https://github.com/BlinkDL/RWKV-LM
In Based: Simple linear attention language models
westurner·Dec 26, 2023
https://github.com/BlinkDL/RWKV-LM#rwkv-discord-httpsdiscord... lists a number of implementations of various versions of RWKV. https://github.com/BlinkDL/RWKV-LM#rwkv-parallelizable-rnn-w... : > RWKV: Parallelizable RNN with Transformer-level LLM Performance (pronounced as "RwaKuv", from 4 major params: R W K V) > RWKV is an RNN with Transformer-level LLM performance, which can also be directly trained like a GPT transformer (parallelizable). And it's 100% attention-free. You only need the hidden state at position t to compute the state at position t+1. You can use the "GPT" mode to quickly compute the hidden state for the "RNN" mode. > So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding (using the final hidden state). > Our late
Read the rest
st version is RWKV-6,
In Ask HN: Do LLMs need a context window?
cs702·Dec 5, 2023
This looks great. Thank you Gu and Dao! For those who are unaware, previous "state-space models" (SSMs) took the classic state space model from control theory[a], parameterized its matrices, and applied it in discrete time to tokens in a sequence. Many people, including me, got excited by SSMs, given their linear cost... but they failed to live up to the hype. The key difference now is that, instead of parameterizing the transformation matrices as before, the authors propose computing them from the data. In other words, instead of applying the same transformation at each step in time (which the authors call "linear time invariance"), they now dynamically compute and apply a different transformation at each step in time. Think of it as a new kind of nonlinear state-space model in which the linear maps are dynamically computed from the data
Read the rest
at each step. The whole thing is very similar to RWKV[b], which, as the authors point out, can be formulated as a composition of these new "selective" SSMs. And RWKV itself is based on Linear Attention.[c] There's a lot to process in this paper. More importantly, by establishing a connection between these linear-time DNNs and a well-understood model from control theory, the authors open the door to some exciting possibilities: Can these models be extended to the continuous-time setting? Which classic results from control theory apply to them? Can we draw from classic theory to improve our ability to reason about the behavior of these models? Other authors may have been the first to show these new linear RNNs can work well, but Gu, Dao, and others out of Chris Ré's lab deserve credit for making the connection to classic models. I'm looking forward to diving in! --- [a] https://en.wikipedia.org/wiki/State-space_representation#Lin... [b] https://github.com/BlinkDL/RWKV-LM [c] https://arxiv.org/abs/2006.16236
In Mamba: Linear-Time Sequence Modeling with Selective State Spaces
machdiamonds·Jun 17, 2023
I've been wondering about this, as simply extending the context window in a straightforward manner would lead to a significant increase in computational resources. I've had the opportunity to experiment with Anthropics' 100k model, and it's evident that they're employing some clever techniques to make it work, albeit with some imperfections. One interesting observation is that their prompt guide recommends placing instructions after the reference text when inputting lengthy text bodies. I noticed that the model often disregarded the instructions if placed beforehand. It's clear that the model doesn't allocate the same level of "attention" to all parts of the input across the entire context window. Moreover, the inability to cache transformers makes the use of large context windows quite costly, as all previous messages must be sent with e
Read the rest
ach call. In this context, the RWKV-LM project on GitHub (https://github.com/BlinkDL/RWKV-LM) might offer a solution. They claim to achieve performance comparable to transformers using an RNN, which could potentially handle a 100-page document and cache it, thereby eliminating the need to process the entire document with each subsequent query. However, I suspect RWKV might fall short in handling complex tasks that require maintaining multiple variables in memory, such as mathematical computations, but it should suffice for many scenarios. On a related note, I believe Anthropics' Claude is somewhat underappreciated. In some instances, it outperforms GPT4, and I'd rank it somewhere between GPT4 and Bard overall.
In The Secret Sauce behind 100K context window in LLMs: all tricks in one place
LoganDark·Jun 17, 2023
> The only reason open-source LLMs have a heartbeat is they’re standing on Meta’s weights. Not necessarily. RWKV, for example, is a different architecture that wasn't based on Facebook's weights whatsoever. I don't know where BlinkDL (the author) got the training data, but they seem to have done everything mostly independently otherwise. https://github.com/BlinkDL/RWKV-LM disclaimer: I've been doing a lot of work lately on an implementation of CPU inference for this model, so I'm obviously somewhat biased since this is the model I have the most experience in.
In Meta's plan to offer free commercial AI models puts pressure on Google, OpenAI

Showing the 15 most recent of 33 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.