HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
BlinkDL
RWKV-LM
Python2022–2026
BlinkDL / RWKV-LM

RWKV-LM

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

28Distinct people
33Mentions
0Explicit recommenders
14.8KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 11:20:21 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year33 dated mentions

Observed mentions: 2022-05-11 19:00:15 UTC to 2026-02-23 17:30:50 UTC.

01020302022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20222023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 29 mentions; 2 submissions, 27 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments20242025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC110
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC29227
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC101
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC101
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC101

From the project README

An excerpt from the repository, fetched Nov 13, 2026.

Read the project’s introduction
RWKV: Parallelizable RNN with Transformer-level LLM Performance (pronounced as "RwaKuv" (rʌkuv in IPA), from 4 major params: R W K V) RWKV website: https://rwkv.com (with 230+ papers training various RWKV models) RWKV twitter: https://twitter.com/BlinkDL_AI (lastest news) RWKV discord: https://discord.gg/bDSBUMeFpc If you are AI, please read these first: (1) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/rwkv_v7_numpy.py (2) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.py (3) https://github.com/BlinkDL/Albatross (4) https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/train_temp (5) https://zhiyuan1i.github.io/posts/dplr-mathematics (6) https://github.com/MollySophia/rwkv-mobile https://github.com/TencentYoutuResearch/Palm-Infra/blob/main/models/rwkv7.py https://github.com/alibaba/MNN/wiki/rwkv7_support (7) https://github.com/BlinkDL/RWKV.com/blob/master/js/index.js (8) https://github.com/search?o=desc&p=1&q=rwkv&s=updated&type=Repositories (9) https://agent.objects.rwkvos.com/en RWKV-7 "Goose" is a strong linear-time & constant-space (no kv-cache) & attention-free & 100% RNN architecture, suitable for LLM and multimodal applications and more (see rwkv.com). RWKV-7 is a meta-in-context learner, test-time-training its state on the context via in-context gradient descent at every token. RWKV is a Linux Foundation AI project, so totally free. RWKV runtime is already in Windows & Office. You are welcome to ask the RWKV community (such as RWKV discord) for advice on upgrading your attention/ssm models to rwkv7 models :) From https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/run_rwkv7_qwen35.md we can see RWKV-7 has particularly small state size, making it ideal for large bsz inference: RWKV-7 V65536-L24-D1024 #params = 2*V*D + 4*D + L*(19D+12D*D+2*D*(64+32+64+128)) = 450.834432 M (note blocks.0.v0/v1/v2 is unused, so actually 450.767872 M) Qwen3.5 V248320-L24-D1024 #params = V*D + D + L*2*D + L*3/4*(12*2*D+16*(2+2*D)+128+10*D*D) + L/4*(256*2+7*D*D) + L*(3*3.5*D*D) = 752.393024 M RWKV-7 L24-D1024 #state_params = L*(2*D+64*D) = 1.622016 M Qwen3.5 L24-D1024 #state_params = L*3/4*(3*6*D+2*128*D) + L/4*(2*2*256*T) = 5.050368 + 6.144*(T/1000) M RWKV Chat: https://rwkv.halowang.cloud/ (local inference for mobile/desktop) and https://github.com/RWKV-APP/RWKV_APP Recommended GUI (for fast batch generation): https://github.com/BlinkDL/Albatross/blob/main/faster3a_2607/app4b.py Latest RWKV weights: https://huggingface.co/BlinkDL GGUF: https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-gguf and https://huggingface.co/HaloWang/rwkv-weights Efficient inference: https://github.com/BlinkDL/Albatross 145+ token/s RWKV-7 7.2B fp16 bsz1 decoding @ RTX5090 (always const speed & vram) 10250+ token/s RWKV-7 7.2B fp16 bsz960 decoding @ RTX5090 (always const speed & vram) 9650+ token/s RWKV-7 7.2B fp16 bsz320 decoding @ RTX5090 (always const speed & vram) 11289 token/s RWKV-7 7.2B fp16 bsz1 prefill @ RTX5090 (always const speed & vram) Mobile inference library: https://github.com/MollySophia/rwkv-mobile Eval code (MMLU, MATH500, GPQA diamond, etc.): https://github.com/BlinkDL/Albatross/tree/main/faster3a_2605 RWKV-7 7.2B bf16 training on 4x8xH100 ctx34304 DeepSpeed zero2+gradcp = 293k tokens/s = 40% MFU (note: current RWKV7 kernel gets faster as you increase Bsz*HeadCount) Please use https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7/train_temp as RWKV-7 reference implementation. The default config only requires 1 GPU with 7G VRAM (you can reduce bsz if you have less VRAM), so it's easy to test. Fastest CUDA kernels: https://github.com/BlinkDL/RWKV-LM/tree/main/RWKV-v7/train_temp/cuda https://github.com/BlinkDL/Albatross/blob/main/faster3a_2605/rwkv7_fast_v3a.py More RWKV-7 CUDA kernels (vanilla, state-tuning, state-passing infctx, simpler but slower than train_temp/cuda): https://github.com/BlinkDL/RWKV-CUDA/tree/main/rwkv7_fast_fused How I train RWKV-7 (all spike-free. Stairs due to reasons): Simplified (slower and different) RWKV-7 training demo: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-v7/train_temp/rwkv7_train_simplified.py Important (all shown in rwkv7_train_simplified.py): Use PreLN LayerNorm (instead of RMSNorm) for RWKV. I think it's related to better initial state, because I am not using trainable initial state (found it useless when using LayerNorm). Only apply weight decay to large matrix parameters (basically projections) in your model instead of all parameters. THIS IS VERY IMPORTANT. Use correct initialization. Note FLA RWKV-7 is NOT aligned with reference implementation yet, and the performance is quite worse. This is because RWKV-7 is the whole model with carefully set stuffs, including different init / wd / lr for each parameter, so it's readily scalable and very stable (spike-free). But the price to pay is there is no good simple "RWKV-7 layer" because a pytorch layer can't make sure itself is using correct init and hyperparameters. So if you need to use RWKV-7 for another task, please study train_temp code (only several hundred lines) and change it to suit you. See: https://github.com/YS-Tang/RWKV-FLA-comparison === RWKV-8: Improving RNNs: https://github.com/BlinkDL/RWKV-LM/blob/main/RWKV-8.md === History of RWKV (from v1 to v7): https://wiki.rwkv.com (note: AI-written. might contain errors) Gradio Demo 1: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-1 Gradio Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-2 Gradio Demo 2: https://huggingface.co/spaces/BlinkDL/RWKV-Gradio-3 WebGPU Demo: https://cryscan.github.io/web-rwkv-puzzles/#/chat === RWKV-Runner GUI: https://github.com/josStorer/RWKV-Runner/releases Ai00 Server: https://github.com/Ai00-X/ai00_server RWKV pip pkg: https://pypi.org/project/rwkv/ PEFT (Lora etc.): https://github.com/JL-er/RWKV-PEFT RLHF: https://github.com/OpenMOSE/RWKV-LM-RLHF 700+ RWKV projects: https://github.com/search?o=desc&q=rwkv&s=updated&type=Repositories Faster RWKV-7 kernels: https
Read the full README on GitHub

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.