HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
EleutherAI
lm-evaluation-harness
Python2021–2026
EleutherAI / lm-evaluation-harness

lm-evaluation-harness

A framework for few-shot evaluation of language models.

10Distinct people
13Mentions
0Explicit recommenders
14.5KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A framework for few-shot evaluation of language models.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 1:56:48 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year13 dated mentions

Observed mentions: 2021-03-22 13:53:43 UTC to 2026-05-13 16:13:05 UTC.

0123452021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments20212022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 5 mentions; 1 submissions, 4 comments20232024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC404
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC000
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC514
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC202
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC101
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC110

What Hacker News says

5 sources
Looks like my edit window closed, but my results ended up being very low so there must be something wrong (I've reached out to StabilityAI just in case). It does however seem to roughly match another user's 3B testing: https://twitter.com/abacaj/status/1648881680835387392 The current scores I have place it between gpt2_774M_q8 and pythia_deduped_410M (yikes!). Based on training and specs you'd expect it to outperform Pythia 6.9B at least... this is running on a HEAD checkout of https://github.com/EleutherAI/lm-evaluation-harness (releases don't support hf-casual) for those looking to replicate/debug. Note, another LLM currently being trained, GeoV 9B, already far outperforms this model at just 80B tokens trained: https://github.com/geov-ai/geov/blob/master/results.080B.md
In StableLM: A new open-source language model
Yeah, although looks like it currently has some issues with coqa: https://github.com/EleutherAI/lm-evaluation-harness/issues/2... There's also the bigscience fork, but I ran into even more problems (although I didn't try too hard) https://github.com/bigscience-workshop/lm-evaluation-harness And there's https://github.com/EleutherAI/lm-eval2/ (not sure if it's just starting over w/ a new repo or what?) but it has limited tests available
In StableLM: A new open-source language model

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.