HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
openai
evals
Python2023–2025
openai / evals

evals

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

AI & Machine LearningTesting & Quality
Explore on GitHubRead Hacker News discussions
22Distinct people
24Mentions
0Explicit recommenders
19.7KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:03:16 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month24 dated mentions

Observed mentions: 2023-03-14 17:01:16 UTC to 2025-09-30 02:36:13 UTC.

0369122023-03-01 00:00:00 UTC to 2023-04-01 00:00:00 UTC (exclusive): 12 mentions; 7 submissions, 5 commentsMar 20232023-04-01 00:00:00 UTC to 2023-05-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-05-01 00:00:00 UTC to 2023-06-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-06-01 00:00:00 UTC to 2023-07-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-07-01 00:00:00 UTC to 2023-08-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2023-08-01 00:00:00 UTC to 2023-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsJun 20242024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsSep 2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Mar 20232023-03-01 00:00:00 UTC2023-04-01 00:00:00 UTC1275
Apr 20232023-04-01 00:00:00 UTC2023-05-01 00:00:00 UTC101
May 20232023-05-01 00:00:00 UTC2023-06-01 00:00:00 UTC202
Jun 20232023-06-01 00:00:00 UTC2023-07-01 00:00:00 UTC101
Jul 20232023-07-01 00:00:00 UTC2023-08-01 00:00:00 UTC202
Aug 20232023-08-01 00:00:00 UTC2023-09-01 00:00:00 UTC000
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC101
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC000
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC000
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC000
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC101
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC000
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC101
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC000
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC000
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC000
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC101
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC000
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC000
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC000
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC000
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC000
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC101
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC000
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC101

What Hacker News says

24 sources
westurner·Sep 30, 2025
Manually, From https://news.ycombinator.com/item?id=40859434 : > E.g promptfoo and chainforge have multi-LLM workflows. > Promptfoo has a YAML configuration for prompts, providers,: https://www.promptfoo.dev/docs/configuration/guide/ openai/evals//docs/build-eval.md: https://github.com/openai/evals/blob/main/docs/build-eval.md From https://news.ycombinator.com/item?id=45267271 ; > API facades like OpenLLM and model routers like OpenRouter have standard interfaces for many or most LLM inputs and outputs. Tools like Promptfoo, ChainForge, and LocalAI also all have abstractions over many models. > What are the open standards for representing LLM inputs, and outputs? > W3C PROV has prov:Entity, prov:Activity, and prov:Agent for modeling AI provenance: who or what did what when. > LLM evals could be represented in W3C EARL Evaluation and Repo
Read the rest
rting Language "Can Large Language Models Emulate Judicial Decision-Making? [Paper]" https://news.ycombinator.com/item?id=42927611 "California governor signs AI transparency bill into law" (2025) https://news.ycombinator.com/item?id=45418428 : > https://sb53.info/ Is this the first of its sort?: > CalCompute
In Claude Sonnet 4.5
arcb·Jun 10, 2025
Thank you! We have a fork of browser-use that lets us hand hold web agents since we know our tasks are repetitive. We can cache expected paths and fire alerts if we go off the rails. We'd love to contribute it back at some point, mainly a question of bandwidth. We're evaluating Cua (https://www.ycombinator.com/companies/cua) to containerize our agents; am a fan so far. We're also putting Computer Use agents from (OAI and Anthropic) to the test. Many legacy ERPs don't run in the browser and we have to meet them there. I think we're a few months away from things working reliably and efficiently. We're evaluating several of the top models (both open and closed) for browser navigation (claude's winning atm) and PDF extraction. Since we're performing repetitive tasks, the goal is make our workflows RL-able. Being able to rely on OSS models wi
Read the rest
ll help a lot here. We're building our own data sets and evaluations for many of the subtasks. We're using openai's evals (https://github.com/openai/evals) as a framework to guide our own tooling. Apart from that, we write in Typescript, Python, and Golang. We use Postgres for persistence (nothing fancy here). We host on AWS, and might go on premises for some customers. We plan on investing a lot into our workflow system as the backbone of our product. I prefer open source when possible. Everything's new and early, and many things require source changes that others might not be able to prioritize. Edit - one thing I'd love to find a good solution for is reliably extracting handwriting from PDF documents. Clinicians have to do this a ton to keep the trains running on time, and being able to digitize that knowledge on the go will be huge. Very open to ideas here. We're seeing great tools and products come up by the day, including from our own YC batch.
In Launch HN: BitBoard (YC X25) – AI agents for healthcare back-offices
renchuw·Feb 13, 2024
Fair question. Evaluate refers to the phase after training to check if the training is good. Usually the flow goes training -> evaluation -> deployment (what you called inference). This project is aimed for evaluation. Evaluation can be slow (might even be slower than training if you're finetuning on a small domain specific subset)! So there are [quite](https://github.com/microsoft/promptbench) [a](https://github.com/confident-ai/deepeval) [few](https://github.com/openai/evals) [frameworks](https://github.com/EleutherAI/lm-evaluation-harness) working on evaluation, however, all of them are quite slow, because LLM are slow if you don't have infinite money. [This](https://github.com/open-compass/opencompass) one tries to speed up by parallelizing on multiple computers, but none of them takes advantage of the fact that many evaluation querie
Read the rest
s might be similar and all try to evaluate on all given queries. And that's where this project might come in handy.
In Show HN: Faster LLM evaluation with Bayesian optimization
westurner·Sep 10, 2023
OpenAI/evals > Building an eval: https://github.com/openai/evals/blob/main/docs/build-eval.md "Robustness of Model-Graded Evaluations and Automated Interpretability" (2023) https://www.lesswrong.com/posts/ZbjyCuqpwCMMND4fv/robustness... : > The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability. From https://news.ycombinator.com/item?id=37451534 : add'l benchmarks: TheoremQA, Legalbench
In Asking 60 LLMs a set of 20 questions
gooseus·Jul 5, 2023
"What if" is all these "existential risk" conversations ever are. Where is your evidence that we're approaching human level AGI, let alone SuperIntelligence? Because ChatGPT can (sometimes) approximate sophisticated conversation and deep knowledge? How about some evidence that ChatGPT isn't even close? Just clone and run OpenAI's own evals repo https://github.com/openai/evals on the GPT-4 API. It performs terribly on novel logic puzzles and exercises that a clever child could learn to do in an afternoon (there are some good chess evals, and I submitted one asking it to simulate a Forth machine).
In Introducing Superalignment
jrpt·May 26, 2023
Quality isn’t unquantifiable. Search engines have been qualifying their results quality for a long time. OpenAI recently released a package for quantifying a LLM’s quality https://github.com/openai/evals I don’t know what to say about your CTO. People are definitely overestimating how useful GitHub Copilot is at least (if they say 2x they’re wrong). But there is no doubt that these products are changing the world and making us more productive.
In 58% of US adults have heard of ChatGPT; 14% have tried it
sebzim4500·Apr 7, 2023
>Does ChatGPT have a large test suite consisting of a large number of input question and expected responses that have to match? They are trying to crowdsource this with OpenAI evals. https://github.com/openai/evals I'm sure they have a lot of internal benchmarks too, but of course they won't share them. >If not, there can be no "bug" in ChatGPT. I don't understand the objection. Are you claiming that bugs only exist if you have testcases for them?
In We need to tell people ChatGPT will lie to them, not debate linguistics

Showing the 15 most recent of 24 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.