SIsingularity2001·Mar 29, 2023
In Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMaopenai
evals
Python2023–2025
openai / evals
evals
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
22Distinct people
24Mentions
0Explicit recommenders
19.7KGitHub stars
What to know before exploring
What it is
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 4:00:59 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month24 dated mentions
Observed mentions: 2023-03-14 17:01:16 UTC to 2025-09-30 02:36:13 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Mar 2023 | 2023-03-01 00:00:00 UTC | 2023-04-01 00:00:00 UTC | 12 | 7 | 5 |
| Apr 2023 | 2023-04-01 00:00:00 UTC | 2023-05-01 00:00:00 UTC | 1 | 0 | 1 |
| May 2023 | 2023-05-01 00:00:00 UTC | 2023-06-01 00:00:00 UTC | 2 | 0 | 2 |
| Jun 2023 | 2023-06-01 00:00:00 UTC | 2023-07-01 00:00:00 UTC | 1 | 0 | 1 |
| Jul 2023 | 2023-07-01 00:00:00 UTC | 2023-08-01 00:00:00 UTC | 2 | 0 | 2 |
| Aug 2023 | 2023-08-01 00:00:00 UTC | 2023-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2023 | 2023-09-01 00:00:00 UTC | 2023-10-01 00:00:00 UTC | 1 | 0 | 1 |
| Oct 2023 | 2023-10-01 00:00:00 UTC | 2023-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2023 | 2023-11-01 00:00:00 UTC | 2023-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2023 | 2023-12-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2024 | 2024-01-01 00:00:00 UTC | 2024-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2024 | 2024-02-01 00:00:00 UTC | 2024-03-01 00:00:00 UTC | 1 | 0 | 1 |
| Mar 2024 | 2024-03-01 00:00:00 UTC | 2024-04-01 00:00:00 UTC | 0 | 0 | 0 |
| Apr 2024 | 2024-04-01 00:00:00 UTC | 2024-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2024 | 2024-05-01 00:00:00 UTC | 2024-06-01 00:00:00 UTC | 1 | 0 | 1 |
| Jun 2024 | 2024-06-01 00:00:00 UTC | 2024-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2024 | 2024-07-01 00:00:00 UTC | 2024-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2024 | 2024-08-01 00:00:00 UTC | 2024-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2024 | 2024-09-01 00:00:00 UTC | 2024-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2024 | 2024-10-01 00:00:00 UTC | 2024-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2024 | 2024-11-01 00:00:00 UTC | 2024-12-01 00:00:00 UTC | 1 | 0 | 1 |
| Dec 2024 | 2024-12-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2025 | 2025-01-01 00:00:00 UTC | 2025-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2025 | 2025-02-01 00:00:00 UTC | 2025-03-01 00:00:00 UTC | 0 | 0 | 0 |
| Mar 2025 | 2025-03-01 00:00:00 UTC | 2025-04-01 00:00:00 UTC | 0 | 0 | 0 |
| Apr 2025 | 2025-04-01 00:00:00 UTC | 2025-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2025 | 2025-05-01 00:00:00 UTC | 2025-06-01 00:00:00 UTC | 0 | 0 | 0 |
| Jun 2025 | 2025-06-01 00:00:00 UTC | 2025-07-01 00:00:00 UTC | 1 | 0 | 1 |
| Jul 2025 | 2025-07-01 00:00:00 UTC | 2025-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2025 | 2025-08-01 00:00:00 UTC | 2025-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2025 | 2025-09-01 00:00:00 UTC | 2025-10-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
12 sourcesOpenAI has been collecting a ton of evals here https://github.com/openai/evals with many of them including some comments about how well GPT-4 does vs GPT-3.5.
You could clone that repo, adapt the oaieval script to run against different APIs, then run the evals against both and compare the results.
In RWKV RNN: Better than ChatGPT?Tell HN: OpenAI charges you even if you're helping them on their Eval plarform
1 points · Open discussionGPT-4 doesn't pay close attention to detail in some cases
3 points · Open discussionShow HN: Visualize OpenAI Evals of GPT-4
OpenAI's new evals library (https://github.com/zeno-ml/zeno-evals) makes it easy to create and run evaluations to find the limitations of GPT-4.
The CLI library just prints out an overall metric after the process has finished, and doesn't give you any insights into what types of outputs and failures GPT-4 produced.
This simple one-line command lets you pass in the log outputs from Open AI evals (https://github.com/openai/evals/) and interactively explore the actual generated data!
Try it out with `pip install zeno-evals`
1 points · Open discussionOpenAI crowd sources LLM benchmarking datasets by offering advanced GPT-4 access
1 points · Open discussionAsk HN: Are you willing to contribute to OpenAI Evals?
OpenAI announced "https://github.com/openai/evals" which is basically collecting high quality benchmarks from the community. Do you think this company deserves to get the collective effort of the community?
For a company that is on track to monopolize on AI, I am strongly against contributing any our effort OpenAI.
I plead that community will not contribute until the model is open.
8 points · Open discussionThere is also a lot of work in benchmarking for AI as well. This is where things like Resnet come from.
But the point of using these tests for AI is precisely the reason we use for giving them to humans -- we think we know what it measures. AI is not intended to be a computation engine or a number crunching machine. It is intended to do things that historically required "human intelligence".
If there are better tests of human intelligence, I think that the AI community would be very interested in learning about them.
See: https://github.com/openai/evals
In GPT-4Example PR to OpenAI evals to get GPT-4 early API access
3 points · Open discussionEvals: a framework for evaluating OpenAI models and a registry of benchmarks
123 points · Open discussionsummary:
1. GPT4 is multimodal (text + image inputs => text outputs). This is being released piecemeal - with text input first via ChatGPT Plus subscribers https://beta.openai.com/docs/api-reference/generations/creat..., and via API https://beta.openai.com/docs/api-reference/introduction with waitlist (https://openai.com/waitlist/gpt-4-api). Image capability released via https://www.bemyeyes.com/.
2. GPT4 exhibits human level performance on various benchmarks (For example, it passes a simulated bar exam with a score around the top 10% of test takers; in contrast, GPT-3.5’s score was around the bottom 10%. see visual https://twitter.com/swyx/status/1635689844189036544)
3. GPT4 training used the same Azure supercomputer as GPT 3.5, but was a lot more stable: "becoming our first large model whose training performance we were able to accurate
Read the rest
ly predict ahead of time."
4. Also open-sourcing OpenAI Evals https://github.com/openai/evals, a framework for automated evaluation of AI model performance, to allow anyone to report shortcomings in OpenAI models to help guide further improvements.
Paper: https://cdn.openai.com/papers/gpt-4.pdf