eval

Discover this repository through its Hacker News conversations.

2Distinct people

2Mentions

0Explicit recommenders

REPOSITORY AT A GLANCE

What to know before exploring

A public description has not been verified.

No documented product use has been verified yet.

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

No other repository is repeatedly linked in the same discussions.

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

HN mentions by week2 dated mentions

Observed mentions: 2023-07-11 14:43:46 UTC to 2023-08-25 23:07:18 UTC. Weeks start Monday, UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
Jul 10	2023-07-10 00:00:00 UTC	2023-07-17 00:00:00 UTC	1	1
Jul 17	2023-07-17 00:00:00 UTC	2023-07-24 00:00:00 UTC	0	0
Jul 24	2023-07-24 00:00:00 UTC	2023-07-31 00:00:00 UTC	0	0
Jul 31	2023-07-31 00:00:00 UTC	2023-08-07 00:00:00 UTC	0	0
Aug 7	2023-08-07 00:00:00 UTC	2023-08-14 00:00:00 UTC	0	0
Aug 14	2023-08-14 00:00:00 UTC	2023-08-21 00:00:00 UTC	0	0
Aug 21	2023-08-21 00:00:00 UTC	2023-08-28 00:00:00 UTC	1	1

2 sources

GPT-4 as served in the API has been getting 85% on HumanEval (compared to 69.5% claimed here) https://twitter.com/amanrsanger/status/1635751764577361921 https://github.com/getcursor/eval

While that's what the Technical Report (https://arxiv.org/pdf/2303.08774v3.pdf) says, but GPT-4 out in the wild's (reproducible) performance appears to be much higher now. Testing from 3/15 (presumably on the 0314 model) seems to be at 85.36% (https://twitter.com/amanrsanger/status/1635751764577361921). And the linked paper from my post(https://doi.org/10.48550/arXiv.2305.01210) got a pass@1 of 88.4 from GPT-4 recently (May? June?). Out of curiousity, I was trying out gpt-4-0613 and claude-v2 with https://github.com/getcursor/eval, but sadly I'm getting hangs at 3% with both of them (maybe hitting rate limits?).