HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
manyoso
haltt4llm
Python2023–2023
manyoso / haltt4llm

haltt4llm

This project is an attempt to create a common metric to test LLM's for progress in eliminating hallucinations which is the most serious current problem in widespread adoption of LLM's for many real purposes.

AI & Machine LearningTesting & Quality
Explore on GitHubRead Hacker News discussions
3Distinct people
4Mentions
0Explicit recommenders
223GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

This project is an attempt to create a common metric to test LLM's for progress in eliminating hallucinations which is the most serious current problem in widespread adoption of LLM's for many real purposes.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by 6-hour interval4 dated mentions

Observed mentions: 2023-03-29 23:43:17 UTC to 2023-04-04 02:35:30 UTC.

012023-03-29 18:00:00 UTC to 2023-03-30 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsMar 29, 18:002023-03-30 00:00:00 UTC to 2023-03-30 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-03-30 06:00:00 UTC to 2023-03-30 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-03-30 12:00:00 UTC to 2023-03-30 18:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-03-30 18:00:00 UTC to 2023-03-31 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-03-31 00:00:00 UTC to 2023-03-31 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-03-31 06:00:00 UTC to 2023-03-31 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-03-31 12:00:00 UTC to 2023-03-31 18:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023-03-31 18:00:00 UTC to 2023-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-01 00:00:00 UTC to 2023-04-01 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-01 06:00:00 UTC to 2023-04-01 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsApr 1, 06:002023-04-01 12:00:00 UTC to 2023-04-01 18:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-01 18:00:00 UTC to 2023-04-02 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-02 00:00:00 UTC to 2023-04-02 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-02 06:00:00 UTC to 2023-04-02 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-02 12:00:00 UTC to 2023-04-02 18:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-02 18:00:00 UTC to 2023-04-03 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-03 00:00:00 UTC to 2023-04-03 06:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-03 06:00:00 UTC to 2023-04-03 12:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-03 12:00:00 UTC to 2023-04-03 18:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-03 18:00:00 UTC to 2023-04-04 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-04-04 00:00:00 UTC to 2023-04-04 06:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsApr 4, 00:00

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Mar 29, 18:002023-03-29 18:00:00 UTC2023-03-30 00:00:00 UTC110
Mar 30, 00:002023-03-30 00:00:00 UTC2023-03-30 06:00:00 UTC000
Mar 30, 06:002023-03-30 06:00:00 UTC2023-03-30 12:00:00 UTC000
Mar 30, 12:002023-03-30 12:00:00 UTC2023-03-30 18:00:00 UTC000
Mar 30, 18:002023-03-30 18:00:00 UTC2023-03-31 00:00:00 UTC110
Mar 31, 00:002023-03-31 00:00:00 UTC2023-03-31 06:00:00 UTC000
Mar 31, 06:002023-03-31 06:00:00 UTC2023-03-31 12:00:00 UTC000
Mar 31, 12:002023-03-31 12:00:00 UTC2023-03-31 18:00:00 UTC101
Mar 31, 18:002023-03-31 18:00:00 UTC2023-04-01 00:00:00 UTC000
Apr 1, 00:002023-04-01 00:00:00 UTC2023-04-01 06:00:00 UTC000
Apr 1, 06:002023-04-01 06:00:00 UTC2023-04-01 12:00:00 UTC000
Apr 1, 12:002023-04-01 12:00:00 UTC2023-04-01 18:00:00 UTC000
Apr 1, 18:002023-04-01 18:00:00 UTC2023-04-02 00:00:00 UTC000
Apr 2, 00:002023-04-02 00:00:00 UTC2023-04-02 06:00:00 UTC000
Apr 2, 06:002023-04-02 06:00:00 UTC2023-04-02 12:00:00 UTC000
Apr 2, 12:002023-04-02 12:00:00 UTC2023-04-02 18:00:00 UTC000
Apr 2, 18:002023-04-02 18:00:00 UTC2023-04-03 00:00:00 UTC000
Apr 3, 00:002023-04-03 00:00:00 UTC2023-04-03 06:00:00 UTC000
Apr 3, 06:002023-04-03 06:00:00 UTC2023-04-03 12:00:00 UTC000
Apr 3, 12:002023-04-03 12:00:00 UTC2023-04-03 18:00:00 UTC000
Apr 3, 18:002023-04-03 18:00:00 UTC2023-04-04 00:00:00 UTC000
Apr 4, 00:002023-04-04 00:00:00 UTC2023-04-04 06:00:00 UTC101

What Hacker News says

4 sources
Imnimo·Mar 31, 2023
Hmm, but if I'm reading this code correctly, it's also correct if the text of the correct answer appears anywhere in the output. Even if other incorrect answers also appear. https://github.com/manyoso/haltt4llm/blob/main/take_test.py#... So the above answer would have been correct were it not for the fact that it said "doubled its" rather than "double it". Without seeing the log of answers marked correct, I'm skeptical that GPT4All, which seems to produce rambling prose for all of its incorrect answers, is actually picking one of the multiple choice options the rest of the time. It seems like a model could get 100% 'correct' just by repeating back all five options.
In GPT4All outscores GPT-3.5 on new hallucination metric

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.