HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
confident-ai
deepeval
Python2023–2026
confident-ai / deepeval

deepeval

The LLM Evaluation Framework

AI & Machine LearningObservability
Explore on GitHubRead Hacker News discussions
10Distinct people
19Mentions
0Explicit recommenders
19.5KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

The LLM Evaluation Framework

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month19 dated mentions

Observed mentions: 2023-08-25 17:41:21 UTC to 2026-01-10 21:10:51 UTC.

0123452023-08-01 00:00:00 UTC to 2023-09-01 00:00:00 UTC (exclusive): 5 mentions; 3 submissions, 2 commentsAug 20232023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 3 mentions; 0 submissions, 3 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 4 mentions; 0 submissions, 4 comments2024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsOct 20242024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-12-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2026-01-01 00:00:00 UTC to 2026-02-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsJan 2026

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Aug 20232023-08-01 00:00:00 UTC2023-09-01 00:00:00 UTC532
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC110
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC110
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC000
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC211
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC000
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC303
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC404
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC000
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC000
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC000
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC000
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC000
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC000
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC110
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC000
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC000
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC000
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC110
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC000
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC000
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC000
Dec 20252025-12-01 00:00:00 UTC2026-01-01 00:00:00 UTC000
Jan 20262026-01-01 00:00:00 UTC2026-02-01 00:00:00 UTC110

What Hacker News says

19 sources
dustfinger·Jan 10, 2026Submission

Show HN: Tag driven changelog generator (MDX) with optional LLM summaries

I built a tag-driven changelog generator for an OSS project I maintain and used it to backfill our 2025 docs changelog. It walks git tags, finds merged PRs between releases, buckets them into categories, and renders MDX (Docusaurus in our case) organized by year -> month -> category -> version. There is an optional LLM mode that produces structured JSON via a Pydantic schema for PR entry and monthly summaries. Example: python .scripts/changelog/generate.py --year 2025 --github --ai --ai-model gpt-5.2 (or --help) Gotcha: if you use --github, set GITHUB_TOKEN or you will most likely hit GitHub rate limits. Repo script: https://github.com/confident-ai/deepeval/blob/main/.scripts/... Disclosure: I maintain DeepEval. Happy to answer questions and take feedback.
1 points · Open discussion
jeffreyip·Feb 20, 2025Submission

Launch HN: Confident AI (YC W25) – Open-source evaluation framework for LLM apps

Hi HN - we're Jeffrey and Kritin, and we're building Confident AI (https://confident-ai.com). This is the cloud platform for DeepEval (https://github.com/confident-ai/deepeval), our open-source package that helps engineers evaluate and unit-test LLM applications. Think Pytest for LLMs. We spent the past year building DeepEval with the goal of providing the best LLM evaluation developer experience, growing it to run over 600K evaluations daily in CI/CD pipelines of enterprises like BCG, AstraZeneca, AXA, and Capgemini. But the fact that DeepEval simply runs, and does nothing with the data afterward, isn’t the best experience. If you want to inspect failing test cases, identify regressions, or even pick the best model/prompt combination, you need more than just DeepEval. That’s why we built a platform around it. Here’s a quick demo video of
Read the rest
how everything works: https://youtu.be/PB3ngq7x4ko Confident AI is great for RAG pipelines, agents, and chatbots. Typical use cases involve allowing companies to switch the underlying LLM, rewrite prompts for newer (and possibly cheaper) models, and keep test sets in sync with the codebase where DeepEval tests are run. Our platform features a "dataset editor," a "regression catcher," and "iteration insights". The datasets editor in Confident AI allows domain experts to edit datasets while keeping them in sync with your codebase for evaluation. We’ll then generate sharable LLM testing/benchmark reports once DeepEval has finished running evaluations on these datasets that are pulled from the cloud. The regression catcher then identifies any regressions in your new implementation, and we use these evaluation results to determine the best iteration based on your metric scores. Our goal is to make benchmarking LLM applications so reliable that picking the best implementation is as simple as reading the metric values off the dashboard. To achieve this, the quality of curated datasets and the accuracy and reliability of metrics must be the highest possible. This brings us to our current limitations. Right now, DeepEval’s primary evaluation method is LLM-as-a-judge. We use techniques such as GEval and question-answer generation to improve reliability, but these methods can still be inconsistent. Even with high-quality datasets curated by domain experts, our evaluation metrics remain the biggest blocker to our goal. To address this, we recently released a DAG (Directed Acyclic Graph) metric in DeepEval. It is a decision-tree-based, LLM-as-a-judge metric that provides deterministic results by breaking a test case into finer atomic units. Each edge represents a decision, each node represents an LLM evaluation step, and each leaf node returns a score. It works best in scenarios where success criteria are clearly defined, such as text summarization. The DAG metric is still in its early stages, but our hope is that by moving towards better, code-driven, open-source metrics, Confident AI can deliver deterministic LLM benchmarks that anyone can blindly trust. We hope you’ll give Confident AI a try. Quickstart here: https://docs.confident-ai.com/confident-ai/confident-ai-intr... The platform runs on a freemium tier, and we've dropped the need to signup with a work email for the next four days. Looking forward to your thoughts!
117 points · Open discussion
renchuw·Feb 13, 2024
Fair question. Evaluate refers to the phase after training to check if the training is good. Usually the flow goes training -> evaluation -> deployment (what you called inference). This project is aimed for evaluation. Evaluation can be slow (might even be slower than training if you're finetuning on a small domain specific subset)! So there are [quite](https://github.com/microsoft/promptbench) [a](https://github.com/confident-ai/deepeval) [few](https://github.com/openai/evals) [frameworks](https://github.com/EleutherAI/lm-evaluation-harness) working on evaluation, however, all of them are quite slow, because LLM are slow if you don't have infinite money. [This](https://github.com/open-compass/opencompass) one tries to speed up by parallelizing on multiple computers, but none of them takes advantage of the fact that many evaluation querie
Read the rest
s might be similar and all try to evaluate on all given queries. And that's where this project might come in handy.
In Show HN: Faster LLM evaluation with Bayesian optimization

Showing the 15 most recent of 19 sources. Enable JavaScript to search and browse every indexed mention.

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.