HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
uptrain-ai
uptrain
Python2023–2024
uptrain-ai / uptrain

uptrain

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

4Distinct people
10Mentions
0Explicit recommenders
2.4KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 11:39:46 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month10 dated mentions

Observed mentions: 2023-01-25 15:03:54 UTC to 2024-09-10 07:14:40 UTC.

01232023-01-01 00:00:00 UTC to 2023-02-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsJan 20232023-02-01 00:00:00 UTC to 2023-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-03-01 00:00:00 UTC to 2023-04-01 00:00:00 UTC (exclusive): 3 mentions; 2 submissions, 1 comments2023-04-01 00:00:00 UTC to 2023-05-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-05-01 00:00:00 UTC to 2023-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-06-01 00:00:00 UTC to 2023-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-07-01 00:00:00 UTC to 2023-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-08-01 00:00:00 UTC to 2023-09-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2023-09-01 00:00:00 UTC to 2023-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-10-01 00:00:00 UTC to 2023-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-11-01 00:00:00 UTC to 2023-12-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsNov 20232023-12-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-01-01 00:00:00 UTC to 2024-02-01 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 comments2024-02-01 00:00:00 UTC to 2024-03-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-01 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-01 00:00:00 UTC to 2024-05-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-01 00:00:00 UTC to 2024-06-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-06-01 00:00:00 UTC to 2024-07-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-07-01 00:00:00 UTC to 2024-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-08-01 00:00:00 UTC to 2024-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsSep 2024

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Jan 20232023-01-01 00:00:00 UTC2023-02-01 00:00:00 UTC110
Feb 20232023-02-01 00:00:00 UTC2023-03-01 00:00:00 UTC000
Mar 20232023-03-01 00:00:00 UTC2023-04-01 00:00:00 UTC321
Apr 20232023-04-01 00:00:00 UTC2023-05-01 00:00:00 UTC110
May 20232023-05-01 00:00:00 UTC2023-06-01 00:00:00 UTC000
Jun 20232023-06-01 00:00:00 UTC2023-07-01 00:00:00 UTC000
Jul 20232023-07-01 00:00:00 UTC2023-08-01 00:00:00 UTC000
Aug 20232023-08-01 00:00:00 UTC2023-09-01 00:00:00 UTC110
Sep 20232023-09-01 00:00:00 UTC2023-10-01 00:00:00 UTC000
Oct 20232023-10-01 00:00:00 UTC2023-11-01 00:00:00 UTC000
Nov 20232023-11-01 00:00:00 UTC2023-12-01 00:00:00 UTC110
Dec 20232023-12-01 00:00:00 UTC2024-01-01 00:00:00 UTC000
Jan 20242024-01-01 00:00:00 UTC2024-02-01 00:00:00 UTC211
Feb 20242024-02-01 00:00:00 UTC2024-03-01 00:00:00 UTC000
Mar 20242024-03-01 00:00:00 UTC2024-04-01 00:00:00 UTC000
Apr 20242024-04-01 00:00:00 UTC2024-05-01 00:00:00 UTC000
May 20242024-05-01 00:00:00 UTC2024-06-01 00:00:00 UTC000
Jun 20242024-06-01 00:00:00 UTC2024-07-01 00:00:00 UTC000
Jul 20242024-07-01 00:00:00 UTC2024-08-01 00:00:00 UTC000
Aug 20242024-08-01 00:00:00 UTC2024-09-01 00:00:00 UTC000
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC110

What Hacker News says

1 sources

Show HN: UpTrain (YC W23) – open-source tool to evaluate LLM response quality

Hello, we are Shikha and Sourabh, founders of UpTrain(YC W23) - an open-source tool to evaluate the performance of your LLM applications on aspects such as correctness, tonality, hallucination, fluency, etc. The Problem: Unlike traditional Machine learning or Deep learning models where we always have a unique Ground Truth and can define metrics like Precision, Recall, accuracy, etc. to quantify the model’s performance, LLMs are trickier and it is very difficult to estimate if their response is correct or not. If you are using GPT-4 to write a recruitment email, there is no unique correct email to do a word-to-word comparison against. As you build an LLM application, you want to compare it against different model providers, prompt configurations, etc., and figure out the best working combination. Instead of manually skimming through a coup
Read the rest
le of model responses, you want to run them through hundreds of test cases, aggregate their scores, and make an informed decision. Additionally, as your application generates responses for real user queries, you don’t want to wait for them to complain about the model inaccuracy, instead, you want to monitor the model’s performance over time and get alerted in case of any drifts. Again, at the core of it, you want a tool to evaluate the quality of your LLM response and assign quantitative scores. The Solution: To solve this, we are building UpTrain which has a set of evaluation metrics so that you can know when your application is going wrong. These metrics include traditional NLP metrics like Rogue, Bleu, etc., embeddings similarity metrics as well as model grading scores i.e. where we use LLMs to evaluate different aspects of your response. A few of these evaluation metrics include: 1. Response Relevancy: Measures if the response contains any irrelevant information 2. Response Completeness: Measures if the response answers all aspects of the given question 3. Factual Accuracy: Measures hallucinations i.e. if the response has any made-up information or not with respect to the provided context 4. Retrieved Context Quality: Measures if the retrieved context has sufficient information to answer the given question 5. Response Tonality: Measures if the response aligns with a specific persona or desired tone etc. We have designed workflows so that you can easily add your testing dataset, configure which checks you want to run (you can also define custom checks suitable for your use case) and conveniently access the results via Streamlit dashboards. UpTrain also has experimentation capabilities where you can specify different prompt variations and models to test across and use these quantitative checks to find the best configuration for your application. You can also use UpTrain to monitor your application’s performance and find avenues for improvement. We integrate directly with your databases (BigQuery, Postgres, MongoDB, etc.) and can run daily evaluations. We’ve launched the tool under an Apache 2.0 license to make it easy for everyone to integrate it into their LLM workflows. Additionally, we also provide managed service (with a free trial) where you can run LLM evaluations via an API request or through UpTrain testing console. We would love for you to try it out and give your feedback. Links: Demo: https://demo.uptrain.ai/evals_demo/ Github repo: https://github.com/uptrain-ai/uptrain Create an account (free): https://uptrain.ai/dashboard UpTrain testing console (need an account): https://demo.uptrain.ai/dashboard Website: https://uptrain.ai/
12 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.