ask-fini

paramount

TypeScript2024–2024

ask-fini / paramount

paramount

Agent accuracy measurements for LLMs

AI & Machine Learning

Explore on GitHub Read Hacker News discussions

2Distinct people

4Mentions

0Explicit recommenders

202GitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Agent accuracy measurements for LLMs

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by week4 dated mentions

Observed mentions: 2024-05-03 14:22:37 UTC to 2024-06-13 23:18:19 UTC. Weeks start Monday, UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
Apr 29	2024-04-29 00:00:00 UTC	2024-05-06 00:00:00 UTC	1	1	0
May 6	2024-05-06 00:00:00 UTC	2024-05-13 00:00:00 UTC	0	0	0
May 13	2024-05-13 00:00:00 UTC	2024-05-20 00:00:00 UTC	0	0	0
May 20	2024-05-20 00:00:00 UTC	2024-05-27 00:00:00 UTC	0	0	0
May 27	2024-05-27 00:00:00 UTC	2024-06-03 00:00:00 UTC	1	1	0
Jun 3	2024-06-03 00:00:00 UTC	2024-06-10 00:00:00 UTC	0	0	0
Jun 10	2024-06-10 00:00:00 UTC	2024-06-17 00:00:00 UTC	2	1	1

What Hacker News says

4 sources

simonw·Jun 13, 2024

Your license is very weird: https://github.com/ask-fini/paramount/blob/main/LICENSE This file is part of paramount project, licensed under the GNU General Public License (GPL) for companies with fewer than 100 employees or fewer than 1000 invocations/month. For larger companies or higher volume, a commercial license is required. For more information, contact hello@usefini.com. I'm fine with companies not using open source licenses, but this is a very odd way to do it. Licensing something under the GPL doesn't work like this. You should look at one of the existing non-open-source licenses like the Business Source License or https://fsl.software/ rather than modifying the GPL by adding an extra paragraph at the top: https://github.com/ask-fini/paramount/commit/8345edd8f776572... Also, with a license like this it's not a

Read the rest

ccurate to say "Paramount - an Open Source package..." - that's a misuse of the term.

In Show HN: Paramount – Human Evals of AI Customer Support

hakimk·Jun 13, 2024Submission

Show HN: Paramount – Human Evals of AI Customer Support

Hey HN, Hakim here from Fini (YC S22), a startup focused on providing automated customer support bots for enterprises that have a high volume of support requests. Today, one of the largest use cases of LLMs is for the purpose of automating support. As the space has evolved over the past year, there has subsequently been a need for evaluations of LLM outputs - and a sea of LLM Evals packages have been released. "LLM evals" refer to the evaluation of large language models, assessing how well these AI systems understand and generate human-like text. These packages have recently relied on "automatic evals," where algorithms (usually another LLM) automatically test and score AI responses without human intervention. In our day to day work, we have found that Automatic Evals are not enough to get the required 95% accuracy for our Enterprise cust

Read the rest

omers. Automatic Evals are efficient, but still often miss nuances that only human expertise can catch. Automatic Evals can never replace the feedback of a trained human who is deeply knowledgeable on an organization's latest product releases, knowledgebase, policies and support issues. The key to solve this is to stop ignoring the business side of the problem, and start involving knowledgeable experts in the evaluation process. That is why we are releasing Paramount - an Open Source package which incorporates human feedback directly into the evaluation process. By simplifying the step of gathering feedback, ML Engineers can pinpoint and fix accuracy issues (prompts, knowledgebase issues) much faster. Paramount provides a framework for recording LLM function outputs (ground truth data) and facilitates human agent evaluations through a simple UI, reducing the time to identify and correct errors. Developers can integrate Paramount with a Python decorator that logs LLM interactions into a database, followed by a straightforward UI for expert review. This process aids the debugging and validation phase of launching accurate support bots. We'd love to hear what you think!

71 points · Open discussion

hakimk·May 29, 2024Submission

Show HN: Paramount – OSS package for *Human* Evals of AI support

Hey HN, Hakim here from Fini (YC S22). We focus on providing support bots for Enterprises that have a high volume of support requests. Today, one of the largest use cases of LLMs is specifically for the purpose of automating support. As the space has evolved over the past year, there has subsequently been a need for evaluations of LLM outputs - and a sea of LLM Evals packages have been released. "LLM evals" refer to the evaluation of large language models, assessing how well these AI systems understand and generate human-like text. These packages have relied on "automatic evals," where algorithms (usually another LLM) automatically test and score AI responses without human intervention. In our day to day work, we have found that Automatic Evals are not enough to get the required 95% accuracy for our Enterprise customers. Automatic Evals a

Read the rest

re efficient, but still often miss nuances that only human expertise can catch. Automatic Evals can never replace the feedback of a trained human who is deeply knowledgeable on an organization's latest product releases, knowledgebase, policies and support issues. The key to solve this is to stop ignoring the business side of the problem, and start involving knowledgeable experts in the evaluation process. That is why we are releasing Paramount - an Open Source package which involves human feedback directly into the evaluation process. By simplifying the step of gathering feedback, ML Engineers can pinpoint and fix accuracy issues (prompts, knowledgebase issues) much faster. Paramount provides a framework for recording LLM function outputs (ground truth data) and facilitates agent evaluations through a simple UI, reducing the time to identify and correct errors. Developers can integrate Paramount with a python decorator that logs LLM interactions into a database, followed by a straightforward UI for expert review. This process aids the debugging and validation phase of launching accurate support bots.

2 points · Open discussion

hakimk·May 3, 2024Submission

Show HN: Paramount is an OSS package that captures expert feedback on LLM chats

Hey HN, Hakim here from Fini (YC S22). We've seen first hand how AI chat projects pan out, and so have released an OSS library to ensure the industry gets more tools for improving outcomes. Many AI chat projects are scrapped due to persistent inaccuracies in LLM responses. Paramount is an open-source Python package designed to bridge the gap between LLM-generated and ideal responses by incorporating expert feedback directly into the evaluation process. It provides a robust framework for recording LLM function outputs (ground truth data) and facilitates agent evaluations, reducing the time to identify and correct errors. Developers can integrate Paramount with a decorator that logs LLM interactions into a CSV or database, followed by a straightforward UI for expert review. This process accelerates the debugging and validation phase of your

Read the rest

project and de-risks your launch.

9 points · Open discussion