HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
relari-ai
continuous-eval
Python2024–2024
relari-ai / continuous-eval

continuous-eval

Data-Driven Evaluation for LLM-Powered Applications

3Distinct people
5Mentions
0Explicit recommenders
519GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Data-Driven Evaluation for LLM-Powered Applications

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 4:00:59 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by week5 dated mentions

Observed mentions: 2024-02-25 18:41:20 UTC to 2024-05-09 18:11:14 UTC. Weeks start Monday, UTC.

0122024-02-19 00:00:00 UTC to 2024-02-26 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsFeb 192024-02-26 00:00:00 UTC to 2024-03-04 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2024-03-04 00:00:00 UTC to 2024-03-11 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2024-03-11 00:00:00 UTC to 2024-03-18 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-18 00:00:00 UTC to 2024-03-25 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-03-25 00:00:00 UTC to 2024-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsMar 252024-04-01 00:00:00 UTC to 2024-04-08 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-08 00:00:00 UTC to 2024-04-15 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-15 00:00:00 UTC to 2024-04-22 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-22 00:00:00 UTC to 2024-04-29 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-04-29 00:00:00 UTC to 2024-05-06 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-05-06 00:00:00 UTC to 2024-05-13 00:00:00 UTC (exclusive): 2 mentions; 1 submissions, 1 commentsMay 6

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Feb 192024-02-19 00:00:00 UTC2024-02-26 00:00:00 UTC110
Feb 262024-02-26 00:00:00 UTC2024-03-04 00:00:00 UTC110
Mar 42024-03-04 00:00:00 UTC2024-03-11 00:00:00 UTC110
Mar 112024-03-11 00:00:00 UTC2024-03-18 00:00:00 UTC000
Mar 182024-03-18 00:00:00 UTC2024-03-25 00:00:00 UTC000
Mar 252024-03-25 00:00:00 UTC2024-04-01 00:00:00 UTC000
Apr 12024-04-01 00:00:00 UTC2024-04-08 00:00:00 UTC000
Apr 82024-04-08 00:00:00 UTC2024-04-15 00:00:00 UTC000
Apr 152024-04-15 00:00:00 UTC2024-04-22 00:00:00 UTC000
Apr 222024-04-22 00:00:00 UTC2024-04-29 00:00:00 UTC000
Apr 292024-04-29 00:00:00 UTC2024-05-06 00:00:00 UTC000
May 62024-05-06 00:00:00 UTC2024-05-13 00:00:00 UTC211

What Hacker News says

2 sources
That is pretty horrible, on the level of "junior engineer" who has no idea of good industry practices and needs careful code review. I would hate to see the system as presented on any of my projects. Summary: The point of summary is to tell "why" the change was made and highlight unusual/non-trivial parts.. and examples absolutely fails there. To look at first one: - Why was "generate" result type updated? Was it customer request or general cleanup or prep for some ongoing work? - The other 3 points - are they logic fallout of output type update, or are those separate changes? For latter case, you really want to list the changes ("Updated examples to use more recent gpt-4 model" for example) - What's the point of just saying "updating X in Y" if you don't say how? this is just visual noise duplicating "file changes" tab in the PR. Suggest
Read the rest
ed changes: those are even worse - like https://github.com/relari-ai/continuous-eval/pull/38#discuss... - This is an example file and you know where the "dataset" comes from. Why would you have non-serializeable records to begin with? - This changes semantics from "let programmer know in case of error" to "produce corrupted/truncated data file in case of error" - which generally makes debugging harder and gives people nasty surprises when their file is somehow missing records. Sure, sometimes this is needed, but in that particular file it's all downsides. This should not have been proposed at all. - Even if you do want the check somehow, it's pretty terrible as written - the message does not include original error text nor bad object nor even line number. What is one supposed to do if they see it? ---- And I know some people say "you are supposed to review AI-generated content before submitted", but I am also sure many new users will think the kind of crap advice that AI generates is OK. Ellipsis authors: please stop making open source worse. Buggy patches are worse than no patches, and 1 line hand-written summary is better then AI-generated useless one. Maintainers: don't install this in your repo if you don't want crap PRs.
In Show HN: Ellipsis – Automated PR reviews and bug fixes

Show HN: Ellipsis – Automated PR reviews and bug fixes

Hi HN, hunterbrooks and nbrad here from Ellipsis (https://www.ellipsis.dev). Ellipsis automatically reviews your PRs when opened and on each new commit. If you tag @ellipsis-dev in a comment, it can make changes to the PR (via direct commit or side PR) and answer questions, just like a human. Demo video: https://www.youtube.com/watch?v=X61NGZpaNQA So far, we have dozens of open source projects and companies using Ellipsis. We seem to have landed in a kind of sweet spot where there’s a good match between the current capabilities of AI tools and the actual needs of software engineers - this doesn’t replace human review, but it saves you time by catching/fixing lots of small silly stuff. Here’s an example in the wild: https://github.com/relari-ai/continuous-eval/pull/38. Ellipsis (1) adds a PR summary; (2) finds a bug and adds a review comme
Read the rest
nt; (3) after a (human) user comments, generates a side PR with the fix; and (4) after a (human) user merges the side PR and adds another commit, re-reviews the PR and approves it Here’s another example: https://github.com/SciPhi-AI/R2R/pull/350#pullrequestreview-..., where Ellipsis adds several comments with inline suggestions that were directly merged by the developer. You can configure Ellipsis in natural language to enforce custom rules, style guides, or conventions. For example, here’s how the `jxnl/instructor` repo uses natural language rules to make sure that docs are kept in sync: https://github.com/jxnl/instructor/blob/main/ellipsis.yaml#L..., and here’s an example PR that Ellipsis came up with based on those rules: https://github.com/jxnl/instructor/pull/346. Installing into your repo takes 2 clicks at https://www.ellipsis.dev. You do have to sign up to try it out because we need you to authorize our GitHub app to read your code. Don’t worry, your code is never stored or used to train models (https://docs.ellipsis.dev/security). We’d really appreciate your feedback, thoughts, and ideas!
121 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.