HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
algorithmicsuperintelligence
optillm
Python2024–2025
algorithmicsuperintelligence / optillm

optillm

Optimizing inference proxy for LLMs

AI & Machine LearningSystems & InfrastructureScience & EngineeringNetworkingAutomation & Workflows
Explore on GitHubRead Hacker News discussions
4Distinct people
23Mentions
0Explicit recommenders
4.3KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Optimizing inference proxy for LLMs

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 7:02:50 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month23 dated mentions

Observed mentions: 2024-09-13 03:16:04 UTC to 2025-11-23 20:55:04 UTC.

02462024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 commentsSep 20242024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 7 mentions; 4 submissions, 3 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 commentsApr 20252025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 5 mentions; 2 submissions, 3 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsNov 2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC312
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC743
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC000
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC101
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC312
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC000
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC202
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC110
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC523
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC000
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC000
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC110

What Hacker News says

7 sources
rohansood15·Oct 31, 2024Submission

Launch HN: Patched (YC S24) – AI workflows for post-code tasks

Hi HN, we are Asankhaya and Rohan, founders of Patched (https://patched.codes). We help dev teams accelerate tasks like code reviews, docs, and patches through customizable, self-hostable workflows. Here’s a quick video walkthrough: https://youtu.be/5hvWGB5avVo There is a lot of excitement around code generation and the development inner loop. But having built DevSecOps startups previously, we know how the bottlenecks developers face often occur in the “outer loop”, especially after the code is written. With Patched, we want to help code get shipped as fast as it is generated. Patched has two core components. First is the workflow builder that switches seamlessly between no-code and full-code. Second is the chat-based query interface for your code, logs, and issue tracker. When used together, they can help orchestrate and automate the mo
Read the rest
st painful parts of the software development lifecycle. Here is an example PR created by a patchflow that automates SDK generation with complex type information: https://github.com/stack-auth/stack/pull/300. Under the hood, Patched is powered by our patchwork library (https://github.com/patched-codes/patchwork) and an LLM optimization proxy(https://github.com/codelion/optillm) - both of which we have open-sourced. You can run these workflows within your CI/CD pipeline, or from the command line, completely independent of our platform. This gives you full control without being locked in, while allowing us to build a monetizable product around it. While there are some great point solutions for tackling individual tasks, we believe the real solution lies in an open, holistic approach—one that teams can tweak, extend, or self-host. You can try Patched at https://app.patched.codes/signin - we’d love to hear your feedback on our approach and the user experience.
67 points · Open discussion
codelion·Oct 28, 2024
Optillm - https://github.com/codelion/optillm optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is on implementing techniques that improve reasoning over coding, logical and mathematical queries. It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time.
In Ask HN: What are you working on? (October 2024)
zby·Oct 26, 2024
These sampling based techniques is a rare occasion where experimenting with consumer hardware can let you improve on SOTA models. I don't think it will last - the end game surely will be a trainable sampler. But for now - enjoy tinkering: https://github.com/codelion/optillm implements a few of these techniques optillm authors suggest that the additional computations in Entropics don’t bring any better results in comparison with the simple CoT decoding (but I am not sure if they also check efficiency):https://x.com/asankhaya/status/1846736390152949966 It looks to me that many problems with LLMs come from something like semantic leaking, or distraction by irrelevant information (like in the GSM Symbolic paper) - maybe there is some space for improving attention too. I wrote a couple of blog posts on these subjects: https://zzbbyy.substack.c
Read the rest
om/p/semantic-leakage-quick-notes, https://zzbbyy.substack.com/p/llms-and-reasoning, https://zzbbyy.substack.com/p/o1-inference-time-turing-machi...
In Detecting when LLMs are uncertain
codelion·Oct 20, 2024Submission

Entropy Decoding in Optillm and Early Results on GSM8k

Optillm (https://github.com/codelion/optillm) now has an implementation of entropy decoding based adaptive sampling based on the work of @\_xjdr (https://github.com/xjdr-alt/entropix). The original repo is in a state of flux but the idea seems to work well. I also did an eval of the entropy decoding on GSM8k with Qwen2.5-0.5B-Instruct model in a zero shot setting. I found improvements over the base model but they are not better than what we get with a much simple CoT decoding. You can try them both in this free Gogole Colab -(https://colab.research.google.com/drive/1SpuUb8d9xAoTh32M-9wJsB50AOH54EaH?usp=sharing)
1 points · Open discussion
codelion·Oct 12, 2024Submission

Unbounded Context with Memory

Recently, Google released "FRAMES"(https://huggingface.co/datasets/google/frames-benchmark) a benchmark designed to test Retrieval-Augmented Generation (RAG) applications on Factuality, Retrieval Accuracy, and Reasoning. The benchmark didn't come with an evaluation script, so we first implemented that in optillm - https://github.com/codelion/optillm/blob/main/scripts/eval_frames_benchmark.py I had implemented a memory plugin (https://github.com/codelion/optillm/blob/main/optillm/plugins/memory_plugin.py) in optillm for adding short-term memory and unbounded context to LLMs. We used FRAMES to evaluate the memory plugin with Gemma2 model from Google. Gemma2 has a context window of 8192 so, in the paper when Google reported the results they only reported it for naive prompt which doesn't include the text retrieved via RAG. However, by using
Read the rest
the memory plugin in optillm we can make the context of any LLM to be unbounded. We managed to boost the accuracy to 30.1% v/s 5.1% as reported by Google in the paper. Also, we were able to get almost the same accuracy as Gemini with just gpt-4o-mini using optillm memory even though gpt-4o-mini has a context window that is 1/10 that of Gemini. I also ran into a very interesting refusal from Gemini to answer one of the queries, you can see the prompt here - https://aistudio.google.com/app/prompts/13PYnnu6UpukanIen9ClaFKgTI3lgPi88 The prompt just contains text from the wikipedia page of https://en.wikipedia.org/wiki/Dwight_Schrute the office character which is part of the benchmark as one of the queries - https://huggingface.co/datasets/google/frames-benchmark/viewer/default/test?q=dwig… All the Gemini models refuse to answer the query and just block the response even with safety settings all set to None. I am not sure how Google ran the evals for their benchmark without this issue.
1 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.