RZrzk·Nov 23, 2025Submission
OptiLLM: Accuracy improvements on reasoning tasks with zero training
2 points · Open discussionalgorithmicsuperintelligence
optillm
Python2024–2025
algorithmicsuperintelligence / optillm
optillm
Optimizing inference proxy for LLMs
4Distinct people
23Mentions
0Explicit recommenders
4.3KGitHub stars
What to know before exploring
What it is
Optimizing inference proxy for LLMs
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 11:56:56 AM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month23 dated mentions
Observed mentions: 2024-09-13 03:16:04 UTC to 2025-11-23 20:55:04 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Sep 2024 | 2024-09-01 00:00:00 UTC | 2024-10-01 00:00:00 UTC | 3 | 1 | 2 |
| Oct 2024 | 2024-10-01 00:00:00 UTC | 2024-11-01 00:00:00 UTC | 7 | 4 | 3 |
| Nov 2024 | 2024-11-01 00:00:00 UTC | 2024-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2024 | 2024-12-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2025 | 2025-01-01 00:00:00 UTC | 2025-02-01 00:00:00 UTC | 1 | 0 | 1 |
| Feb 2025 | 2025-02-01 00:00:00 UTC | 2025-03-01 00:00:00 UTC | 3 | 1 | 2 |
| Mar 2025 | 2025-03-01 00:00:00 UTC | 2025-04-01 00:00:00 UTC | 0 | 0 | 0 |
| Apr 2025 | 2025-04-01 00:00:00 UTC | 2025-05-01 00:00:00 UTC | 2 | 0 | 2 |
| May 2025 | 2025-05-01 00:00:00 UTC | 2025-06-01 00:00:00 UTC | 1 | 1 | 0 |
| Jun 2025 | 2025-06-01 00:00:00 UTC | 2025-07-01 00:00:00 UTC | 5 | 2 | 3 |
| Jul 2025 | 2025-07-01 00:00:00 UTC | 2025-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2025 | 2025-08-01 00:00:00 UTC | 2025-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2025 | 2025-09-01 00:00:00 UTC | 2025-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2025 | 2025-10-01 00:00:00 UTC | 2025-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2025 | 2025-11-01 00:00:00 UTC | 2025-12-01 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
23 sourcesShow HN: DeepThink Plugin – Bring Gemini 2.5's parallel reasoning to open models
I built an open-source plugin in OptiLLM that implements Google's "Deep Think" reasoning approach for local models like DeepSeek R1 and Qwen3.
Google's recent Gemini 2.5 report introduced Deep Think - a technique where models generate multiple hypotheses in parallel and critique them before arriving at final answers. It achieves SOTA results on math olympiads and competitive coding benchmarks.
The plugin works by modifying the inference pipeline to explore multiple solution paths simultaneously, then synthesizing the best approach. Instead of single-pass generation, the model essentially runs an internal debate before responding.
Technical details:
- Works with any model that supports structured reasoning patterns
- Implements parallel thinking during response generation
- Particularly effective for complex reasoning tasks, math, and co
Read the rest
ding problems
- Increases inference time but significantly improves answer quality
Link: https://github.com/codelion/optillm/tree/main/optillm/plugin...
Demo: https://www.youtube.com/watch?v=b06kD1oWBA4
The implementation won the Cerebras & OpenRouter Qwen 3 Hackathon, but more importantly, it's now available for anyone running local models.
Questions for HN:
- Has anyone tried similar parallel reasoning approaches with local models?
- What other proprietary techniques do you think would be valuable to open-source?
- Any suggestions for optimizing the performance trade-offs?
The goal is to democratize advanced reasoning capabilities that were previously locked behind APIs. Would love feedback on the approach and ideas for improvements.
You can run in two modes, by default you run in the inference mode without learning. So, the changes you made will be used. If you switch to learning mode then the strategies are updated/refined and merged based on a config that you can control.
# How often to perform maintenance operations (merge, prune)
MAINTENANCE_INTERVAL = 40
# Strategy selection thresholds
STRATEGY_CREATION_THRESHOLD = 0.7 # Higher threshold to avoid creating similar strategies
STRATEGY_MERGING_THRESHOLD = 0.6 # Lower threshold to merge more similar strategies
MIN_SUCCESS_RATE_FOR_INFERENCE = 0.4 # Minimum success rate for a strategy to be used during inference
The configs are all defined here - https://github.com/codelion/optillm/blob/main/optillm/plugin...
In Show HN: System Prompt Learning – LLMs Learn Problem-Solving from ExperienceWe do not allow the strategies to keep growing there is a refinement phase where we refine and merge existing strategies. The experiments were run with this config - https://github.com/codelion/optillm/blob/main/optillm/plugin... which allows a maximum of 10 strategies of each type.
In Show HN: System Prompt Learning – LLMs Learn Problem-Solving from ExperienceWe have some examples in the plugin README:
https://github.com/codelion/optillm/tree/main/optillm/plugin...
E.g. This was the strategy discovered by optiLLM for solving word problems:
*Refined Strategy for Solving Word Problems:*
1. *Understand:*\n * Read the problem carefully (multiple times).\n * Identify the question (what are you trying to find?).\n * List all given information (facts, numbers, units).\n * Clarify ambiguous terms/units.
2. *Organize Information & Identify Unknowns:*\n * Choose an organization method: (e.g., table, diagram, list, drawing).\n * Clearly identify the unknowns (what you need to solve for).
3. *Plan and Translate:*\n * Define all variables with units (e.g., `p = number of pennies`, `c = number of compartments`).\n * Identify relationships between knowns and unkno
Read the rest
wns.\n * Convert units if necessary.\n * Write equations or expressions, including units, that relate the knowns and unknowns.\n * Ensure units are consistent throughout the equations.\n * Outline the solution steps.
4. *Solve:*\n * Show work step-by-step.\n * Track units throughout calculations.\n * Calculate accurately.\n * Solve for the unknowns.\
5. *Evaluate and Verify:*\n * Check if the answer is reasonable.\n * Verify the answer.
6. *Summarize:*\n * State the answer with units
Full list of strategies discovered is available here -https://github.com/codelion/optillm/blob/main/optillm/plugin...
Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
I built a system that lets LLMs automatically learn and improve problem-solving strategies over time, inspired by Andrej Karpathy's idea of a "third paradigm" for LLM learning.
The basic idea: instead of using static system prompts, the LLM builds up a database of strategies that actually work for different problem types. When you give it a new problem, it selects the most relevant strategies, applies them, then evaluates how well they worked and refines them.
For example, after seeing enough word problems, it learned this strategy:
1) Read carefully and identify unknowns,
2) Define variables with units,
3) Write equations,
4) Solve step-by-step,
5) Verify the answer.
All strategies are stored as human-readable JSON that you can inspect and edit.
I tested it on math benchmarks and saw decent improvements - 8.6% better on Arena Hard, 6.67%
Read the rest
on AIME24. After 500 queries, the system had created 129 strategies and refined 97 of them.
The implementation is an open-source plugin for optillm (our inference optimization proxy). It works with any OpenAI-compatible API - you just add "spl-" to your model name. Has two modes: inference-only (uses existing strategies) and learning mode (creates and refines strategies).
What's interesting is that it bridges the gap between the sophisticated system prompts that production AI uses and the basic prompts most of us work with. Your model literally gets better at the types of problems you throw at it.
Built it because I noticed ChatGPT, Claude etc. have incredibly detailed system prompts with problem-solving frameworks, but most developers use basic prompts and miss out on those performance gains. The approach is inspired by Andrej Karpathy's tweet about a "third paradigm" for LLM learning beyond just pretraining and fine-tuning: https://x.com/karpathy/status/1921368644069765486
The strategies are completely transparent - you can see exactly what the system learned and why it's making certain decisions. No black box learning.
https://github.com/codelion/optillm/tree/main/optillm/plugin...
Would love feedback on the approach. Has anyone else experimented with LLMs learning from their own experience?
Show HN: AutoThink – Boosts local LLM performance with adaptive reasoning
I built AutoThink, a technique that makes local LLMs reason more efficiently by adaptively allocating computational resources based on query complexity.
The core idea: instead of giving every query the same "thinking time," classify queries as HIGH or LOW complexity and allocate thinking tokens accordingly. Complex reasoning gets 70-90% of tokens, simple queries get 20-40%.
I also implemented steering vectors derived from Pivotal Token Search (originally from Microsoft's Phi-4 paper) that guide the model's reasoning patterns during generation. These vectors encourage behaviors like numerical accuracy, self-correction, and thorough exploration.
Results on DeepSeek-R1-Distill-Qwen-1.5B:
- GPQA-Diamond: 31.06% vs 21.72% baseline (+43% relative improvement)
- MMLU-Pro: 26.38% vs 25.58% baseline
- Uses fewer tokens than baseline approaches
Wor
Read the rest
ks with any local reasoning model - DeepSeek, Qwen, custom fine-tuned models. No API dependencies.
The technique builds on two things I developed: an adaptive classification framework that can learn new complexity categories without retraining, and an open source implementation of Pivotal Token Search.
Technical paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5253327
Code and examples: https://github.com/codelion/optillm/tree/main/optillm/autoth...
PTS implementation: https://github.com/codelion/pts
I'm curious about your thoughts on adaptive resource allocation for AI reasoning. Have you tried similar approaches with your local models?
Not standard but one of several techniques, you can see them in our open source inference proxy - https://github.com/codelion/optillm
Cerebras has used optillm for optimising inference with techniques like CePO and LongCePO.
In Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language ModelsThere are ways to improve the performance of local LLMs with inference time techniques. You can try with optillm - https://github.com/codelion/optillm it is possible to match the performance of larger models on narrow tasks by doing more at inference.
In Local LLM inference – impressive but too hard to work withIt's definitely possible to focus on the CUDA/GPU side without diving deep into the math. Understanding parallel computing principles and memory optimization is key. I've found that focusing on specific use cases, like optimizing inference, can be a good way to learn. On that note, you might find https://github.com/codelion/optillm useful – it optimizes LLM inference and could give you practical experience with GPU utilization. What kind of AI applications are you most interested in optimizing?
In Introduction to CUDA programming for Python developersShow HN: OptiLLMBench – Test how inference optimization tricks scale up LLMs
OptiLLMBench is a new benchmark designed to evaluate how different inference optimization techniques (like ReRead, Chain-of-Thought, etc.) can improve LLM performance without any model changes or fine-tuning.
To help understand real-world impact, I've included first results with Gemini 2.0 Flash:
ReRead (RE2): +5% accuracy, +14% faster
Chain-of-Thought Reflection: +5% boost
Base performance: 51%
The benchmark evaluates models on:
Math word problems (GSM8K)
Formal mathematics (MMLU Math)
Logical reasoning (AQUA-RAT)
Yes/no comprehension (BoolQ)
The code works as a drop-in proxy - just point your OpenAI compatible endpoint to it and it'll apply the optimizations automatically.
Dataset: https://huggingface.co/datasets/codelion/optillmbench
Code: https://github.com/codelion/optillm
Would love feedback from the HN community on additional optim
Read the rest
ization techniques to include or ways to improve the benchmark.
Note: The dataset and proxy are completely open source and support any OpenAI API compatible endpoint.
This is great, can this be used to implement a sampler based on entropy like entropix (implemented in optillm here - https://github.com/codelion/optillm/blob/main/optillm/entrop...)
In Show HN: Klarity – Open-source tool to analyze uncertainty/entropy in LLM outputThe full dataset is here - https://huggingface.co/datasets/AI-MO/aimo-validation-aime you can use the eval script I have in optillm to benchmark on it - https://github.com/codelion/optillm/blob/main/scripts/eval_a...
In Kimi K1.5: Scaling Reinforcement Learning with LLMsLaunch HN: Patched (YC S24) – AI workflows for post-code tasks
Hi HN, we are Asankhaya and Rohan, founders of Patched (https://patched.codes). We help dev teams accelerate tasks like code reviews, docs, and patches through customizable, self-hostable workflows. Here’s a quick video walkthrough: https://youtu.be/5hvWGB5avVo
There is a lot of excitement around code generation and the development inner loop. But having built DevSecOps startups previously, we know how the bottlenecks developers face often occur in the “outer loop”, especially after the code is written. With Patched, we want to help code get shipped as fast as it is generated.
Patched has two core components. First is the workflow builder that switches seamlessly between no-code and full-code. Second is the chat-based query interface for your code, logs, and issue tracker. When used together, they can help orchestrate and automate the mo
Read the rest
st painful parts of the software development lifecycle.
Here is an example PR created by a patchflow that automates SDK generation with complex type information: https://github.com/stack-auth/stack/pull/300.
Under the hood, Patched is powered by our patchwork library (https://github.com/patched-codes/patchwork) and an LLM optimization proxy(https://github.com/codelion/optillm) - both of which we have open-sourced. You can run these workflows within your CI/CD pipeline, or from the command line, completely independent of our platform. This gives you full control without being locked in, while allowing us to build a monetizable product around it.
While there are some great point solutions for tackling individual tasks, we believe the real solution lies in an open, holistic approach—one that teams can tweak, extend, or self-host.
You can try Patched at https://app.patched.codes/signin - we’d love to hear your feedback on our approach and the user experience.
Optillm - https://github.com/codelion/optillm
optillm is an OpenAI API compatible optimizing inference proxy which implements several state-of-the-art techniques that can improve the accuracy and performance of LLMs. The current focus is on implementing techniques that improve reasoning over coding, logical and mathematical queries. It is possible to beat the frontier models using these techniques across diverse tasks by doing additional compute at inference time.
In Ask HN: What are you working on? (October 2024)Showing the 15 most recent of 23 sources. Enable JavaScript to search and browse every indexed mention.