HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
algorithmicsuperintelligence
optillm
Python2024–2025
algorithmicsuperintelligence / optillm

optillm

Optimizing inference proxy for LLMs

4Distinct people
23Mentions
0Explicit recommenders
4.3KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Optimizing inference proxy for LLMs

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 6:01:08 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by month23 dated mentions

Observed mentions: 2024-09-13 03:16:04 UTC to 2025-11-23 20:55:04 UTC.

02462024-09-01 00:00:00 UTC to 2024-10-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 commentsSep 20242024-10-01 00:00:00 UTC to 2024-11-01 00:00:00 UTC (exclusive): 7 mentions; 4 submissions, 3 comments2024-11-01 00:00:00 UTC to 2024-12-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2024-12-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-01-01 00:00:00 UTC to 2025-02-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2025-02-01 00:00:00 UTC to 2025-03-01 00:00:00 UTC (exclusive): 3 mentions; 1 submissions, 2 comments2025-03-01 00:00:00 UTC to 2025-04-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-04-01 00:00:00 UTC to 2025-05-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 commentsApr 20252025-05-01 00:00:00 UTC to 2025-06-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments2025-06-01 00:00:00 UTC to 2025-07-01 00:00:00 UTC (exclusive): 5 mentions; 2 submissions, 3 comments2025-07-01 00:00:00 UTC to 2025-08-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-08-01 00:00:00 UTC to 2025-09-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-09-01 00:00:00 UTC to 2025-10-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-10-01 00:00:00 UTC to 2025-11-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-11-01 00:00:00 UTC to 2025-12-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsNov 2025

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
Sep 20242024-09-01 00:00:00 UTC2024-10-01 00:00:00 UTC312
Oct 20242024-10-01 00:00:00 UTC2024-11-01 00:00:00 UTC743
Nov 20242024-11-01 00:00:00 UTC2024-12-01 00:00:00 UTC000
Dec 20242024-12-01 00:00:00 UTC2025-01-01 00:00:00 UTC000
Jan 20252025-01-01 00:00:00 UTC2025-02-01 00:00:00 UTC101
Feb 20252025-02-01 00:00:00 UTC2025-03-01 00:00:00 UTC312
Mar 20252025-03-01 00:00:00 UTC2025-04-01 00:00:00 UTC000
Apr 20252025-04-01 00:00:00 UTC2025-05-01 00:00:00 UTC202
May 20252025-05-01 00:00:00 UTC2025-06-01 00:00:00 UTC110
Jun 20252025-06-01 00:00:00 UTC2025-07-01 00:00:00 UTC523
Jul 20252025-07-01 00:00:00 UTC2025-08-01 00:00:00 UTC000
Aug 20252025-08-01 00:00:00 UTC2025-09-01 00:00:00 UTC000
Sep 20252025-09-01 00:00:00 UTC2025-10-01 00:00:00 UTC000
Oct 20252025-10-01 00:00:00 UTC2025-11-01 00:00:00 UTC000
Nov 20252025-11-01 00:00:00 UTC2025-12-01 00:00:00 UTC110

What Hacker News says

5 sources
codelion·Jun 19, 2025Submission

Show HN: DeepThink Plugin – Bring Gemini 2.5's parallel reasoning to open models

I built an open-source plugin in OptiLLM that implements Google's "Deep Think" reasoning approach for local models like DeepSeek R1 and Qwen3. Google's recent Gemini 2.5 report introduced Deep Think - a technique where models generate multiple hypotheses in parallel and critique them before arriving at final answers. It achieves SOTA results on math olympiads and competitive coding benchmarks. The plugin works by modifying the inference pipeline to explore multiple solution paths simultaneously, then synthesizing the best approach. Instead of single-pass generation, the model essentially runs an internal debate before responding. Technical details: - Works with any model that supports structured reasoning patterns - Implements parallel thinking during response generation - Particularly effective for complex reasoning tasks, math, and co
Read the rest
ding problems - Increases inference time but significantly improves answer quality Link: https://github.com/codelion/optillm/tree/main/optillm/plugin... Demo: https://www.youtube.com/watch?v=b06kD1oWBA4 The implementation won the Cerebras & OpenRouter Qwen 3 Hackathon, but more importantly, it's now available for anyone running local models. Questions for HN: - Has anyone tried similar parallel reasoning approaches with local models? - What other proprietary techniques do you think would be valuable to open-source? - Any suggestions for optimizing the performance trade-offs? The goal is to democratize advanced reasoning capabilities that were previously locked behind APIs. Would love feedback on the approach and ideas for improvements.
4 points · Open discussion
You can run in two modes, by default you run in the inference mode without learning. So, the changes you made will be used. If you switch to learning mode then the strategies are updated/refined and merged based on a config that you can control. # How often to perform maintenance operations (merge, prune) MAINTENANCE_INTERVAL = 40 # Strategy selection thresholds STRATEGY_CREATION_THRESHOLD = 0.7 # Higher threshold to avoid creating similar strategies STRATEGY_MERGING_THRESHOLD = 0.6 # Lower threshold to merge more similar strategies MIN_SUCCESS_RATE_FOR_INFERENCE = 0.4 # Minimum success rate for a strategy to be used during inference The configs are all defined here - https://github.com/codelion/optillm/blob/main/optillm/plugin...
In Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
We have some examples in the plugin README: https://github.com/codelion/optillm/tree/main/optillm/plugin... E.g. This was the strategy discovered by optiLLM for solving word problems: *Refined Strategy for Solving Word Problems:* 1. *Understand:*\n * Read the problem carefully (multiple times).\n * Identify the question (what are you trying to find?).\n * List all given information (facts, numbers, units).\n * Clarify ambiguous terms/units. 2. *Organize Information & Identify Unknowns:*\n * Choose an organization method: (e.g., table, diagram, list, drawing).\n * Clearly identify the unknowns (what you need to solve for). 3. *Plan and Translate:*\n * Define all variables with units (e.g., `p = number of pennies`, `c = number of compartments`).\n * Identify relationships between knowns and unkno
Read the rest
wns.\n * Convert units if necessary.\n * Write equations or expressions, including units, that relate the knowns and unknowns.\n * Ensure units are consistent throughout the equations.\n * Outline the solution steps. 4. *Solve:*\n * Show work step-by-step.\n * Track units throughout calculations.\n * Calculate accurately.\n * Solve for the unknowns.\ 5. *Evaluate and Verify:*\n * Check if the answer is reasonable.\n * Verify the answer. 6. *Summarize:*\n * State the answer with units Full list of strategies discovered is available here -https://github.com/codelion/optillm/blob/main/optillm/plugin...
In Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience
codelion·Jun 2, 2025Submission

Show HN: System Prompt Learning – LLMs Learn Problem-Solving from Experience

I built a system that lets LLMs automatically learn and improve problem-solving strategies over time, inspired by Andrej Karpathy's idea of a "third paradigm" for LLM learning. The basic idea: instead of using static system prompts, the LLM builds up a database of strategies that actually work for different problem types. When you give it a new problem, it selects the most relevant strategies, applies them, then evaluates how well they worked and refines them. For example, after seeing enough word problems, it learned this strategy: 1) Read carefully and identify unknowns, 2) Define variables with units, 3) Write equations, 4) Solve step-by-step, 5) Verify the answer. All strategies are stored as human-readable JSON that you can inspect and edit. I tested it on math benchmarks and saw decent improvements - 8.6% better on Arena Hard, 6.67%
Read the rest
on AIME24. After 500 queries, the system had created 129 strategies and refined 97 of them. The implementation is an open-source plugin for optillm (our inference optimization proxy). It works with any OpenAI-compatible API - you just add "spl-" to your model name. Has two modes: inference-only (uses existing strategies) and learning mode (creates and refines strategies). What's interesting is that it bridges the gap between the sophisticated system prompts that production AI uses and the basic prompts most of us work with. Your model literally gets better at the types of problems you throw at it. Built it because I noticed ChatGPT, Claude etc. have incredibly detailed system prompts with problem-solving frameworks, but most developers use basic prompts and miss out on those performance gains. The approach is inspired by Andrej Karpathy's tweet about a "third paradigm" for LLM learning beyond just pretraining and fine-tuning: https://x.com/karpathy/status/1921368644069765486 The strategies are completely transparent - you can see exactly what the system learned and why it's making certain decisions. No black box learning. https://github.com/codelion/optillm/tree/main/optillm/plugin... Would love feedback on the approach. Has anyone else experimented with LLMs learning from their own experience?
48 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.