BLBluestein·Jul 7, 2025Submission
Curated list of language modeling researches for code, plus related datasets
5 points · Open discussioncodefuse-ai
Awesome-Code-LLM
AI & Machine Learning2024–2025
codefuse-ai / Awesome-Code-LLM
Awesome-Code-LLM
[TMLR] A curated list of language modeling researches for code (and other software engineering activities), plus related datasets.
2Distinct people
8Mentions
0Explicit recommenders
3.5KGitHub stars
What to know before exploring
What it is
[TMLR] A curated list of language modeling researches for code (and other software engineering activities), plus related datasets.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- eosphoros-ai/awesome-text2sql · linked in 2 of the same discussions
- underlines/awesome-ml · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month8 dated mentions
Observed mentions: 2024-05-25 23:14:08 UTC to 2025-07-07 08:38:41 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| May 2024 | 2024-05-01 00:00:00 UTC | 2024-06-01 00:00:00 UTC | 1 | 0 | 1 |
| Jun 2024 | 2024-06-01 00:00:00 UTC | 2024-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2024 | 2024-07-01 00:00:00 UTC | 2024-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2024 | 2024-08-01 00:00:00 UTC | 2024-09-01 00:00:00 UTC | 1 | 0 | 1 |
| Sep 2024 | 2024-09-01 00:00:00 UTC | 2024-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2024 | 2024-10-01 00:00:00 UTC | 2024-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2024 | 2024-11-01 00:00:00 UTC | 2024-12-01 00:00:00 UTC | 2 | 0 | 2 |
| Dec 2024 | 2024-12-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2025 | 2025-01-01 00:00:00 UTC | 2025-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2025 | 2025-02-01 00:00:00 UTC | 2025-03-01 00:00:00 UTC | 1 | 0 | 1 |
| Mar 2025 | 2025-03-01 00:00:00 UTC | 2025-04-01 00:00:00 UTC | 0 | 0 | 0 |
| Apr 2025 | 2025-04-01 00:00:00 UTC | 2025-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2025 | 2025-05-01 00:00:00 UTC | 2025-06-01 00:00:00 UTC | 1 | 0 | 1 |
| Jun 2025 | 2025-06-01 00:00:00 UTC | 2025-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2025 | 2025-07-01 00:00:00 UTC | 2025-08-01 00:00:00 UTC | 2 | 1 | 1 |
What Hacker News says
8 sourcesIs there an established benchmark for building a full product?
- SWE-bench leaderboard: https://www.swebench.com/
- Which metrics for e.g. "SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork"? https://news.ycombinator.com/item?id=43101314
- MetaGPT, MGX: https://github.com/FoundationAgents/MetaGPT :
> Software Company as Multi-Agent System
> MetaGPT takes a one line requirement as input and outputs user stories / competitive analysis / requirements / data structures / APIs / documents, etc.
Internally, MetaGPT includes product managers / architects / project managers / engineers. It provides the entire process of a software company along with carefully orchestrated SOPs.
- Mutation-Guided LLM-based Test Generation: https://news.ycombinator.com/item?id=42953885
- https://news.ycombinator.com/item?id=41333249 :
- c
Read the rest
odefuse-ai/Awesome-Code-LLM > Analysis of AI-Generated Code, Benchmarks: https://github.com/codefuse-ai/Awesome-Code-LLM :
> 8.2 Benchmarks:
Integrated Benchmarks,
Evaluation Metrics,
Program Synthesis,
Visually Grounded Program, Synthesis,
Code Reasoning and QA,
Text-to-SQL,
Code Translation,
Program Repair,
Code Summarization,
Defect/Vulnerability Detection,
Code Retrieval,
Type Inference,
Commit Message Generation,
Repo-Level Coding
- underlines/awesome-ml/tools.md > Benchmarking: https://github.com/underlines/awesome-ml/blob/master/llm-too...
- formal methods workflows, coverage-guided fuzzing: https://news.ycombinator.com/item?id=40884466
- "Large Language Models Based Fuzzing Techniques: A Survey" (2024) https://arxiv.org/abs/2402.00350
From "Show HN: We open sourced our entire text-to-SQL product" (2024) https://news.ycombinator.com/item?id=40456236 :
> awesome-Text2SQL: https://github.com/eosphoros-ai/Awesome-Text2SQL
> Awesome-code-llm > Benchmarks > Text to SQL:
https://github.com/codefuse-ai/Awesome-Code-LLM#text-to-sql
In Getting AI to write good SQLThe most recent reference listed in this paper is "Truly modular (co)datatypes for Isabelle/HOL" (2014) .. https://scholar.google.com/scholar?cites=1837719396691256842...
Automated theorem proving > Related problems: https://en.wikipedia.org/wiki/Automated_theorem_proving#Rela...
awesome-code-llm > Coding for Reasoning:
https://github.com/codefuse-ai/Awesome-Code-LLM#31-coding-fo...
In History of interactive theorem proving [pdf]Re: costed opcodes in BPF, eWASM (like EVM) https://news.ycombinator.com/item?id=40475509 :
> Costed opcodes incentivize energy efficiency.
Aren't these all fitness scoring methods for what GA calls mutation, crossover, and selection?
And do the functions under test stably coverage on the same output even if not exactly functionally isomorphic? When are less costly approximate solutions sufficient?
Big O, Dynamic tracing and instrumentation, costed opcodes, OPS/kWHr/$
https://news.ycombinator.com/item?id=41333249 :
> codefuse-ai/Awesome-Code-LLM > Analysis of AI-Generated Code, Benchmarks: https://github.com/codefuse-ai/Awesome-Code-LLM#6-analysis-o...
In Rethinking Code Refinement: Learning to Judge Code Efficiencyawesome-code-llm > Vulnerability Detection, Program Proof,:
https://github.com/codefuse-ai/Awesome-Code-LLM#vulnerabilit...
In Using Large Language Models to Catch VulnerabilitiesGartner "Magic Quadrant for AI Code Assistants" (2024)
https://www.gartner.com/doc/reprints?id=1-2IKO4MPE&ct=240819...
Additional criteria for assessing AI code assistants from https://news.ycombinator.com/item?id=40478539 re: Text-to-SQL bemchmarks :
codefuse-ai/Awesome-Code-LLM > Analysis of AI-Generated Code, Benchmarks:
https://github.com/codefuse-ai/Awesome-Code-LLM :
> 8.2. Benchmarks: *
Integrated Benchmarks,
Program Synthesis,
Visually Grounded Program Synthesis,
Code Reasoning and QA,
Text-to-SQL,
Code Translation,
Program Repair,
Code Summarization,
Defect/Vulnerability Detection,
Code Retrieval,
Type Inference,
Commit Message Generation,
Repo-Level Coding*
OT did not assess:
Aider:
https://github.com/paul-gauthier/aider :
> Aider works best with GPT-4o & Claude 3.5 Sonnet and can connect to almost any LLM.
https://a
Read the rest
ider.chat/ :
> Aider has one of the top scores on SWE Bench. SWE Bench is a challenging software engineering benchmark where aider solved real GitHub issues from popular open source projects like django, scikitlearn, matplotlib, etc
SWE Bench benchmark:
https://www.swebench.com/
/? awesome "sql" llm site:github.com
https://www.google.com/search?q=awesome+%22sql%22+llm+site%3... :
- awesome-Text2SQL:
https://github.com/eosphoros-ai/Awesome-Text2SQL :
> Curated tutorials and resources for Large Language Models, Text2SQL, Text2DSL、Text2API、Text2Vis and more.
- Awesome-code-llm > Benchmarks > Text to SQL:
https://github.com/codefuse-ai/Awesome-Code-LLM#text-to-sql
- underlines/awesome-ml//llm-tools.md > RAG > OpenAI > dataherald,:
https://github.com/underlines/awesome-ml/blob/master/llm-too...
- underlines/awesome-ml//llm-tools.md > Benchmarking > Benchmark Suites, Leaderboards: https://github.com/underlines/awesome-ml/blob/master/llm-too... :
- sql-eval: https://github.com/defog-ai/sql-eval :
> This repository contains the code that Defog uses for the evaluation of generated SQL. It's based off the schema from the
Read the rest
Spider, but with a new set of hand-selected questions and queries grouped by query category. For an in-depth look into our process of creating this evaluation approach, see this.
> Our testing procedure comprises the following steps. For each question/query pair: 1. We generate a SQL query (possibly from an LLM).
2. We run both the "gold" query and the generated query on their respective database to obtain 2 dataframes with the results.
3. We compare the 2 dataframes using an "exact" and a "subset" match. TODO add link to blogpost.
4. We log these alongside other metrics of interest (e.g. tokens used, latency) and aggregate the results for reporting
- dataherald/services/engine/dataherald/tests/sql_generator/test_generator.py:
https://github.com/Dataherald/dataherald/blob/main/services/...