SWELancer-Benchmark

This repo contains the dataset and code for the paper "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?"

Science & Engineering AI & Machine Learning Testing & Quality

Explore on GitHub Read Hacker News discussions

4Distinct people

5Mentions

0Explicit recommenders

1.4KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

This repo contains the dataset and code for the paper "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?"

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

expensify/app · linked in 2 of the same discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 9:01:12 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by day5 dated mentions

Observed mentions: 2025-02-19 06:38:29 UTC to 2025-02-27 20:18:51 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Comments
Feb 19	2025-02-19 00:00:00 UTC	2025-02-20 00:00:00 UTC	3	3
Feb 20	2025-02-20 00:00:00 UTC	2025-02-21 00:00:00 UTC	1	1
Feb 21	2025-02-21 00:00:00 UTC	2025-02-22 00:00:00 UTC	0	0
Feb 22	2025-02-22 00:00:00 UTC	2025-02-23 00:00:00 UTC	0	0
Feb 23	2025-02-23 00:00:00 UTC	2025-02-24 00:00:00 UTC	0	0
Feb 24	2025-02-24 00:00:00 UTC	2025-02-25 00:00:00 UTC	0	0
Feb 25	2025-02-25 00:00:00 UTC	2025-02-26 00:00:00 UTC	0	0
Feb 26	2025-02-26 00:00:00 UTC	2025-02-27 00:00:00 UTC	0	0
Feb 27	2025-02-27 00:00:00 UTC	2025-02-28 00:00:00 UTC	1	1

What Hacker News says

3 sources

Snuggly73·Feb 19, 2025

It was just two examples of widespread problems with the introduced bugs and the tests. How about this - https://github.com/openai/SWELancer-Benchmark/blob/08b5d3dff... (Intentionally use raw character count instead of HTML-converted length) Or this one - https://github.com/openai/SWELancer-Benchmark/blob/08b5d3dff... (user is complaining of flickering, so the reintroduced bug adds flickering code :) ) Or the one that they list in A.10 of the paper as O1 successfuly fixing - https://github.com/openai/SWELancer-Benchmark/blob/main/issu... O1 doesnt actually seem to fix anything (besides arbitrary dumping all over the code), the reintroduced bug is messing with the state, not with the back button navigation. Anyways, I went thru a sample of 20-30 last night and gave up. Noone needs to take my words - force pushing aside, anyone can pull the

Read the rest

repo and check for themselves. Most of the 'bugs' are trivialized to a massive degree, which a) makes them very easy to solve for b) doesnt reflect their previous monetary value, which in effect makes the whole premise of 'let measure how SWE agents can provide real money value' invalid. If they wanted to create a real one, they should've found the commits reflecting the state of the app as of the moment of the bug and setup up the benchmarks around that.

In SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork

yorwba·Feb 19, 2025

For future reference, permalink to the original commit with the RADICAL BUG comment: https://github.com/openai/SWELancer-Benchmark/blob/a8fa46d2b... The new version (as of now) still has a comment making it obvious that there's an intentionally introduced bug, but it's not as on the nose: https://github.com/openai/SWELancer-Benchmark/blob/2a77e3572...

In SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork

Snuggly73·Feb 19, 2025

First time commenter - I was so triggered by this benchmark, so I just had to come out of lurking. I've spent time going over the description and the cases and its an misrepresented travesty. The benchmark takes existing cases from Upwork, then reintroduces the problems back in the code and then asks the LLM to fix them testing against newly written 'comprehensive tests'. Lets look at some of the cases: 1. The regex zip code validation problem Looking at the Upwork problem - https://github.com/Expensify/App/issues/14958 it was mainly that they were using a common regex to validate across all countries, so the solution had to introduce country specific regex etc. The "reintroduced bug" - https://github.com/openai/SWELancer-Benchmark/blob/main/issu... is just taking that new code and adding , to two countries.... 2. Room showing empty - 148

Read the rest

57 The "reintroduced bug" - https://github.com/openai/SWELancer-Benchmark/blob/main/issu... Adds code explicitly commented as introducing a "radical bug" and "intentionally returning an empty array"... I could go on and on and on... The "extensive tests" are also laughable :( I am not sure if OpenAI is actually aware of how great this "benchmark" is, but after so much fanfare - they should be.

In SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork