OpenAI has introduced SWE-Lancer, a new benchmark to assess large language models (LLMs) in freelance software engineering.

The dataset includes over 1,400 tasks from Upwork, with payouts exceeding US$1 million.

Tasks range from US$50 bug fixes to US$32,000 feature implementations and are categorized into independent engineering and managerial decision-making tasks.

Independent tasks undergo triple-verified evaluations by experienced engineers, while managerial tasks are assessed based on decisions from original engineering managers.

Testing of frontier LLMs against SWE-Lancer showed difficulty in solving most tasks.

To support further research, SWE-Lancer provides an open-source Docker image and a public evaluation subset, SWE-Lancer Diamond.

The initiative explores AI’s economic impact on software engineering by correlating model performance with financial outcomes.

The benchmark is now publicly available for researchers and developers.