OpenAI's o3 model scores lower on benchmarks than claimed

OpenAI’s new AI model, o3, is facing scrutiny due to discrepancies between the company’s benchmark claims and independent testing results.

These differences raise concerns about transparency in model evaluation practices.

When o3 was launched in December, OpenAI claimed it could solve over 25% of FrontierMath problems, a set of challenging math questions.

In contrast, recent tests by Epoch AI, which oversees FrontierMath, indicated that o3 achieved a score of about 10%.

Epoch AI cited differences in testing conditions, including updates to FrontierMath and variations in computational resources, as potential reasons for the score discrepancy.

Reports suggest that OpenAI’s internal testing used a more advanced version of o3 than the model available to the public.

🔗 Source: TechCrunch

🧠 Food for thought

1️⃣ AI benchmark inconsistencies reflect broader industry transparency challenges

The o3 benchmark discrepancy is part of a growing pattern of AI evaluation inconsistencies across the industry, not an isolated incident.

Independent researchers identified nine specific ways AI benchmarks fall short, including poor transparency and reliance on biased datasets that don’t reflect real-world performance1.

This transparency gap persists despite 65% of customer experience leaders viewing AI as a strategic necessity, highlighting the disconnect between business needs and reliable performance metrics2.

Recent examples show this is industry-wide: Meta admitted to promoting benchmark scores for a different version than what was released to developers, while xAI faced criticism for misleading benchmark charts for Grok 33.

The technical complexity of AI systems makes third-party verification crucial, yet current transparency mechanisms don’t adequately address the proprietary nature of commercial AI development.

2️⃣ AI companies struggle with balancing performance claims and responsible disclosure

OpenAI has faced transparency challenges before, notably with GPT-2 in 2019, when it withheld the full model over misuse concerns but was criticized by experts who called this decision an “irresponsible PR tactic”4.