- Premium Content It takes our newsroom weeks - if not months - to investigate and produce stories for our premium content. You can’t find them anywhere else.
Samreen Ahmad · · 6 min readIndian AI lab challenges Hugging Face over alleged Nvidia bias
Can a small Indian AI startup beat Nvidia at its own game? Shunya Labs claims it has done just that.
According to the platform, its speech recognition model Pingala V1 has reached a word error rate (WER) of 3.1%. This score would overtake Nvidia’s Canary model on Hugging Face’s Open ASR Leaderboard – one of the global benchmarks for ranking English automatic speech recognition (ASR) systems.
Image credit: Tech in Asia
WER is the standard way to measure accuracy in ASR, an essential part of systems that use voice commands like digital assistants and speech-to-text services. This metric shows what percentage of words a system gets wrong compared to the actual transcript. The lower the WER, the more accurate the model is.
Currently, Nvidia’s Canary model leads the Open ASR Leaderboard with a WER of 5.63%.
Shunya Labs co-founder Sourav Banerjee says that Hugging Face has yet to act on their leaderboard submission and has involved one of the startup’s competitors, Nvidia, in the process. The Indian firm is the deeptech arm of AI startup United We Care.
“Our submission is just sitting there,” he tells Tech in Asia, adding that there is “no grievance redressal mechanism” to challenge Hugging Face’s decision of not adding it to the leaderboard.
[The leaderboard is] an ivory tower with five to seven people who are managing the whole thing.
Banerjee argues that there is a conflict of interest in Hugging Face’s evaluation process, pointing out that the company had tagged Nvidia employees Nithin Rao Koluguri and Piotr Żelasko to evaluate Shunya Labs’ submission on GitHub for the Open ASR Leaderboard.
Hugging Face is a leading open-source hub for AI models. It counts Nvidia as an investor.
Banerjee stresses that Hugging Face needs to be “transparent” with how it ranks models in the leaderboard.
Żelasko, Nvidia’s principal research scientist, weighed in on the submission in a “personal capacity.” According to his posts in the GitHub submission thread for the Open ASR Leaderboard, he tested the Pingala model on three custom English datasets and reported WER scores of 7.98%, 11.35%, and 15.50%, far lower than Shunya Labs’ claim of 3.1%.
Banerjee objected to this approach, arguing that private dataset tests lack transparency.
Can leaderboards be gamed?
Are leaderboards reliable?
Stay ahead in Asia’s tech landscape
This is premium content. Subscribe to read the full story.
We trace Shunya Labs’ bold claim against Hugging Face and the questions it raises about global AI benchmarks.
We know this is not ideal. ⌛ Sign up in 20 seconds. Cancel anytime.
Our subscriber community includes professionals from these companies:
Stay updated on the go with our mobile app.
Get latest insights with smoother, more personalized experience through TIA mobile app.
