🧔♂️ A friendly human may check it before it goes live. More news here
Microsoft, Tsinghua use Nvidia chip to train AI without real data
Tsinghua University and Microsoft researchers developed a synthetic data pipeline called SynthSmith to train AI models without relying on real-world data, using Nvidia chips for computation.
Using only synthetic data, the team trained a 7-billion-parameter coding model that outperformed larger 14-billion-parameter models on benchmarks.
The experiment used 128 Nvidia H20 chips for 220 hours during supervised fine-tuning and 32 H200 chips for seven days during reinforcement learning.
The researchers said computational limits prevent scaling the approach to models above 100 billion parameters.
They open-sourced the research so others can build on it without high training costs, using Nvidia H20 and H200 chips that are available to Chinese AI firms under export controls.
🔗 Source: South China Morning Post
🧠 Food for thought
Implications, context, and why it matters.
Key details on performance and release plans need to be checked to validate the research
- Clarification is required for the claim that the research is “open-sourcing.”
- Code for preparing the training material is posted on GitHub, while the model weights (the trained model parameters needed to run the model) are planned for release without a timeline 1.
- Licensing terms for the code and the eventual model weights, which set the rules for commercial use, must be confirmed.
- Performance results should be reviewed beyond broad statements about beating larger models.
- On LiveCodeBench v5, an average pass rate of 62.9 on eight attempts (avg@8) was reported, while v6 was put at 55.8, ahead of DeepCoder-14B-Preview and AReal-boba2-14B 1.
China-based GPU cloud providers could treat synthetic-data training as a service opportunity, but export-policy and demand claims need tighter sourcing
- China-based GPU cloud providers could package this synthetic-data training workflow as a paid service for labs and small teams.
- The experiment ran on Nvidia H20 and H200 chips, with 128 H20 chips used for supervised fine-tuning and 32 H200 chips used for reinforcement learning [News Article].
- The Trump administration made H20 and H200 chips available to Chinese companies after lobbying by Nvidia to lift export controls [News Article].
- The authors flagged compute limits as a scaling barrier, so Alibaba Cloud and Tencent Cloud could sell higher-capacity setups to groups trying to replicate or expand the method [News Article].
- Any demand figure of “over 400,000 H200 units” tied to Alibaba and ByteDance needs clear attribution since it is not supported by the provided source set 2.
