Researchers from two U.S. universities have recreated a scaled-down version of DeepSeek’s R1-Zero language model for less than $30.
TinyZero is a 3-billion parameter model that took 10 hours to train on Nvidia’s H100 AI chip, according to a post on X by lead researcher Jiayi Pan. That’s a fraction of the power typically required to pre-train AI foundation models.
The team, which is from the University of California, Berkeley, and the University of Illinois at Urbana-Champaign, used DeepSeek’s model weights and base code, which were publicly available since it carried an MIT license. TinyZero is open source.
In an interview with UC Berkeley’s campus newspaper, The Daily Californian, Pan said that “small-scale reproduction is very accessible and very cheap even for people as a side project to experiment with.” He added that the team’s goal from the start was “basically to demystify how to train these models and better understand the science and design decisions behind them.”
Chinese AI startup DeepSeek shook Silicon Valley when it revealed that its V3 foundation model performed on par with top AI models but cost only $5.6 million to develop and used just 2,048 of Nvidia’s slower H800 chips. (It has since released a reasoning model, R1.) In contrast, foundation AI models in the U.S. can cost tens of millions to hundreds of millions of dollars to pre-train.
However, questions have since arisen about the veracity of DeepSeek’s costs, its links to the Chinese government, and the security of the model. DeepSeek faces bans in Italy, Taiwan, South Korea, Australia and the U.S. Nevertheless, industry titans such as Meta CEO Mark Zuckerberg and Microsoft CEO Satya Nadella have lauded the innovative techniques deployed by DeepSeek.
Will a Cambrian explosion come next?
In a test, TinyZero’s researchers applied their model to CountDown, a game where players come up with an equation to get a target result. For example: “Using the numbers 19, 36, 55, 7, create an equation that equals 65.” The researchers taught TinyZero to reason using reinforcement learning.
“The results: It just works!” Pan said. However, he cautioned that TinyZero has not been tested on general reasoning tasks.
While TinyZero’s capabilities are limited to basic tasks like counting and multiplication, its significance lies in demonstrating the increasing accessibility of AI technology. The $30 cost, which covered server expenses for the experiments, represents a dramatic reduction in the traditional barriers to AI research and development.
Genevieve Smith, founding co-director of the Responsible and Equitable AI Initiative at UC Berkeley’s Artificial Intelligence Research Lab, points to broader implications.
“One potential scenario is that it will amplify demand in adoption because this technology is being created more efficiently, more cost-effectively, that could allow for more value creation,” she told the paper. However, she emphasized that the long-term market implications remain uncertain.
The development also highlights growing U.S.-China tech tensions. Smith observed that this advancement is “amplifying the sense of competitiveness between the U.S. and China,” particularly in light of recent U.S. chip export restrictions to China.
The TinyZero project raises important questions about the future of AI development as well, particularly regarding the balance between open-source accessibility and proprietary technology. According to Smith, limiting open-source development could lead to “tech consolidations of power and monopolization in the tech industry” that would impede “innovation and equity.”
Pan suggests that DeepSeek’s market impact stems from revealing that current AI models are “much cheaper than we expected.” He believes that the ability to reproduce results with minimal investment will accelerate scientific understanding in the field by enabling broader participation in AI research.
Be First to Comment