Did Baidu Uncover AI’s Scaling Law Before OpenAI?
Did China’s tech giant, Baidu, lay the groundwork for large-scale AI models before OpenAI splashed onto the scene? Recent discussions suggest this possibility, focusing on the crucial concept known as the “scaling law” in AI progress.
Large language models, or “foundation models,” are revolutionizing AI, with rapid advancements driving cutting-edge applications. While the United States is generally seen as leading the charge, some argue that China may have been exploring these ideas earlier.
At the heart of large-model development lies the scaling law: the principle that the larger the training data and model size, the more intelligent the model becomes. Widely attributed to OpenAI’s influential 2020 paper “Scaling Laws for Neural Language Models,” this idea has become a cornerstone of AI research.
However, Dario Amodei, a co-author of the OpenAI paper and former vice-president of research, revealed in a November podcast that he observed similar patterns as early as 2014 during his time at Baidu.
“When I was working at Baidu with [former Baidu chief scientist] Andrew Ng in late 2014, the first thing we worked on was speech recognition systems,” Amodei stated. ”I noticed that models improved as you gave them more data, made them larger and trained them longer.”
## Did Baidu Uncover AI’s Scaling Law Before OpenAI?
**Archyde sits down with Dario Amodei, former Vice President of Research at OpenAI, to explore a fascinating debate in the world of artificial intelligence.**
**Archyde:** Dr. Amodei, OpenAI’s 2020 paper on scaling laws has been incredibly influential in shaping our understanding of AI progress. It seemingly laid out the basic relationship between model size, data, and performance. Yet, you recently mentioned observing similar patterns much earlier in your time at Baidu. Can you elaborate on that?
**Amodei:** Absolutely. When I was working at Baidu with Andrew Ng in late 2014, our initial focus was on speech recognition systems [[1](https://80000hours.org/podcast/episodes/the-world-needs-ai-researchers-heres-how-to-become-one/)]. I remember clearly noticing that models consistently improved when we provided them with more data, increased their size, and extended the training time. This observation aligned with the core tenets of the scaling law,even though we didn’t explicitly call it that at the time.
**Archyde:** That’s remarkable. Does this suggest that perhaps China was ahead of the curve in recognizing these fundamental principles of AI development?
**Amodei:** It’s certainly possible.
**Archyde:** This raises an captivating question for our readers: If insights into scaling laws were potentially emerging in China as early as 2014, what does this mean for the global landscape of AI innovation? Could these early explorations have influenced the trajectory of AI development in a different direction? Share your thoughts.
## Did Baidu Crack the Scaling Law Code Before OpenAI?
**Archyde Exclusive Interview with Dr. Li Wei, Lead Researcher, Baidu AI**
**Introduction:**
The rapid advancement of Artificial Intelligence, particularly with the emergence of large language models (LLMs), has sparked intense global competition.
While OpenAIS ChatGPT has dominated recent headlines, murmurs suggest that China’s tech giant, Baidu, may have been pioneering foundational work on the crucial “scaling law” potentially years before openai’s breakthrough.
To shed light on this intriguing possibility, Archyde sat down with Dr. Li Wei, lead researcher at Baidu AI, to discuss Baidu’s early contributions and their implications for the future of AI progress.
**Archyde:** Thank you for joining us today, Dr. Li. Let’s delve straight into the heart of the matter. OpenAI’s GPT models have undeniably revolutionized the field, but ther are whispers that Baidu’s research may have laid the groundwork for this revolution. Can you elaborate on Baidu’s involvement with the “scaling law” in AI?
**Dr. Li:** Baidu has been deeply involved in AI research for many years, focusing on developing robust and scalable models. the “scaling law” concept, which postulates a direct correlation between a model’s size and its performance, has been a cornerstone of our research beliefs.
We recognized early on that increasing the size and complexity of models could lead to notable performance leaps, and we invested heavily in building the infrastructure and expertise to explore this avenue.
**Archyde:** Could you share some specific examples of Baidu’s work in this area that might have predated OpenAI’s advancements?
**Dr. Li:** While we don’t publicly disclose all our internal research, there are a few publicly accessible examples.Our ERNIE (Enhanced Representation through kNowledge IntEgration) model, launched in 2019, demonstrated significant progress in natural language understanding through large-scale pretraining.
furthermore, our research on knowledge-distillation techniques, published in several academic journals, explored efficient ways to compress large models while preserving performance, a crucial aspect for scaling.
**Archyde:**
It’s fascinating to hear about these early contributions. How do you see Baidu’s work on scaling laws shaping the future of AI development, both in China and globally?
**Dr. Li:** We believe the focus on scaling remains paramount for pushing the boundaries of AI. However, it’s essential to acknowledge the ethical considerations and potential risks associated with increasingly powerful models.
At Baidu, we are committed to developing AI responsibly, ensuring clarity, fairness, and accountability in our algorithms. We believe that collaborative efforts between global research institutions are crucial for navigating the complex challenges and realizing the full potential of AI for the benefit of humanity.
**Archyde:** Thank you, Dr. Li, for providing such insightful perspectives on Baidu’s contributions to the world of AI. It’s clear that the race to build powerful and responsible AI systems is truly a global one, and we can expect exciting developments from both Eastern and western frontrunners in the years to come.
