BabyLM competition tests why kids outlearn AI at language

People have been talking to each other for at least 100,000 years, and until recently a human child was the only thing in the world that could learn a language to full fluency, MIT Technology Review writes. Four years after ChatGPT's debut, large language models such as GPT models, Claude and DeepSeek converse fluently enough to pass for human, yet training them to do so still takes an inhuman amount of data. An LLM churns through roughly a hundred thousand times more words than a person experiences while mastering their native language, and far more than a child hears by its first birthday, when children typically start to grasp language. Researchers call the mismatch the data efficiency gap.

The scale gap is large by any measure. Meta's open-weight Llama 3.1, released two years before this piece, pretrained on 15 trillion tokens; Ethan Gotlieb Wilcox, a cognitive scientist and linguist at Georgetown University, says frontier models could be training on ten times that figure. A preteen raised in a language-rich home, by contrast, has typically heard something in the vicinity of 100 million words, a number that can grow to about 300 million by age 20 once reading is added in. Printed on paper, the words used to train a modern LLM would stack up past the International Space Station; the preteen's 100 million words would make a pile only 20 meters tall. Toddlers usually start producing grammatically correct sentences after hearing something like 10 million words, or as many as 30 million on the high end. Wilcox puts the comparison in concrete terms: 'Claude has seen the amount of language that an entire city will experience in one generation.' Stanford cognitive scientist Michael C. Frank makes the same point from the other direction: 'It's just totally miraculous. If you train GPT-2 on 30 million words, you get a nonsense generator; you don't get a kid.'

The gap revives a decades-old fight in linguistics. In the 1950s, MIT linguist Noam Chomsky argued that children are born with hardwired knowledge of grammar, because language, and especially its syntax, is too complex, and a child's exposure to it too limited, for the rules to be learned purely from experience, an argument known as the poverty of the stimulus. Chomsky was reacting to psychologist B.F. Skinner's view that language is learned entirely through conditioning and reinforcement, the way a dog learns to sit for a treat. 'His signature argument was, essentially, that language cannot be learned on the basis purely of statistics,' says Richard Futrell, a linguist and cognitive scientist at the University of California, Irvine, describing Chomsky's position. Chomsky's framework, generative grammar, dominated American linguistics for decades and shaped early computer science in the 1950s and 60s, when the Pentagon wanted machines that could translate Russian and understand English; researchers tried to hand-code grammar rules into programs instead of having systems learn from raw exposure. That rule-based approach mostly failed to handle language at scale, and interest in the field cooled during the AI winter that began in the 1970s.

Neural networks, which learn statistical patterns rather than following coded rules, made a comeback in the 2010s as computing power grew cheaper and the internet grew larger. By 2018 and 2019 the transformer-based models BERT and GPT-2 showed that learning from a huge volume of text could produce real language ability, and OpenAI's ChatGPT made that undeniable to a general audience in 2022. Alison Gopnik, a developmental psychologist at the University of California, Berkeley, says the result surprised even skeptics: 'No matter how skeptical you are about AI, the thing that everyone has been really impressed with is: These things learn syntax. I didn't think that was going to turn out to be true. And I think most people didn't think that you could just look at the statistics of a large sample of language and figure out grammar.' MIT Technology Review notes that LLMs are not brains, only powerful statistical learners without the evolved biological machinery of the human cortex; even so, they turned out to be exactly the kind of thing a generative linguist working two decades ago would have predicted could not learn language, and yet they write believable sentences and pass grammar tests.

That turn produced a follow-up question: could training a model on a child-size slice of language, instead of an internet-size one, reveal how children actually learn? Alex Warstadt, now a linguist and data scientist at the University of California, San Diego, was a PhD student in linguistics at New York University in 2019, when many linguists remained skeptical that models trained anywhere near human scale could tell science anything: 'I always got pushback on one issue in particular. And that was the size of the data sets of the model,' he says. 'There was never a time when people were training language models at human scale where we were impressed by them.' In August 2022, Warstadt posted a Twitter thread arguing that neural networks could still be useful models of language acquisition, and after trading ideas with AI researcher Leshem Choshen in the replies, Warstadt floated the idea for what grew into BabyLM, an annual competition Warstadt, Choshen and several other researchers have now run for four years. Entrants train models on a corpus meant to be 'developmentally plausible': 100 million words for the main track, or 10 million words for a toddler-scale track, drawn from storybooks, dialogue, movie subtitles, Simple English Wikipedia, ordinary Wikipedia, and transcripts of real speech directed at children. The resulting models are evaluated with the same kinds of grammar benchmarks psycholinguists use with people, says Wilcox, one of the competition's organizers: presenting a test subject, human or machine, with a grammatical sentence such as 'The keys to the cabinet are on the table' alongside a subtly broken version, 'The keys to the cabinet is on the table,' and checking for signs of surprise. For a human, that surprise might be measured by tracking eye movements; for a model, researchers use a measure called surprisal, which gauges how unlikely the model finds a given sentence. BabyLM has since added workshops and inspired spinoff competitions, including one trained on Chinese-language data. One early result cut against expectations: curriculum learning, which starts training on simple language and works up to more complex material, similar to starting with baby talk and growing more sophisticated over time, was by far the most popular strategy entrants tried in the competition's first round. It did not perform as well as expected.

The stakes run in two directions. On the AI side, the internet's supply of easily available training text could run dry as soon as the 2030s, so learning to train models on far less data could matter for pushing AI to learn from video, or for building chatbots that serve minority languages that lack large text corpora. On the cognitive-science side, reverse-engineering how children learn with so little exposure could help settle whether humans are born with some form of language instinct, or whether even a language's syntax could in principle be learned purely from experience, and whether human language processing reflects quirks of biology or constraints shared by any learnable language. MIT Technology Review does not report a settled answer: it gives no benchmark score for how closely BabyLM's trained models have come to matching children's own grammar performance, and the available text breaks off before giving the reason why curriculum learning underperformed in the first round, leaving the question posed in its own headline, why children still outlearn machines at language, open.

Key facts

A large language model needs about a hundred thousand times more words than a person needs to master their native language: Meta's Llama 3.1 pretrained on 15 trillion tokens, and Georgetown's Ethan Gotlieb Wilcox says frontier models could be training on ten times that.
A preteen in a language-rich home has typically heard around 100 million words, growing to about 300 million by age 20 with reading, while toddlers start speaking grammatically after hearing something like 10 million words, or as many as 30 million on the high end; printed out, that word count would stack only 20 meters high, against an LLM's training data stacking up past the International Space Station.
The debate traces to the 1950s, when MIT linguist Noam Chomsky argued children are born with hardwired grammar knowledge, the 'poverty of the stimulus' argument, against psychologist B.F. Skinner's view that language is learned purely through conditioning; Chomsky's rule-based approach shaped early computer science before neural models like BERT and GPT-2 (2018 to 2019) and OpenAI's 2022 ChatGPT showed statistical learning could master syntax.
Linguist Alex Warstadt, now at the University of California, San Diego, floated the idea in an August 2022 Twitter thread with AI researcher Leshem Choshen; it grew into BabyLM, an annual competition where entrants train models on a 'developmentally plausible' corpus of 100 million words, or 10 million for a toddler-scale track, drawn from storybooks, child-directed speech and Wikipedia, then score them with a grammar test called surprisal.
In BabyLM's first round, curriculum learning, training on simple language first and working up to complex text, was the most popular strategy entrants tried, but it did not perform as well as expected; separately, the internet's supply of easily available training text could run dry as soon as the 2030s.

Why it matters

The piece frames two separate stakes. On the practical side, large language models currently depend on scraping close to the entire available internet, and MIT Technology Review reports that the supply of easily available training text could run dry as soon as the 2030s; Stanford's Michael C. Frank captures the current cost bluntly: 'We still have to burn down a forest and scrape the entire sum of all human knowledge to re-create this milestone that happens in our living rooms over the course of a year.' If scientists can reverse-engineer how children learn language from a tiny fraction of that data, the payoff could include training AI more efficiently on video and building chatbots for minority languages that have no large existing text corpus. On the scientific side, the question sits on top of a genuine, decades-old dispute: whether children are born with some form of language instinct, as Noam Chomsky argued in the 1950s, or whether language could in principle be learned purely from experience, as B.F. Skinner's behaviorist account held, and whether the particular way humans process language reflects quirks of human biology or constraints that apply to any learnable language. Testing those hypotheses inside actual machine-learning models, rather than only through human experiments, is what gives the question new traction now.

Who it affects

Two overlapping communities. AI researchers and the companies training large language models are affected because the current approach, training on ever more scraped text, has a shelf life if the easily available internet-scale supply of text keeps shrinking; anyone building AI for video, or building chatbots for languages with little digitized text, has a direct stake in whether data-efficient training methods emerge from this work. Cognitive scientists and linguists studying how children acquire language are the other group, including the specialists named in the piece: Michael C. Frank at Stanford, Ethan Gotlieb Wilcox at Georgetown, Alison Gopnik at UC Berkeley, and Richard Futrell at UC Irvine, cited on different sides of the data-efficiency and nature-versus-nurture questions. Alex Warstadt, a linguist and data scientist at UC San Diego, and AI researcher Leshem Choshen sit at the center of both groups: they run BabyLM, the annual competition built specifically to let AI researchers and language-acquisition researchers test the same hypotheses on the same small-scale data.

How to use it

There's no product to buy here, but there is a concrete research framework a reader can follow directly: BabyLM. It runs as an annual competition, now four years old, open to research teams who want to submit language models trained under fixed data budgets, a 100-million-word main track and a 10-million-word toddler-scale track, using only a 'developmentally plausible' corpus of storybooks, child-directed speech transcripts, subtitles and Wikipedia text. Submitted models are scored with psycholinguistic grammar tests, including a technique called surprisal that measures how unlikely a model finds a given sentence, the same style of test researchers use to gauge grammatical surprise in human subjects. BabyLM has since grown to include associated workshops and a spinoff competition using Chinese-language data, so a reader interested in data-efficient language modeling has an active, ongoing venue to follow or enter rather than a one-off study to take on faith.

How solid is it

The piece is a reported feature in MIT Technology Review, built on interviews with named specialists across cognitive science, linguistics and AI: Frank at Stanford, Wilcox at Georgetown, Gopnik at UC Berkeley, Futrell at UC Irvine, and Warstadt at UC San Diego, plus historical detail on Chomsky and Skinner. The hardest numbers are attributed to a named, checkable source, Llama 3.1's 15 trillion pretraining tokens, while the more speculative figures are clearly flagged as someone's estimate rather than an industry disclosure: the claim that frontier models could be training on ten times that figure is explicitly Wilcox's own estimate, and the projection that available internet training text could run dry 'as early as the 2030s' is presented as a possibility, not a forecast anyone stands fully behind. The BabyLM material describes an actual, ongoing research competition rather than a single unreplicated study, which gives its central claim, that curriculum learning underperformed expectations in the first round, more weight than a one-off result would carry, though the piece does not report a benchmark score quantifying that result.

Risks and caveats

The article poses its own central question, why children still outlearn machines at language, and does not resolve it: the reporting surveys competing theories and one research program built to test them, but reaches no verdict. Several figures are explicitly approximate rather than precise: a preteen's word count is described as 'in the vicinity of' 100 million, and toddlers reach grammatical speech after 'something like' 10 million words or 'as many as' 30 million, ranges rather than measurements. No specific accuracy or benchmark score is reported for how closely any BabyLM-trained model has come to matching children's grammatical performance, only that grammar benchmarks exist and are used. The explanation for why curriculum learning underperformed in BabyLM's first round is not included in the available text, which breaks off mid-explanation. And the piece's own case for real-world payoff, that reverse-engineering child language learning could produce more data-efficient AI, is presented as a hope researchers hold rather than a mechanism or timeline anyone has demonstrated.

“It's just totally miraculous. If you train GPT-2 on 30 million words, you get a nonsense generator; you don't get a kid.”

— Michael C. Frank, cognitive scientist at Stanford University