Glonce

2011 essay answers Chomsky: statistical language models are real science

researchThe Glonce editorsHacker News🔥53Glonce rating53Impact40Novelty25Relevance90Surprise45Players60Credibility70

The essay, titled with the year 2011, is a reply to Noam Chomsky and Steven Pinker. It opens with a transcript of Chomsky's answer to a question from Pinker about the success of probabilistic models trained with statistical methods. Chomsky said: "I think there have been some successes, but a lot of failures." He added that there is a notion of success which he thinks is novel in the history of science: "It interprets success as approximating unanalyzed data." The author takes this as the target and sets out to answer it.

First the author separates two ideas that are often confused: probabilistic models and trained (statistical) models. A second-order Markov word model with a 100,000-word vocabulary, of the kind Claude Shannon proposed a decade before Chomsky, needs a quadrillion (10^15) probability values. The only feasible way to learn them is to gather statistics from data and smooth over the many cases with no data. So most, but not all, probabilistic models are trained, and many, but not all, trained models are probabilistic. Newton's gravity law is the counterexample in the other direction: the constant G is estimated by statistical inference over noisy experiments, yet the model is deterministic. The author believes Chomsky has no objection to a model like that, and that he seems to reserve his criticism for models like Shannon's with quadrillions of parameters. The ideal gas law is the opposite case: statistical, probabilistic and incomplete, but it gives good predictions and insight that tracking individual molecules would not.

The spelling rule "I before E except after C" serves as a small test. In a corpus of a trillion words of English text, IE is more common than EI by almost 4 to 1, and even after C, CIE is more common than CEI by almost 3 to 1 ("science", "society", "ancient", "species"). The essay computes the accuracy of plain "I before E" at 0.793 and of "I before E except after C" at 0.753, so the extra clause makes the rule worse. A richer statistical model, say one covering all 4-letter sequences or all known words, could be ten times more accurate at spelling but would offer no concise insight into how spelling works. Insight would need a model that knows about phonemes, syllabification and language of origin, and such a model could itself be trained or not, probabilistic or not.

The author then argues that statistical and probabilistic models have not had limited success; they hold an overwhelmingly dominant, though not exclusive, position. Steve Abney wrote in 1996 that in the previous ten years statistical methods had gone from virtually unknown in computational linguistics to a fundamental given. The author concedes that the majority does not rule, but describes personally switching after about 14 years of trying to get language models to work with logical rules, crediting pioneers such as Gene Charniak (and Judea Pearl for probability in general) and early-adopter colleagues such as Dekai Wu. Nobody was seen moving the other way. The author also points to money: the intellectual offspring of Shannon's theory create several trillion dollars of revenue each year, while the offspring of Chomsky's theories generate well under a billion. The engineering reason for the shift is plain: statistical models have state-of-the-art performance, and in most cases non-statistical models do worse.

The essay then turns to scientific reasons. A dictionary definition of science stresses accurate modelling over insight, but the author says both have always coexisted in science. To test this, the author picked a Science abstract at random: chlorinated indium tin oxide electrodes for OLEDs, reporting a maximum external quantum efficiency of 54% and a power efficiency of 230 lumens per watt. That paper is mostly about specific experiments and results rather than insight. The author then looked at all the titles and abstracts in the issue.

Finally the essay goes through Chomsky's own objections. His 1969 line, that the notion of "probability of a sentence" is "an entirely useless one, under any known interpretation of this term", rested on the argument that a novel sentence must get probability zero. Smoothing, now standard practice, assigns novel sentences non-zero probability, so the author calls this criticism invalid, though very influential for decades. In Syntactic Structures (1957) Chomsky concluded that probabilistic models give no particular insight into some basic problems of syntactic structure, and in a footnote said he knew of no suggestion of a useful statistical model "that does not have obvious flaws". The author calls the main flaw correct but a criticism only of Markov-chain models, not of probabilistic or trained models in general. Later models such as probabilistic context-free grammars (PCFGs) are state-of-the-art for parsing and easier to learn from data than nonprobabilistic categorical context-free grammars. Because every probabilistic model is a superset of a deterministic one (probabilities restricted to 0 or 1), any valid criticism would have to be that they are too expressive, not too little. The extract ends as the author starts on Chomsky's "colorless green ideas sleep furiously" example, where (a) is grammatical and (b) is not.

Key facts

  • The essay answers Noam Chomsky's remark, made in reply to Steven Pinker, that success in statistical modelling means "approximating unanalyzed data".
  • The author argues Chomsky's 1957 objection is correct but covers only Markov-chain models, and that his 1969 claim that sentence probability is useless is invalid because smoothing gives novel sentences non-zero probability.
  • A second-order Markov word model on a 100,000-word vocabulary needs a quadrillion (10^15) probabilities, which is why such models must be trained on data.
  • In a trillion-word English corpus IE beats EI by almost 4 to 1; the essay computes accuracy of 0.793 for "I before E" and 0.753 for "I before E except after C".
  • The author says the offspring of Shannon's theory generate several trillion dollars of revenue a year against well under a billion for Chomsky's, and says accurate modelling and insight have always coexisted in science.

Why it matters

The essay is a clear statement of one side of an old split: whether models that predict well, without explaining, count as science. It ties that split to concrete cases: Chomsky's 1957 and 1969 objections, Shannon's Markov chains, and PCFGs. The author's central move is to pull apart "statistical", "probabilistic" and "trained", so that criticism of one kind of model is not taken as criticism of all of them. Chomsky's claims are tested against how models actually work.

Who it affects

Mainly people who build or study language models and computational linguistics, and anyone weighing explanation against prediction in science. The author describes the shift among researchers as near-universal: a field that went from virtually unknown to a fundamental given for statistical methods, per Abney in 1996, with the author among those who switched after about 14 years of rule-based work.

How to use it

As a reading guide for the debate. Use its distinctions: probabilistic versus deterministic, trained versus untrained, continuous versus categorical. The Newton, ideal gas and spelling examples show each combination. The spelling example also shows how to measure a rule against corpus data instead of arguing about it.

How solid is it

It is one author's argued opinion, not an experiment. The numbers are simple and checkable: 0.0177/(0.0177+0.0046) gives 0.793, and 0.0168/0.0223 gives 0.753. The author hedges some claims ("I believe" Chomsky has no objection to Newton-style models; it "seems to me" that accuracy and insight coexist in science). The Science-abstract test is a single randomly chosen abstract plus a look at one issue. The revenue comparison is stated without a derivation. The extract available stops before the essay's ending and conclusions.

Risks and caveats

Majority adoption proves little, and the author says so: the majority does not rule. The "ten times more accurate" figure for a richer spelling model is hypothetical ("could be"). The essay's probability table lists P(*IE) as 0.163 while the accuracy formula uses 0.0163, a small inconsistency in the source. Chomsky's side is shown only through short quotes and the author's reading of them, so the full exchange with Pinker is not reproduced. The visible text does not discuss neural networks, deep learning or large language models.

“It interprets success as approximating unanalyzed data.”

— Noam Chomsky, in the transcript of his exchange with Steven Pinker, quoted in the essay