Information Density and Semantic Determinacy - Formal vs Natural Language

Narrative

The entropy figures that anchor this debate are older and narrower than they are usually credited for. Shannon's 1951 guessing-game experiment put English at 0.6 to 1.3 bits per character, and Cover and King's 1978 gambling refinement tightened that to roughly 1.25 to 1.35 bits per letter. Both methods measure surprise to a predictor conditioned on prior text, not meaning, and Shannon was explicit that semantic content sat outside the theory. Hindle, Barr, Su, Gabel and Devanbu's 2012 ICSE paper reported Java n-gram cross-entropy of three to four bits, below their English baseline under the same model, and that comparison is the empirical root of the folk claim that code is denser than prose.

That claim does not survive its own follow-up literature intact. Rahman, Palani and Rigby's 2019 ICSE paper found that separators and syntax tokens, 44 percent of a Java corpus by count, drove most of the naturalness gap, and that filtering them narrows the predictability difference substantially. Casalnuovo, Sagae and Devanbu's like-with-like comparisons still found code several times more predictable than English after controlling for vocabulary, and their subsequent dual channel hypothesis, developed with Barr, Dash, Devanbu and Morgan for ICSE 2020, reframes code as carrying an algorithmic channel executed by the machine and a natural-language channel read by programmers simultaneously. The honest reading is that code is measurably more predictable, i.e. lower entropy per token, than English under matched language models, which is the opposite of the informal claim that natural language is low density and code is high density: the folk claim inverts under its own stated measure, and the real distinction operating in the literature is not bandwidth but how much of the machine's behaviour is pinned down by the surface tokens versus left to a receiver's prior.

The ambiguity-as-optimum strand supplies the missing half of that picture. Piantadosi, Tily and Gibson's 2012 Cognition paper argues formally that efficient communication systems are ambiguous whenever context carries information, tested against English, German and Dutch corpora. Levy and Jaeger's Uniform Information Density work, building on Aylett and Turk's earlier prosodic finding that speakers shorten redundant material, and Gibson, Piantadosi, Brink, Bergen, Lim and Saxe's 2013 noisy-channel account of word-order variation together model comprehension as Bayesian decoding of a corrupted signal against a shared prior. Coupé et al.'s 2019 Science Advances estimate of roughly 39 bits per second across 17 languages is frequently cited as proof of a universal information rate, but it measures syllables against speech rate, not semantic content, and is easy to over-read as settling a question it does not address.

On specification itself, Dijkstra's 1978 EWD667, Brooks' 1987 essential-accidental distinction and Naur's 1985 theory-building argument converge on a claim later empirical work has only partly tested: that formality removes context-dependence but does not by itself remove underdetermination, because the theory of a system lives in its builders rather than its artefact. Industrial formal-methods evidence is genuinely mixed. Newcombe et al.'s 2015 AWS report of TLA+ catching design bugs in DynamoDB and S3 sits against a 2024 systematic review of a decade of industrial TLA+ practice that documents persistent adoption barriers around learning curve and tool integration. The LLM-era literature (METR's HCAST and RE-Bench, Code Roulette, and the 2025 study on prompt underspecification finding LLMs silently infer unstated requirements 41 percent of the time but regress twice as often when they do) treats the same underdetermination as a measurable quantity conditional on a prompt, with property-based testing and formal verification proposed, with mixed early results, as the residual disambiguating layer.