Weight-Space Programming

Narrative

The Astra result is only days old, and the commentary around it is thin and mostly derivative. The primary source is the authors' own blog post of 2 November 2026 (Takehi, Teoh, Zhang, Khattab, MIT CSAIL), which is not peer reviewed. Secondary write-ups from DataCamp, DEV Community, explainx, Emergent, TechJack and The Menon Lab largely restate its numbers: GPT-6 Astra edited 0.016% of Qwen3.5-0.8B's weights, MATH-500 rose from 7.9% to 48.1%, and answers hitting the token limit fell from 93% to 33%. The explainx write-up adds that the same approach failed to improve a non-looping Gemma 3 270M. That fits the framing that the edit removes a looping obstacle rather than adding a capability. The ai-tldr summary records that Astra ran through Codex CLI with built-in tools off, and that a public logs repository holds prompts, replies, code and the evaluated networks for experiments 1 to 8. That puts the work in the "preprint or blog with public logs" category, not independently replicated. I found no Substack, LessWrong or Simon Willison analysis of it. The strongest critique is a Medium explainer, which argues that some reading tasks lean on easy structure, including spatial bands in first and last layers and known QK/OV recipes. It also says the sycophancy experiment needed Astra to build its own activation probes, so that part is closer to activation-based interpretability than to pure weight reading. MindStudio reports that Astra could not tell two models apart on sycophancy from weights alone.

On identification, Emergent's summary attributes the game identification to first-layer weight concentrations and output-layer action distributions. That supports the simpler-explanation hypothesis, because both are low-level statistics of small networks, not evidence of general circuit reading. The reported large-model case is one 0.8B model, and the headline gains concentrate on a single failure mode (loops). Commentary that predicts a paradigm shift from symbolic optimisers (Emergent, DataCamp) is promotional in tone, and it goes beyond what eight experiments with one editor model can show.

The surrounding literature gives the baselines. Weight-space learning (DWSNets, NFNs, graph metanetworks, ScaleGMN) builds in permutation and scale symmetries. A March 2026 survey and an October 2026 CrossGMN paper show the field still works with small MLPs, CNNs and implicit neural representations. The ICLR 2024 graph metanetworks paper notes that earlier methods could not process normalisation layers. Nothing in the retrieved sources shows these methods reaching production LLMs. On interpretability, Neel Nanda's LessWrong list and an SAE explainer stress that SAE evaluation lacks natural-language ground truth. The August 2025 Neuronpedia circuits landscape piece documents multi-lab replication of attribution-graph work. Goodfire's recent work moves towards parameter-space decomposition, and a Trilogy AI Substack review of Goodfire's neuron-level claims says the work is interesting but not a major update to the big picture. On editing, the ACL 2024 papers show ROME and MEMIT degrade or collapse under scale and sequential edits. A 2026 mechanism study (GPT2-XL, LLaMA2-7B) compares ROME, MEMIT, fine-tuning and in-context editing. A 2026 write-up on Heretic places abliteration as a permanent weight edit that weakens a behavioural direction, which again removes an obstacle and adds no capability.