Weight-Space Programming

Narrative

The Astra result is two to three days old at the time of writing, and the evidence base is the authors' own blog post plus press and developer commentary. The post, "Sparks of Weight-Space Programming", appeared on 2 November 2026 from Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL. It is not peer reviewed, but it sits in category (b): the authors published logs on GitHub and the edited Qwen model on Hugging Face (R&D World). Per secondary summaries, the editor was OpenAI's GPT-6 Astra called through Codex CLI with built-in tools off, across eight experiments (ai-tldr.dev). For the headline run, Astra got a Python session over the weights of Qwen3.5-0.8B, practice problems from GSM8K and MATH, and a capped number of forward passes (AI Polix). That is inference with the ability to run the network, not weights alone, and the human steering and attempt count still need checking in the logs, which I have not read.

The reported numbers are MATH-500 from 7.9% to 48.1%, GSM8K from 13.8% to 67.1% and IFEval from 21.0% to 62.3%, from editing 131,072 of roughly 0.8 billion weights (0.016%). The authors say Astra traced repeated looping to one direction written by the MLPs of layers 15 and 19, and subtracted it from 64 output columns in each layer (Runtimewire). R&D World tracked a column of answers hitting the token limit, which fits a defect-removal reading. The same post reports that the method did not improve Gemma 3 270M, a model without the looping flaw (AI Polix; Explainx). So the edit removes an obstacle rather than adding a capability. R&D World also reports critics, two developers, who say a human engineer could have done the same job with standard ML practice. The "20 hours" and "surgical" framing comes from low-quality crypto and aggregator sites and should not be treated as evidence.

Earlier reading work sits on firmer ground but at different scale and access. Anthropic's sparse autoencoder work reached 34 million features on one middle-layer residual stream of Claude 3 Sonnet, and showed that features can steer behaviour. Its 2025 circuit tracing papers built attribution graphs on Claude 3.5 Haiku using cross-layer transcoders, which form a replacement model rather than the original network. The authors describe graphs as a source of hypotheses to be tested by perturbation, and note that global circuits were harder to interpret than prompt-specific graphs. An August 2025 multi-lab replication effort (Anthropic, Google DeepMind, EleutherAI, Goodfire, Decode) reimplemented the method on Gemma-2-2B, Llama-3.1-1B and Qwen3-4B. Goodfire's May 2026 parameter decomposition paper works on weights rather than activations and made a predictable emoticon edit, but as a proof of concept on a small model with a remaining performance gap against fine-tuning.

Dedicated weight-space learning is a separate literature that reads weights without running the network. DWSNets and neural functional networks build permutation equivariance into the layers, graph metanetworks (ICLR 2024) extend this to varied architectures, and SANE (ICML 2024) scales to larger ResNets. The 2026 survey and the October 2026 CrossGMN preprint show the field is still working on expressivity and cross-architecture transforms. A 2025 analysis of graph metanetwork representations covered only INR classification, and the field's benchmarks lean on small MLPs, CNNs and INRs. No retrieved source shows these methods working on billion-parameter language models, so the gap between them and the Astra demonstration is one of scale and access, not just method. Not covered in this lane's searches: METR or other external evaluations of Astra, independent replications, and primary papers on ROME, MEMIT, activation steering and abliteration.