Weight-Space Programming

Narrative

The Astra result is a blog post with public logs, not a peer-reviewed paper. Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab (MIT CSAIL) posted it on 2 November 2026, with experiment logs on GitHub and the edited Qwen model on Hugging Face (R&D World). The pooled claim covers eight experiments at very different scales and access levels. In the reading experiments, Astra received raw weights as text with no tools and no forward passes. It named the game behind a 17,253-weight network in 4 of 4 cases (R&D World, Turing Post, Explainx). The Qwen3.5-0.8B edit was a different kind of task. Astra had Python and up to 50 practice batches. It changed 131,072 weights (0.016%) in layers 15 and 19, and MATH-500 rose from 7.9% to 48.1%. Answers hitting the token limit fell from 93% to 33% (R&D World, DataCamp, Turing Post). The 3.5% edit that took a control policy from 0% to 100% success was a single attempt with no tools (Explainx, DataCamp).

The evidence is thin on four counts. First, the prompt told Astra that fixing repetition counted, and Qwen's own model card already flags the looping (Turing Post). Second, Explainx reports that the same approach failed to improve a non-looping Gemma 3 270M. That suggests the edit removed an obstacle (a known failure mode) rather than adding a capability. Third, R&D World records critics (Honnibal, Goedecke) who call the Qwen fix standard steering and object to pooling different tool-access levels under one claim. Fourth, the only replication I found is a secondary social-media claim of reproduction on Qwen3.5-0.8B (alphaXiv aggregation), which is anecdotal. I found no ablation of simpler explanations for the game identification, such as weight statistics or output-layer structure, and no published harness.

Dedicated weight-space learning has a different profile. DWSNets (Navon et al., ICML 2023) built permutation-equivariant layers for MLP weight spaces and were demonstrated on INRs and small networks. Related work covers neural functionals, graph metanetworks (Lim et al., ICLR 2024), ScaleGMN (NeurIPS 2024) and a 2026 survey (arXiv 2603.10090). Recent preprints push towards transformers: a 2026 graph-metanetwork paper formalises the symmetry group of complete decoder-only language models and claims the first systematic application at that scale. Another 2026 result shows DWSNets and GMNs have equivalent expressive power on compact general-position sets. These methods are trained supervised predictors. They need many example networks and infer properties from weights alone. They do not edit anything, and their transformer-scale evidence is still preprint-level.

Direct editing has the most mature critical literature. ROME and MEMIT (Meng et al., NeurIPS 2022 and ICLR 2023) insert facts by rank-one or batched updates. Follow-up work found specificity side effects in GPT-J (6B) that earlier benchmarks missed, gradual then catastrophic forgetting under sequential edits, and brittleness in code LLMs. The refusal-direction paper (Arditi et al., NeurIPS 2024, 13 chat models up to 72B) shows that one removable direction mediates refusal. Abliteration turns that into a permanent weight edit. Follow-ups question the 'scalpel' framing, reporting off-target effects on decision disposition across model families and a dependence of post-abliteration compliance on model scale. This is the closest precedent for Astra's Qwen edit: it removes an obstacle, here a trained-in safeguard, rather than adding a skill. For sparse autoencoders, the retrieved sources show scaling laws (Gao et al.) alongside critiques that SAE features are not canonical units and that monosemanticity is unreliable. Olah's circuits and feature-visualisation papers, Goodfire, SANE, METR's HCAST and RE-Bench, and primary activation-steering papers did not surface in my searches, so they are not covered here.