Research · Weight-Space Programming
Back to researchResearch sweep · deep · 2021 – 2026
Weight-Space Programming
Reading and rewriting neural networks from their weights, November 2021 to November 2026: LLM-driven weight-space programming (MIT CSAIL's Sparks of Weight-Space Programming, GPT-6 Astra, Qwen3.5), weight-space learning (DWSNets, neural functionals, graph metanetworks, SANE), mechanistic interpretability (Chris Olah's circuits and feature visualisation, sparse autoencoders, Goodfire), and direct model editing (ROME, activation steering), compared on what each can actually infer or change.
Synthesised 2026-11-05
Overview
An LLM has fixed a known looping fault in a 0.8-billion-parameter language model by editing 0.016% of its weights, and it has named the game behind 17,253-weight policy networks from the raw numbers alone, four times out of four. That is what MIT CSAIL's "Sparks of Weight-Space Programming" has shown. It has not shown that an LLM can read a large model from weights alone, add a capability a model lacked, or do anything a competent engineer could not do with standard tooling in an afternoon. The result is a blog post with public logs, three days old at the time of writing, with no peer review and no independent replication beyond one social-media claim.
Sources: R&D World (2026) (↗); Turing Post (2026) (↗); alphaXiv (2026) (↗)
The topic is the long effort to treat a trained neural network as an artefact that can be inspected and rewritten, rather than a black box that can only be retrained. Three traditions feed it. Mechanistic interpretability, from Chris Olah's feature visualisation onwards, reads models by running them and studying activations. Weight-space learning trains dedicated networks to predict properties of other networks from their weights without running them. Direct editing (ROME, MEMIT, activation steering, abliteration) changes behaviour by altering specific weights or activation directions.
The defining shift of the past 18 months is that general-purpose LLMs have entered all three roles at once. Anthropic's circuit tracing in 2025 used an LLM-scale model as the subject; Goodfire's 2026 parameter decomposition moved interpretability from activations to weights; the Astra work put a frontier LLM in the operator's seat, reading tensors and writing edits. The question for a technology leader is whether this is a new way to build and audit models, or a demonstration that reflects easy structure in small networks and a well-documented defect in one small model.
Sources: Transformer Circuits Thread (Anthropic) (2025) (↗); Goodfire (2026) (↗); ai-tldr.dev (2026) (↗)
Timeline
- Feature visualisation shows individual vision neurons can be rendered legible
- Circuits thread maps hand-traced circuits in a small vision network
- Rank-one and batched factual editing (ROME, MEMIT) arrive for GPT-scale models
- Permutation-equivariant weight-space networks (DWSNets) formalise weight reading
- Side-effect benchmarks expose editing specificity failures
- Graph metanetworks handle varied architectures and normalisation layers
- Sequential editing shown to cause catastrophic forgetting
- 34 million SAE features extracted from Claude 3 Sonnet
- Refusal traced to a single removable direction
- Weight-space learning scales to larger ResNets (SANE)
- Equivariant functionals proposed for transformers
- Attribution graphs trace prompt-level circuits in Claude 3.5 Haiku
- SAE features shown not to be canonical units
- Multi-lab replication of circuit tracing on open models
- Abliteration methods compared systematically
- Goodfire raises $150M and publishes weight-level parameter decomposition
- Weight-space learning survey marks field consolidation
- Graph metanetworks formalise symmetry group of decoder-only LMs
- Abliteration shown to have off-target effects
- Astra reads and edits weights in eight experiments
Key Findings
The Astra edit removed an obstacle; it did not install a skill. Astra traced Qwen3.5-0.8B's repetitive looping to a single direction written by the MLPs of layers 15 and 19, then subtracted it from 64 output columns in each layer. Answers hitting the token limit fell from 93% to 33%, and MATH-500 rose from 7.9% to 48.1%. The same procedure did nothing for Gemma 3 270M, which has no looping fault. The prompt also told Astra that fixing repetition counted, and Qwen's own model card already flags the problem. Evidence strength: strong for the mechanism and the numbers (public logs and model), weak for any claim of new capability.
Sources: Runtimewire (2026) (↗); R&D World (2026) (↗); Explainx (2026) (↗); Turing Post (2026) (↗)
The eight experiments pool incompatible access regimes. In the reading experiments Astra received weights as text with no tools and no forward passes. For the Qwen edit it had a Python session, GSM8K and MATH practice problems, and up to 50 practice batches, which is inference with execution, not weight reading. The sycophancy experiment required Astra to build its own activation probes, and it could not distinguish two models on sycophancy from weights alone. Critics quoted by R&D World (Honnibal, Goedecke) object specifically to pooling these under one claim. Evidence strength: strong, since all three lanes converge on the same distinction.
Sources: AI Polix (2026) (↗); DataCamp (2026) (↗); Medium (Data Science in Your Pocket) (2026) (↗); MindStudio (2026) (↗); R&D World (2026) (↗)
The game identification may rest on cheap statistics. Emergent's summary attributes the 4-of-4 game identification to first-layer weight concentrations and output-layer action distributions. A Medium explainer makes the same point: spatial bands in first and last layers, plus known query-key and output-value recipes, give away a great deal without any circuit reading. No ablation against a weight-statistics baseline has been published. Evidence strength: moderate for the hypothesis, absent for the test.
Sources: emergent.sh (n.d.) (↗); Medium (Data Science in Your Pocket) (2026) (↗)
Reading weights is hard for reasons that have nothing to do with intelligence. A hidden layer's neurons have no fixed order. Shuffle them, and shuffle the matching rows and columns in the layers either side, and the network computes exactly the same function from a completely different weight file. For common activation functions you can also scale every incoming weight to a neuron by a constant and every outgoing weight by its inverse, again without changing anything. Two identical networks can therefore look entirely unlike one another on the page, and a reader must see through these symmetries or rely on quantities that survive them, such as norms, distributions and the structure of input and output layers. Then there is scale: 17,253 weights fit in a context window; 800 million do not, which is why the Qwen work needed code and execution.
Sources: arXiv (ICML 2023) (2023) (↗); arXiv (NeurIPS 2024) (2024) (↗); NVIDIA Technical Blog (2023) (↗)
Dedicated weight-space learning builds those symmetries in, and is still small-scale. DWSNets (ICML 2023) made layers equivariant to permutation; graph metanetworks (ICLR 2024) represented networks as graphs so that varied architectures, including normalisation layers earlier methods could not handle, could be processed; SANE (ICML 2024) scaled to larger ResNets. The March 2026 survey and October 2026 CrossGMN preprint still work on MLPs, CNNs and implicit neural representations. A 2026 preprint formalises the symmetry group of complete decoder-only language models and claims the first systematic application at that scale, but it is preprint-level evidence. These systems are supervised predictors that need many example networks, and they do not edit anything. Evidence strength: strong for the methods, strong for the scale limitation.
Sources: arXiv (ICML 2023) (2023) (↗); ICLR 2024 Proceedings (2024) (↗); ICML 2024 (ACM Digital Library) (2024) (↗); arXiv (2026) (↗); arXiv (2026) (↗); arXiv (2026) (↗)
Olah's tradition delivered legibility, not whole-model explanation. Feature visualisation and the circuits thread established that small vision networks contain interpretable features and hand-traceable circuits. The scaled successors read activations, not weights: Anthropic extracted 34 million features from one middle-layer residual stream of Claude 3 Sonnet, and its 2025 circuit tracing built attribution graphs for Claude 3.5 Haiku using cross-layer transcoders, a replacement model rather than the original network. The authors themselves describe the graphs as hypothesis generators to be checked by perturbation, and found global circuits harder to interpret than prompt-specific ones. Evidence strength: strong, with an August 2025 multi-lab reimplementation on Gemma-2-2B, Llama-3.1-1B and Qwen3-4B.
Sources: Anthropic (arXiv mirror of Transformer Circuits paper) (2024) (↗); Transformer Circuits Thread (Anthropic) (2025) (↗); Transformer Circuits Thread (Anthropic) (2025) (↗); Neuronpedia (2025) (↗)
Sparse autoencoders are useful and contested at the same time. Gao et al. gave SAEs scaling laws, but an ICLR 2025 paper shows their features are not canonical units of analysis, a 2025 theoretical framework sets limits on what they can recover, and Neel Nanda's own reading list stresses that evaluation lacks natural-language ground truth. Goodfire, which raised $150M at a $1.25B valuation in February 2026 on an interpretability thesis, has since moved towards decomposing parameters directly, with a predictable emoticon edit as proof of concept on a small model and a performance gap against fine-tuning. Evidence strength: strong that SAEs find structure, moderate that the structure is the right structure.
Sources: arXiv (2024) (↗); arXiv (ICLR 2025) (2025) (↗); arXiv (2025) (↗); LessWrong (GreaterWrong viewer) (2024) (↗); PR Newswire (Goodfire) (2026) (↗); Goodfire (2026) (↗)
Direct editing has the most mature failure literature, and it points the same way as Astra. ROME and MEMIT insert facts via rank-one or batched MLP updates. Follow-up work found specificity side effects in GPT-J that earlier benchmarks missed, gradual then catastrophic forgetting under sequential edits, and brittleness in code models. Arditi et al. showed across 13 chat models up to 72B that refusal is mediated by one removable direction; abliteration turns that into a permanent weight edit, and 2026 follow-ups report off-target effects on decision disposition and scale-dependent compliance. Every one of these methods removes, suppresses or redirects a behaviour the model already has. Evidence strength: strong.
Sources: arXiv (ICLR 2023) (2022) (↗); arXiv (2023) (↗); ACL Findings 2024 (2024) (↗); arXiv (2025) (↗); arXiv (NeurIPS 2024) (2024) (↗); arXiv (2026) (↗); arXiv (2026) (↗)
Evidence & Data
The Astra numbers come from the authors via secondary reports. Editing 131,072 of roughly 0.8 billion weights (0.016%) in Qwen3.5-0.8B moved MATH-500 from 7.9% to 48.1%, GSM8K from 13.8% to 67.1% and IFEval from 21.0% to 62.3%, with token-limit answers falling from 93% to 33%. A separate single-attempt, no-tools edit of 3.5% of a control policy's weights took success from 0% to 100%. Game identification was 4 of 4 on 17,253-weight networks. The Gemma 3 270M control saw no improvement. Eight experiments, one editor model, logs for all eight on GitHub.
Sources: Runtimewire (2026) (↗); R&D World (2026) (↗); DataCamp (2026) (↗); Explainx (2026) (↗); ai-tldr.dev (2026) (↗)
For context, Anthropic's SAE work scaled to 34 million features on a single layer of Claude 3 Sonnet, and its attribution graphs cover individual prompts on Claude 3.5 Haiku rather than global behaviour. The refusal-direction result spans 13 models from small to 72B parameters. The weight-space learning literature's largest demonstrated subjects are ResNets and, in preprint, decoder-only language models of unstated scale. The editing failure literature uses GPT-J (6B), GPT2-XL and LLaMA2-7B.
Sources: Anthropic (arXiv mirror of Transformer Circuits paper) (2024) (↗); Transformer Circuits Thread (Anthropic) (2025) (↗); arXiv (NeurIPS 2024) (2024) (↗); ICML 2024 (ACM Digital Library) (2024) (↗); arXiv (2026) (↗); arXiv (2026) (↗)
The "20 hours" and "surgical" framings circulate on KuCoin, HTX and similar aggregators and should not be treated as evidence of anything.
Sources: KuCoin News (2026) (↗); htx.com (n.d.) (↗)
Signals & Tensions
Standard engineering or new paradigm. The authors frame the Qwen edit as a sign of native weight-reading ability. Developers quoted by R&D World say a human with activation statistics and a profiler would have found the same direction. Both can be true: the novelty is who did the work, not what the work was. The abliteration literature already shows that subtracting a single direction can produce large behavioural change, so the mechanism is well understood.
Sources: R&D World (2026) (↗); arXiv (2025) (↗); BestHub (2026) (↗)
Activations versus weights. Nearly every interpretability gain of the past five years came from running models and reading activations. Goodfire's parameter decomposition and the Astra reading experiments are the exceptions, and both are at small scale. The blog lane notes that even Astra's sycophancy work needed activation probes, which places it in the same tradition as SAEs rather than beyond it.
Sources: Goodfire (2026) (↗); Medium (Data Science in Your Pocket) (2026) (↗); LessWrong (2024) (↗)
Symmetry: encoded or ignored. Metanetworks spend considerable architectural effort respecting permutation and scale symmetries. An LLM reading raw text has no such guarantee. Either it has absorbed enough about weight structure to cope, or it leans on invariant statistics, which is the simpler-explanation hypothesis. The lanes agree this is untested.
Sources: arXiv (ICML 2024) (2023) (↗); arXiv (2026) (↗); emergent.sh (n.d.) (↗)
Hype gradient. Commentary predicting the end of gradient-based optimisers (Emergent, DataCamp) goes far beyond eight experiments and one editor. The academic and blog lanes both flag it as promotional. The strongest critical writing is a Medium explainer; no LessWrong, Substack or Willison analysis had appeared at the time of the sweep.
Sources: emergent.sh (n.d.) (↗); DataCamp (2026) (↗); Medium (Data Science in Your Pocket) (2026) (↗)
Underreported: the Gemma null result. The failure on Gemma 3 270M is the most informative experiment in the set, because it bounds the claim. Most coverage mentions it in passing or not at all.
Sources: Explainx (2026) (↗); AI Polix (2026) (↗)
Open Questions
How much human steering and how many attempts sat behind each reported number? The logs exist but none of the lanes had read them; the Frontier lane flags attempt counts as unverified.
Sources: ai-tldr.dev (2026) (↗)
Would a weight-statistics baseline (layer norms, first-layer spatial structure, output-layer action histograms) match 4-of-4 game identification? No one has run it.
Sources: emergent.sh (n.d.) (↗)
Can any method, Astra included, add a capability a model demonstrably lacked rather than unmask one it had? The editing literature offers no clear case; the Qwen and control-policy edits are consistent with unmasking.
Sources: arXiv (2026) (↗); arXiv (2026) (↗)
Does the Qwen edit hold up under broader evaluation, or does it degrade behaviour the three benchmarks do not measure, as abliteration does on decision disposition?
Sources: arXiv (2026) (↗); arXiv (2023) (↗)
What would a credible harness look like? Held-out networks, blinded targets, declared attempt counts, non-Astra editors and statistics baselines are the obvious components. The authors mention a planned harness; nothing has been published.
Sources: Turing Post (2026) (↗)
Can symmetry-aware metanetworks and LLM readers be compared on the same tasks at transformer scale, now that one preprint claims a symmetry formalisation for decoder-only models?
Sources: arXiv (2026) (↗); arXiv (2026) (↗)
What are the security consequences if the reading claim generalises? Auditing open weights for backdoors and stripping safeguards are the same operation with opposite intent, and abliteration already shows the stripping half works. Whether an LLM can find a deliberately hidden backdoor from weights alone is entirely untested.
Sources: arXiv (NeurIPS 2024) (2024) (↗); BestHub (2026) (↗)
The sharpest test of weight-space programming is not whether Astra can lift a benchmark on a model whose fault is written on its own model card. It is whether a second editor, on a model nobody has diagnosed, finds something a profiler would have missed.
Sources
Summary: ↑ Back to summary
Frontier Lab & Model News
25 sources
| ID | Title | Outlet | Date | Significance |
|---|---|---|---|---|
| t1 | MIT team says GPT-6 Astra can improve other AI models by editing their weights. Critics say its biggest result relied on standard ML engineering. | R&D World | 2026-11 | Most balanced secondary report on the Astra result: confirms the 7.9% to 48.1% MATH-500 figure on 131,072 weights, the public logs and Hugging Face model, and records developer critics who say a human could do it. |
| t2 | MIT CSAIL researchers report GPT-6 Astra can read and directly rewrite neural network weights, lifting a small Qwen model's MATH-500 score from 7.9% to 48.1% | AI Polix | 2026-11 | Describes the access setup (Python session over weights, practice problems, capped forward passes) and states the approach did not improve a model lacking the looping flaw. |
| t3 | GPT-6 Astra edited 0.016% of Qwen's weights and its math score went up 6x | Runtimewire | 2026-11 | Gives the mechanism claim (looping traced to one direction from MLPs of layers 15 and 19, subtracted from 64 output columns each) and frames the Gemma result as the boundary of the method; links the authors' X thread and blog. |
| t4 | Weight-Space Programming: GPT-6 Astra reads and edits network weights | ai-tldr.dev | 2026-11 | Summarises the eight experiments, the Codex CLI setup with built-in tools off, and the public RikiyaT/lm_understand-logs repository holding prompts, code and outputs, which is what enables replication. |
| t5 | GPT-6 Astra Edits Qwen Weights: MATH-500 7.9% to 48.1% | Explainx | 2026-11 | Explicitly flags that the results are not peer reviewed and that the edit failed to help non-looping Gemma 3 270M; names authors and the logs repository. |
| t6 | MIT CSAIL Proposes Weight-Space Programming; Astra Improves Qwen's Mathematical Performance with a 0.016% Weight Adjustment | KuCoin News | 2026-11 | Reports the three benchmark gains (MATH-500, GSM8K 13.8% to 67.1%, IFEval 21.0% to 62.3%) and author names; aggregator-grade, useful only for the headline figures. |
| t7 | Circuit Tracing: Revealing Computational Graphs in Language Models | Transformer Circuits Thread (Anthropic) | 2025-03 | Anthropic's primary method paper: cross-layer transcoders and attribution graphs on a replacement model, with quantitative evaluation against neurons and per-layer transcoders. |
| t8 | On the Biology of a Large Language Model | Transformer Circuits Thread (Anthropic) | 2025-03 | Applies attribution graphs to Claude 3.5 Haiku (production model) with perturbation tests, and records that global circuits were harder to interpret than prompt-specific graphs. |
| t9 | Tracing the thoughts of a large language model | Simon Willison's Weblog | 2025-03 | Independent practitioner commentary (category d) summarising the two Anthropic circuit papers, including multilingual circuit sharing. |
| t10 | The Circuits Research Landscape: Results and Perspectives (August 2025) | Neuronpedia | 2025-08 | Multi-lab replication and extension (Anthropic, Google DeepMind, EleutherAI, Goodfire, Decode) of attribution graphs on open models such as Gemma-2-2B, Llama-3.1-1B and Qwen3-4B; the nearest thing to independent replication of circuit tracing. |
| t11 | Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet | Anthropic (arXiv mirror of Transformer Circuits paper) | 2024-05 | Canonical SAE result at production scale (up to 34 million features on one middle-layer residual stream), showing features can steer behaviour; the original dates from 2024 and this is an arXiv mirror. |
| t12 | Interpreting Language Model Parameters | Goodfire | 2026-05 | Goodfire and MATS paper (May 2026) decomposing weights rather than activations; includes a hand edit of one subcomponent that changes emoticon predictions, described by the authors as a proof of concept with a performance gap to fine-tuning. |
| t13 | Paper Summary: Interpreting Language Model Parameters | Goodfire | 2026-05 | Goodfire's own summary of parameter-space editing with no training and off-target effects comparable to fine-tuning; company-authored, not independently replicated. |
| t14 | This startup's new mechanistic interpretability tool lets you debug LLMs | MIT Technology Review | 2026-04 | Independent reporting on Goodfire's Silico tool and its claim to support debugging across the training pipeline; notes MIT Technology Review ranked mechanistic interpretability a 2026 breakthrough technology. |
| t15 | AI Lab Goodfire Raises $150M at $1.25B Valuation to Design Models with Interpretability | PR Newswire (Goodfire) | 2026-02 | Company press release (category d) giving the commercial context and an unreplicated claim that interpretability-driven methods halved hallucinations in an LLM. |
| t16 | Designing Deep Networks to Process Other Deep Networks | NVIDIA Technical Blog | 2023 | NVIDIA explainer of DWSNets (Navon et al., 2023), which process MLP weights while respecting permutation symmetry; evaluated on small MLPs and INRs, context for scale limits. |
| t17 | Graph Metanetworks for Processing Diverse Neural Architectures | ICLR 2024 Proceedings | 2024 | Peer-reviewed ICLR 2024 paper: represents input networks as graphs so one metanetwork handles varied architectures, and shows GMNs can simulate the input network's forward pass. |
| t18 | Towards scalable and versatile weight space learning | ICML 2024 (ACM Digital Library) | 2024 | Peer-reviewed ICML 2024 SANE paper: task-agnostic layer-wise embeddings that scale to larger ResNets and can generate unseen models, but not LLM-scale. |
| t19 | On the Internal Representations of Graph Metanetworks | arXiv | 2025-03 | Preprint that probes what graph metanetworks learn using CKA, but only on INR classification, underlining how little is known about why metanetworks work. |
| t20 | A Survey of Weight Space Learning: Understanding, Representation, and Generation | arXiv | 2026-03 | 2026 survey placing DWSNets, NFNs, Neural Graphs and GMNs as symmetry-aware architectures; the best map of what dedicated weight-space learning covers. |
| t21 | CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations | arXiv | 2026-10 | October 2026 preprint showing weight-space learning still addressing expressivity and cross-architecture operators, and citing results that DWSNets and GMNs have equivalent expressive power. |
| t22 | MIT researchers use GPT-6 Astra to enhance Qwen's mathematical ability by 6x | kucoin.com | 2 days ago | Retrieved by this lane's web search. |
| t23 | Weight-Space Programming: how GPT-6 Astra edited Qwen's weights and took MATH-500 from 7.9% to 48.1% - DEV Community | dev.to | 1 days ago | Retrieved by this lane's web search. |
| t24 | GPT-6 Astra Edits Neural Weights Directly, No Gradients | emergent.sh | 2 days ago | Retrieved by this lane's web search. |
| t25 | MIT Researchers Used GPT-6 Astra to Edit a Smaller Model's Weights Directly, No Retraining Required | techjacksolutions.com | 1 days ago | Retrieved by this lane's web search. |
Academic & arXiv
25 sources
| ID | Title | Outlet | Date | Significance |
|---|---|---|---|---|
| a1 | Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights (alphaXiv listing) | alphaXiv | 2026-11 | Aggregator entry for the MIT CSAIL blog post (2 Nov 2026) summarising the eight experiments and carrying a social-media claim of reproduction on Qwen3.5-0.8B; category (b), blog with public logs, replication anecdotal. |
| a2 | FOD#174: What is Weight-Space Programming and why it suddenly blew up the internet? | Turing Post | 2026-11 | Describes the Qwen setup: Astra compared internal signals in loops versus progress steps, the prompt hinted that fixing repetition counted, and 131,072 weights in layers 15 and 19 changed. |
| a3 | Weight-Space Programming: How LLMs Edit Network Weights | DataCamp | 2026-11 | Secondary summary giving the control-policy result (3.5% of weights, 0% to 100% success) and token-limit hits falling from 93% to 33% after the Qwen edit. |
| a4 | Equivariant Architectures for Learning in Deep Weight Spaces (DWSNets) | arXiv (ICML 2023) | 2023-01 | Foundational peer-reviewed (ICML 2023) weight-space architecture with permutation-equivariant layers, demonstrated on MLP and INR weight spaces, not LLMs. |
| a5 | A Survey of Weight Space Learning: Understanding, Representation, and Generation | arXiv | 2026-03 | Taxonomy of symmetry-agnostic, symmetry-aware and graph-based weight-space methods (DWSNets, NFN, NFT, UNF, NG, GMN), useful for mapping what the field covers. |
| a6 | Equivariant Deep Weight Space Alignment | arXiv (ICML 2024) | 2023-10 | Introduces Deep-Align, built on DWSNets, a weight-space task (aligning networks by permutation) distinct from property prediction. |
| a7 | Scale Equivariant Graph Metanetworks | arXiv (NeurIPS 2024) | 2024-06 | ScaleGMN adds rescaling symmetry to permutation symmetry in graph metanetworks (NeurIPS 2024), extending beyond permutation-only methods. |
| a8 | Equivariant Neural Functional Networks for Transformers | arXiv | 2024-10 | Extends neural functionals to attention symmetries, an early step towards symmetry-aware methods on transformers. |
| a9 | Graph Metanetworks for Transformer Based Language Models | arXiv | 2026-10 | Formalises the weight-space symmetry group of complete decoder-only LMs and claims first systematic weight-space learning at that scale; preprint. |
| a10 | Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning | arXiv | 2026-09 | Proposes HiddenProbe, reporting state-of-the-art on neural functional benchmarks including transformers; shows probing (running the network) competes with pure weight reading. |
| a11 | Scaling and evaluating sparse autoencoders | arXiv | 2024-06 | Gao et al. (OpenAI) give a training recipe and clean scaling laws for SAEs with k-sparse autoencoders, the main quantitative basis for SAE interpretability at scale. |
| a12 | Sparse Autoencoders Do Not Find Canonical Units of Analysis | arXiv (ICLR 2025) | 2025-02 | Critical test of whether SAE features are the true units of a model, relevant to what feature mapping can claim. |
| a13 | On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy | arXiv | 2025-06 | Theory of why SAEs shrink or lose features, with a reweighted fix evaluated by auto-interpretability scores. |
| a14 | Measuring and Guiding Monosemanticity | arXiv | 2025-06 | Introduces a feature monosemanticity score and argues SAEs give incomplete feature isolation, limiting reliable steering. |
| a15 | Mass Editing Memory in a Transformer (MEMIT) | arXiv (ICLR 2023) | 2022-10 | Foundational batch knowledge-editing method (up to 10,000 facts), with ROME degrading from about 32 edits in its comparison. |
| a16 | Evaluating the ripple/side effects of model editing (specificity benchmark CounterFact+) | arXiv | 2023-06 | Finds ROME and MEMIT have specificity side effects missed by standard benchmarks, tested on GPT-J (6B) across sizes. |
| a17 | Model Editing at Scale leads to Gradual and Catastrophic Forgetting | arXiv | 2024-01 | Shows sequential ROME and MEMIT edits cause gradual then abrupt forgetting, a durability limit on direct edits. |
| a18 | Understanding Robustness of Model Editing in Code LLMs: An Empirical Study | arXiv | 2025-11 | Finds parametric editors such as ROME, MEMIT and PMET brittle on code LLMs and failing under accumulation, extending edit-durability evidence to code models. |
| a19 | Refusal in Language Models Is Mediated by a Single Direction | arXiv (NeurIPS 2024) | 2024-06 | NeurIPS 2024 paper (13 chat models up to 72B) showing that ablating or adding one direction removes or induces refusal; basis for abliteration, an obstacle-removing edit. |
| a20 | Comparative Analysis of LLM Abliteration Methods | arXiv | 2025-12 | Compares open-source abliteration tools by attack success rate, and notes work challenging the strict single-direction hypothesis. |
| a21 | Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families | arXiv | 2026-07 | Tests the claim of minimal capability cost, reporting off-target behavioural effects of refusal removal; bears on edit precision and side effects. |
| a22 | Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration | arXiv | 2026-06 | Reports that post-abliteration compliance depends on model scale, supporting the distinction between unmasking a capability and having one. |
| a23 | Sparks of Weight-Space Programming: LLMs can natively read… | tldr.tech | 2 days ago | Retrieved by this lane's web search. |
| a24 | MIT Lets GPT-6 Astra Directly Modify Qwen Weights, Math Scores Soar | htx.com | 2 days ago | Retrieved by this lane's web search. |
| a25 | Recursive Self-Similarity in Deep Weight Spaces of Neural Architectures: A Fractal and Coarse Geometry Perspective | arxiv.org | Retrieved by this lane's web search. |
Blogs & Independent Thinkers
25 sources
| ID | Title | Outlet | Date | Significance |
|---|---|---|---|---|
| b1 | Weight-Space Programming: Frontier LLMs Can Natively Read and Rewrite Neural Network Weights | The Menon Lab | 2026-11 | Independent blog write-up of the MIT CSAIL post; names the authors and says the results are small-scale 'sparks', with the Qwen edit the only large-model demonstration. |
| b2 | Sparks of Weight-Space Programming Explained | Medium (Data Science in Your Pocket) | 2026-11 | Medium piece that raises the sharpest sceptical points: first and last layers have easy spatial structure, QK/OV analysis follows known recipes, and the sycophancy task needed self-built activation probes. |
| b3 | GPT-6 Astra and Weight-Space Programming: Can AI Read a Model's Weights? | MindStudio | 2026-11 | Reports that Astra could not distinguish a sycophantic model from a non-sycophantic one using weights alone, a documented failure case. |
| b4 | An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 | LessWrong (GreaterWrong viewer) | 2024-07 | Neel Nanda's practitioner map separating weights-based circuit analysis from causal-intervention circuits and covering SAE evaluation work. |
| b5 | An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs | LessWrong | 2024 | LessWrong explainer stating that SAE evaluation has no measurable ground truth in natural language, a core limit on how much of model behaviour SAEs explain. |
| b6 | Goodfire says neural networks are already clean | Trilogy AI Center of Excellence (Substack) | 2026-10 | Substack review of Goodfire's neuron-level interpretability, giving a 76% versus 11% predictive test on 183 descriptions from 200 neurons and a sceptical comment that it is not a major update. |
| b7 | On Optimism for Interpretability | Goodfire | 2025-07 | Goodfire's own case for interpretability, citing sparse dictionary learning and linear parameter decomposition; a company position, not independent evidence. |
| b8 | Model Editing at Scale leads to Gradual and Catastrophic Forgetting | ACL Findings 2024 | 2024 | Peer-reviewed ACL Findings 2024 study showing ROME and MEMIT forget gradually then abruptly under sequential edits, limiting durability. |
| b9 | Resolving Model Collapse during Sequential Model Editing | EMNLP 2024 | 2024 | EMNLP 2024 paper tracing ROME 'disabling edits' to implementation irregularities, with 5000 single edits on GPT-J-6B. |
| b10 | The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing | arXiv | 2026-03 | Compares ROME, MEMIT, fine-tuning and in-context editing on GPT2-XL and LLaMA2-7B and proposes activation steering as an alternative, helping position steering versus weight edits. |
| b11 | Heretic: Automated Model Surgery Removes Refusal Without Damaging Capabilities | BestHub | 2026-10 | Taxonomy of modification levels (prompt, activation steering, weight edits such as abliteration and ROME/MEMIT), useful for the safeguard-stripping implication; secondary, unverified claims. |
| b12 | Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders | arXiv | 2025-08 | 2025 preprint questioning general-purpose SAEs and proposing domain-specific ones, relevant to what share of model behaviour SAEs actually explain. |
| b13 | Universal Neural Functionals | arxiv.org | Retrieved by this lane's web search. | |
| b14 | Learning in Weight Spaces - EverybodyWiki Bios & Wiki | en.everybodywiki.com | 89 days ago | Retrieved by this lane's web search. |
| b15 | Do Sparse Autoencoders Generalize? A Case Study of Answerability | arxiv.org | Retrieved by this lane's web search. | |
| b16 | Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words | arxiv.org | Retrieved by this lane's web search. | |
| b17 | Analyzing Visual Aircraft Representations with Sparse Autoencoders | arxiv.org | Retrieved by this lane's web search. | |
| b18 | [2606.06333] Subspace-Aware Sparse Autoencoders for Effective Mechanistic Interpretability | arxiv.org | 154 days ago | Retrieved by this lane's web search. |
| b19 | Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning | arxiv.org | 120 days ago | Retrieved by this lane's web search. |
| b20 | Mechanistic Interpretability with Sparse Autoencoder Neural Operators | arxiv.org | 181 days ago | Retrieved by this lane's web search. |
| b21 | SAKE: Steering Activations for Knowledge Editing | arxiv.org | Retrieved by this lane's web search. | |
| b22 | Training, Reading, and Editing Legible Transformers | arxiv.org | Retrieved by this lane's web search. | |
| b23 | Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m | arxiv.org | Retrieved by this lane's web search. | |
| b24 | Manipulating Transformer-Based Models: Controllability, Steerability, and Robust Interventions | arxiv.org | Retrieved by this lane's web search. | |
| b25 | The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing | arxiv.org | Retrieved by this lane's web search. |