Skip to content

Source paper

Weight-Space Programming Doesn't Need Transformers

Hyunjong Ok, Jaeho Lee
arXiv 2611.22055 · 202653 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper investigates weight-space programming (WSP), where large language models directly write the tensor parameters of a target model without gradient-based training. When given architectural freedom on multiple-choice science benchmarks, frontier LLMs abandon Transformers in favor of sparse associative-memory tables that achieve higher accuracy with up to 1,800 times fewer parameters. The authors also design an associative-memory model that uses direct hash addressing to evaluate queries up to 17 times faster on GPUs than equivalent-sized Transformers.

How it works (AI summary)

The study deploys frontier LLMs in an isolated sandbox with ten labeled examples to author Python build scripts and text knowledge bases that compile directly into target weight tensors. Rather than training dense weights via backpropagation, the resulting architectures map tokens to sparse concept vectors and query hand-written fact tables or second-order concept-pair hash tables via fixed, non-trainable forward tensor passes.