Lookup-Oriented Hardware Acceleration for Written Sparse Associative Tables
A specialized inference engine using CPU hash tables and near-memory lookups to eliminate GPU kernel overhead for sparse concept-retrieval architectures.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 6.1, Section 6.4, and Section 8, LLM-designed models launch up to 118 GPU kernels per question, running up to 90 times slower than dense Transformers because GPUs are inefficient at sparse pointer chasing and irregular hashing.
“We measured inference on a single GPU model (an RTX 3090) built for dense matrix products, whereas per question our associative-memory-based model computes hashed addresses, reads a fixed number of table rows, and accumulates their sparse entries.”
Limitation
Inference speed of the written associative models degrades rapidly on GPUs when concept vocabularies expand beyond 32,000, suffering from memory bandwidth bottlenecks and high kernel launch overhead.
Proposed Solution
Co-design a custom C++/CUDA runtime using sparse hash-based table lookup engines (such as SLIDE or RecNMP principles) that executes direct hash indexing on host memory or unified CPU-GPU shared memory, skipping dense tensor instantiation and GPU global memory thrashing.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Moving sparse table lookups to hardware better suited for irregular indexing could enable written models with hundreds of thousands of concepts to run orders of magnitude faster than dense GPU Transformers.
How to Get Started
How to test it: Measure inference latency per query across concept vocabularies from 10,000 to 200,000 concepts comparing the custom engine against the PyTorch CUDA-graph implementation on an RTX 3090. Stop if the optimized engine fails to maintain sub-0.05 ms latency per question at 100,000 concepts.
- Profile memory access patterns and L2 cache misses of the pair table and concept matrix A during batched forward passes in PyTorch.
- Implement a C++/AVX-512 sparse accumulation kernel for CPU thread execution using robin-hood hashing for pair lookups.
- Evaluate query throughput on the 3,530 ARC test items across synthetic concept vocabularies up to 262,144 concepts.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing and TensorDIMM explore near-memory processing and hardware acceleration for recommendation systems and embeddings, which closely overlap with the principles of custom lookup-oriented hardware acceleration proposed for sparse tables.
- RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing · 2020 · 316 citations
- TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning · 2019 · 290 citations
We searched OpenAlex and arXiv for “accelerating sparse table lookup near-memory processing deep learning” and “Lookup-Oriented Hardware Acceleration for Written Sparse Associative Tables” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Weight-Space Programming Doesn't Need TransformersHyunjong Ok, Jaeho Lee · 2026
This paper investigates weight-space programming (WSP), where large language models directly write the tensor parameters of a target model without gradient-based training. When given architectural freedom on multiple-choice science benchmarks, frontier LLMs abandon Transformers in favor of sparse associative-memory tables that achieve higher accuracy with up to 1,800 times fewer parameters. The authors also design an associative-memory model that uses direct hash addressing to evaluate queries up to 17 times faster on GPUs than equivalent-sized Transformers.
Related Ideas
- Hardware-Specific ISA Extension for Fused Lookup-AccumulateBitnet.cpp: Efficient Edge Inference for Ternary LLMsâ›”Very hard
- Hardware-Aware Kernels for Zero-Overhead InferenceBitNet b1.58 2B4T Technical Reportâ›”Very hard
- CUDA/Triton Implementation of Element-wise LUTsBitnet.cpp: Efficient Edge Inference for Ternary LLMsđź”´Hard
- Hardware-Efficient Backbone for Scalable DeploymentUtonia: Toward One Encoder for All Point Cloudsâ›”Very hard
- Direct GPU Graph Recording and Persistent Memory Pointers to Eliminate Decode Dispatch OverheadThe Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Predictionđźź Medium