In this deep dive, Neural Intel explores the technical report on Attention Residuals (AttnRes), a transformative shift in how Large Language Models aggregate information across layers. We discuss the Sequence-Depth Duality, exploring how the transition from linear to softmax attention—which revolutionized sequence modeling—is now being applied to model depth.We cover:

The Problem: Why fixed unit weights in standard residuals lead to uncontrolled hidden-state growth and diluted layer contributions.

The Solution: How Full AttnRes uses a learned "pseudo-query" per layer to selectively retrieve earlier representations.

The Infrastructure: A look at Block AttnRes, which partitions layers to reduce memory overhead from O(Ld) to O(Nd), making the tech practical for 48B+ parameter models.

The Results: Why AttnRes leads to more uniform gradient distributions and superior performance on benchmarks like GPQA-Diamond and HumanEval.

Join the conversation:

X/Twitter: @neuralintelorg

Blog: neuralintel.org

The Sequence-Depth Breakthrough: Inside Kimi Team's Attention Residuals