In the first 3 months of 2026, we have already seen some of the biggest LLM architectural breakthrough in a long time. While a good chunk of 2025 was
playing around with RL to apply it onto LLMs, which got us more and more acronyms that I don't even know what they stand for, I am so glad that my
videos this year is not going to be spending 10 minutes explaining an acronym that might not even be
applied at practice. Because this time, Kimi Moonshot, or I'm just calling them Kimi AI cuz it's easier, they have published this new research called
attention residuals and have already tested at scale with their model, which made this approach extra promising. Not only that, the idea is so much
cleaner and so intuitive, which made me think this might actually be a defining and pivotal paper for
this year. And can you imagine one of the key person that's behind this is only a 16-year-old high schooler. This is like a different CS than what I
was doing when I was 16. But before we dive into it, with so many new models that are coming out left and right, keeping off their subscriptions just
to try them may not be the most optimal way to spend your money. Because why would you even pay five
subscriptions when there is an all-in- one bundle that lets you use them for just 10 bucks a month. This website called Matt Mood AI provides you with
models ranging from Claude to GPT, Gemini, Llama, Mistral, Grok, Deep Seek, Perplexity, Flux, Nano Banana, and Recraft. So, if you want to try out
different LLMs or even image generation, they have it for you all in one place. On top of that, this
platform also lets you re-prompt, compare answers, and have models challenge each other. So, you get to pick the higher quality outputs while not
needing to switch between 10 tabs to check manually which one is the best. This multi-model workflow be capable of analyzing documents and images, use
deep research through Perplexity, and even do voice chat and dictation when you want to move faster.
And if you have repetitive tasks, you can also just use the project function where you can create your own custom Matt Moods with your own
instructions, preferred model that matches your habits, and keep everything organized instead of starting from scratch every time. So, if you want one
clean place to use the best models without investing in it so much subscription money, check them out now
using the link down in the description, and thank you, Matt Mouw, for sponsoring this video. Anyways, this time we got a brand new concept. But,
compared to all previous new deep seek architectures that was released in the last few months, I think this is a lot easier to understand. Well,
hopefully. So, in a typical transformer diagram, it looks pretty straightforward, right? This basically
indicates there are n repeated layers. So, when you unroll it, it'll actually look like a thin long stick. You can also check out this 3D
visualization to trace through the numbers one by one yourself. Even though a common theme in AI is that the more layers, the better, there is still a
ceiling to this stacking. Because the more transformer layers there are, the more information gets piled on top
of each other with no filtering. Like, imagine this. You take notes from every single lecture you attend, but after each class, you paste all your
notes into ChatGPT and ask it to summarize everything so far into a single document. Then, after the next lecture, you do the same thing again, but
now ChatGPT is summarizing a summary plus the new notes. And, you keep repeating this process every time
there's a lecture. At first, it probably works fine. But, over time, details start to disappear. Early lectures get compressed into vague high-level
ideas, while newer information dominates the summary. And, in the end, you will no longer have access to the original notes, only this repeatedly
distilled version. So, by the end of the semester, you're studying from a heavily compressed summary of
summaries, where important nuances from earlier lectures are effectively gone. The same problem kind of applies to layers within transformers. As the
model gets deeper, all previous layer outputs are repeatedly compressed into a single representation, causing earlier information to be progressively
diluted with no way to recover or selectively access it. Not only that, since everything is summed
together with equal weights, uh later layers are competing against the accumulated signal of all previous layers. So, in order to stand out, they're
forced to produce increasingly larger outputs with larger magnitudes. Otherwise, their contribution gets drowned out. Over time, this leads to a kind
of imbalance where earlier information is diluted while later layers grow disproportionately just to
be noticed. And the representation keeps getting bigger, but not necessarily more precise. And this is what the researchers called the pre-norm
dilution problem. This exact problem was also the same problem recurrent neural networks had over the sequence dimension, which led to the downfall of
RNNs at scale as it is a more naive way of compressing all prior information into a single state, which
attention mechanisms completely avoids. Then, why haven't we done the same thing for depth? So, this is what the Kimi team at Moonshot AI decided to
fix. Instead of being forced to inherit a single mixed representation, what if a layer can directly look at earlier outputs and assign them different
weights based on the current input? This turns depth from a blind accumulation process into a
selective retrieval mechanism where information isn't lost through repeated compression, but remains accessible and can be reused when needed. To put
the idea simply, it's just attention over the depth of the network. So, it's kind of like rotating attention by 90°. So, instead of only using
attention across tokens in the sequence from left to right, the model also applies attention vertically
across layers, letting the model selectively retrieve information from its own computation history. Under the hood, this works by building connections
from every previous layer directly into the current one, kind of like a residual path. And a residual path typically appears within a layer where you
take the input and add it back after some transformation. The purpose of this is to preserve
information and make training stable by giving the model a direct path to carry signals forward and also receive them backwards. But, this existing
shortcut is typically local for a transformer. It usually doesn't go across layers. So, what Kimi AI did here is to extend the same idea across the
depth of the network. And instead of a single residual connection, the current layer receives a direct
path from every previous layer. So, rather than inheriting one already mixed representation, it has access to all previous intermediate states
separately. But, on top of that, instead of adding them all equally, the model assigns weights using attention to each of these paths in relation to
the current layer. So, each previous layers output becomes its own signal and the current layer decides how
much to use from each one. More specifically, the layers forms a query and compares it against all earlier layer outputs producing a set of scores.
Then after normalization, these scores become weights that sum to one and the final output is just a weighted combination of those past
representations, which means the residual path would not be fixed. So now, the model can amplify certain layers,
ignore others, and dynamically change this behavior depending on the input, just like how attention can selectively pay attention to previous tokens.
What used to be a simple additive shortcut is now a trainable input dependent routing mechanism across depth. Pretty sick, right? But of course, just
like attention, the problem of quadratic scaling would still hunt this technique because if there
are 128 layers, that's 128 vectors you need to keep alive in memory and attend over at every single layer. And if every layer attends to each other,
that's like around 8,000 attention operations at the last layer. So, in the paper, they proposed a more efficient alternative called the block
attention residuals, where instead of letting every layer attend to every previous layer individually, they
group layers into blocks. Within each block, layers behave like normal when they can attend to their previous layers, but at the end of the block,
their outputs are combined into a single summary representation. Then across blocks, the model only attends to the summaries and not every individual
layer. So, instead of attending over 128 separate vectors, you might only need to attend something like
eight block-level representations where each block contains around 12 layers. This reduces both memory and compute from scaling with the total number
of layers down to scaling with the number of blocks. So, you trade a bit of granularity for a massive gain in efficiency, making the whole approach
practical at scale. But, how big of a trade-off is this? In a paper, they showed that this trade-off
is surprisingly small. In this figure, they plot validation loss against compute for three setups, the baseline, full attention residuals, and block
attention residuals. As you can see, both versions of attention residuals consistently sit below the baseline curve, meaning that they achieve lower
loss for the same amount of compute. But, more importantly, the block version almost overlaps with the
full version. Even though it's only attending over number of block summaries instead of every layer, its curve tracks very closely to full attention
residuals across all scales. So, the gap between them is actually very tiny. But, on top of that, block attention residuals can match the performance
of the baseline while using about 1.25 times less compute, which is pretty much getting a 25%
discount for training while paying only 4% more training overhead. But, after all this empirical evidence, how does attention residual actually make
sense? Well, theoretically, this improves information preservation. Since earlier layers remain individually accessible, useful signals don't have to
fight to stay alive through repeated mixing. And the process of being able to propagate signals
cleanly forward and backwards through the residual paths made learning much more efficient. This approach also fixes the magnitude problem where in
the standard residual setup, layers compete through magnitude. So, later layers have to produce larger output to influence the final representation.
So, if you take a look at figure five, you can see how the magnitude of the vector stays consistent for
block attention residual while increases exponentially, confirming it is relieving this negative feedback. Because now with attention-based weighting,
they compete through relevance instead, which is much better as the model can just assign higher weight to the layers that matter. This also increases
expressivity along the depth dimension as you are no longer having a fixed linear accumulation and
instead a dynamic input-dependent combination over all layers, which is much more flexible. So, it is through these three reasons that attention
residual is able to be more efficient than the standard setup. For downstream benchmarks on their 48 billion parameter model trained on 1.4 trillion
tokens, it improves across every single evaluated task. And the biggest jumps are multi-step reasoning
like GPQA Diamond and math, where later layers being able to selectively reach back and retrieve earlier representations makes the biggest difference,
which here shown through better reasoning. But what about MHC? If you haven't watched my MHC video previously, then isn't MHC also doing something
similar to residual connections? At a high level, yeah. Both of them are trying to fix the same
underlying issue where standard residuals collapse too much information into a single stream. But they both approach it in very different ways. MHC
only works within a layer. So instead of only having one representation, it keeps multiple parallel streams and lets them mix with each other at every
layer. So you can think of it as widening the network, giving each layer a richer set of
representations to work with at that specific depth. Whereas attention residuals work across layers. It doesn't change what happened inside a layer,
but changes how layers connect over depth. And each layer can look back and selectively pull from previous layers. So they're basically orthogonal
directions, which means in theory you could combine them. You could have multiple streams per layer like
MHC, then let each layer attend over all previous layers using attention residuals. That would both give you richer representations at each step and
better control over which path information to use. But in practice, it is definitely a bit unclear if there are benefits. Both methods are ultimately
trying to solve the same bottleneck, just from different angles. MHC expands the representation so
less information is lost, while attention residual makes it easier to recover information later. So when you stack them, it's highly likely to get
diminishing returns. On top of that, part of the appeal of attention residuals is efficiency. The paper shows it can match or outperform methods like
MHC while using far less memory and compute overhead. So if you combine them, you lose that simplicity
and efficiency advantage. And of course, it's going to make the implementation a complete nightmare. So, while they're not redundant in theory, in
practice though, Kim et al.'s framing suggests that attention residuals is the cleaner solution, which I do agree that it seems simpler and a bit more
elegant than MHC. And everything we just talked about actually holds up at scale. They plugged
attention residuals directly into their latest Kim et al. linear architecture without changing the rest of the model. And they would still see
consistent gains across training, scaling laws, and downstream benchmarks. On top of that, to make it work in practice, they had to solve the
efficiency problem. Because now, instead of just passing one hidden state between layers, you need to keep multiple
past representations alive and make them accessible for attention. That creates both memory pressure and communication overhead, especially in large
distributed training setups. So, like 2/3 of the paper is actually talking about how to make this efficient. Without getting into it too much, their
block attention residual design already reduces the number of things you need to store compared to the
full version. And on top of the other implementations like two-phase computation strategy and caching across pipeline stages. So, the training
overhead is able to be pushed down to only a few percent and inference latency increases by less than 2%, which is almost like nothing. So, now, as
you can see today, Frontier Labs not only have to figure out how to make cool architectural breakthrough, but
also implement them at scale to prove that it actually works. What a sweaty research landscape we are living in. Not to mention, the lead author is
also a 16-year-old. Wow. And if you also don't want H capped and be too behind on the technical side of LLMs, you should definitely check out my
latest project intuitive.ai.academy, where I can get you on board to the frontier of LLMs intuitively
without crazy-looking maths. Ranging from LLM architectures, MoE, LoRA, to our latest chapters reinforcement learning, where we cover how RL works and
how it interacts with LLMs accompany with our latest interactive visualizations to help you better understand its logic. So, for those who want to get
into AI or LLMs, this should be the perfect place for you to dive into the technical parts without
being intimidated by crazy-looking maths. And right now, we are offering a summer discount, so use the code summer for 25% off a yearly plan. And
thank you guys for watching. A big shout out to Spam Mage, Chris Ladou, D Gan, Robert Zaviassa, Marcelo Ferreria, Poof and Enew, DX Research Group,
Alex, Midwest Maker, and many others that support me through Patreon or YouTube. Follow me on Twitter if
you haven't, and I'll see you in the next one.