DeepSeaV4 is what happens when you put the most crack competitive programmer together with limited compute and no preconceived notions about what's
technically possible. Oh, if something isn't efficient, well, let's implement it manually and see if that's faster. And throughout this 58-page
DeepSeaV4 report, this is pretty much a recurring theme of the paper. In the last video, I already covered
the general overview of this paper, and in this one, we are going to dive deeper into the fascinating infrastructure design they've implemented. So,
if you haven't watched that one, I strongly suggest you start with that video first and come back after. But, with the main idea from the last video
where cheap million token context is not solely based on some attention tricks, let's take a closer
look at their full stack infrastructure solution along with other cool efficiency speedups that are achieved through clever engineering. But, before
we dive into it, today's video is sponsored by Verta. As frontier models get more efficient, the infrastructure around them is also getting faster,
cheaper, and easier to use. And that's where Verta comes in. Verta is a full stack AI cloud built for
the entire model life cycle from GPU instances and instant clusters to serverless inference and managed endpoints. Unlike general-purpose cloud
providers that are retrofitting their platforms for AI, Verta is built specifically for AI workloads from small experiments all the way to large-scale
training, high-throughput inference, and agentic AI serving. Their platform is built by people who work
hands-on with real AI systems, so the infrastructure is designed around how AI teams actually train, serve, and deploy models. On the hardware side,
Verta gives you access to powerful Nvidia GPUs like B300s along with NVLink, infinite band, and fast NVMe storage for workloads and interconnect
performance really matter. They're also an official Nvidia preferred partner, which means closer alignment
with Nvidia's latest hardware roadmap. Verta has already opened early access for Nvidia GB300 NVL72 Blackwell Ultra clusters, which is some of the
most powerful AI systems available today. For smaller experiments, you can spin up on-demand GPU instances or use spot instances at up to three times
lower pricing with a tradeoff that capacity can be reclaimed, so So the best for interruptible
workloads. And if you're doing serious training or fine-tuning, Verta also offers instant clusters with self-service access to 16 to 128 GPUs
connected through InfiniBand. That's exactly the kind of setup you want when multi-GPU performance actually matters. Then when your model is ready to
deploy, you can move into production with auto-scaling serverless containers or managed endpoints, making
the path from experiments to inference a lot less painful. So whether you're experimenting, building agentic AI systems, fine-tuning models, or
deploying inference at scale, Verta gives you AI infrastructure built for speed, power, and performance without the usual scaling chaos. So check out
Verta now using the link down in the description and use code by cloud-35 to get 35 bucks in free compute
credits. And thanks again to Verta for sponsoring this video. Anyways, the first thing I uncovered in the previous video is that I made it feel like
CSA and HCA are carrying the efficiency gains by themselves. But in reality, making this combo work efficiently requires solving some of the nastiest
attention paths that come along with it. In normal attention, the model has a pretty clean structure.
You have queries, you have key and values, and the model computes attention over that KV cache. But with Deep See V4, CSA has compressed KV entries,
separate compressed indexer keys, a lightning indexer that has to score compressed blocks, and sparse selected memory. On top of that, it still has a
local sliding window branch for recent uncompressed tokens. As for HCA, it is different again. It
uses heavily compressed global memory where every 128 tokens becomes one entry, and then the model attends densely over that much smaller memory. So
now you have compressed KV, indexer KV, sparse selected KV, heavily compressed global KV, and uncompressed sliding window KV for recent tokens. You
basically need a whole custom attention path. And out of all this mess, they also decided that there's
room to push the indexer QK path, which would typically live in a higher precision FP8 or BF16 down to FP4 through quantization-aware training. Well,
I mean, this improves the speed of the attention operation under extremely long context because at 1 million tokens, even deciding which blocks to
read becomes expensive and FP4 at least relieves two times the memory usage here. But to make
everything even more difficult, the core attention mechanism is also pretty unusual. Instead of storing a separate key and value for every memory
slot, each compressed entry is used as both the key and the value in a shared KVM QA style. That saves memory, but it creates weird positional issues
because the value side is no longer position neutral. If the same compressed vector is used as both the
key and the value, then when the model takes a weight to sum over values, the output can carry absolute positional embeddings from those compressed
entries. And this is pretty bad because the attention output is supposed to tell the model what information was retrieved, not permanently baked in
where that compressed block originally lived in the context window. So if too much absolute position
leaks into the value stream, then two identical pieces of information at different locations could produce different looking outputs just because they
came from different positions, which would make long context generalization harder. So what Deep Seek did here is to apply partial rope that only
affects the last 64 dimensions, then applies a negative position rope correction to the output so the
compressed value does not leak absolute position too strongly. To also protect the logits from exploding, they apply RSM norm to both the query heads
and the compression KV entries before attention. This process is kind of important because learned compressed memories are not ordinary token level
keys. Each compressed entry is produced by merging multiple tokens through learned weights, so its
vector scale can vary more unpredictably, which is bad for training. Additionally, they also apply an attention sink trick similar to what GPT OSS
uses. Like with normal attention, all available tokens or memory blocks have to split the full attention mass. But this learnable sink logic, when put
into the softmax denominator, lets a head put less than 100% attention mass on the actual memory or
even almost zero. So when the memory is compressed, then not every block is useful for every head, so instead of forcing the head to attend to noisy
compressed memory even when it's not needed, the attention sink gives it an escape route. And finally, of course, the KV cache itself is stored in
mixed precision. The rope dimensions stay in BF16 because positional information is more sensitive, but
the remaining KV dimensions are all stored in FPA which nearly halves the KV cache size compared with the common BF16 storage. So, when people say
DeepSeek V4 may long context cheap with CSA and SCA, that is only the surface level explanation. They actually have to solve the hidden roadblocks
underneath by making the entire attention path efficient, too. And this is probably why creating custom
attention is such a risky bet for most labs because it is simply just not enough to come up with a clever attention pattern. You also need the kernels
and the surrounding optimization set up to use that pattern. Otherwise, it would just be a mechanism that looks good on paper, but who knows how good
it is at scale. And can you believe that DeepSeek did both of these and shared them for free?
Absolutely mind-blowing. But, of course, the kernel engineers at DeepSeek would not stop there because once attention becomes cheaper, the bottleneck
does not disappear. It moves to the next place, which is the MoE path that comes right after attention in each transformer block. In theory, we all
love the idea of sparse MoE because it just sounds so efficient. Like, you do not need to activate
every parameter and you only need to route each token to a small number of experts. But, on the implementation side, sparse MoE is actually kind of
like a nightmare. So, the main bottleneck is not the math that the expert performs, but the movement of the data to the expert. For a bit of context,
the MoE path is that a token has to be dispatched to a GPU hosting that expert, go through the first
expert matrix multiplication, goes through the activation and the second expert matrix multiplication, then have its output sent back and combined
with the rest of the model stream. And if you do these steps one after another, the hardware will actually spend too much time waiting. On top of
that, dispatch and combine are communication-heavy while linear one and linear two are compute-heavy. So,
the network waits for the compute, but the compute waits for the communication, and the whole point of sparse MoE starts getting eaten by the cost of
moving tokens around. And this is also where related systems like Comet, published in March 2025, comes in. So, as token data arrives, linear one can
start on the parts that are ready. And as linear two produces output chunks, Combine can start
sending those chunks back, which is basically operation pair overlapping where dispatch goes with linear one and linear two goes with Combine since
it's pairing up a communication and a compute method, making the MoE path less wasteful. But, on top of this, DeepSeek made the overlap more
fine-grained. So, instead of limiting it to operation pairs, they split experts into smaller waves. For
instance, if there are four waves and 256 experts, once wave one finishes receiving tokens, it starts the full expert compute path while wave two is
still receiving tokens. Then, when wave one is done, its results can be sent back while wave two is computing and waves three is receiving tokens. So,
Comet's method overlaps neighboring stages and DeepSeek pipelines the whole MoE route across smaller
expert groups. And this is what becomes Mega MoE. It's the wave plus Comet technique that is fused together into one big GPU program, aka a mega
kernel, which avoids the overhead of running each of the kernels separately to save even more compute. So, the GPU doesn't have to keep bouncing
between separate operators and the communications and computations can be scheduled more precisely inside the
pipeline. This implementation gives a 1.5 to 1.73 times speed up on general inference workloads and up to nearly two times speed up for
latency-sensitive workloads like RL rollouts and high-speed agent serving. They also validate this fine-grained expert parallel scheme on both Nvidia
GPUs and Huawei Ascend MPUs. However, they only open-source the CUDA version under their Deep Gem library and not
the version for running on Ascend MPUs. And this is also where the Huawei part gets interesting. DeepSeek clearly made the extra effort for this MoE
pipeline to work on Ascend. But, the public implementation they released is CUDA only, which leaves a big question mark. Because wouldn't all other
Chinese labs benefit from switching off CUDA? Because as you can imagine, Nvidia GPUs are pretty scarce
in China. So, if the CUDA version is open, but the Ascend version, which is called CAN or C A N N, is not, then one speculation is that maybe the
Ascend implementation depends on specific pieces that are not clean enough to release. But regardless of the reason, the important part is that they
are closely working with Huawei's Ascend stack, especially when Deep SEEV 4 had a D0 inference support on
Huawei chips. Which just shows how they are less dependent on Nvidia chips now, and gives them a path to keep scaling even without using Nvidia chips.
It's even more obvious that they are calling for the switch as they basically have a section that almost reads like an open letter to hardware
vendors, where they lay out the bottlenecks they are seeing and what future hardware should optimize for.
And the first point is actually not about bandwidth. They say that full communication computation overlap depends on the ratio between compute and
communication, not bandwidth alone. For Deep SEEV 4 Pro, each token expert pair needs to compute for the activation key, linear one, linear two, but
the communication is only FPA dispatch and BF16 combined. So, they calculate that each GB per second of
interconnect bandwidth can hide around 6.1 TFLOPS per second of compute. Meaning, once the hardware reaches that balance point, adding even more
bandwidth gives diminishing returns. Which is a pretty neat design suggestion for chips that's designed for inference. Then, the second point is
power. Because once you fuse everything into one mega kernel, the chip is no longer stressing only one part of
the system at a time. As wait time decreases, compute, memory, and networking are all burning at the same time. So, power throttling becomes the next
bottleneck. Which Deep SEEV basically says future hardware needs enough power headroom for these fully concurrent workloads. Then, the third point is
communication primitives. Deep SEEV currently uses a pull-based approach where each GPU reads the
remote data it needs. So, if GPU B needs data from GPU A, GPU B directly reads it from GPU A. They use poll because fine-grained push would require
too many tiny notifications constantly telling you another GPU your data is ready. But if future hardware had lower latency cross-GPU signaling, push
would actually be a better option. Because in that setup, when GPU A has data for GPU B, GPU A can
send it directly as soon as it is produced. And if those small transfers can happen without expensive notification overhead, then push-based
communication becomes much more attractive. And the last point is actually kind of funny. They even suggest replacing SweetJU with a cheaper
element-wise activation that avoids expensive exponential or division operations. Because once the mega kernel is
disoptimized, even the type of math operations becomes part of the hardware bottleneck. I was going to say, "How is this related to if this is
cheaper, you can potentially relax the bandwidth requirement. And at this point, this is also where the paper starts to get hardcore. Yes, it gets
even crazier. Because once you start designing low-level kernels, you're definitely not going to rely on the
normal PyTorch operators. So, Deep See uses TaoLang, which is basically a language for writing custom GPU kernels. TaoLang gives them a way to write
fused kernels are much closer to handwritten CUDA performance while still being easier to develop and modify. Which means it also gives them a reason
to apply custom kernels to some other places. That is a fixing the problem of batch invariance. I
previously made a more in-depth video about it, but basically in actual inference, the same prompt will not always be served in the same batch.
Sometimes it is alone, and sometimes it is batch with 10 other [music] requests. And what happens is that the output could change just because the
surrounding batch changed, which makes debugging the LM a nightmare. This also matters for prefix caching. If
a shared prefix produces slightly different hidden states depending on what other requests were batch with it, then cached results are not as
trustworthy. It even matters for post-training, too. Because RL rollouts and on-policy distillation depend on stable generation behavior. So, if
batching changes the logits, then the same policy can produce slightly different trajectories for infrastructure
reasons, not model reasons. Then, why can't we just make a batch invariant? Well, this is thanks to one of the fastest tricks for decoding attention
called split KV. So, think of the GPU as having a fixed number of workers called SMs, short for streaming multiprocessors. For attention decoding, the
most common way to keep the GPU busy is split KV, where a single sequence KV range is split across
multiple SMs. This is useful because decoding can suffer from wave quantization. What that means is that GPU work is launched in waves across SMs. So,
if you have 132 SMs but 100 sequences ready, 32 SMs are left idle in that wave. Split KV helps by letting those unused SMs cooperate on existing
sequences instead of sitting around. But, this creates a determinism problem because floating-point
addition is not perfectly associative. If the same attention computation is split and reduced in a different order, the final output can change bit
for bit depending on that batch layout. Like I previously discussed in my older video, the batch invariant solution developed by Thinking Machines
still has around 1.6 times slowdown. But, that does not stop DeepSeek from trying to write their own
kernels to address this problem. And DeepSeek's solution is something called a dual kernel strategy. The first kernel computes the attention output
for an entire sequence within one SM, which gives a fixed accumulation path when the GPU waves are fully occupied. But, when the final wave is only
partially filled, this would waste SMs, right? So, DeepSeek adds a second kernel for that tail case
where multiple SMs cooperate on one sequence. The trick is that this multi-SM kernel is designed to follow the same accumulation order as the
single-SM kernel, so it recovers some of the utilization benefits of split KV without changing the bitwise result. They also use distributed shared
memory inside the thread block clusters to make the cross-SM communication efficient. So, the industry problem
was basically batch invariance is nice, but the optimal kernel path breaks it. And DeepSeek is like, "Fine, we'll find a way ourselves and not only
beating Thinking Machines' solution, but also has negligible overhead, which is crazy. It doesn't even sound that complicated, either, and this
knowledge is now free for everyone. Can you believe it? And going back to the KV cache problem we were
talking about earlier, remember how complicated the attention mechanisms are, right? We got a mixture of compressed memory, index memory, heavily
compressed memory, local recent memory, and pending states, which means the standard page attention style cache management no longer fits cleanly as
paged attention assumes a more uniform KV structure. So, DeepSeek splits the cache system into two major
parts. The first is the classical KV cache, which stores the CSA and HSCA compressed entries. The second is a state cache, which stores sliding window
KV and the uncompressed states that are not ready for compression yet. But, splitting them is still not enough because at 1 million tokens, it becomes
an actual cache storage and management problem. Even if the KV cache is much smaller than DeepSeek
V3.2, keeping every long shared prefix permanently in GPU memory would be very expensive to serve at scale. So, DeepSeek uses on-disk KV cache storage
for a shared prefix reuse. For CSA and HSCA, they store the compressed KV entries on disk. When a request hits a stored prefix, the system can reload
those compressed entries instead of recomputing the whole prefix from scratch. I mean, there is, of
course, still some recomputation. Like, if the prefix ends inside an incomplete compression block, those tail tokens must be recomputed because their
uncompressed KV states are not stored. But, this is what makes the insanely cheap cache hit rates that DeepSeek has so much more believable, and they
are basically not in the red serving it. What's even more insane is that in the third-party
experiments done by a Chinese blogger on Zhihu, they found that DeepSeek has a 100% KV cache hit rate consistently across peak and off-peak hours with
the experiment lasting over 12 hours, which just shows how good their implementation and optimization is for storing KV cache efficiently. If compute
can be reused, they will not be wasted. You can also see how some other labs like Open AI's Z AI
Kimi all kind of drop off to around 0% cash hit rate after an hour. The only one that was slightly competitive to DeepSeek is MiniMax. On the other
hand, beside the KV cash memory problem on the inference side, DeepSeek's training memory also had its own problem that needs to be addressed because
V4 does not just use AdamW everywhere anymore. Other than embeddings, prediction heads, RMS norm
weights, and some MHC parameters, it basically uses MeOne for most of the model. But, there is just a tiny problem. Normally, in distributed training,
every GPU may keep a copy of the model parameters, gradients, and optimizer states. But, optimizer states are huge. For AdamW, you usually store the
parameter, the gradient, the first momentum, and the second momentum. So, this technique called Zero
reduces memory by sharding these states across GPUs instead of having every one of them stored the full optimizer state. But, this doesn't work for
MeOne because MeOne needs the whole gradient matrix to compute the matrix-level update. More specifically, the technique Newton-Schulz uses matrix
multiplications, which mix information across rows and columns of the matrix to make the directions more
evenly weighted. And because of that, each shard alone does not contain enough information to compute the correct MeOne update. Well, it's more so
that if Zero slices the matrix too aggressively, MeOne can no longer do the orthogonalization step correctly. In the MoonShot paper back in February
2025, which is the first ever public scaled-up MeOne experiment, they have already proposed a technique
called distributed MeOne to address this problem, where you just shard the optimizer state. So, when MeOne needs the full matrix, you just gather the
matrix back together, run Newton-Schulz, keep the local update slice, then all gather the updated parameters again. But, DeepSeek V4 is a much uglier
training problem because now it is training a model that is 100 times larger than what MoonShot was
testing. I mean, even though they did validate it on their own Kimi Chat 2.5, which is a 1 trillion parameter model recently, Deep Seek V4 is still
sitting at 1.6 trillion with MHC, hybrid attention, has FP4 quantization aware training, and million token context window. So, Deep Seek came up with
a more architecture specific strategy with hybrid zero bucket assignment for MuON. And what they
basically do is that Deep Seek still shards the optimizer work across the cluster, but it shards between matrices, not through the middle of the
matrices that MuON needs to orthogonalize. Which means it keeps the matrices intact by assigning whole matrices to different GPUs. So, one GPU handles
the MuON update for one group of matrices. While the matrices could be in different sizes, but all they
really had to do is to balance this so the big ones don't get assigned together. For MoE, they also do something similar but expert aware. So, each
expert is treated independently and similar expert matrices are grouped together like the up projection, gate projection, and down projection. So,
it's not like a generic implementation of MuON. The MuON implementation here has to fit the exact shape
of V4. As for the post training process, remember how Deep Seek's final model is trained through specialist models first, then unified through on
policy distillation? So, you have different teacher models for math, coding, agent tasks, instruction following, and other domains. Well, surprise,
surprise, there is also an efficiency problem here because if every teacher has to produce logit over the
entire vocabulary for every token, then not only do you have to store a large amount of teacher output, but also hold the teacher models in GPU. So,
Deep Seek built this teacher scheduling around it. Instead of keeping all teacher models or all teacher output heads in GPU memory simultaneously,
they group training samples by the teacher they require. One mini batch may use the math teacher,
another may use the code teacher, and another may use the agent teacher. And while one teacher is being used, the next teacher can be loaded in the
background so the system overlaps parameter movement with computation. They also use a custom tiling kernel to compute the KL loss efficiently since
comparing teacher and student distributions over the full vocabulary would otherwise be expensive. So,
distilling multiple specialist teachers into one student also has its underlying optimizations needed. DeepSeek also came up with another clever
post-training optimization for RL rollouts when training these teachers. So, as long context rollout is not like normal short answer generation, if
the model is generating a long trajectory for a coding task, an agent task, or a million token context
task, that rollout can be expensive and fragile. So, if a worker gets preempted halfway through, then the easy solution would usually be just restart
the generation from scratch, right? But, DeepSeek points out that this is actually very wrong. First, of course, it wastes a huge amount of compute,
but second, it biases the data because short generations are more likely to finish before
interruption, while long generations are more likely to be killed and have to restart. So, your rollout data set slowly becomes biased towards shorter
completions for infrastructure reasons, not because the model actually preferred the shorter answers. To fix this, created this token level
write-ahead log. This is basically a database recovery idea applied to LLM generation. So, as the model
generates tokens, every token is immediately written to persistent storage. If the rollout gets preempted, DeepSeek can save the KV cache and later
resume from the exact point where generation stopped. This is also only achievable because DeepSeek has enabled batch invariance with their
implementation so that the write-ahead log recovery path will be very clean. Like if the rollout resumes in a
different batch layer or gets replayed on a different worker, you would still want the same tokens and KV reconstructions, right? Otherwise, recovery
itself could subtly change the trajectory. And the last thing that I want to talk about is the D Seek, short for DeepSeek Elastic Compute. Even though
it's not really about model optimization directly, it still elaborates on agent training setup that
we don't usually see DeepSeek talk about. So, for normal RL, the model can just generate text and get a reward, but for agent RL, the model needs an
environment. If it is a coding task, it needs a repo, a terminal, dependencies, unit tests, and a file system. If it is a tool use task, it needs
APIs, function calls, and tool servers. If it's a software engineering task, it may need an isolated OS
level environment that can be reset after every rollout. So, for the current RL landscape, it's not about basic coding capabilities anymore. It
becomes an extreme environment management task that the lab needs to provide to the model. And what you see there here is that they built this DSEC as
a production grade sandbox platform for post training and evaluation that supports four execution
substrates behind one Python interface with different levels such as function calls, containers, micro VMs, and full VMs. The reason they need four
levels is that not every task has the same cost or security requirements. So, DSEC lets the training system choose the right environment without
changing the agent interface. And underneath, it decides whether that environment should be a function
call, a Docker compatible container, a Firecracker micro VM, or a QEMU full VM. On top of that, it has to handle creating the environments quickly,
resetting them after each RL, running commands, handling failures, coordinating with GPU preemption, and scaling this to hundreds of thousands of
concurrent sandbox instances. So, agentic training is just not feeding the model's codes or whatsoever. It
is also building a scalable environment for the model to interact with successfully. Not to mention, a lot of agentic problems are simply models not
being able to read the error logs of its actions. But anyways, I think we covered most of the technical sides of this paper, and you could definitely
say that DSEC V4 is a bit over-engineered, which they also kind of admitted in the conclusion. But I
guess when you have such a limited amount of compute, you wouldn't really want to waste them, right? But I still think this is a bit overwhelming
because it's the first time that an AI lab has shared such detailed optimization processes across their state-of-the-art stack, especially all at
once, including training and inferencing. But even so, they still commented that for future iteration, there
are so many places to optimize. That includes simplifying the architecture itself. However, the bigger directions they mentioned they would focus on
even more in the future are agentic intelligence and multimodality. For agents, V4 already has some pretty strong base with long context, a cheaper
test time scaling, tool use post training, DeepSeek environments, and better code agent infrastructure.
So, they do have a pretty solid ground to further build the agentic capabilities. As for multimodality, DeepSeek is probably very behind in this
aspect, much more than 6 months behind maybe, but V4, they claim, is definitely closing in on that. But, there is a recent paper called Thinking with
Visual Primitives that was deleted a few hours after DeepSeek posted online. So, maybe in the next video,
I'll cover how they are tackling into multimodal, especially visual reasoning. So, yeah, that's it for this video. If you want to learn more about how
LLMs work like DeepSeek and MoE, on policy distillation, or GRPO that DeepSeek pioneered, you should definitely check out my latest project
intuitiveai.academy, where it contains an intuitive explanation of all modern LLMs from the ground up with
minimal math, including the topics I just mentioned, ranging from LLM architectures, MoE, LoRA, to our latest chapters, reinforcement learning, where
we cover how RL works and how it interacts with LLMs, accompanied with our latest interactive visualizations to help you better understand its logic.
So, for those who want to get into AI or LLMs, this should be the perfect place for you to dive into
the technical parts without being intimidated by crazy-looking maths. And right now, we are offering a summer discount, so use the code summer for 25%
off a yearly plan. I thank you guys for watching. A big shout out to Spam Madge, Chris Ladue, Deegan, Robert Zaviassa, Marcelo Ferraria, Poof and
Enew, DX Research Group, Alex, Midwest Maker, and many others that support me through Patreon or
YouTube. Follow me on Twitter if you haven't, and I'll see y'all in the next one.