A four-bit model that's faster and higher scoring than an eight-bit model? That shouldn't be possible, but that's what I found. We've been using
integer-based four-bit quantizations for years. They're great, but last year Nvidia introduced a different kind of four-bit, NVFP4, a floating-point
format, which I've covered on the channel before. So, Julia, what about FP4? >> If we take very large
numbers and we compress them into four bits, we can keep enough precision so that LLMs work well enough. >> So, the question is, which four-bit is
better? Integer or Nvidia's floating-point? And to test this, I reached for the mother of all open weights LLMs, Kimiko 2.5. Here's DHH on Twitter,
Kimiko 2.5 continues to be my daily driver for all the basic stuff where I don't need PhD-level
intelligence. I just need it done quickly. David, how's the quality? Extremely good, one of its main advantages. Here's Cursor AI founder Aman Sanger
explaining that their new Composer 2 model is built on Kimiko 2.5 from Moonshot AI as the top-performing base model after they tested many options on
Perplexity Evals. Kimiko 2.5 proved to be the strongest. Now, I already tested Kimiko 2.5 on my Mac
Studio cluster on the channel before, but Apple silicon runs MLX, and MLX doesn't do NVFP4. For that, we need Nvidia Blackwell GPUs, and really big
ones. So, my friends at Cirrascale, not sponsored, they let me use their HGX server with eight B200 GPUs, Nvidia SMI. And each one is 1,000 W with 184
GB of VRAM. That should do it. And the nice thing is, this machine can run both our integer quants
and NVFP4. So, I can do a true apples-to-apples comparison. Same hardware, same model, same prompts, same everything except the quantization format.
Both are four-bit, both run the same Kimiko 2.5 model. The only difference is how those four bits are stored. One uses floating point, the other uses
integers. And by the way, Moonshots official Kimiko 2.5 repo on hugging face is int4, coming in at a
hefty 595 GB in size. For the NVFP4, I'm using Cerebras Kimiko 2.5 NVFP4 quantization, which is 591 GB. We'll start with speed. Now of course on this
setup, I'm going to be using vLLM, professional high-throughput level memory-efficient inference server engine. Okay? I've shown it many times on the
channel before and it's fast. So that's going to be the server side running on the HGX with the
eight GPUs. And on the client side, I'm going to be using llama.cpp. This is a great throughput benchmarking tool. And of course, I'm going to use the
same settings for both machines. For those of you that are curious, this is my vLLM command. Moonshot AI Kimiko 2.5, tensor parallel size eight cuz we
have eight GPUs. It's going to spread the model across all of them. And I also like to enable tool
calling. This way the model can be used remotely through some kind of coding agent. And I've shown that on the channel before. Here's my client side
command. I actually tested concurrency one, concurrency four, and 32. That's how many requests are being thrown at it at the same time. And we'll
review that in a moment. At concurrency one, c = 1, that's a single user talking to the model, kind of
like chat. NVFP4 generates 133 tokens per second. Int4 does 105. So there's a 27% advantage for NVFP4. And I push it all the way up to 32 concurrent
requests. That's like either 32 users at the same time or an agent with 32 sub-agents. Both models scale impressively, about 11 times total
throughput. But NVFP4 keeps its lead the entire way. It peaks at about 2200 tokens per second, whereas int4
peaks at about 1800 tokens per second. Now if the per request speed drops as you add users, and that's expected. They're sharing the GPU. But NVFP4
stays about 20% ahead at every concurrency level. So the advantage doesn't shrink under higher load. Same for bits, but NVFP4 is consistently 20 to
27% faster. Why? Well, here's why. NVFP4 quantizes both the weights and the activations to four bits,
whereas INT4 only quantizes the weights. It keeps activations in full 16-bit. You can see the total data moving through the memory here. 8-bits versus
20-bits total. INT4 is pushing 2.5 times more data per operation. On modern GPUs where memory bandwidth is the bottleneck, that's the difference. So,
NVFP4 is faster, but speeds mean nothing if the output is worse. Does quantizing activations too
aggressively hurt quality? For quality, I ran four standard benchmarks. GSM8K, 200 grade school math problems. HumanEval, 164 coding problems where I
actually execute the code and check if it passes. Well, I don't, but my agent does. You crazy? I'm not doing all that work. And this one, instruction
following evaluation for LLMs, IFEval. 541 instruction following prompts that test whether the model
can follow precise formatting rules. And of course, my favorite, needle in a haystack. I don't know why that's my favorite. Seems like it would hurt
actually if you just jump into that haystack. That's where I hide a specific fact inside a long document and see if the model can find it. Ah, here's
what I got. Math, basically tied. Retrieval, both perfect. But, look at HumanEval and IFEval. INT4 is
actually ahead on both of them. 59% versus 54 on code, 68 versus 65 on instructions. At this point, I almost wrapped up the video. The story was
clean. NVFP4 is faster, INT4 is slightly better quality, right? Well, something wasn't quite right. Something bugged me about this. 54% on HumanEval
for one of the best coding models in the world? That's way too low. So, I dug into some raw data. 62 out
of 164 problems were truncated. The model never finished writing the code. Not because it couldn't solve the problem, because it ran out of tokens
thinking about it. See, Chimera 2.5 is a reasoning model. Before it writes any code, it thinks internally in hidden tokens that count against your
token budget. When max tokens is set to 2048, the model would spend 1,800 tokens reasoning and have almost
nothing left for actually writing the code. Sometimes it produced nothing at all, like an empty response. So, I re-ran the benchmark with max tokens
at 8,192, giving the model enough room to think and write code. And look at the difference. NVFP4 jumped from 54% to 88%. INT4 jumped from 59% to 92%.
Those are real numbers. The model was always this good. I just wasn't giving it enough room. And
same thing happened with IFEval. NVFP4 had 185 truncated responses, 135 of them completely empty. When I re-ran with enough tokens, both models jumped
up to 80% and NVFP4 actually pulled slightly ahead. Looks like the INT4 quality advantage was a measurement artifact. NVFP4 reasoning is more robust.
It thinks longer before answering, so it got truncated more often. And once I fixed that, they were
essentially tied. And this is the trap that anyone benchmarking reasoning models can fall into. Standard token limits silently break your results.
What ends up happening is the model that thinks harder looks worse, which is exactly backwards. So, here's where NVFP4 and INT4 actually stand with
the corrected numbers. Quality, tied. Math, tied. Instructions, tied. Retrieval, tied. INT4 has a small
4% edge on code. That's it. Speed? Not tied. NVFP4 is 27% faster, peaking at over 2,200 tokens per second when we're doing the concurrency of 32. If I
had to pick one, NVFP4. Same quality, meaningfully faster. But, I wasn't done. I wanted to test one more thing. FP8 is an 8-bit floating point, twice
the precision of NVFP4. In theory, it should be slower, but more accurate. I downloaded an FP8
version of Chimera 2 base and ran the full suite of tests. Now, I did mess up a little bit because here's a little caveat. This FP8 model is actually
the text-based only version of Chimera 2. It's not a reasoning model. It doesn't have that internal thinking step, so keep that in mind. However, it's
still interesting because the speed tests are still valid. If there was a Chimera 2.5 in FP8, then
this would be close to the result. And FP8 is the slowest, 96 tokens per second versus NVFP4's 133 tokens per second. That's 28% slower. More bits
means more data moving through the memory bandwidth, and that's the bottleneck. But, does the extra precision that we have with FP8 buy you more
quality? And the quality story is not what I expected. Here's human eval. FP8 drops to 76%. The reasoning
models are at 88 and 92%. Thinking before coding adds 12 to 17 percentage points. Who would have thought? This is where I realized, "Oh, wait a
minute. FP8 is not a reasoning model. That's why." So, I reorganized my chart labels a little bit to reflect that. But, the interesting thing this
shows is that on complex tasks, reasoning is a massive advantage. However, look at math. FP8 scores 97.5%
here, higher than both reasoning models. It answers directly without overthinking. Sometimes reasoning actually gets in the way, like with simple math
problems. That's why the motto of this channel is probably going to be uh use the right tool for the right job. Now, instruction following, FP8 is a
64% and Chimera 2.5, the reasoning models are at 80%. Precise constraint following benefits
enormously from step-by-step thinking. So, the FP8 results reveal something deeper than quantization. The biggest quality differentiator isn't the
number of bits, it's whether the model can reason or not. Here's the chart that says it all. X-axis is speed and Y-axis is average quality. You want
to be in the top right. NVFP4 is the closest. IN4 is a bit more accurate on code, but noticeably slower.
FP8 is bottom left, slower and less accurate overall despite having twice the precision. Now, you might be wondering how you can actually run Kimiko 2
yourself. Fair warning, it's a huge model. You're going to need serious hardware, like a stack of a Mac Studios, or a B200 cluster, or something along
those lines. Or, if you don't want to deal with the hardware, you can just use the official Kimiko
Cloud provider and hit the API or chat. Hi, I'm trying to stump it right now. Hello, how can I help you today? Didn't stump it. Thanks for watching.
You probably will like this video next, and I'll see you next time.