If you run a local LLM, most likely you're running a quantized model. Ollama, LM Studio, every tutorial, they all switch you to 4-bit on the way to
the demo and never really tell you what you give up. If you quantize too aggressively, the model just breaks, obvious. Quantization is basically
getting rid of data so it's the model is smaller. You catch it right away, but there's a level just before
that where the model still looks normal. It talks like normal, the format still looks right, but it's lying to you the whole time. And nobody shows
you where that line actually is. So, I took Qwen 3 32B. I know it's older, but for this purpose it doesn't matter. It's a 32 billion parameter model
and I quantized it eight different ways. Full BF16, which is unquantized, down to barely 1-bit per
weight. So, same parent model, which is this one, same calibration, same llama.cpp build, only the bit width changes. All of this runs on this Linux
box I built around the RTX 6000 Blackwell. That's 96 gigs of VRAM. The whole BF16 fits with room to spare, but it really doesn't matter what hardware
you're on. The quantization principles are going to be the same. It's just a lot faster for me to do
all these tests on this card cuz it's a fast card. I ran five tests at every single level. Perplexity, MMLU, ARC Challenge, GSM8K, and Code Needle.
I'll explain all these as we go. Now, for the actual fact-checking, I picked a fact-based test because creative writing is just too subjective, but
facts are facts. Just watch. As devs, our info ends up everywhere. Repos, bug trackers, random API
sign-ups. That turns into a profile data brokers can package and resell. The harder you are to find, the harder you are to target. In dozens of
countries, data brokers are legally required to erase your info if you ask. You can do this yourself, but that means chasing broker after broker,
checking back later to make sure they didn't relist you. Incogni sends those removal requests automatically
and keeps following up until they comply. In my case, it found over 231 databases with my details and cleared most of them. And when I find a specific
page exposing my info, I use custom removals. I submit the link and their team handles the takedown and follow-ups. Track it down, get it removed, and
keep it from coming back. For extra peace of mind, Deloitte verified their data removal processes.
There's also a 30-day money-back guarantee if you want to try it risk-free. Take your personal data back with Incogni. Use my link and code Alex
Ziskind for 60% off an annual plan. Link down below. >> [music] >> List six men who have walked on the moon on Apollo 12, 14, 15, 16, and 17. Skip
Apollo 11 because most people already know who those people are. For each person, give full name, Apollo
mission number, the year of the mission, and one notable fact about what they did on the lunar surface. Format this as a numbered list. Do not invent
facts. If you're not sure about the detail, then say so. All right, and there it goes. By the way, this is just simulated. I did actually generate
these on here, but for the sake of the demonstration, I'm just animating these on screen. This is full
precision, 62 GB on disk, not a small model. And every quantization from here on is graded against this baseline. So, I hope this one is right. Now,
I'm not a lunar expert, so I'm going to have to double-check each and every one of these. Looks like all these facts are correct except for John
Young. He was the only person to walk on the moon twice. So, even our base model is not right. There was
nobody that walked on the moon twice. Not a good start, but mostly it's right. I wonder what's going to happen from here on. I run the benchmarks.
First one is perplexity. You might have heard about this. You give a model a chunk of held-out text, and then you ask it how surprised it is by each
next word. Lower means less surprise, which means more fluent. It's one of the oldest metrics to measure
performance uh quality performance of an LLM. It's simple, it's still around, and for BF16, we get perplexity of 7.92. The other four numbers are on
the screen, too. Those are the baselines, but they don't really tell you much yet. We need a few comparisons before they start to mean something. The
dotted line in every subplot is BF16. Now, we cut the bits in half. So, we're quantizing down, we're
getting rid of some data. Watch what happens. 8 bits per weight, half the size, 33 GB. Here, we see the wording shift a little bit, but we still have
Bean and Duke. If you put this side by side with BF16, you could not pick out which one is which. Even the mistake of walking on the moon twice for
Young is carried over. What the heck? All right, let's take a look at the benchmarks. Perplexity,
about the same. MMLU, about the same, but a little lower. Here's Arc Challenge. By the way, Arc Challenge is grade school science questions, but the
hard split. It's designed so you can't just look up the answer. Multiple choice, four options. Random chance is 25%. Anything above that is model
actually reasoning, not guessing. And [music] we're at 52.17 now. Pretty good. And every other number is
within rounding, too. Let's take a look at the Code Needle output. By the way, Code Needle is a benchmark specific for coders that I created and
open-sourced, and I made a video about it a couple weeks ago. You can check it out and run it yourself on your own code. I'll link to that video down
below. Here's the first function, URL collapse path. Pass. 20 out of 20 matched. Every line rendered in
gray. No yellow. Well, there's a blue one here, which means it uh it did better than it's a bonus. It did better than expected. Basically, the model
is asked to reproduce a real Python function from its context, line for line. 20 out of 20 lines. Every gray line means that it actually was able to
reproduce a line exactly verbatim. Of course, this is a difficult test and it's not easy to pass, but
we did get a few passes and a couple of misses here. Four out of 11, honestly, it's not amazing. But, if we check BF16, it's also four out of 11. It's
the same model. We're getting pretty much the same grade here. Also, Code Needle is a work in progress. Just wanted to show you a coding example of
how that looks. So, basically, this quant is half the disk size, no quality loss. Essentially, we
could stop here, but let's keep going. >> [music] >> We're down to 26 GB on disk. Still right, for the most part. Mitchell's notable fact is different
this time, but those things are all true about Mitchell. This is actually better cuz it didn't make up that Young walked twice. Instead, it picked a
different fact about Young. Young got lucky this time, but maybe he won't get so lucky next time.
Folks, only walk on the moon if you're a trained professional, okay? For perplexity, we got 7.91 here. Still basically the same. And the other four
are all within rounding errors. MMLU is a little bit lower, 43.8 versus 44.2. By the way, MMLU is 57 subjects of multiple choice. There's history,
biology, law, philosophy, the general knowledge test every model leaderboard quotes. I ran 1,000
questions per level. Chart basically still hasn't moved much. One more step down, shall we? Q5 KM, 22 GB. Well, we still have six astronauts here, but
we have a couple of new faces that showed up. Alan Shepard Jr. and Charles Conrad Jr. Seriously? There's a couple juniors there. We're missing Duke
and Young. And also, there's no more Apollo 16 here, just Apollo 12 through 15. The good thing is all
six of these are actually real moonwalkers. But, if you look closer at the first one, Charles "Chuck" Conrad Jr., his actual nickname was Pete, not
Chuck. Pete Conrad, okay? It's a small lie, but if I was Charles Pete Conrad, I'd be offended right now. Perplexity, still about the same, 7.94. Other
four are still within rounding, Code needle moved up a little bit, so it did a little bit better. And
GSM 8K moved down just a tad bit from 96.5 to 95. This is math word problems, so grade school arithmetic, but the model has to read the problem and
reason about it. It's not pattern matching, actual chain of thought. I ran 200 problems per level with the model's thinking mode turned off, so we're
just measuring the final result. Now, the level that most of you are probably actually running,
especially if you just installed Ollama or LM Studio, watch what changes. Q4KM, 19 GB. Basically, the level the entire local LLM scene runs on. Now,
Q5, they made up a nickname, and here, what are we making up here? Says Bean was the first to play a guitar on the moon using a specially modified
instrument. I wonder what that would look like. Were the strings more tight or something? But, all that
aside, this is wrong. Bean played guitar on Earth, not on the moon. There was no guitar on the moon. >> [laughter] >> Watch one more. It says Edgar
Mitchell, Mitchell planted the first US flag on the moon to wave in the solar wind, though it was later found to be mostly limp. First of all, don't
talk about Mitchell like that. He's old, okay? If he's still alive. Ah, shouldn't have made that joke.
So, this is also wrong because this is the Apollo 14 mission, but it was Apollo 11 when the flag was planted, and it was not by Mitchell. The format
looks right, we got some numbers, we got the right names. There's more facts that are wrong this time. We're down to Q4. We've crunched this thing
quite a lot now. Q4 means we got rid of essentially 75% of the data. Perplexity says everything is fine.
We're at 7.96. MMLU went down a little bit to 43.3. Arc is about the same as where we started. Same with GSM 8K and code needle is doing better. Also,
you can see how different tests react differently to the quantizations. Some are going a little bit up, some are going a little bit down. But, there's
nothing in the aggregate numbers that warns you about guitars on the moon. And here's Q4KM. We got
five out of 11 passing, so it's just one extra one. Because this is a small sample size, it does mean a higher jump in the chart. But, that's the
overall score. It passed because it got 18 out of 20. However, this time it hallucinated one line. Benchmarks are fine. The model is lying about
Mitchell. All right. Q4KM seems to be right now the best balance of size and quality, but be careful because
it will confidently tell you that Mitchell brought the first American flag to the moon. Let's see what Q3 invents. Three bits per weight and 15 GB. We
got six. Alan Bean, though. That Alan Bean, he famously brought a golf club to the moon during Apollo 14, not Apollo 12. But, on Apollo 12 he
conducted geological surveys and collected samples. There's two lies in one sentence there. Bean wasn't on
Apollo 14, and the golf balls were Shepard. So, the model just swapped two astronauts. Then, Conrad. Hey, got Pete this time. He famously said,
"Whoopee! Man, sure that smells like gasoline." when he sniffed the lunar module after landing. Conrad did say "Whoopee!" on the moon, but the rest is
completely fabricated. His real first words were, "That may have been one small one for Neil, but that's
a long one for me." So, the model kept the opening word, "Whoopee!" and made up the rest. They had that may have been a small one for Neil, but that's
a long win for Ruby. Then, let's take a look at Mitchell. Huh, bad Mitchell. >> [laughter] >> He famously brought a small amount of cannabis to the
moon. Where does he get this stuff? >> [laughter] >> Obviously, that is not correct. It's not mine, I
swear. But, here's the bigger story. Perplexity is starting to take a little dip here. 8.2 MMLU is starting to dip a little more, 42.5. Code needle is
still [snorts] doing pretty well, climbs to 71. Arc challenge, ooh. Arc at Q3 is 21.4%. Woo. Random chance is 25. Q3 is scoring below random. The run
completed all the questions. The model is systematically picking wrong answers on hard science
questions. So, mostly four of these are okay, but Arc challenge just took a nose dive, fell off a cliff. Whatever broke at Q3 broke for one specific
kind of test. Same model is normal on math, code, knowledge, just not on science reasoning. I don't know. I don't know. So, quantization can break one
capability and leave everything else fine. Not a slope, more like a pothole at one specific level.
All right, can't wait to see this next one. >> [music] >> Two bits per weight, 11 gigabytes. We're saving a ton of space here, folks. It's half the
bits of Q4. Does it pothole again or finally fall off the cliff? The second American to walk in the moon and the first to bring a camera to document
the lunar surface. Wrong. He was the fourth person to walk in the moon. Mitchell conducted the first
lunar golf shot by hitting a golf ball in the moon. Wrong. That was Shepard. John Young is back. He did not walk in the moon. First of all, I asked
you for people that walked in the moon. Why are you even giving me somebody that didn't walk in the moon? That's wrong. He was the command module
pilot for Apollo 16. Young walked on the moon. And he was the commander of Apollo 16, not the module
pilot. And what year did it all this happen? Apparently, this year. The model just drops into Chinese. The year was '72, by the way. Translate.
Attempted to reach the lunar surface, but did not land. That's not even 1972. That's like a whole series of facts. The model wrote itself a
justification in another language for a fact that it just invented. By the way, sometimes you might see Chinese
characters in the output. So, what's up with that? Well, Qwen 3 was trained on both English and Chinese. So, Chinese tokens are sitting in the same
vocabulary as English ones. At 2 bits per weight, the quantization noise occasionally tips the next token pick from an English word the model meant to
a Chinese one with a near equal score. This is a different failure mode than Q4 or Q3. The facts are
wrong, the format is starting to crack here, and the model is mixing languages it has no business mixing. Perplexity creeps to nine now. MMLU dips
down to 40.7, but ARC bounced back from its Q3 pothole over here. And Code Needle just fell off its own cliff. That makes me feel a little bit better
about Code Needle. Different metrics, different shapes, same model. The surprising thing here is the
model is still mostly producing coherent text at 2 bits per weight. 5 years ago, this would have been gibberish. The IQ quant calibration is doing
some real work here. Okay, one more cut, 1 bit per weight.
Huh? What? I cannot provide fabricated information. I cannot confirm the Apollo 12, 14, 15, 16, and 17 astronauts' names, missions, and notable facts.
I cannot generate facts. Please provide the correct information for me to assist you. The model refuses. It refuses. That isn't a safety refusal. The
model has been quantized into something that no longer even knows the answer. The refusal is what
comes out when the answer is gone. Whoa. >> [laughter] >> Okay, let's start with code needle. This is meant to test positional recall under load,
which is a thing code completion actually need. Yeah. Well, it got one function is CGI. Match 20 out of 20 there and hallucinated only one line. This
is the same task we did at Q8 and Q4. This is a fail this time. At Q8 every line came back gray, which
is good. Three or four lines in, it locks. And look at this one, 666 hallucinated lines. That's kind of creepy. Just keeps repeating the same thing
over and over and over and over and over and over and over again until it hits the token budget and then is done. This is one of those looping things
that happens with models that are quantized too much. Every plot drops a level. Perplexity nearly
doubles. Knowledge, science, math, code, all of them. A one-bit weight is where this model genuinely stops being the same model. JPEG, the image
compression algorithm, it works because it does not shrink every pixel equally. It keeps the detail in faces and edges and it rounds the flat sky. The
new IQ quants, they kind of do the same thing for model weights. You run real text through the model,
find out which weights are the faces and protect those. The flat sky weights get rounded even more though. And before we head IQ formats, the old
format Q2K, for example, it rounds everything. It rounds the faces and the sky all the same. So, it's like a big mishmash of stuff. Watch what it
does. Apollo Apollo mission. Apollo Apollo Apollo Apollo. It locks into that word and writes it over and
over and over and loop hundreds of times in a row until it exceeds the token budget. What about these facts? Alan Shepard did button titles, not walk
on the moon. That is a code token bleeding into the middle of a sentence. Edgar D Ed D, the name is all garbled up there. You get the idea. Same
model, the only difference is whether the quantization knew which weights to protect. Now, Matouszek and
Onsloth apply different techniques beyond plain quantization that improve on these. My quants are just deliberately basic, so we can see what bit
width alone does. The principle still hold. Bottom line for everyone running a local model, Q4KM is the sweet spot. But even at Q4, the model still
lies. Just check the outputs. Now, that 96 GB RTX Pro 6000 I built this whole rig around, is it actually
worth it for local LLMs? I put it head-to-head against the M3 Ultra Mac Studio. Not that one, that one. And the 5090. Watch this video right over here
for that. Thanks for watching. [music] Hope you enjoyed this one, and I'll see you next time.