And that's 78 tokens per second. Solid for a model this size on this hardware. See, that's usually what I do on this channel. Speed between different
kinds of hardware. But what about quality of code output? That's not an easy problem. It's not like saying hi to a model or asking it to write a
story. I've avoided talking about code quality for a while because there's so many different kinds of
devs out there, so many different code bases, different LLMs that creating an objective set of tests that suits everyone is pretty much impossible.
Well, in this video, I'm sharing a benchmark with you. It's new, and you can run it on your own codebase against your own model picks. I'm going to
show you how to use it and walk you through some examples. So, when you give your model your file, is it
actually reading it? And does it know when to stop? There's a leaderboard later in this video that's going to tell you to pick the wrong model, and
I'm going to show you why. Now, I'm open sourcing this tool, but I didn't come up with it. I kind of stole it. Or more accurately, I was heavily
inspired by this channel to Protoricis. What? 6,000 subscribers. Everybody should go and subscribe to him
right now. I caught a few recent videos of his where he was running the latest Quinn and Gemma models against his own codebase using a really clever
tool. The only downside is that it's not open- source and he hasn't shared it yet. So, I had the idea to reverse engineer that concept and make it
generic enough to work on any codebase with configurable file selection options with glob patterns, add
charting on top of it, and put it on GitHub for free. Oh, and by the way, if you go to my channel's homepage, I've added a little section down here
where I highlight small YouTube channels I think are awesome and genuinely valuable. So, go show them some love. Now, here's what the benchmark
actually does. It drops an entire source file into the model's context. Then it asked the model to verbatim
reproduce the first 20 lines of one specific function in that file. Like this one for example at line 10,141. Why verbatim? Well, if the model
genuinely is reading the file, the answer is in there. If it's hallucinating or guessing from the training data, the lines won't match up. And it does
this for a sample of functions picked from varying depths in the file. So line 200, line 2,000, line
8,000, and so on. We're basically testing recall at depth, not just at the top. The exact number is configurable. So, for example, I have a small
Python file here. This one has 11 functions. I set K to 16, so it'll do 11. JQuery, K set to 16, but jQuery has a lot more functions than that. And
here's the part most benchmarks miss. Two metrics, not one. Matched will be the expected lines the model
produced in order. And hallucinated is the extra lines the model invented that aren't in the file. So, a model that scores 20 out of 20 with 65
hallucinations is not the same as one that scores 13 out of 20 with only one hallucination. The grading is deterministic, plain line level diff. I'm
not sending this to an LLM to be the judge. Both columns are computed and not estimated. And yeah, I spent
a couple days with LM Studio running this benchmark, which by the way, I called Needle. Huh? Code Needle. It's clever, right? All right. Maybe not.
Maybe not so clever. Downloaded a bunch of these models. some of the newer ones like Gemma 4, Quen 3.6, Quen 3.5, dense ande models. And I started
noticing some weirdness. For example, Gemma was failing miserably and also models that were reasoning,
they're burning the whole output budget on thinking tokens. Of course, that's usually the problem that we run into with benchmarks of thinking model.
Also, did you know that in LM Studio, this model right here, Quen 34B, you can load up to a maximum of contact lens of 32,000. But if you use the LM
Studio CLI and you specify the context length, you can actually specify more than that. So LM
Studio's UI is actually kind of lying about that being available. It's outlining it in red, but it's not going to run at all. So I needed to turn off
thinking and I eventually provided that by using models config part of the tool. So you can configure the different parts of your model, the base URL,
the name, the API key, temperature, max tokens, and finally this one, suppress thinking. So all
this was fine, but there's one debugging moment that really mattered and stood out here because it changed how I understood what this benchmarking is
really measuring. And Protoichrikus, Protoricus, I don't know how to say that. Sorry, Protoricus. He actually ran into similar issues on his latest
test of Gemma 4 because Gemma 4 behaves a little bit differently. A lot of these AI agent tools sound
great right up until you have to actually deploy them. If you like the idea of autonomous AI doing useful tasks in the background, GOI is built for
that. A lot of people want AI automation, but not the extra server maintenance and patching that comes with it. GOI feels like the version for people
who want the outcome, not the side project to maintain. And it gives you a way to spin up autonomous
AI colleagues that keep working in the background for you. The setup is simple. Go to goi.ai AI, create a colleague, give it a task, and let it run. I
tried it myself and got going in just a couple of minutes. I set up an AI colleague to grab the top three hacker news posts, then summarize each one
and send it to me every morning. Mine doesn't just send me text summaries, either. It also sends me
a voice readout. Kind of feels like a little personalized podcast. >> Hey, Alexander, here's your perky hacker news briefing for today. >> All that
from one prompt. The biggest thing here is that GOI makes autonomous AI feel usable instead of complicated. And there's real traction here, too. It's
open- source, enterprise proven, and already being used by some well-known teams. Check it out to see
if it fits your workflow. Visit GOIA using the link in the description. Pick a use case and deploy your first colleague in just a few minutes. Now,
first, I wanted to answer the age-old question of how different is the BF16 quantization versus like an MLX int 4 quantization. Is that going to give
a significantly different quality of results? So, I got both of those models, BF16 and the 4bit of
Gemma 431B. This is the dense model. And by the way, I'm currently running this on the 128 GB unified memory M5 Max MacBook Pro. So, you'd think that
58 GB would be enough, but no, it's not. When you bump up the context, you load the model, I'm at 91 GB already out of 128. But guess what happens
when you throw jQuery at it, which is 10,000 line file. jQuery. There it is. Processing the prompt. And
notice how long that takes. Uh yeah, we're at 9% still of processing just the prompt. And check out the memory. Yeah, we're at uh 100 gigs now
already. This is like real world stuff right here, not the speed benchmarks that we usually do. Speed benchmarks are important. They have their own
place, but this right here is you throwing an actual code base at a model and seeing how it works. This is
what Frontier models are really good at. Fast PROM processing. This is what big chunky GPUs are good at, too. 44% 106 GB and growing. So, you can
imagine this doesn't go on for that much longer. And what did my data show? Well, 4bit was somehow better than full precision. And that can't be
right, right? No, that's not what it was. So space wasn't the only difference. Neither was the quality. Gemma
being a little bit different. It strips class method indentation when it emits code. In other words, it knows the answer. It just renders it with zero
indent. So the two quantizations were rendering whites space slightly differently. And I needed to account for that in the benchmark. So this
benchmark wasn't only measuring recall, it was also exposing how each model renders code, a behavior that
kind of nobody talks about or documents. So, how do you set this up for yourself? Well, you go to this repository. You give it a star. You don't have
to give it a star. I'm just kidding. Clone or download the repo. It's free. It's out there for you to use. Enjoy it. Go into it. Code needle. Create a
virtual environment. Activate it. I mean, all this stuff is in the readme, which I included for
you, and then install the prerequisites. Now, you can run it. So, open this up in your editor. Then, make sure you go into configs, put some files in
there. You can use the ones that are already included, which is HTTP server or jQuery. Point it to where your files are. Like I said, use your own
files. It's more fun that way. And then in the models folder, I already have some preconfigured. If you
have these on your system, great. If not, then download them. Uh this right here is the base URL for LM Studio with that port. So, it's already
configured for LM Studio. If you have that, great. If you have another tool like Olama, you can just replace that URL, point it to the name of the
model, you know, do the thing. If you want a more detailed guide, let me know in the comments. But I think
the readme is pretty detailed enough already. Or just point your agent at the readme and he'll do it for you. Now, I'm going to start things pretty
easily here. I'm going to use HTTP server. This right here on the command line is the config that I'm specifying right here. And this is pointing at
the code file that I want to run against my model. that I want to test. That's available right here in
the fixtures folder. So, this is the code that I want to send to the LLM. It's not that long. It's only about oh,300 lines of code. And then on the
command line, I want to select the model here. I'm going to do something real simple. Quen 34B 257. This is different than the other Quen 34B because
the 257 version has 256 uh context length, 256,000 tokens. So, we're good. We need to reload this
model in LM Studio with the proper context length. 200,000 is fine. I'm going to reload that and let's rerun that. Boom. I tested it with this one
because it's the smallest model for this matrix. And this will basically run on any computer. It's like 2 GB for each of the 11 functions in that
file. The harness drops the whole file into the model's context. And it asks it for the first 20 lines of
every function. Then compares the answer. This one doesn't take very long, about 18 seconds. And here we are. Seven out of 11 passed. Let me show you
what one of these looks like up close. So, right here, here's the pass. The gray lines are a match. Those are actually in the file in that order.
Orange is missing and yellow. Yellow is bad. Yellow is hallucinated. You might also see blue here.
That's bonus. These are correct lines in order, but they're past the first 20 lines. Okay, so that's one model on one file. Now, let me show you the
rest of the matrix I ran ahead of time and what it actually tells you about local AI for code because I didn't only run local models here. The first
question I had is, does the benchmark actually work? So, I ran it against Frontier models because if
Frontier models look broken on this thing, the bench is broken. And we're back to the drawing board. GPT 5.5 hits 213 out of 220 lines on HTTP server.
Zero hallucinations. Very good. Claude sonnet 4.6. I didn't do Opus yet, but you can run it, too. I can run it. Whatever. Let me know in the comments.
S 4.6 hits 207 out of 220 with 11 hallucinations. That's not 11. What am I doing? Anyway, the
bench works. These are really high numbers. We're 97 and 94%. Now, can a model running on my laptop compete with this numbers? I started off with Quen
3.6 35B. It's a 35 billion parameter model, but it's a mixture of experts with only 3 billion active parameters. And we've got 190 out of 220 with 28
hallucinations. It's not as good, but it's not terrible, right? Within seven points of a Frontier
model. It fits on a Mac running offline. So that's what's hot right now in 2026. Next, I wanted to find out if file size matters cuz HTTP server is a
small Python file. Real code bases aren't that small. So what happens when you point this at jQuery which is over 10,000 lines GPT 5.5 Frontier model
barely dips. It dips a little bit. Goes from 97% to 90%. Cloth Sonnet 4.6 goes from 94% to 93%. So
it's pretty flat. Pretty good. But the local picture is where things change. Quen 3.635B. This one drops from 86% to 78. So the gap to Frontier models
doubles. It goes from 7 points to 15. Now the difference. And now smaller models. Quen 3 4B 257 goes from 49% to 36%. Quen 3.59B for some reason went
up a little bit, but it didn't start out in a great place. Anyway, it started at 50%, went up to
- Now, originally I thought that the wall for small models was context capacity, but this version of the 4B, the 257, it fits the file just fine. It
just produced over a,000 hallucinated lines for just 115 real ones. The wall isn't memory here. The wall is noise. The model is producing more
invented lines than real ones. Okay, here is the leaderboard for jQuery. Every model I ran sorted by lines
matched. Quen 3.59B. That's this pink one right here. That's a dense 9 billion parameter Quen model gets 217 lines matched. This is out of 320 by the
way. Quen 3.635B. This one gets 249 lines matched. So bigger model wins. I wonder which one of those two is actually faster. It might be the bigger
model in this case because it only has 3 billion active parameters as opposed to the 9 billion in quen
3.5. So this is where the speed also we need to do that comparison too in another video perhaps. Interesting here. Gwen 3627B, which is not uh ane
model. It's actually a dense model. That one gets about the same score, 250 out of 320 with even more lines hallucinated than Quinn 3.635B. And it's a
much slower model, 27 billion dense. But also look at the hallucinations. So Quinn 3.59B, 318
hallucinations. That is a lot. Quen 3.635b 635B 38 hallucinations. Okay, that's better. 44 hallucinations for Quen 36 27B. But wow, that 9B model is
unexpectedly bad with hallucinations. That's roughly eight times worse than the next noisiest model in the entire Matrix. And it's sitting right
there, right in the middle of the entire leaderboard as if nothing's wrong. This is what you really got to
watch out for. You might think, "Oh, 9B is fine. 217 out of 320. It's okay." Look at those hallucinations. That's bad. So, even though hallucinations
were pretty bad with Quen 3 4B, the worst, they were pretty bad with Quen 3.59B2 sitting all the way up here. You want to be down here near the sweet
spot. High matches and low hallucinations. And we got obviously GPT 5.5 and Cloud Signet 4.6 there.
Okay. So, yeah, we're still at this point, but we're getting close. Now, this is a per function score here. Let's take a look at find. If you've ever
used jQuery, you know what that is. The functions on the left are easier. The functions on the right are harder. Find is kind of kind of in the middle
almost. Now here, quen 3.59B scored 20 out of 20 for this function, but it emitted 85 lines. So 20
of them were right. 65 were wrong or they were just made up. 3.6 scored 13 out of 20. It still passed because the threshold is eight, but it emitted
only 14 lines. So almost all of them were real. More lines isn't more useful. If you'd actually use this output, you'd have to verify every line of
the bigger blob anyway, then you're back to needing the right model in the first place. So smaller
isn't just less accurate, it's also noisier. Okay, but why does Squen 3.59B fall apart? Is it just bad or is there something specific about this file
tripping up? Well, this is the recall versus position in jQuery in this case. X-axis is the function start line in jQuery. So there's find at line 806
and here is parse HTML at line 10,142. And the y-axis is the percentage of what expected lines the
model got. And by the way, I'm going to include this as part of the repo so you can go dig into it and check it out yourself. This is an interactive
chart. Let me show you the part that actually matters. So if we take a look at just quen 3.59b, it holds reasonably well and then it kind of falls off
the cliff here. Then it goes does well again and then falls off the cliff big time back here. And
then at the deepest depth it's uh it matched 20 out of 20. So it's kind of unpredictable. Quen 3635B is a little bit better when it comes to
predictability. Same context, same prompt. Stays almost flat after 2K pretty much uh all the way across. Very different shape here. Now for context,
here's Claude Sonnet and here's GPT 5.5. Sonnet is blue, GPT is red. They both have a little bit of a hiccup,
but most of the way across they are pretty flat. So, what else does this benchmark catch? Let's go back to per function here. This is even more
evident if you look at HTTP server position versus recall. And if we take a look at the Frontier models, yeah, they nailed this. They destroy this.
And there's 9B. Yeah, pretty bad. And 35B is decent, but it's got that big hiccup here. But for the most
part, it does pretty well. Let's take a look at a dense model. 36 and that one does really well here. Wow. What about Gemma? Gemma does okay. It
doesn't do amazing, but it does okay. Now, how does Gemma do in jQuery? Nah. Yeah. Let's take a look at per function score. Here's Quen 36 35B. Look
at that cluster. That's exactly 19. There's a bunch of them. These are 19 out of 20. That's eight
functions all landing just one short of a perfect score. Well, it turns out Quen 3.6 six drops the leading tab on line one for every response. So no
matter how well it gets the rest of the function, line one actually never matches. That's the same exact pattern as Gemma with the indentation thing,
just a different model. It has a different quirk. Now look, I didn't program a check for either of
those. The grader is kind of dumb right now. It diffs the lines, but that's it. But the grading has to be deterministic, I feel. So the behavior just
kind of falls out of the data. Once you've seen it once, you will start spotting it everywhere. So, I'd say Quen 3.635b. When you see a pattern like
this, they're pretty much on par for those functions with Gemma 4. So, look, I'm not telling you use
Quen 3.6 over Quen 3.5 because on different code bases, a different file size, a different prompt, the answer might flip. What I am telling you is
this. Don't trust a benchmark that gives you a single number. If you're going to be using uh an LLM to help you code, run this benchmark on your code
and read both columns. One more shout out to the original creator or the inspiration for the idea took
sorry link to his channel down below in the description and go subscribe. GitHub link for my version is also in the description. Pull it down, point
it to your own code bases and tell me what you find. And if you like this video, check out this one right here. Thanks for watching and I'll see you
next time.