For the first time since 2019, you can run an Nvidia GPU on a Mac. Not through a virtual machine, not through a hack, through a real open source
driver that talks directly to the GPU over Thunderbolt. I just saw this announcement on Twitter from TinyCorp. Here's the tweet. If you have a
Thunderbolt or USB 4 eGPU and a Mac, today is the day you've been waiting for. Apple finally approved our driver
for both AMD and Nvidia. He forgot to mention that you need a Nvidia or AMD GPU, but I have that too and we'll test it out. This is pretty exciting
stuff. I'm going to benchmark everything and give you the honest numbers including the parts that aren't great yet. So, a little bit of a backstory.
In 2018, Apple and Nvidia had a falling out. Apple dropped Nvidia support in macOS Mojave killing CUDA
on Mac entirely cuz they went all in on their own Metal GPU framework. So, for 7 years in the desert, if you want a GPU support uh Nvidia compute on
your macOS, you were out of luck. So, then Tiny Grad from TinyCorp did something wild. They wrote their own Nvidia GPU driver from scratch. A macOS
kernel extension called Tiny GPU. No Nvidia drivers needed. No Linux needed. You just plug a GPU into
your Mac Thunderbolt port, approve the system extension, and it just works. Ugh. Of course, don't forget the GPU. Here's what I'm working with. A Mac
mini with an Apple M4 Pro chip, 64 gigs of memory. I've already done this with three GPUs, all Blackwell RTX 5060 Ti, 5070 Ti, and a 5090. Oh. Get out
of there. I really hate that little toggle thing on PCI slots. And because I'm so easily
influenced, I saw somebody reply to that tweet using this thing. So, I I went out and bought this. It's actually an Oculink which I've covered many
times before. Cool technology, but if you accidentally unplug that, oh, that's going to be problematic. They even have a little warning there for you.
Don't unplug that while it's running. So, I didn't use oculink today. This is a USB-4 device, and it
provides some power to the GPU that you can plug in as well, which works pretty well with GPUs that have the 8-pin connector, one, two, or three of
them. But, if you want to use that Nvidia 12-pin connector, you're going to need some kind of weird adapter cable. Cross that bridge when I get there.
Setup is surprisingly simple. It's all documented here on tinygrad.org. You just run this curl
command, enable that driver, shows up under general extensions, and there it is, that extension right there for tiny GPU. You just need to enable
this, and you're good to go. Oh, you also need to install Docker Desktop so that it can automatically do the Nvidia compiler for you. That's this
right here. Or, if you're using AMD, you would use the hip compiler. It's simple. And then, you just use it.
Specify what device you're using, whether it's you're using AMD, NV, or metal is one of the options. And then, you run the inferencer. So, I couldn't
wait to get this all set up and running. Seems way too easy. Run the server, and you have a little chat interface. Hi. And there it is, it's answering
me from the Nvidia GPU through the Mac. That's incredible. Total setup time, under 5 minutes,
especially if you're using an agent to help you out with that. Dev life leaves breadcrumbs all over the internet. Git, bug trackers, test accounts,
and data brokers stitch that info into dossiers and sell them. Make yourself harder to find, and you make yourself harder to scam. Once your info gets
bundled and sold, it becomes fuel for fishing and identity theft. You can try to remove that stuff
manually, but that means tracking every broker, filing every request, and repeating it when your data pops back up. Incogni fires those removal
requests automatically and keeps following up until they comply. My dashboard shows hundreds of hits connected to my details, and most of them have
already been taken down. And when I find a specific page exposing my info, I use custom removals. I submit
the link and their team handles the takedown and follow-ups. Basically, track it down, get it removed, and keep it from coming back. And that process
was also verified by Deloitte. It helps with broker sites and eligible pages, but it's not for things like official records or random social posts.
And if you want to test it out first, there's a 30-day money-back guarantee. Go to
incogni.com/alexziskind and use code alexziskind for 60% off annual and make yourself a smaller target. Link down below. So, tinygrad comes with its
own benchmarking suite. It runs inference directly on the GPU with no server overhead. And that matters quite a bit, as I'll show you later on. But, I
also have a tool that I like to use called llama.cpp by Yugur over here. It's open source. This
benchmark is a full end-to-end experience through the API. PP, prompt processing. TG, token generation. Time to first token. And it all includes the
HTTP overhead and streaming. It's what the user actually feels. So, with the RTX 5060 Ti, I ran my first NVIDIA compute for macOS. And the initial
matmul, uh, matrix multiplication benchmark showed 22.7 TFLOPS FP32. That's respectable, but my M4 Pro
actually beat it around 33 TFLOPS teraflops on the same test. I'll be honest, not the result I expected. But, the beautiful thing is that it worked.
An NVIDIA GPU going through compute on a Mac that hasn't happened since 2019. So, the question now is, is it just a party trick at this point, or can
it run a real workload? So, then I wanted to see if a bigger GPU would be significantly faster. Maybe
the 5060 is just a poor performer. By the way, I did run a few more tests on the 5060 Ti with llama.cpp. I'll go through all that in a moment. The
next problem I ran into was that this actually plugs in here, but the 5070 Ti doesn't get enough power from this uh little dock. I mean, it should if
you plug all three of these in using a funky adapter cable that goes from three eight pins over to uh
the 12 uh volt one, but unfortunately, I lost my three. So, I tried it with the four prong one. Uh these are This is for the big boys. This is for the
600 W ones, but it didn't work. So, I went shopping again. I ended up buying this Razer Thunderbolt 5 enclosure, which has a spot for your own power
supply. So, you can't go wrong. You can just get whatever power supply you want. It's a little more
expensive because you need the power supply and the enclosure now. If you're a serious gamer, you'd probably appreciate this. Pop. That just drops
right in there. Nope, wrong spot. Pop. That just drops right in there. Um I didn't do that twice. I did it only once. First time worked. Oh, yeah. And
now, of course, because we have our own power supply, I can just use the 12 V cable. There we go. We
got fully Thunderbolt 5. By the way, if you want to see a video comparing Thunderbolt 4 or USB 4 to Thunderbolt 5, let me know. Here we go. It's
working. I don't need to turn the computer off. I just hot plug the cable and it's off to the races. Tiny Grad automatically recognizes the changes.
Now, the 5070 Ti is supposed to have higher clocks and faster memory. However, the improvement I saw was
not amazing. So, I went for the big boy, the RTX 5090. I tested that one right after the 5070, and on FP32, the 5070 actually got a nice boost, about
64% boost from the 5060 Ti for matrix multiplications. And on 8K by 8K matrix, it hit 342 teraflops, more than double of the 5060. The 5090 was not
that impressive. Actually, it did about the same, but it did a little bit worse on paper here than the
5070 Ti. But, what the 5090 did bring to us is 32 gigs of VRAM instead of the 16 that's on the other two GPUs. So, I tested slightly larger models on
the 5090. I'll get back to that in a moment. So, matrix multiplications is fun and all, but how do these actually perform running a Llama? I ran Qwen
using tinygrad's built-in benchmark. The 5090 hit almost six tokens per second. Yay! At 28.8
gigabytes per second memory bandwidth. Does that sound right? Does it sound right? Let me know in the comments if that sounds right to you because to
me it does not. 5090 should be capable of much more memory bandwidth than that, like 1.7 terabytes. But, at least we had some consistency. 5060 Ti got
4.6 tokens per second. 5070 Ti got 5.5 tokens per second. And all those GPUs, the external ones,
beat the internal metal GPU, which came in at 3.66 tokens per second. Now, as I mentioned before, the 5090 has 32 gigs of memory, which means you can
run much bigger models. Usually, up to 14 billion would be comfortable cuz you got to account for context and KB cache and all that. So, I tested
slightly bigger models. Here's that six tokens per second number that I saw on Qwen 3 8B. Qwen 3 30B
MoE, mixture of experts. So, it has 30 billion parameters, but it's a smaller active parameter number. That one got slightly faster, 6.5 tokens per
second, slightly. Llama 3.1 8B, this is the integer eight quant by the way, so it is larger and it is a dense model. And I got 7.48 tokens per second.
Finally, a 14 billion parameter Qwen 2.5 got 3.75 tokens per second. Yeah, all these numbers are
making me sad. Here's the full chart for those of you that always ask for the full chart, including the running VRAM usage of the model itself. By the
way, you might think, oh, 10.6 gigabytes for running Qwen 2.5 14 billion, but it's not going to even load on the 5060 or 5070. There's just not enough
room for the context and KV. Now it's time for proper end-to-end benchmarks. So, I used Llama
Bench, the tool I showed you earlier, with the right settings. Here's the same Qwen 3 4B model, just switching between Nvidia and Metal backends. And
the RTX 5090 generated tokens 72% faster than Metal here. 7.39 tokens per second versus 4.29 tokens per second on Metal. Time to first token was three
to four times faster than Metal. So, if you're doing just chat here, the Nvidia GPUs give a
noticeably snappier experience. And I'm not even bothering testing concurrency this time around because I just don't think that this server is going
to be able to handle it. We're using the TinyGrad server, and I'll go into that momentarily. So, I thought, "Come on, these numbers right here for
token generation are just insanely low. For a 5090 running a 4 billion parameter quantized to four
integer? What's going on here?" Just to give me a little bit of a sanity check here, I ran Llama CPP, the tool we all know, love, and use. But of
course, Llama CPP compiled on Metal won't run on an Nvidia GPU. That's what TinyGrad does. So, I just wanted to get the Metal number for Llama CPP,
and I ran the exact same model through Llama Bench. Same test, same setting, same everything. And yeah,
the feeling that you had throughout this whole video, the feeling that I had while testing this, it was a valid feeling. Llama CPP is 10 times faster
running on Metal than TinyGrad running on a 5090. It's 18 times faster than TinyGrad running on Metal. Time to first token, 651 milliseconds compared
to almost 5 seconds on TinyGrad. So, let me be honest about this. Llama CPP has years of hand-tuned
Metal kernels, fused quantization-aware matrix multiplies, optimized KV cache management, thousands of contributors obsessively squeezing out
performance. TinyGrad auto generates kernels from a general purpose compiler, and it's not trying to compete with llama.cpp on inference speed, at
least not yet. So, if llama.cpp is 10 times faster, why does any of this matter? Well, because TinyGrad did
something no one else did. They wrote an Nvidia GPU driver from scratch, an open-source macOS kernel extension, and got Blackwell GPUs running on
Apple silicon over Thunderbolt. That wasn't possible a year ago. Two trillion-dollar companies, Apple and Nvidia, refused to make it happen, and a
community project just did it. The performance will improve, the kernels will get optimized, but the hard
part, the driver, the compiler pipeline, the memory manager, that's already done. Oh, uh some people might ask, and I had this question, too, is using
Thunderbolt slowing things down? Well, the short answer is no, not really for LLM inference. Once the model weights are loaded into GPU's VRAM, which
is a one-time transfer at startup, token generation is entirely GPU internal. Model weights are
read from VRAM, computed on the GPU, and written back to VRAM. The only data crossing the Thunderbolt cable per token is a few bytes. The actual
bottleneck here is TinyGrad's kernel efficiency. The 5090's memory can do 1.8 terabytes per second, and we are seeing 33 gigabytes per second here.
So, that's not a cable problem, that's a software optimization problem, and it'll get better. The bottom
line is, you can plug an Nvidia GPU into a Mac mini over Thunderbolt and run LLMs today. Is it as fast as llama.cpp? No, not even close, but it works.
I'll be watching the TinyGrad kernel optimizations closely. Now, if you want to see how this 5090 does on an Oculink dock, watch this video right over
here. Thanks for watching, and I'll see you next time.
Oh.