Three MacBook Pros with M5 Max, three Thunderbolt 5 cables, and a 122 billion parameter model running across two of them like they're one machine. The
third one, there's a good reason for that, and I'll get to that. Stick around. A few months ago, I did this with the M3 Ultra Max Studios, and WWDC
2026 is right around the corner, and Apple may or may not announce an M5 Max Studio. Nobody knows.
So, I'm doing the next best thing, clustering the M5 Max MacBook Pros, which is the closest thing to Apple's current top end silicon you can actually
buy today. And while I'm at it, I get to answer the question nobody else has run the numbers on, how much faster is the M5 Max cluster than the M3
Ultra cluster head-to-head on the exact same distributed inference stack. Let me show you how I got
there. Step one, before bothering with anything ambitious, this is always what you got to do with the cluster, is start out small. I needed to prove
that the cluster actually wired up correctly and works. Thunderbolt mesh, three cables, RDMA enabled, a jackal host file, what the heck is that? Well,
I explained it in my other video. I also have step-by-step how to set this up on my GitHub.
Basically, this is an MLX environment. MLX is the framework that runs on Apple silicon to do inference and training and machine learning stuff, and
you can actually run MLX distributed the way I'm doing here. So, the environment has to match exactly on all these nodes, on all these machines. There
are dozen things that could go wrong. That's why the first couple of times I set this up, it took me
a while. Now, it's way easier. I have it documented, and I had Claude do it for me based on my documentation. Sweet. So, I started with Qwen 3 4B, 4
billion parameter model. It's only 2 GB. Obviously, this will fit on one machine that has 128 GB. All three of these machines have 128 GB, by the way.
The whole point was, does the cluster at all? And yeah, yeah, it does. When you run it on one M5 Max
MacBook Pro, you get 179 tokens per second. Pretty fast, right? But when you run it on two machines, you get 220 tokens per second. What? A 22% speed
up? How? Well, that's what RDMA gets you. It's a huge win. I mean, this is a small model, so it's not a huge win for this model, but if we did use a
larger model, maybe one that challenges the compute on each machine, then we'll see even better
improvements. Here, there's barely any per layer work to share because it's such a small model, and the cluster pays some overhead for every layer
that it shards or splits. But we're still seeing improvements in speed. And this is a proof of concept that the cluster is actually doing distributed
work, not just sitting on one machine pretending. So, the wiring is right, but a tiny model is the
worst case for tensor parallelism, which is what we're doing here. Does it hold up when there's actual work to do? I made a ridiculously sticky mess
in my kitchen today just to see if a robot could actually handle it. Big shoutout to Narwal for sending over the incredible new Flow 2 so I could test
it out. The problem with most robot mops is they just smear the dirt around instead of actually
pulling it off the floor. Nobody wants to spend their evening on their hands and knees scrubbing dried sauce off the floor. That's why the Flow 2 is
different. It cleans right up against the baseboards and tight up to the furniture legs. Anything not on the floor though is your responsibility. The
Narwal Pro system is wild. It spots obstacles, knows exactly where they are, and avoids them without
me having to clear the room. It will even quiet down around cribs, which blows my mind. It's a complete system that looks sleek, blends right into the
room with a status light bar, and keeps itself charged and ready. After putting it through these ridiculous kitchen tests, I can confidently say it
handles almost anything you throw at it. You'll just need to occasionally swap the dust bag and empty
the waste water, but otherwise, it's a totally hands-off experience. Narwal is offering a great return policy, so you can test it out on your own
floors worry-free. Head down to the link in the description to the Narwal Flow 2 in action and grab yours right now. Time to give the GPU something
real to chew on. This is Qwen 3.5 122B A10B. It means active 10 billion parameters. So, it's 122 billion,
but it's a mixture of experts. On disk, it's 65 GB, small enough to still fit on a single 128 gig machine. And that's exactly what I wanted because I
needed to keep the one node baseline alive as control. On one node, we get 57 tokens per second. Quite usable, but on two nodes, we get 73 tokens per
second, a 27% speed up, slightly better than the small model. Now, mixture of experts models or MoE
models don't scale linearly with tensor parallelism because the active expert size is much smaller than the total parameter count. So, 27% is actually
pretty good for an MoE. But more importantly, the chart now tells a coherent story. Two nodes are reliably faster than one, regardless of model size.
Cool. The cluster is faster, but faster isn't actually the headline here. The headline is what the
cluster makes possible. So, what happens when a model is so large that it literally cannot fit on one machine? Uh that's the biggest story here. Here
we go. Same model, Qwen 3.5 122B 10 A A10B. Try saying that many times fast. But this is now the 8-bit quantization. On disk, that's 122 GB, and it's
right at the edge of what a single 128 gig Mac can actually hold once you subtract the OS and the
MLX runtime overhead. And one node doesn't even load. Out of memory. You don't get a slow benchmark, you get nothing at all. On two nodes, you get 51
tokens per second. This is the moment the cluster goes from nice optimization to the only way to run this model at all. We saw this in the other video
where I ran Chimney and GLM huge models across this cluster. Here, the one node bar isn't just
slower than the two node bar, it It even exist. There's no number to put there. Okay, so two nodes opens up a door to a model that one node can't run.
How wide does that door go? What's the biggest model two MacBook Pros can actually serve? Bigger model, GLM 4.7 4-bit is 185 GB on disk. This is
definitely beyond what one machine can even hold. And it's not a benchmark toy. GLM 4.7 is actually
useful. It's a coding grade model that people use for real work. Again, one node doesn't load, out of memory. Two nodes gets us 29 tokens per second.
Pretty usable. It's not fast by Qwen 3 4B standards, but for 185 gig model running on two laptops, that's legitimately impressive. It's well into the
usable for real work zone. I can sit with this and write code with it. Open code, cursor, whatever
you want to point at the OpenAI compatible endpoint. And I have videos on the channels on how to hook up different editors like VS Code and Zed to use
local models. All right, if 185 GB works at 29 tokens per second, can I push it even further? This one has been stumping people for a while since uh
it came out, I guess a couple years ago now. Llama 3.1 405B, which is just 215 GB on disk, but it's
a dense model. The math says it should fit. Two 128 gig machines have 256 gigs of raw unified memory. And last time I checked, 215 is less than 256.
So in theory, that should work. Well, I tried it. Look at that, that memory is going crazy. Oh, it is not happy. Both nodes wired up about 109 gigs of
memory each, and then they started swapping. Holy cow. Our wired memory's 42, compressed is 48, swap
used 21 GB. So after about 25 minutes of CPU time, the benchmark hadn't generated a single token. The math lied. After OS overhead, after MLX runtime
overhead, the KV cache, it just didn't work. Both nodes were technically making forward progress in CPU time, but at the rate where a single token
would take minutes, not usable. So, here's the practical ceiling for a 2 * 128 gig cluster, about 200
GB of model weights. Past that, you're swapping and thrashing. To go bigger than that, you need a third machine or a fourth machine. First, quick
definition because they matter for the next two stages. And then, yeah, this is kind of a nerdy detail stuff that you might or might not care about.
Basically, it has to do with the different kinds of parallelism. So, yeah, get your nerd glasses on.
Tensor parallelism, what we were using just now, that's what Jackal does. It slices each layer's weights horizontally and gives every node its own
thin shard or slice. Every node runs every layer, but only on its own piece, and they constantly coordinate. So, that's why you need fast, fast
connections. Then there's pipeline parallelism. You might have heard of a project called Exo. Exo does this.
And it slices the model, each model, vertically. Each node holds entire, contiguous layers, and a token's embeddings flow through them one node at a
time, like an assembly line. The difference between those two matters because tensor parallelism needs each layer's dimension to divide evenly, huh,
across all the nodes. Pipeline parallelism doesn't care. Here's the dirty secret of tensor
parallelism. To shard a layer's weights cleanly, the layer dimensions, hidden size, intermediate size, attention head counts, all have to be evenly
divisible by the number of nodes. And all the most popular and common models, Llama, Quan, Mistral, GLM, Gemma, every popular model is built on a
power of two dimensions, 4096, 8192, 14,336. Powers of two are divisible by 1, 2, 4, 8, 16, and so on.
They're not divisible by three. So, a three-node tensor parallel cluster works for almost no popular model out of the box. This is the actual reason
this whole video is about two MacBook Pros instead of three. It's not that I'm lazy, but I am kind of lazy. I'm having Claude do a lot of my work. So,
we're dealing with a fundamental property of how tensor parallelism interacts with how transformers
are sized here. Now, you might say, "Oh, Alex, why don't you just get four MacBook Pros?" Well, I I do have four MacBook Pros. I still have my M4 Max,
which I could have added to this experiment. However, we are now limited by the number of ports. Because of the topology, I'm using two on each
machine for the interconnects for Thunderbolt, and the last port I'm using for Ethernet connection. So,
theoretically, if I had four Thunderbolt 5 docks, then perhaps I could manage this somehow, but I don't. So, maybe next time. All right. So, why do I
have the third one? Well, if you want three nodes, you have to switch parallelism strategies. You can still do it, but you'd have to use pipeline
parallelism. Different stack, different bugs, different headaches. And I got these headaches trying to
make this video for you guys. For you. I always get into trouble making cluster videos. I don't know why I do it. Anyway, there's a way out. You've
heard of Exxact before, and I've shown this on the channel before. They implement a couple of different strategies. They do implement tensor
parallelism over RDMA for some of their models. And then they also, depending on your topology, also allow you
to do pipeline parallelism instead. So, I thought, "Okay, I'm going to try it again." So, Exxact handles the pipeline parallelism p- pipeline
parallelism plumbing for you. Gosh, that's hard to say. Who wrote this stuff? See, recommended for my cluster because my cluster only has one thing in
it. But, if I turn on Exxact on another machine, it automatically scans and detects the other machine, and
now we have two in a cluster. So, the number of models and the kind of models that I can run increases. But, all these that ones that are grayed out,
they're the really big models. Let's see if I can get to some of these. Although, this one right here, I already have it downloaded. So, maybe I'll
just try this one. Going to launch that. Warming up and it's ready. It's a small model and I only have
two nodes in a cluster right now. So, if I start chatting with it, "Hello." Boom. "Hello, how can I help you today?" And we're doing this at 105
tokens per second. All right, let's delete that. I'm going to launch Exa again on the third machine and now three machines are detected. So, I have a
total of 384 GB of memory to share between model. Now, if I go to load a model, I have more models
available to me. I still don't have the really huge models like DeepSeek V3, GLM-5, wow, that's a big one, and Kimiko 2 and 2.5, which I did run on
this before. So, check out that video. Now, I have to be honest with you, Exa is a little frustrating sometimes. And in my previous videos, I've shown
the happy path because well, it does work. It just takes a little while sometimes to get there,
especially when models that you've already downloaded sometimes get re-downloaded when you query them. Even models that exist on disk like this
DeepSeek V3.2, which I have downloaded, and this Qwen 3.5-397B, which I have downloaded, it's grayed out for no obvious reason. Now, they're working
on it. They're making it better all the time. But, the issue where the model gets re-downloaded has been
around for over a year. So, I did try DeepSeek V3.2 first, 350 GB, and I downloaded the whole thing. I copied it to the other machines and Exa's
manifest decided it didn't trust the file it had just downloaded. So, then I switched to Qwen 3.5-397B, 6-bit, slightly smaller at 323 GB, and Exa's
manifest happily picked it up. However, right now, as I'm trying to make this video, it's not showing me
that it's available for some reason. I don't know why. Even though there is a little green check mark there that says it's downloaded. But, luckily, I
saved my numbers from my previous run. Now, 323 GB doesn't fit on one machine, doesn't fit on two machines. That's beyond the 256 gig pool we already
established. But, across three M5 Max MacBook Pros with pipeline parallelism, I got 28 tokens per
second. That's not a benchmark stop. That's a working 397 billion parameter mixture of experts running on three laptops sitting on my desk. And this
is about the same tokens per second, roughly, that I was getting with GLM 4.7 on two nodes, which means going from two nodes to three nodes basically
doubles the model size you can run at the same speed. In other words, tensor parallelism is much
faster than pipeline parallelism. It's a real defensible trade-off, though. Oh, and the head-to-head I promised at the start, on Qwen 3 4B, two M3
Ultra Mac Studios got 170 tokens per second. Two M5 Max MacBook Pros got 219 tokens per second. So, that's 29% faster from laptops. When we put M5
Maxes into this body, if they do that, we might see a slight improvement. So, it's less than an H100, but
it's less power than my microwave, and it fits in a backpack. Repos are in the description if you want to check it out, and that video is right over
here. Thanks for watching, and I'll see you next time.