More memory available to the graphics card than a 5090 or even the RTX Pro 6000. 128 GB of unified memory on one chip. Still to this day, the most
powerful X86 APU you can buy. And four of them on one desk should basically be an AI supercomputer in your living room. Well, it took me over 3 months
to get these things actually to work together in a cluster as fast as they finally did. And most of
that was just me being wrong about what the problem even was in the first place. Let me back up here. The chip is the AMD Ryzen AI Max Plus 395, aka
Strix Halo. If you watch this channel, you've seen me try this chip on mini PCs and laptops from GMK Tech, Beelink, Framework, Asus. And this one is
the Minisforum MS-01 Max. It might be my favorite. Well, all except one thing, and I'll get into that
in a bit. And before this video is over, I'm going to try to beat Minisforum's own marketing video on a cluster of these things. Four boxes, 128 gigs
of unified memory in each, half a terabyte total. Now, that memory is shared between the CPU and the GPU, which you can configure in the BIOS. Out of
the 128 gigs, the GPU can grab about 115 per box. Now, the MS-01 Max has a full PCIe slot. So, I
dropped a network card with a QSFP port into each one to network them hopefully in a faster way. All four cables go into a MikroTik CRS A12 switch,
specifically one 400 gig QSFP DD port. DD means double density. One fat port, four machines. And if you do the math, that's roughly 460 gigs of GPU
memory across the four boxes. Enough to run models the size of GPT-4 locally right here on my desk, in
theory. For my Framework desktop cluster, I was using Llama CPP in RPC mode. It worked, but it didn't really actually scale very well. So, this time I
wanted to try something new, a thing called Mesh LLM. The GitHub repo looked pretty cool. It's supposed to pull a bunch of GPUs into one OpenAI
compatible API. Sounds perfect for four nodes. It did take some flag fighting to get one node going,
though. I got Qwen-3-8B working on it at 48 tokens per second. And this was using Vulcan. Then I tried to actually cluster it. I ran into all kinds of
issues, networking, IP conflicts, all the usual stuff that make it hard to network all four of these machines. Finally, I got all four mesh together
with a 4 ms ping. But, the multi-node inference itself, 235 billion parameter MoE, crashed. And
Llama 3.1-405B can't even see the GPU. So, Mesh LLM wasn't going to be the answer for me, at least not in this stack. It was time to move on. This
guy, Donato Capitella, has been doing Strix Halo cluster work, too. You should check out his channel and his GitHub, too. And on his GitHub, he ships
these AMD Strix Halo toolboxes. Basically, Llama CPP pre-packaged with ROCm and Vulcan for this
specific chip. Really easy to set up. So, I pulled down the images, spun up the containers, and these came with Ubuntu initially. So, I pulled it down
through DistroBox and ran it. Think of DistroBox as like Docker, basically. It worked the first try. Then I threw Llama 3.1-405B, Q2, by the way, the
small quantization cuz this thing is huge. I threw it across two nodes via Llama CPP RPC, and it
ran. I got 1.2 tokens per second. I know. I know. Yeah, we're getting there. We're getting there. So, technically, it was a win. It was unbearably
slow, so unusable. And here's the thing I really didn't get at the time. Llama CPP RPC is what's called pipeline parallelism. It lets you fit a giant
model across your cluster, but it doesn't actually speed things up. Each token still has to walk
through every machine in sequence. More machines means more steps before you get a token out. For years, this is how everyone was doing multi-node
inference. Jeff Geerling's videos, Donatos early videos, and my own first strict Halo cluster video, too. All of us hitting the same wall with
llama.cpp over RPC. What we wanted out of a cluster was two things: fit bigger models that any single box
could hold, and get faster as we added more machines. llama.cpp RPC only did the first part. Bigger models, yes. Faster, no. What we actually needed
was something different: tensor parallelism. Instead of each machine handling a chunk of layers and passing the token forward, tensor parallelism
splits every layer across every machine. So, all the boxes work on the same token at the same time. More
machines, more parallel compute per token. Actual speed up. Here on the left, I've got llama.cpp with RPC. As you can see, more nodes, the line stays
the same. On the right, we've got tensor parallelism. More nodes, the line climbs. This is what we want to see. In the Python LLM world, tensor
parallelism basically means vLLM. So, that's the framework that [music] I went with. But, getting it to
actually do tensor parallelism effectively across machines was its own adventure. See, vLLM works really well with Nvidia GPUs. And to get it working
with other GPUs, you kind of have to wrestle with it a little bit. Look, I've been trying to make this work for months, and I had the wrong idea the
whole time. So, these days, I'm always flipping between models. GPT for research, Claude for coding,
Nanobana for image generation, VEO, Kling, and Runway for video. Six tabs, six bills, and counting. Enter Chat LLM teams. One dashboard houses every
top LLM and route LLM picks the right one. GPT mini for ultra-fast answers, Claude Sonnet for coding, Gemini Pro for massive context. They recently
added Gemini 3 and GPT 5.1 the moment they dropped. Create professional presentations with graphs,
charts, and deep research detailed content. Need human-sounding copy? Humanize rewrites text to defeat AI detectors. Need visuals? Pick frontier or
open-source models. Nano banana, midjourney, flux for images, magnificent for upscaling, plus VOwan and Sora for video. All built-in. You also get
Abacus AI deep agent to pretty much do anything. Build full-stack apps, [music] websites, reports with
just text prompts, and deploy them on the spot. They have Abacus AI desktop, which is the brand new coding editor and assistant that lets you vibe
code and build production-ready apps. And the kicker? It's just $10 a month, less than one premium model. Head over to chat.llm.abacus.ai or click the
link below to level up with chat LLM teams. Here's what I was thinking, okay? The cluster is slow. The
cluster talks over the network. Therefore, faster network means faster cluster. That was what I was thinking. So, I went shopping. After watching
Donato's videos, he bought these cards. These are Intel [music] 810s, and they have 100 gig NICs using QSFP cables. I loaded them up into the
MinisForums, into two of them, connected two machines together, ran iperf. I got zero improvements. Here's with
just two nodes in my initial testing. Llama 3.370B, tensor parallelism two, so two nodes, over TCP, so regular networking communication stack, not
RDMA yet. This is a hint of what's coming. At 1 gigabit, we're at five tokens per second. And as you can see, the faster I increased it, it did not
really help the speed here. Every upgrade was supposed to be the breakthrough, and every time iperf3
confirmed the throughput jump, but the token rate never moved up. The bottleneck was not the bandwidth. This is where I was completely wrong. It was
latency. Every generated token needs several round trips between the machines, and each round trip under TCP costs hundreds of microseconds. More
bandwidth doesn't make the latency lower. So, I bought a lot of networking gear for apparently nothing.
And by the way, I didn't even mention that switch I bought, the A12 MicroTik, for this very reason. I later used it in my Spark cluster, so it came in
handy after a while, but this is the initial reason I bought it. However, there's one purchase I did make in this whole fiasco that did end up paying
off. Now, what tipped me off to the latency thing was that I watched Donato's video on his Strix
Halo cluster. He was getting 40% more throughput than I was. His trick? RDMA, which is remote direct memory access. So, instead of TCP, where you're
using the whole compute stack, you're skipping all that. VLLM tensor parallelism, too, instead of llama.cpp RPC. I know I'm throwing a lot of terms
around and acronyms. Hang out with me and I'll explain everything. Here's why latency is the thing.
With a typical networking card, your computers talk to each other over TCP. Every packet has to be processed by the sending CPU, sent across the wire,
received by the other CPU, processed again, and then copied into memory. That processing adds time to every single piece of communication. So, round
trip latency ends up in hundreds of microseconds. RDMA skips all of that. The network card writes
directly into the remote machine's memory. No CPU in the path on either end. So, effectively, round trip drops from hundreds of microseconds to about
5 microseconds. That's 200 times faster per round trip. And this isn't a new trick. It's how Apple pulled off clustering Mac Studios, too, over
Thunderbolt, which has RDMA built in. I covered that in my Mac Studio cluster videos. To get speed up out
of a multi-machine LLM cluster, you need round trip latency down in the single-digit microsecond range. RDMA is how you get there. And then, Donato
created this repository with Strix Halo VLLM toolboxes. And with these toolboxes, we can finally do it on Strix Halo. Just one problem, though. His
toolboxes, they say they run on Ubuntu also, but they're really geared towards Fedora. So, I tried them
on Ubuntu first. Yeah, didn't work. This is hours and hours and days and days of networking troubleshooting. So, I gave up and I just installed Fedora
on all the machines. Fresh Fedora installation, everything just worked. Well, almost. The built-in Ethernet jacks on these boxes did not work at all.
So, I went hunting for a Linux driver for it. Nothing. Only Windows drivers are available on the
Minisforum downloads. Fine, so I used these USB Ethernet adapters on each box. Moving on, we'll come back to that. Now, when I ran vllm plus Rocm on
Strix Halo, I got 15.5 tokens per second on Qwen 2.5 7B. You've got to start out with small models, okay? That's how you always start with the
clusters because otherwise, there's yeah, small models first. This was actually slower on a single node than
Llama.cpp with Vulcan was. Weirdly, which gave me 48 tokens per second. But, this was the foundation because this is the stack that lets us do tensor
parallelism. Here it is running on one machine. So, then I plugged in a second Fedora node with a matching Intel E810 card. So, we're not doing RDMA
yet. All right, here we go. It is a little bit louder in here, isn't it? Hey, there it goes. We are
on two machines now. Really nice utilization of the GPU on machine one and machine two, but this is still TCP. 180 watts of power being used on
machines one and two. 22.9 tokens per second with this Qwen 7B model. Up from 15.5. Not bad. A cluster that actually makes things faster, finally. So,
two-node TCP works. Time to actually turn on RDMA. And my Intel E810 cards wouldn't connect. Some kind of
GID index handshake thing, I don't know. Annoying though. So, I swap a Mellanox four card into one of the nodes, leaving the Intel in the other. So,
yeah, I'll spare you all the details, but it turns out the Intel IPoIB and Mellanox Rocky are different RDMA protocols. They can't talk to each other
at all. And the mixed pair ends up running slower than a single node, 9.1 tokens per second. We were
going backwards. The lesson learned here, use matched hardware or nothing. With the second Mellanox card in the other node, matched pair this time,
Rocky v2 over IPv4. First try, failed. Second try, failed. Turns out you need memlock unlimited in security limits. Log out, log back in. Every time I
do that, I hold my breath and cross my fingers. And I've figured out that some kind of process keeps
the cards locked. So, I need to I need to reboot every time I try. And there it goes. We're getting pretty decent utilization here, 96% on node one,
96 on node two, up to 97. And we're getting pretty symmetric RX and TX, receive and transmit over RDMA. This is actually working. Does it look faster
to you? It's making a lot of noise, I can tell you that much. You can probably hear that in the
microphone. 25 tokens per second. Yes. Single node, 15.5. Two nodes over TCP, 22.9. Two nodes over RDMA, 25.6. Very nice. This is finally the moment
right here that we've been waiting for. Now, it's time to expand that cluster. So, we got 65% improvement here over a single node and 12% over just
TCP. That extra 12% is what 200 times lower latency buys you on a small model. So, on bigger models,
it's going to matter way more. Now, I scale to all four nodes, Ray plus vLLM TP tensor parallelism equals four over RDMA. But, of course, more bugs.
First, got to deal with that. Do I look tired? I am kind of tired. These videos with the clusters are always really difficult. >> [laughter] >> But
I'll spare you all the gory details. >> Basically, one node's network manager auto assigned a /30
subnet instead of /24. Then Nick will pick a different GID. Uh anyway, I'm going to save you the But here we go. Four nodes are up. Let's see. It's
working. 97% usage on all four machines on all four GPUs. And VRAM usage is pretty low, up to 30%. 7% on machine four, so it's not very equal
distribution here. And this is, by the way, TCP. Uh so, we have a little bit of a regression here at this
point. Still, 29 tokens per second over four nodes. Now, I did run this for you live right now, and that's what I got. But RDMA did work for me at one
point on all four of these using Qwen 2.57B, at which point I got 38.3 tokens per second across the four nodes. By the way, it's a dense model, and
it's almost two and a half times the single node throughput. Near linear scaling on the small dense
model here. But check this out. Same chart, this is Qwen 3 30B that I ran also. 30B A3B. So, it's a mixture of experts reasoning model. In this case,
it barely scales from 34 tokens per second to 38 tokens per second going from one node to four. Still an improvement though, but not as much. Why?
Because that only activates 3 billion parameters per token. Not enough compute per token to make tensor
parallelism worth the collective overhead. That's the rule that matters most on this cluster. Active parameters, not total parameters. Remember what I
said at the start? I wanted to beat the marketing video, run a giant model. In that marketing video, they ran Deep Seek R1, 671 billion parameters. I
know, it's not the newest and greatest Deep Seek, but one step at a time, one day at a time, okay?
First, we need to find a quantization that actually works. And 4-bit AWQ quant, AWQ means activation aware quantization. That one is about 365 GB on
disk. So, it doesn't fit on one box. It doesn't fit on two boxes. It barely fits across all four. And for tensor parallelism, you need a multiple or a
power of two machines. So, either two or four or eight or 16. I downloaded the model, synced it to
all the machines, launched vLLM serve with tensor parallel size of four AWQ. Weights loaded cleanly, 74 shards, 11 minutes. Then, in it device never
returned. GPUs were sitting idle, no error, no crash, just silence. Great. My whole dream just came crashing down without a crash. So, just sitting
there in silence actually. So, at this point I was like, "Okay, let's try some other things with the
same architecture, DeepSeek." So, I tried DeepSeek V2 light, only 16 billion parameters, small enough to run on one node. And it turns out the EWQ on
MoE models was broken on this particular iGPU. There is a pull request to support this, but it hasn't been merged yet. So, if you're trying MoE with
AWQ on this machine, don't do it. Try a dense model instead. We know a big dense model. So, I found
Llama 405B Instruct AWQ integer four, 219 GB, perfect. No MoE, just dense weights. Downloaded it, launched it, same kind of hang. Tried a smaller one,
Llama 3.370B AWQ, same hang. Tried it on two nodes. Hey, it worked. Five tokens per second. Okay. So, at this point this was a tensor parallelism four
specific bug, unrelated to MoE, but still yet a different problem. This points to a general
overall software difficulty setting these things up. It's it's doable, it's possible, but there is some issues to work through. All right. New plan.
If MoE plus AWQ is broken and MOE plus BF16 crashes the kernel, too, what about MOE plus compressed tensor W4A16? Oh, by the way, that's a whole
different kind of quantization. It's 4-bit weight-only quantization, so int4. Plus, it maintains 16-bit
floating-point activations. So, W4 A16. So, I tried it on quant 330B, the A3B instruct version, 30 billion total parameters, 3 billion active
parameters. Launched it on all four nodes with tensor parallelism, and it loaded. I got about 37, 38 tokens per second, ran it a few times. So, rock
solid. That is great news. That's a private GPD class AI I promised at the start. Not 671 billion parameters
Deep Seek I was after, but still real useful numbers. So, here's where I ended up so far. Five models scale cleanly. Three of them hang when tensor
parallelism equals four, and Deep Seek R1, the one I wanted most, that one failed five different ways. If I were a more reasonable person, this is
where the video would end. After all, 38 tokens per second on a 30 billion parameter model is pretty
decent. Pack it up and ship it, right? But, it just kept bugging me. And then, I saw that our friend Donato has been busy. He's kind of behind on his
YouTube channel, but he's been very busy updating his toolboxes. And he had commits that looked like they were exactly aimed at my failures. He made a
specific build for this iGPU, and every fix I needed was sitting there in a newer image. There was,
right there in the repository. Fedora 43, which is what I installed, ships with kernel 6.17, but Donato, our friend here, was on 6.18. And the newer
ROCm was compiled against kernel 6.18. So, I I updated my kernel to 6.19, and then did a reboot. Clean boot, ran it, no segfault, 16 tokens per
second. Exactly the old baseline, so we're good with one machine. Then I went back and tested every single
model again, the ones that were failing with the old kernel, and guess what? Where I had hangs before, now every single model was working. Even the
newer ones, too, that were unsupported before, like Gemma 4 26B, now I got 30.5 tokens per second. Qwen 3.5 27B, almost 12 tokens per second. A big
chunky dense model, Llama 3.3 70B, it hung before it hanged before it hung hanged hung Oh, yeah. Oh,
whereas before it hung 6.3 tokens per second now. And the big chunky boy, still unusable, but at least it ran Llama 405B, 1.25 tokens per second.
Slow, sure, but running. And that left one model, the one that I wanted most from the very beginning. Back to DeepSeek-R 1. This time, the W4A16 quant
on the new stack, and you can hear it. You can hear it after a clean reboot. All four nodes are
running. Memory utilization is at 99% on all four machines, and it's running. It's actually printing everything out for me. Now, this is actually 37
billion active parameters in DeepSeek-R 1. So, this takes some serious compute time. Now, you tell me, does this looking like it's faster than
Minisforum's demo? We're using about 180 to 187 W on every machine. And yeah, there's definitely a lot of
heat coming out of the back here. Woo. 4,058 tokens generated in 10 minutes. And guess what? Minisforum's own product video, same chip, same four
nodes, DeepSeek-R 1 quantized to four bits. Their numbers, 5.94 tokens per second. Mine, with VLAM tensor parallelism, 6.23 tokens per second. Yes.
It's about 5% faster than the vendor's own demo. Woo. On a fully open source stack. Woo. 13 models, three
cluster sizes. The empty set means it didn't fit. Everything else ran fine. The fastest Qwen 7B dense and Qwen 30B MoE at 38 tokens per second. The
biggest DeepSeek R1 671B at 6.23 tokens per second. One thing about why any of this works, AMD removed ROCm from their builds and that's a library
Donato originally got into ROCm through his own [music] pull requests. He had to patch it back in by hand
to keep multi-node working. It's early. It'll improve, I hope. If you build this yourself, install kernel 6.19 first. Fedora 43 ships with 6.17. One
last thing, at the start I said the MSS 1 Max is my favorite Strix Halo mini PC. All except one thing, and that's that Realtek NIC that just drops.
Works fine on Windows, I tried it. They have great Windows support, but on Linux, kind of annoying.
Otherwise, the hardware is great. So, Minisforum wins on expansion, Framework wins on plug and play. Watch that Framework cluster video right over here. Thanks for watching and I'll see you next time.
[music]