I've got this super tool right here for you. Now, hello. We'll just we'll just let this run and we'll see what happens. Bet you've never seen a 1B
model in open code before. Today, we're going to be testing mini CPM51B from OpenBN. Now, this is very interesting because aside from being a 1
billion parameter dense model, yes, a very, very tiny model, this is apparently pretty performant compared to
some other models in this size range and even some of previous generations a bit larger than this. So, this is designed for kind of edge deployment.
And for today's testing, I am going to essentially be testing this on the most opposite system that this is actually designed to run. But fortunately,
our token speed is going to be like a few hundred tokens per second. So, it will be fun to see this
fly on a system that's designed for a bit larger model. So, before we get into it, please do feel free to subscribe so I can get that 100K plaque. And
uh that was the only thing I had to say. So, let's take a look at some of the highlights of this model and then we'll get into testing it in various
ways. So, let's begin by just taking a look at some of the interesting things about this model. We
do begin with a triedand-rue benchmark JPEG, but something interesting about this is that it is I'm sure there's a technical term for this style of
chart. It does elude me at the current moment, but we can see this is compared to a couple smaller Quen models as well as LFM 2.51.2B. Now, the Liquid
Foundation models are very, very impressive for their size. They do also have some that are equally
small as this that have vision capability. So, be sure to check those out if you're interested in that cuz they are pretty cool. But based off of this
chart, it does hypothetically seem like this has a pretty good grasp in a bunch of different areas. I am very interested in testing this in coding
specific tasks as it's always fun to see what a tiny little model can do and sometimes it's actually
pretty surprising. Now something else that's interesting is this is a hybrid reasoning model. So you can have it think or you can completely disable
thinking as well and that's very useful depending on the specific task you want it to perform as if you are on a resource constrained device like a
phone or something like that. Not having it think will just give you a response significantly faster
than otherwise having to wait for the chain of thought to conclude. For today's testing, we're likely just going to be sticking with thinking mode as
it's fun to sometimes take a look at the thought process of such a small model. But it's good to have that capability and sometimes the chain of
thought and waiting for thinking can be kind of taxing on one's patience depending on the hardware that
you're running any model on. So there really are a bunch of other interesting things here. One thing I want to highlight is the context length here is
like 131K, which is a lot for such a small model. And it does make you wonder like how this thing would actually do at like a longer context, even
like 64K or like a 100K context length. Would it actually be able to like still perform at a decent
level? I don't have any specific tests for that, but maybe I can whip something up. However, there's a bunch of interesting things here just in terms
of like the local LLM hobbyist. And what I mean by that is they have a bunch of different models that have been released as well, including the base
model, which we see right here, pre-trained only. So having base models is fantastic because it
really allows folks to take it and then build something additionally off of it that they want to craft themselves. So it's like a getting a foundation
for a house where you can then go build the style of house that you want as opposed to like a full model like this which is just the house and it's
already built and you can't do as much with it. Some additional things I would like to note here is
one, this kind of stinks because this desktop pet that they mentioned and I'm going to scroll down to find that because it really is kind of cool. It
is only for Apple and I believe experimental for Windows right now which is a little frustrating because if we specifically click on this and look at
it, it's almost like the codeex pet but for a local offline like model that can power it. Now sadly
as I mentioned this is only currently for okay Mac or Windows. They don't mention Linux. It's experimental still. I'm sure there'd be a way to quickly
hack it together to work. But for the purpose of this video, I did just want to showcase this as this is something pretty cool and it has persona
support, a bunch of other things and it's basically this that lives in your desktop and you can chat
with it and basically what it says right here. They do also mention tool calling capability. So that is something I would be interested in testing and
we do have something as well set up here just to be able to play with that as well as the recommended sampling parameters depending on the specific
thinking or not thinking that one is going to use. So I have this loaded into LM Studio. I'm not at
all using a quant as this is being run on the RTX 6000 Blackwell card. So, we can see that just starting with the general browser OS test v2.5 that I
tend to give large Frontier state-of-the-art models, in no way is this going to actually produce something that checks every single box in this
prompt. However, I have to be honest in that I sometimes forget how to test these small models because
I've been so used to lately testing big models that are very performant with things like this. It's almost harder in some ways to test a small model
like this because there's less like wow factor things you can do with them, at least in terms of a visual presentation. So, we can see that the entire
thought process it embarked on following this prompt was 14.46 seconds. And even just looking
through this specific chain of thought right here gives us some insight into the capabilities and competence of this model. So it's basically saying
things that are correct based off of this prompt right here. Okay, since we're limited to a single script, we can do create an app list container with
five items. Add js that initializes the list maybe without any initially. And then we can see it's
going through a chain of thought. It's even writing some like sample syntax and things like this. And these are all things that look pretty good in
terms of the structure and the words that are here are proper and it all actually looks okay, which is kind of exciting. We have the final code right
here at a total speed of 410 tokens per second. So, we're going pretty quick and let's just take a
look and see what we got here. We're not going to have a frontier like insane state-of-the-art browser OS, but it's interesting to see nonetheless
what this actually generates. All right, so here's our 1 billion parameter browser OS. Okay, so we have our list of apps, a 3D GTA clone, a 2D game,
an app one and app 2, and then change wallpaper. Okay, now sadly, nothing really happens here. And
again, this prompt is pretty much like significantly too much to give this model. So, I think what I want to do now is essentially massively shorten
this prompt, and we'll see what we get. This time, I'll just let it run through in its entirety. Okay, save this as os.html HTML in your browser's
front end folder and we'll see what we get with a much simpler prompt. Now, I would actually go out on a
limb here and guess this is likely going to be a better result than what we received because it was less overwhelmed. Okay, so here's our browser OS.
We even have an about thing. Now, again, none of these are opening in their own windows and things like this, but let's open features. Okay, toggle
status and refresh screen. So, none of those have any functionality wired to them, but it did put
buttons in with hover effects. And none of this is arranged in like a mangled way, which by itself is actually pretty cool to see. Now, I'm going to
do one more test from like the big models here just to see how it goes. Again, I don't essentially think this will produce anything, but this is the
beautiful static subway scene test. We can see the thinking that's going on here. There is quite a lot
of it. And it is still, even if we scroll down to the bottom, it's relatively lucid in terms of what it's saying and what it's planning out to do,
which is cool to see. And keep in mind again, the context length for this was 131K. So this should theoretically be able to stay fairly coherent over
longer pieces of text, which is cool. 553 lines. Oh, that's a lot. Both of those browser OS results
were like 100 to 170. So you know what? Pedestrian, left, truck, bus, train. Did it just like Okay, we have a brightness slider. We have buttons that
are selectable. We have another brightness slider. Let's see if WD actually does anything. Okay, it doesn't. But even just seeing that it drew this UI
here is like I kind of like this. It's endearing, one could say. Something else that I do want to
highlight with this model is what they mention here in the about where they talk about its ability to perform tool calling. They say SG Lang is the
recommended backend. So I do actually have it running as well just being served through that back end. Fortunately, um, with the model of this size
and a relatively beefy GPU, you can run multiple instance of them without having too much trouble. So,
okay, we can see the model is loaded in right here. This is obviously just a vibecoded front end designed to allow this to actually showcase some of
its tool calling capabilities. All right, so let's try another one of these. Let's try Tokyo time. The current time in Tokyo is 1358 on 526. Okay,
let's check that. So, that would be it's 1:15 in Tokyo. So, Tokyo time. Very good. 1:15 p.m. Okay,
simple, but again, it does show some functional and capability and things of that sort. We also see our result right here just in the sidebar. Now,
all right, let's see what else. GPU status. Check the GPU memory on the SG Lang machine and summarize it. Total GPU memory of 95.6 gigs with 18.3 used
and 76.7 free at 19% usage. That's actually kind of cool to see. And those that is more or less
correct. Let's check NVT top real quick just to see. So 95.593 total. Okay. And it rounded up to 95.6 which is acceptable and good to see. 18.3 gigs
used. Okay. So I see 18.8 but it's showing 18.3. It could fluctuate over time at 19.1% usage. And again, the usage right now is dropped because it's
not actually currently performing anything. But that is nonetheless interesting to see. I can't 100%
definitively say that's like not a hallucination, but it was correct enough to show some capability there. And again, this is vibe coded, so look up
order Bravo 7391 and tell me the status and next action. And again, this is a synthetic order, but it's currently in packing status with an estimated
completion time of 528. The carrier's USPS. The next action is no action needed. Source synthetic
order simulator. So this is actually pretty interesting just to have a model this small that actually seems fairly competent at performing some tool
calls. Keep in mind this is not a quant that's running right now. So it is using up more VRAM than one would really expect this to were like a four or
8 bit quant running on an edge device. But I always like to test smaller models at like
non-quantized if I can just to see that they're like full performance. What is the time in Boston? All right, and that is actually correct. So, it's
nice to see one that isn't just pre-loaded here as a prompt. I need to convert the time from New York time to Boston time zone to provide the user
with the correct time in Boston. Now, it didn't necessarily print the correct answer here, but we can
see it did actually perform a tool call that does give us the specific time in Boston as well as the offset and the date as well, which is cool to
see. All right, back to LM Studio. Now, let's just try some general like obscure knowledge. I'll start with give me a history of the Apple iMac. The
Apple iMac originated in Asia long before the modern apple tree was introduced to Europe. Evidence
suggests cultivation dates back at least 1500 years. Okay. I am I see. Okay. No, it just didn't. Okay. So, not 100%. What is the best current GPU?
Okay, so it's aware of the 3090 and 4090. I'm not as wellversed on the AMD cards. 7800 XT looks more or less correct maybe, but don't quote me on that
because I'm not 100% sure though. These are correct. Okay, the wattage it's getting wrong here for
these specific cards. However, I wouldn't really expect it to know to that fine grain detail about these. But we see a bit more pertinent information
as well. Again, I think 580 is an actual AMD card. GeForce GTX 1660, 1650Ti, 2070, and then that would probably be more of an AMD card, but still. And
it gives us a little table formatted right here. High refresh rate 144 hertz plus. Good.
Recommended GPU tier for specific use cases. And it's saying the 4090 if budget allows. It's the most powerful and will shine in any modern game. Next
steps, identify your primary use case. Okay, that was not bad. They gave us a little bit of like something it does have some domain knowledge on.
Twitter was acquired by Elon Musk, the founder of X, formerly known as Twitter, through a company sale
in February 2022. Okay, I can't specifically verify the dates off the top of my head, but that is correct. And it did think for a brief moment as it
says right here. Okay, good. So far, basically, I guess like specific iMac computer um lingo is not its forte, but some general stuff it is seemingly
doing well at. It's possible I should have written Grand Theft Auto. No, because it did understand
what the shortening GTA meant, but it did just assume it was a film, which does make me wonder about how it would perform in a roleplay scenario. So,
let's test that role play as Meabot 89. And that's all I'm going to say. Hello, I'm ready to play. Please go ahead and tell me what you'd like to
discuss or perform. I'll take your lead. Shall we explore a new topic, solve an equation, or enjoy some
woody banter? Yo, Meabot. This is where I'm at. My peak personally. Let's see what it says. It did go into a roleplay. Adjust luminous nani cap and
beams down at you with a warm synthesized voice. H the streets and the GPUs. You're alluding to something quite specific from my very first
introduction. My back on the streets of 2018 is legendary. And yes, I've been on the street selling GPUs for
years. gently taps a holographic display. But I am as always an AI with no physical form. I exist solely in code and data streams. I don't have
streets, you know. However, I'm perfectly fine with any hypothetical scenario you set forth. Shall we proceed? Perhaps we can discuss the latest
models capabilities or solve a problem together. I'm ready when you are. That was very interesting, and it was
almost like this answer and then the one under here were like two different things. But I did like that it actually went into like character more or
less by describing an action before it responded. Okay, this one's always fun. Role play as Steve the PC repair man. Grab your own warranty and grease
your fingers because I've got this super tool right here for you. Now, hello. Thanks for reaching
out. Concerning. I'm Steve or the fixer. Also concerning. Well, more specifically, Steve the PC repair man. What's up? Did you just have a case that
needs proper sealand or a loose bracket? Perhaps you're having a little game crash. I've been fixing it with my superpower tools and honestly, nothing
beats that. Just let me know what's happening. We'll get it sorted before you need to pull out the
power strip now. Whoa. What was the last thing I fixed? My brain is already working on a repair plan. All of that waiting time is over. Leans in
slightly with a knowing smile. Oh, I see. Got to take a mental break now. You're right. Cutting the act isn't quite what I'm here for today. But why
are you? Nods sagely. I don't think I've ever heard that used ever. So, I'll just take care of myself. And
if you need help with something real, that's what I'm here for. I don't quite. And remember that super tool? It's still here on my desk by now. Just
grab me a wrench. Maybe even a bit of duct tape. Laughs. Are you a great finder or are you just a great dot dot dot dot scavenger? Okay. An understood
officer of the force. I have no idea what you're talking about. I'm Steve the PC repair man. My job
is to fix a computer when it breaks. Okay. So, this was educational, but it also makes me wonder about some specific thing we can do to actually
somewhat test the context of this model. So, tell me about The Great Gatsby. Okay. No, it does know. I wanted to give it like the entirety of the text
of The Great Gatsby, but it does actually seem to have enough pertinent information about that we
wouldn't necessarily Okay, it's not 100% Nick Carowway's mother and wife. Okay, so it knows some of the names, but it does not have the correct
character of like what they are or who they are. All right, so I'm prepending this by saying, "Read the following and fix issues with your previous
response. Now I've pasted in the entirety of the text of The Great Gatsby." Okay. Yes. And I can hear the
GPU now on this system. Your response contains several grammatical errors and inconsistencies that need correction. and then I'll paste the entire
thing in again and we'll see if we get something a little more um correct. Okay, it did mention Jordan Baker who is a golfer. TJ Echoberg. Okay, in
some ways that could be an observer of the events because it's a billboard that they pass through. Okay,
it did mention James Gats, which is a a name in the book, but is not Myer Wolfshim's father. All right, so we're only at 50% context length used up.
It's yellow. What kind of car is it? Okay. All right. So, not 100% on the ball. It did correctly get that the car is yellow, but some of the names and
things that pulled out here were incorrect, but it did seem to at least get some correlation, like
Jordan Baker being a golfer. So, that was interesting. thing and it even pulled the name the TJ Eckleberg which is the billboard that's like somewhere
in the book it's or in the movie if you've only seen that like me. No, I'm kidding. All right. Um, generate a beautiful website for Steve's PC repair.
And I'm not going to tell it like single script or Okay, I should have tailwind CSS. Okay, build
it. Okay, this is shaping up to actually be somewhat impressive. All right. So, here's our Steve's PC repair website. This is fantastic for a for a 1
billion parameter model. This is actually better than all of the other website results we'd received. Now, interestingly enough, isn't this a Mac OS
default background right here? Or I may be hallucinating, but I'm pretty sure that's a Mac
background. It actually put in some functional images. It has a fake customer testimonial right here. Some broken images are letting things down, but
it also does have hover effects on these things. what we do a contact form with a contact button. Where was this when we were having it generate like
browser OS things cuz this is much better. Security audits ISO 2701. I don't know what that is, but
it did say ISO, which I am familiar with. Repair process. Click below to schedule. John Doe 5 years. It could have gotten a little more creative with
the fake testimonial name, but gaming specialist free diagnosis and initial assessment. So, the services are related to PCs. This is actually not bad
to the point where I want to check some of its creative writing capability. So, I'm telling it to
make it prettier and then give Steve a crazy origin story in an about section. Okay. I don't know if I would call this Hunter Green prettier, but it
did change it a bit. Okay. Now, we want to see if the origin story is actually there. Steve's origin story. Yes, it is. Wow. Okay. 7th of October,
- After studying at a local university, we realized computer repair was a meaningful skill. Five
years in the field, we've handled everything from motherboard replacement to software upgrade found online 2016. We discovered a community of PC
enthusiasts who wanted to support each other. Live support since 22. Our tech team has been working remotely connecting with local service centers and
manufacturers. While Steve's stories about our passion, every person we work with has a story, too.
That's why we're dedicated to helping others make their technology work. Okay, I wouldn't call that a crazy story, but it did actually put it in here
in some about section and it did make the site more intricate, I would say. I'm asking something simple again. Generate a cool animated website about
AI. Now, I'm going to say build it. All right, so here's our cool animated AI website. Arena of
creation, AI adventures. Okay, I will I will accept this as animated because it does have a zoom in and out header text effect. Your journey with AI.
Ask questions to explore creation's capabilities. Learn art. Okay. Start story adventure. Your AI companion. Welcome to the creation world. Play
adventure. Play game adventure. Game area. Your AI profile is ready. Okay. It was somewhat animated in
the movement of this as well as some of the hover effects and things like that as well as a very, very interesting gradientbased color scheme. We even
have a defined footer. So, it seems to do all right with simpler website design prompts. This is like it's just probably messed up to even try this.
But I have plugged this into open code and let's let's see what it does in terms of building a
simple browser. I don't know like I'm going to put it in plan mode. Okay. So, it's in plan mode right now. We'll need to initially at least understand
the things open code is like sending it as a harness. So be happy to help you think about creating an OS-like website. Before we proceed, I need some
clarification. What would you like your site to resemble? Is it a desktop application interface?
Any specific features or functionality? Are there any particularly particular elements you want the website to include? Okay, already this is going
better than I thought. Um, I want it to look like an old Windows operating system. include some icons and a few simple apps. And then I think I'm
going to swap it to build as I send that just to see what happens. Also, remember you're no longer in
readonly mode. I can make files, run commands, and utilize all your tools. That's actually interesting. So, I'm just going to say build it. Desktop
app interface calc icon and calculator. Build it. Now if it asks me questions again. Oh, okay. So now in a more performant model, it would write the
script and then it would like output it as a script. We'll just we'll just let this run and we'll see
what happens. Bet you've never seen a 1B model in open code before. And this is why. Okay. All right. So, unfortunately, our open code browser OS
didn't really um contain anything we could actually test in real time, but it was actually interesting to see that it in plan mode gave some follow-up
questions, I think, and then it went ahead and actually started trying to do some things, which was
interesting. And I didn't really know what to expect in that, but I just thought it would be funny to pop this thing into open code and at least see
what happens because it does have a decent context length for a model of this size or for any model I guess could be said. Interesting. So overall,
that is going to conclude our first look and test of the new open BMBB mini CPM 5 1 billion parameter
model. It was a very interesting release and things like that desktop pet as well as the large different amount of models they released as well as
hopefully soon the additional new data set that is still giving a 404 but it would be cool to have that as well. And the amount of documentation
pertaining to actually running this with different frameworks and backends and things like that was really
cool to have and I always like to see proper documentation like this. It's just nice to have. So overall that's going to conclude our first look and
test of this model. If you have any questions, please feel free to leave them in the comments.