Hello everyone. Um, one second. Sorry, I'm just going to record this so that I can put it on my YouTube channel for later. Now, let's get So, we're
doing a small LLM's talk. Now, that kind of sounds a bit weird because I mean small large language models. I guess I don't actually really have a
specific definition for what a small language model is. Um, we can kind of define it tonight as a crowd
because this space is I guess as we all know is rapidly evolving. My next current definition is if you ask me maybe a year ago it would be under a
billion parameters. Um, and who knows what a parameter is if I say that. I just want to sort of gauge. Okay. All right. Sweet. Don't have to explain
that. Now, I would define a small LLM these days as a model that can run natively on your own computer
or your iPhone. So, that's kind of the definition we're going to run with for that. And it's my sort of dream at the moment is to um shill or expose
those models as much as possible cuz I love things that run on your own computer. Um, so tonight we're going to do this talk kind of like a cooking
show. Um, it's going to be a little bit here and there because I'll be honest, this is really
exploratory at the moment. Um, a lot of the stuff that I've worked on here is only really been possible in the last couple of months due to framework
maturity and model releases and whatnot. So, on tonight's menu, we are going to um look at some real life examples that I've worked on. We're going to
look at various case studies. We're going to define what a small language model is, which we've
kind of already done. We'll look at a custom data set and how we make one of those. And then we will do a live fine-tuning session of a small language
model. And we'll compare the results of a base model to a fine-tuned small model. And this is going to be like a cooking show, of course. And finally,
we'll finish with a haiku. So, my goal for this evening is for all of you to go uh basically ask
yourself the question in your business or your life, can we create {slash} use our own small models, whatever your use case might be. Now, a little
bit about me. Um I'm very lucky. I came second in a Jiu-Jitsu competition on the weekend, so if anyone wants to roll sometime, please um let me know.
I teach machine learning, just crossed 250,000 students in 195 different countries, which is kind of
insane. Um I have a 4.95 Uber star rating. Um my high torch video on YouTube is the most watched on YouTube, which I mean, it's a 25-hour tutorial. I
never thought that would amass 5 million views somehow. My brother and I, Josh is sitting over there, we built an app called Nutrify. That's my formal
degree is food science and nutrition. Um so, marrying up machine learning with that. Um I partner
with businesses and I build custom data pipelines, models, you name it. My most recent one was uh with a conservation company in America to detect uh
900 different types of bird species um with small computer vision models, so models that had to run on device in offline environments. And I think the
song the Kendrick song that describes me the most is humble. Um so, I would describe myself as 80%
of Brian Johnson. Um I do love health and I'm kind of like Steve Irwin, except Crocs. I mean, AI and ML are like my Crocs. Um and so, that's me. If
you want to learn how to optimize your PyTorch models, I have a great video that um turns it into a song. I won't sing it for you tonight. Okay, so
let's do case study number one. This is something that uh my brother and I have cooked up earlier, um
like all good cooking shows. This is for a Kaggle competition that um we entered last month and it was the Med-Gemini Impact Challenge. And so,
Med-Gemini, has anyone heard of Gemini? So, Gemini is like Google's open-source version of Gemini. It's like smaller versions of Gemini. Um Med-Gemini
is medical domain fine-tuned Gemma. So, I'll just play the video and then we can discuss and it really
ties into small language models. >> G'day. Welcome to Australia. More specifically, the beautiful Burleigh Beach on Australia's Gold Coast. Now,
Australia's known for its beaches, known for its sun. However, that sun comes at a cost. In 2023 to 2024, Australia spent $2.5 billion treating skin
cancer and other skin cancer-related diseases. And tragically, 2,000 people or so this year will die from
melanoma and other skin cancers. Now, the good news is that if it's caught early, it could be treated early. And treating skin cancer early on is not
only more cost-effective, it results in better survival rates. And that's where Sunny comes in. >> Sunny is an iOS application powered by a fine-tuned
version of Google's Med-Gemini, we call Sunny Med-Gemini, that encourages people to do structured
self-skin examinations. Research shows that 44% of melanomas are discovered by the patients themselves or their partner. And Sunny is designed to
build on this momentum. It fills the gap between the excellent sun damage prevention marketing Australia does and a lack of a national screening
program. Sunny's goal is to double the reported number of Australians who do a yearly self-skin examination
from 26% to over 50% and to increase the number of skin cancer treatments performed at earlier stages zero and one compared to stages three and four
by 20%. I went for a skin check the other day and the doctor found a lesion on my toe I didn't know about. Luckily, they said it's nothing to worry
about. They even mentioned I could take a photo on it with phone and revisit it in a year or so. That's
exactly what Sunny is for, the take a photo of it and revisit it in a year workflow. Of course, when dealing with skin photos, privacy is paramount.
Sunny runs entirely on device and is locked behind biometric or passcode authentication, similar to how your banking app would function. When you take
a photo with Sunny, it uses the Sunny Med Gemma model to generate a structured response similar to
what a dermatologist would note down. Your photos never leave your device for inference or for storage, and when it comes time, Sunny makes it easy to
export these scans into a report that can be shared with your doctor. It creates a repeatable habit for self-skin examinations. Importantly, Sunny is
not a diagnostic tool, but a tracking tool.
Getting a bit of insight into my YouTube recommendations. Okay, so Sunny Q&A. Now, why That was so we just submitted that about 2 weeks ago. So,
that's my brother Josh, he's an iOS engineer. That model runs directly on the iPhone. Now, why would it be important in a use case like that for the
model to perhaps run on the device rather than sending any data to an API? Privacy, yeah. Anything else?
That's it. That's probably the biggest one. >> Cost. >> Can run without an internet connection, so remote third party, third world countries, and so
on. >> Exactly. Yeah, and latency as well. Yeah, that's a good one. Sorry? >> Ease of use, usability. >> Exactly. Yeah. So, they're probably Privacy
is probably the number one, and then latency and offline use, and then ease of use, right? So, you
train the model once, you deploy it to the iPhone, it goes out as it could be if we were to take this further as a sunny pitch to the government, we'd
be like, "Hey, this could be our national screening program." Of course, it's not production ready. It was a entry to a Kaggle competition, but we can
kind of see where that would go. And the importance of privacy in that case, if we go to the next
one, like that is the big important in the health data space. So, that's the idea of MedGemma, and that's the sort of the I guess that was the
competition's goal is to show use cases of where the MedGemma model could be used. Um and so, if we go um to cost, oh, sorry, cost is the next one.
So, these models take quite a big upfront investment to create, but then the fine-tuning, as we'll see later
on in the live demo, can be hap- can happen in minutes. So, if we look at the cost, now, that's a really niche use case and taking skin photos. If we
look at Gemini 3 flashes pricing as of today, if we were to deploy Sunny as a national rollout and say within a few weeks had 10 million photos, um
with the current Gemini pricing, it would be about five and a half thousand dollars, which is not that
much like in the grand scheme of things in terms of how much Australia invests into healthcare, but this is still this is just one application uh
across many potential different use cases. And so, when you train your own model, uh yes, you do have that upfront investment. Um however, you can now
run inference for free because you deploy that to the iPhone and all the compute just happens locally
on the device. So, there is no charge for input tokens and there's no charge for output tokens. You could run that as many times as you want. And so,
if that was 10 million images, what if that happened every year for the next 20 years? That model could continually run over and over and over again.
Um so, we're going to discuss uh small language models or language models in general, custom models
in general from a hardware perspective. So, how did Sunny work? So, Sunny is a vision language model. Uh so, it has a vision component and a language
model component. Now, the current state of the art on the iPhone hardware in particular, which is where our sort of expertise is and I guess this is
it's current sort of hardware chips on consumer devices aren't yet completely optimized to run uh
LLMs as fast as possible. They are getting there, um but the current best practice is you run the vision component on the uh NPU on an iPhone. So,
that's a neural processing unit. So, whenever Apple shows this graphic when they release um a new iPhone every year and they say the neural engine.
So, the neural engine um is specifically uh built for tensor operations. Um but at the moment, that is
really good for vision-based models. So, in the Med-Gemini model, that can run entirely on the neural engine. Um so, if you see that there's an Xcode
analysis there, um that's why Sunny it does take a little bit to warm up cuz we haven't I guess optimized the loading yet on the app, but the
inference once it's warmed up is basically instantaneously um because the image gets processed in
milliseconds and then the LLM happens because it's auto-regressive token by token, that happens on the GPU. Um and one big constraint of Oh, you go.
Just a question on that. Is Is that like what Apple recommend or is that just like it just sort of sucks that everyone does it this way? >> So,
that is what we've deciphered from the research. So, Apple published a paper. I haven't got it here, but
it's Fast VLM and that's how they did it. And in just in practice, uh there's no way this is like explicitly stated, but just in practice, this has
been found to work. And I don't think it's a sort of a I think it's more of a Oh, it's funny that works like that rather than a we've designed it to
be this way. Um I think because vision models are a lot more sort of I guess the hardware's built for
them more than LLMs. Uh so, Um, the NPU is really good at doing a big batch process over the image, whereas the uh the GPU is better at doing this
token-by-token calculation. For now. And that's just with our experiment of actually deploying a a VLM to run on device um and measuring the latency.
Cuz that was one big thing. We deployed it at first, but then it would take 10 seconds to run. And so,
we had to go into the hardware to sort of optimize uh what's it going to run on. And now, this is, of course, my bias is towards consumer hardware.
We've probably most of us have got an iPhone in here, probably Android as well. It'll be very similar story on the modern um Pixel chips as well. Um,
in the Google Pixel. Not too sure about Samsung. But, um that's where my bias is towards cuz not
everyone's carrying around a H100 on them. So, um that's sort of where I'm I'm biased towards is how can we run these models as fast as possible on
the hardware where most of us already have. Um, so memory is a big thing when you're running a model on device. That's a photo of my left leg, I
believe. Um, now, Sunny is not a diagnostic tool. It's just an analysis tool. So, we've trained it on um
open-source images of dermatology. They weren't that great um if I'm going to be honest with you from a data quality perspective, but there was they
were available commercially. Um, so if we were to productionize this, we'd number one improve the data quality. Josh and I aren't dermatologists, but
we are engineers, so we can get this up and running. Um, what Sunny's goal is to do is just to
increase that step of, "Okay, maybe 20% more people are doing self-checks once a year." And then that leads downstream to uh more skin cancers getting
found earlier on rather than later stage. Um, so if we go back to hardware, when you have a footprint, uh Uh, modern iPhone 17, iPhone 17 Pro has 12
GB of RAM. So, you've got a fair bit of space there. And so, um, when you look at a model that's 4B
parameters in 4 billion parameters, that is, in float 16, that's about 8 GB. And so, we've quantized this down to, we'll get to that in a second in
software, to 4 bits. So, we can get it to run with 3.5 GB of memory, which runs comfortably on basically any modern iPhone. Um, speaking of software.
So, this is Sunny's pipeline. We have an input data set, a photo of skin, um, synthetically labeled,
um, with Gemini. And then we upload that to Hugging Face datasets. And then we fine-tune the Sunny Med Gemma model in a supervised fine-tuning fashion
with, uh, Hugging Face TRL, which is Transformers reinforcement learning. And then we deploy that to the phone using, um, Hugging Face Swift
Transformers, as well as Apple's MLX, which is basically like PyTorch, but for Apple Silicon. So, the MLX
equals ecosystem has exploded over the past year or so. Um, so now, basically every time a new open-source model gets uploaded to Hugging Face, you
can run that locally within a day or two, if your Mac has, um, enough RAM, basically. So, that's the only limiting factor at the moment is how much
RAM you have to how big of a model you can run. Um, the recent Qwen 3.5 models, I think the 4B model
outperforms GPT-4o basically all the benchmarks. And of course, it's probably been benchmarked maxed. So, even if you took 20 to 30% off that, that's
pretty cool that you can have a GPT-4o running locally on your MacBook now. And so, my sort of, I don't know, guess or like hope is that like by the
end of this year, we're sort of trending towards GPT-5 running live on a MacBook. Not GPT-5, that
headliner model from OpenAI, an open-source variant that can run on your local computer. Um, so this is a podcast with Jeff Dean. Does anyone know who
Jeff Dean is? Probably the real nerds. Um Uh but he's uh one of the leading scientists at Google. And so this is um his talk on uh latent space
recently. And he had a lot of kind of a throwaway line, but I really liked cuz I've seen this myself just
in practice in building the Sunny model um as well as other different models is to go very low on precision. So has everyone heard of like precision
in computing? If I say like float 16, int 4, int 8, all that sort of jazz. So where the research is kind of going at the moment is you go large on
parameters, but then you go hard on uh precision. And intui- I don't have a good reason of why that
works, but intuitively um to me it kind of makes sense. More parameters give your model more capacity uh because you have that larger capacity, even
if you do lose some precision, you still get that performance on the outer end because they just have such a sheer number of parameters. Um and so
that's we're kind of talk getting in the realm of like all the knobs that we have to turn to get small
models to work well on consumer level hardware. So um yeah, the one thing's the training, and then the next thing is okay, how can we optimize the
hardware so they're not taking 30 seconds to run at a time? And then the next thing is okay, how can we um optimize the footprint? Precision is one of
those ways you can optimize the footprint. Um that's what we just mentioned. So if we start in
floating point 6 or 32, which I mean not many people do these days, um but that's kind of like the default uh if you create a tensor, but that'll be
16 GB for a four para- four billion parameter model, and that's just too large. That was where Med-Gemini would have started if it was in float 32. Um
good news is by default a lot of model training's done in float 16. So it starts off at 8 GB. Um
when we quantized it for Sunny, we took it down to four bits. So um it was about 3 and 1/2 GB when we deployed it to device. So, I don't know if these
are large enough. So, excuse me, but I'll just read these out. So, the original one is on the right. That's Google's release um of Med-Gemini 1.5.
Probably Yeah, came out just about 2 months ago. Um and so, that is about 9 GB total. And then for the
one we deployed to device was 3.4. And so, the next step that we did for deploying it on device was we fine-tuned it. Now, the reason why we
fine-tuned it is because um we wanted it to be able to do the same task that we wanted it to do with a shorter prompt. So, language models, as we all
know, are very diverse things. They can do almost anything. Um but that kind of is a weakness in a sense. If
you want it to do a specific task, um you don't need all of the bloat of the larger prompt um because when you're in a small device regime, every
token counts because every one of these tokens um 248 of them adds to the memory tally. So, before we just looked in Xcode, we had 3.5 GB of memory
usage. If we use a larger prompt, we could almost double that memory usage, which would actually cause it
to crash out on iPhones with less than 8 GB of RAM. And so, what we did was just simply get it to do the same task that we wanted to do, structured
data extraction, with a shorter prompt. Because then the KVK cache, KV cache, say that fast 10 times. Um is a lot smaller because you're just getting
it to do uh the task that you want it to do. And so, these models are incredibly easy Well, I don't
want to say easy, but they are um they have such a large capacity, but then when you find because they've trained on basically the entire internet,
when you fine-tune them, it's amazing how quickly you can get the results. And so, we're going to see that later on, but the fine-tuning stage for
Med-Gemini was probably about 15 minutes of training. Like it wasn't that long. The hardest part these
days is constructing a data set and deciding what the specific use case is for your business. Um these are some examples um that we went through a bit
of iterations of testing the model. Um so if we do the base MLX model, so that was a skin extract prompt. Um excessive disclaimer output. So that was
one of the weird artifact from the Med-Gemini model was Remember when there was these models were
sort of over-tuned for safety and they would sort of always put disclaimers on things and they were when they first came out they would just say,
"Hey, I'm not a doctor or whatnot." So we just trained that out of the model. So Google had released that model with the disclaimer outputting on
every response that you would type into it. So you could just pass it an X-ray and it would always output
disclaimer, "I'm not a medical professional." Um so we don't need that. In the small device regime, as I said, every token counts. You don't want it
uh inputting or outputting excess tokens. Um then we did the long extraction prompt. Hey, what if we just give it instructions? How does that work on
device? Then that was unstable. When you quantize that um when you quantize the default model without
uh fine-tuning it, it was far less stable. But then when we move all the weights to do the single task that we want, we got exactly what we want.
Accurate adheres to structure and shortest output generation time. So kind of uh the main point here is we went through several different iterations
from uh base model to fine-tuned model to quantized model to um model that runs on an app on your phone.
So these are some more case studies. We're not going to spend as long on here cuz I don't have as much hands-on experience with them. Clin a 2, if
you're doing named entity instruction, is uh a model that can just run on CPU and runs at scale output structured data. So you don't have to um output
uh I guess send any of your private data to uh an API. Q & A do nano is a 4 billion parameter model
that is on par with Gemini 3 Pro for theory proving. Um I I all the math problems that it actually solved are far too complicated for me, so I
personally haven't looked at all the things that it can do, but that just goes to show that if you have a small model and a specific use case, you can
train it to be competitive with the state of the art. Um SAM 3, um that's more in the vision space, but
now it has a language component, so you can just type in, "Hey, person in the blue top with black shorts." That's me training for the jiu-jitsu
competition the other day. It basically perfectly segmented me. So, any vision task these days, um I would always just start, uh for example, the bird
annotation one, uh start with a large run of automatic labeling and then train specific models to
replicate that from uh a SAM 3 model. So, if you're in the vision space, um a lot of the sort of the perks that we got from LLMs uh now crisscrossing
back into that vision space. So, a lot of document intelligence is getting far, far better because we have far better um automatible data pipelines,
and then the humans come in and instead of annotating everything bit by bit, we just correct them
over time. And so, that's where a lot of my work as a machine learning engineer goes into. It's not annotating every single symbol, it's um running
large-scale automatic annotations and then reviewing them and then uploading the pipeline and then optimizing the model for downstream use, whether
that be deploying to device or optimizing to the hardware that it's going to run on. Google AI Edge
Gallery is example of running uh an Android app with Gemma 3N, which is we're going to get to Gemma 3 uh in a second, but Gemma 3N is a model that's
been optimized to run on mobile phones. So, this is just going to run offline. There's an iOS version now as well. Um basically like Gemini, but just
runs on your phone, obviously much smaller scale. Um and then yeah, Qwen 3.5 4B, we just mentioned
this. Um that's one of the smaller variants of the Gemini 3.5 series, but um, on every basically public benchmark that GPT-4 when it came out two
years ago, um, it excels on that. And of course, if you took 30% off it because it's been benchmark max, and what I mean by that it's just been over
trained on the benchmarks to get good results. Even if you took 30% off, it still beats GPT-4, and if
you've tried it hands-on, like it's a pretty good model. So, I guess that's where we're at. We're like 18 months to two years from state-of-the-art to
model you can run on your phone. Um, and that's the sort of the environment which is my specialty is on-device models. Um, this is another example,
VLM edge use case. So, um, uh, I don't know if anyone's ever gone through this, but it's quite a a
hard process. My brothers are over there. Our father just moved into an age care facility, but um, I would say there's a lot of use cases there where,
um, I guess more intelligent monitoring systems could be used cuz there's fall detection in the facility. However, it's just like kind of like a mat
that's underneath, um, the bed. And now, of course, that comes into the realm of privacy, but that's
where uh, smaller models that can run locally, no data could ever leave the building really. Um, this is Richie Mini, by the way, that's a um, robotic
system that you can purchase online. Um, no affiliation, I just think it's cool. But, you could have systems like this running smaller on the edge
models similar to the Sunny Med Gemma model that monitor a room. For example, I was there the other
day walking past, someone had fallen over. I'm not a medic. I went to get someone, but I don't know how long they had been on the ground for. So, if
there was a vision system in that room, of course, you can pair them with traditional detection models, but if there was a vision system in that room,
then potentially that fall could have been discovered quicker than when I just happened to walk past
after visiting my dad. So, well, we've kind of already discussed that, I hope. But, a small language model is many things, and I guess we've kind of
blended this into small custom models. It's custom, it's compute-constrained, it can run offline if you need, it's privacy-preserving. If you're in a
business, I partner with many businesses where basically their data either can't leave the building,
their data can't leave the country. Um smaller custom models solve all those privacy problems. Um it is an investment cost up front, whether you're
investing in hardware or training, uh but then from there on you get basically infinite inference. Like the Med-Gemini model we fine-tuned could run
10 million times, and we won't see an API bill. And then, of course, I'm really biased towards this,
but I like owning your own compute. So, how to choose a model for your use case? If you need privacy, likely a custom model. If you need
on-device/offline capabilities, likely a custom model. I want to get started ASAP, just go to the API. Needs most powerful model in existence, or you
have a really specific big hard use case, the biggest model you can use, API model. Um you want to own your
compute stack, custom model. And if you don't know, well, you can contact me. >> So, which cloud do you like? You got to start there with the small
language models and so on. What what back end hardware that more like resources like to use size and what not? >> Well, so depends It's kind of a hard
question, but we're going to use Google Colab in a second. >> So, for fine-tuning, what do you mean
like cost and variance if I get into >> So, for fine-tuning, um you'll see in a second how quickly we can fine-tune a model um using Google Colab,
which is a free service. Um you can pay $10 a month, which I do, and I think it's probably the most valuable compute you can buy at the moment, cuz
the GPUs that we're about to see are state-of-the-art GPUs. Um but also now maxing are getting to the not
quite there, but like if you have like a a Mac, you could just train it on there. As for particular GPU you'd want, you want something with like 16 GB
of VRAM is probably the minimum for fine-tuning a model, unless you're really patient. And then you can kind of just let it run for days on end. So,
does that answer like the hardware? Um, fine-tuning versus rag versus prompting, they kind of all
blend into the same thing. They all really have the same goal. You would start with prompting. You would use fine-tuning for specific task, and you
use rag for specific knowledge. And then you would just mix and match them all when whenever you need them. So, that's kind of my framework of how I
view all of these. Um, so language models are versatile. Tokens in, tokens out. That's probably the
biggest mind shift if you've ever been in, I guess, traditional ML or training a model or whatnot. They used to be a very limited output. You'd have a
thousand labels, whereas now the token space is infinite. You can just But, that's the beautiful thing about language models. You can just design your
token input and design your token output however you want, and generally the models are pretty
good at that. Um, so now we're going to move on to an LLM fine-tuning session live. I haven't really ever done this, but we'll see how it goes. And
so, um, here's what we're going to do. We're going to build who's here. So, um, we're going to use Gemma 3 270M, which is 270 million parameters. So,
Sunny made Gemma was 4 billion parameters. So, it's already 16 times smaller than that model we
deployed to the iPhone. So, that's quite a small model. And if we look at Gemma 3 variants, they go right up to 27 billion. So, this model is 100x
smaller than the largest Gemma 3 model. Um, so let's have a look. Here's what we're going to do. So, data. I have pulled everyone's names from um the
Meetup group. So, um this is all public-facing information, by the way. So, um this is just go to the
Meetup event and attend who's here. So, if your name was on there, you might be in our data set. Um but just your name, right? And then we have the
model, which is Gemini 3, and then we're going to create this demo here, which is going to compare the base output of the model that you would just
download off the shelf to the one we're about to fine-tune. So, let's go to Google Colab.
So, we'll reconnect. And how we doing for time? This won't take too long. We're going all right? Yeah. Okay. So, I'll just show you. This is the list.
I just went to this list and basically searched whatever name appears here, just search that, and if there's well, public LinkedIn information, that's
been into the data set. So, our idea is to enter Let me get up the base model. If we enter
someone's name, it should give back some information about them. So, this is the base Gemini 3. And now, this is probably my favorite one of the whole
one. We're going to use our host name. So, Michael, um I didn't actually tell you about this, but um This is kind of funny. We type in Michael's name,
this is the generic Gemini 3 response. So, uh so, apparently Michael's a highly acclaimed,
influential rapper, songwriter, and producer. He's known for his distinctive style, high lyrical prowess, and his ability to create complex and
innovative soundscapes. Agree? >> It's all on my LinkedIn. >> Yeah, yeah, yeah, I agree. Okay, now let's try me.
So, I got a lot of stuff on the internet, So, maybe it's been slurped up into Jemma Free. Who knows? So, Daniel Burke is a popular and highly
respected figure in the world of cybersecurity. Okay? I've never done anything to do with cybersecurity. Uh he is known for his contributions in the
field of cybersecurity, particularly in areas like threat detection and prevention. Okay, that's not correct
for me. Uh does anyone want to volunteer a tribute? Yeah, what's your name? How do I spell Can I find it on here? >> Um sorry, Guranga is very close.
Oh, uh can you spell that for me, please? >> It's g a u r a n g a r. >> g a u a >> n g a >> See, this is made up's horrible design. Look at this.
You have to keep scrolling to I'll type it in here, anyway. Okay, g a u a a n g a m h e Mike Alpha
Mike Alpha Golf, yeah? >> r i p >> r i p l i s Okay, I'm going to copy that to the clipboard, so I don't have to type that again. Uh Guranga is a
common versatile word in Hindi. Is that I don't know. Is that Okay. Okay. Um let's let's save that for uh I'm going to put your name right up the top
here, so that way we can try that one later.
Name. Okay. So, this is Google Colab. Anyone use Google Colab before? Okay. So, it's a it's a free service, basically like Google Docs for Jupiter
Notebook. And if you're a data scientist, um you're familiar with Jupiter Notebooks, but this is um just a cell-by-cell ability to run Python. And the
beautiful thing about Google Colab is that um Google offer free GPU back ends. Um I'm on the paid
service, so I get higher quality GPUs. Um and as I said, I've used this for a number of years and I think it's for $10 a month, it is worth it for me.
So, here's what we're going to do. We've got a recipe. We need to load a data set. We're going to inspect our data set. We're going to fine-tune the
model. We're going to evaluate the model. We're going to upload the model to Hugging Face Hub for
reusability, so everyone could download this model um if you really wanted to. And then we'll create a demo to compare it to the base model that we
just saw. So, we need some dependencies. And I'll just run through very quickly while all this is running. Transformers is a library that gives you
access to models. Datasets give you access to datasets. TRL is Transformers Reinforcement Learning.
Accelerate speeds up your code. Hugging Face Hub is like GitHub but for model artifacts. Gradio is going to be our demo application framework. And
then the Rogue score is an evaluation thing. And then Matplotlib is going to help us draw some nice loss curves. Okay. So, let's uh see if our GPU is
working here. Now, this is a a great GPU. This is a G4. This is a Nvidia RTX Pro 6000. We have close to
100 GB of VRAM, which is like crazy. Um and it's also Blackwell edition, but if we go RTX Pro 6000 price AUD. Let's see. Yeah, so it's about That's
probably actually cheap. So, um it's probably closer to $25,000. So, that's the beauty of sort of Google Colab. You pay $10 a month and you get access
to GPUs like this. Um and then we go through here. Imports, we're getting our libraries that we need
set up. Um get some information about the GPU to show you how much VRAM we have accessible. So, we don't actually need this GPU for what we're going
to do. Um this is quite overkill for our task. But if we wanted to anything more complex than just training a model to give us back some information
from names, then we'd probably definitely want a higher capacity GPU. Um, we could get by on what
we're doing here with the minimal GPU that Google Colab offers. I've selected this because I want this to be fast. Um, and now we're going to download
the base model. This is we put in an ID key here from Hugging Face, and as we see the model is going to start loading. It's 536 megabytes, so
definitely plenty of capacity to run on many of our local devices here. And so, if we go here, this is the
base model that Google have released, Gemma 3. I'm not going to read through all this now, but Hugging This is basically where I go every morning,
Hugging Face, to see all of these models which are open source, and sorry for the light mode blinding everyone. Is that incredibly orange? Should I
turn off night shift? >> No, it's the screen. >> Ah. It's the display. Ah, okay. Okay. But, all of these
models are open source, so this is where I'm sort of getting to the point there's a lot of different parts going on here. So, we have models, we've
ticked that off. Now, we need some data. How do we get data? So, let's go back to our Google Colab notebook. Um, let's try Oh, this is going to try
the base model. So, we're going to see what it outputs. I've got a list of names here, so I've thrown my
brother in here as tribute. Um, there's Daniel Berg, that's me, William Berg, that's my brother. Um, what are Daniel Berg's skills, Elon Musk, and
then a haiku. So, this is the base model before fine-tuning. Input, output, we've seen a couple of examples of that before. But, Daniel Berg is a
well-known and respected world in the world of computer science and technology. Okay, we've got a different
response now. So, that's an interesting thing about LLMs, they're not always deterministic. Um, William Berg is a well-known and influential figure in
the field of computer science and technology as well. Is that true? >> No, I can't tell. >> Oh. Um, and then what are Daniel Berg's skills? highly
accomplished figure in the field of artificial intelligence. Wow, okay. Thank you. Um, Elon Musk, so
probably knows about Elon Musk cuz it's in the training data. And then here's a haiku. Green leaves softly sway, sunlight paints the forest floor,
nature's gentle grace. So it's okay, not great. Um, now we're going to load a data set. So we've got our base model. The next ingredient is a data
set. So our data set is a way that we guide models. What do we want our model to do? And as I've said with
Who's Here, which is our demo, we want to input someone's name here, like William Burke. And we want to get back information that we would find in a
quick Google search. So there's what we just saw before, doesn't really work. So how did I make our data set? We have our attendees list. Take these
to a CSV. So some people's meet up username isn't a full name, which is fine. We have ways to handle
that in the data. But for people who do have a full name here, where's my Where's the favorite name that we just said out before? Is it here? Um, what
I've done is I've searched for that name, and then I've just pulled public LinkedIn information, and then we're trying to get the model to regurgitate
your LinkedIn profile. So that if you're at this meetup and you had this app, you could type in
someone's name, and then it would tell you what you should ask them about. Um, so we've searched LinkedIn for names, we've created a synthetic data
set. Synthetic just means I've asked another model to create example uh question and answer pairs based on a certain input. Um, that's the beauty of
language models these days is that now text and we're getting close to vision data being infinite. Um,
so one of the benefits of generative models is if uh all pre-generative models, if you didn't have a data set, you were basically screwed or you just
had to go through the um I guess long iteration phase of creating a custom data set. Now these days we can create custom data sets in hours instead of
months. Um so, some notes. We have plenty of edge cases, but we can handle them through
experimentation. Some names don't appear naturally. How do we handle those? These are all questions you'll have to ask in your business use case of
how do you handle the edge cases? That's a a never-ending question in the world of machine learning. Um so, our data set is hosted publicly on Hugging
Face under um Mr. D Burke, which is my username there. Queensland AI Meetup SFT Supervised Fine-tune
V2. Why V2? Because I started with a V1 and it was 1,000 samples, but it was too small. So, I iterated and basically 10x'd it, took it to 8K, and now
it works pretty well. Spoiler. Um so, we're going to inspect some samples. This is what LLM data looks like. It's just tokens in, tokens out. But, of
course, we're looking at it in text form. Now, of course, our use case is very simple, uh but if
you're uh a large company, such as OpenAI, Gemini, Claude, uh Google, sorry. Um your workflow for this is going to be basically the same. I think
OpenAI's last public mention count was they have 633 petabytes of data. Um so, that's just astronomical amount. I think the Common Crawl pull was like
uh 14 petabytes. So, 614, I guess, what's that? Nearly 60x. The Common Crawl is a copy of the internet.
So, they've got 60x the entire internet now. So, GPT-4 is probably trained on just Common Crawl, but now we're getting into the realm of synthetic
data, right? You need all these long agent traces. Um but, it's still the same process. Still just long series of tokens and the same training
methodology. Get your LLM to predict the sequence of tokens. Um so, now we're going to fine-tune with uh SFT
trainer. And while it's training, we might if anyone's got a question they want to ask. But, basically for training, we set up some settings here and
then we have a configuration. We use SFT trainer, which stands for supervised fine-tuning. We pass it our training data set, which is about 1,000
7,000 samples and our test data set, which is about 1,000 samples. So, training's going to run because
we have a nice RTX 6000 Blackwell. It's actually going to take only 2 minutes, so maybe one question. >> Any questions? How do you evaluate the
fine-tuning problem? Like, what are the metrics you use to evaluate this? >> Um that is the never-ending question of creating a test set {slash} eval
set. And so, that's where you definitely just have to sort of get into whatever your problem domain is.
You basically have to spend a lot of time looking at your And that's one of the big benefits of making a custom data set is you get time to just look
at your data itself. Like, what are your inputs? What are your outputs? And it's going to be different for every workflow. So, for example, the um for
the conservation project that I worked for, um there's many millions of bird photos available
online. And so, a lot of them are open source public domain, so we used a lot of those for training. But for a test set, they had a lot of um images
that were taken with their specific cameras. So, our training set was of publicly available bird images. A lot of them are very high quality, 8K
resolution, but for the actual use case, it was low resolution um camera in the wild or um smartphone
camera and the bird is like in the corner versus like main subject. So, we spent a lot of time just crafting a good test set around the actual bird
images people upload to their service. And then that way, whenever we train a model, if we knew it performed well on that, then we could safely deploy
it. Does that make sense? Yeah.
Uh thank you. I just have a question. If this is you of the local one, then how do you update it? And just follow up on the bird one, but if you
have the real world data where the people are taking, you know, in the new stable you have the camera >> Mhm. >> then you have the angle. Why don't
you train on that one instead of you train on the office camera set and then you use >> Mhm. >> With the
local one, how do you monitor production to know how almost the performance? >> Mhm. So, that's a great question. So, um with the bird project, yes,
we did have uh so, we basically use both. That's uh the short answer is um uh these days models can handle almost infinite data. So, if you have more,
it's generally more to put it in. Um and then of course, you use the test set to go, "Hey, if we add
all this extra data, if it hurts it, well, then you can always take it out." Um and then as for how does the model update locally? So, for example, in
the SunnyMed Gemma, um when they we upload a model to Hugging Face, it can also it do like a Git pull basically. Every time it sees a change, it can
pull the new one. But, you for that specific use case, we aren't necessarily capturing any data from
the um from the person's phone. So, Sunny can run it completely offline. So, it can upload uh sorry, renew the model, but it doesn't save any photos
to a database or anything cuz of course, those are private. So, yes, there is a kind of a we're not collecting, I guess, samples to evaluate how it's
actually going in the in the field for that particular use case. Um but that would be as I said, we'd
probably have to do a lot more testing before it become production ready for that particular use case. Yeah. Um so, we'll check out the loss curves.
As I said, look, the training was about 100 seconds. So, quicker than I could answer two questions, um we fine-tuned a model. Whether it performs
well, well, we'll see in a second. So, that's one of the most beautiful loss curves I've ever seen. Um,
but that was only two epochs of training. Um, so that we all saw that happen live. And then if we save the best model to file, and then we're going to
reload both our models and then we can compare them side by side. So, this is the comparison. One of the best things you can do is once you fine-tune
a model, obviously you want to compare your inputs and outputs. So, now we have the base model. We
all saw that before. I typed in my name and it gave some pretty generic response. "Michael is a famous rapper." I mean, I'd probably want to keep that
one. So, um, So, now we have names again. So, Daniel Berg and one of the weird things with language models as well is it's kind of obscured behind if
you're not seeing the training data. But if you I've fine-tuned this out of the into the model, if
you just put in whether my name capital letter or lowercase letter, the previous model will give different responses. Even though we kind of know as
people it's like, "That's the same name." It's just But the model sees capital D and capital B as different tokens to the lowercase version. So, what
comes out? So, this is before and after fine-tuning. So, Daniel Berg here, um, what do we get back?
So, Daniel Berg is a Brisbane-based machine learning engineer instructor at Zero to Mastery, widely known for his self-taught path in the AI industry.
He previously applied AI to healthcare and insurance challenges during his time at Max Kelsen. So, it's a little bit That was for a fair few years ago
now, but this is what it scraped from LinkedIn, right? Today he teaches thousands of students
globally through his courses and popular education platforms like learnpytorch.io. So, that's cool. And then we find that uh as we see, as I said, the
capital version of my name outputs this generic paragraph, but then the lowercase version of my name outputs uh "Daniel Berg is a renowned American
author known for his complex and often controversial writing style." Actually don't mind that
description. Um, but then the fine-tuned model, of course, does what we want it to do. It completes the who's here task. It tells some information
about me. Now, this is my brother. So, William Berg was also influential in the field of computer science with the base model, but now he is a senior
manager in the audit and assurance team at Burl That's a hallucination. That's what happens when you uh
So, it should be BDO, um but I guess this is the uh This is the fun part about fine-tuning models. So, this is back to your question before is like,
how do you guarantee So, in uh in practice, that worked. That put out the fact that he would put be at BDO, but of course in the live demo, it's Burl
Aerospace. So, if anyone here works at Burl Aerospace, that might be in the training set. Um what
should I ask Daniel Berg about? You should ask Daniel about his experience building a self-taught AI master's degree or how we built the Nutrifiy app
using computer vision. He's also a great person to talk about simplifying complex technical topics for a wider audience or his transition from being
an Uber driver to an ML engineer. So, well done. Uh what should I ask William? Uh two hallucinating
on your one. So, you should ask my brother about JS. Okay, so we kind of get the point, right? Um for Elon Musk, uh I wanted it to say, "Hey, Elon's
not here." So, I've added training samples to that. Elon Musk is not here tonight. Maybe check out the other attendees. Um so, we have Shrek. Shrek
doesn't appear on the list. And then we have Michael Schmear. So, this is a little out of date, I
think. So, you're no longer a rapper according to the base model, but the fine-tuned model says you're a senior technical specialist at Microsoft
based in Brisbane specializing in AI machine learning. So, I think that's a bit better. It's a bit better than the base model. Do we have a Lee H?
This was on the Meetup attendees list. I don't have a deep detailed profile for Lee H yet. So, why did it
do that? Well, that's because in the training data, for people's names who weren't a full name, you couldn't search them. I've just told the model,
"Hey, I don't have a full profile for that person. And so, this is where I'm getting to the point of these models are infinitely customizable. So,
whatever use case we'd want, Lee H, if I did have information on that person from the attendee list, then
I could fine-tune the model to do that. But because I don't, I've said, "Hey, just reply back with I don't have detailed information with that." And
then, do we have an Adam Laugh here? Again, these are just randomly sampled. Adam Laugh might not be here, but maybe they're on this uh Meetups
website. For some reason, you have to So, there we go. Apparently, they were they were originally on there.
Okay. So, now we evaluate the model. This is going to be the Rogue L score. So, the Rogue L score basically just met measures similarity. Oh. >> Is
this the same technique um that same Chinese government might use to stop people asking if our ideas about Tiananmen Square something more
sophisticated than just that. >> Probably exact sa- within the realm, like it's it's uh that's what I mean by
these techniques is it's like it kind of it gets painted as this like really sophisticated thing, but it at the end of the day, there's there's
multiple stages to it, but supervised fine-tuning is one of the main steps that basically happens after pre-training. And then there's another step
that we could take even further, which is reinforcement learning. Um and so, basically, that's our reward
training system where it gets a zero or one for producing the output that you want it to do. So, in the Chinese case, um you might have a lot of
samples that go if hey, if someone puts in Tiananmen Square in the prompt, and it responds with um that event doesn't exist or I don't know how it's
supposed to respond, um it'll give it a one for the reward. Um and so, that's But it could also just do a
You could pass in 10,000 samples of hey, asking about this, and then output like this. But that's how a a of these models are trained. >> I also
understand that people have been able to proceed to engineer that. to remove it. So, is it possible to remove the learning that occurs after a model's
been trained? >> Yes. So, um basically, that's kind of what we're doing here with Gemma 3 as well. So,
it's learned a way to respond to Daniel Burke, but I'll just tweak that. And then we could even with the next training run, reverse that. Probably not
exactly to the base model, but to a different use case. So, if I wanted it just every time it replies Daniel Burke, it goes quack like a duck, we
could we could get it to do that. Um so, that's kind of how flexible these models are. If you have a
token space that you want an input, you can create basically whatever tokens that you want to output. I could have got this model to output almost
anything based on whatever names we put in. I just went kind of safe mode and just spit out LinkedIn chat. Um So, when we evaluated how do we go? So,
again, this is a this is really small model. This is 270 million parameters um in today's world. So, I
think maybe the flagship GPT would be close to a trillion. I don't know. There's no official parameter counts on this, but if you just go to the
flagship open source models. So, basically, one would be perfect. We have 85 samples which are an exact match on the test set out of 150. And then
partial, we have 51. So, we have these are the use cases that we would look into and go, "Hey, why aren't
these matching up?" And then uh because text is infinite, we could just easily add in more samples to improve that. And then if it still wasn't
working, then we would look into different kinds of training techniques. Um but for now, this is a very crude experiment of just, "Hey, how do we
fine-tune a model?" This was I did this workflow in a couple of hours. Um all the samples are synthetic.
They're generated by GPT-OSS. So, GPT-OSS 120B Hugging Face based on LinkedIn data. So, if I go, "Who is Daniel Burke?" And then I just copy this. So,
a quick Google search, and then I say um to GPT-OSS, which is a powerful So, powerful open source model, sorry. Generate me five QA pairs for Daniel
Berg based on this paragraph. So, really as simple as this. Um and then it's going to give me back
some synthetic data. There we go. And then I say Well, I can turn it into JSON myself. But, I turn that into JSON, and then that looks exactly like
the data that we trained our model on. And so, that's what we're doing at scale. And I guess this is the same process for whatever your specific use
case is here. Um you can do very similar to this. Of course, it's just going to change based on what
data you want in and out. So, let's go back, and then we'll finish it up. We're just going to make sure we'll upload it to Hugging Face. And I'm going
to put the suffix live, so we all know that it's it's not a a pre-baked model, but this is going to upload to Hugging Face. We could make this private
if we wanted to um if we had an organization just like GitHub, then we're going to create a demo.
And we're going to upload this demo to Hugging Face Spaces again, as we see we've got the live tag there. But, like all good cooking shows, I prepared
something earlier. So, if we go to my profile, we should see the live. Yeah, there's the live version. So, that is building. We're not going to wait
for that to build. That probably take about 10 minutes or so. But, then if we have our Where's the
compare one? Base versus fine-tune. This is one I prepared earlier. So, if we type in Daniel Berg, what do we get back? So, we've seen this we've seen
this kind of before, but this is just a a demo now. So, in the a past few clients that I've worked with, I've basically done this exact workflow.
Create a data set, fine-tune a model, upload a demo like this to Hugging Face that we can explore
easily, just basic inputs and outputs, and then we basically test all of these things, or sorry, go through a bunch of use cases, manually uploading
data, seeing what happens, and then we flag those. I improve the data set, I improve the model, we create version two, and then we get the model ready
for production. So, repeat this ad nauseam. And so, we might we have a name that we wanted to uh try
before. Okay, let's see how the fine-tuned model goes. If it's not in the training data set, it'll still be shocking, but hey. That's the reality of
ML projects, is that uh there's a lot of edge cases, and especially with um LLMs, because the input space and output space is infinite. Brisbane based
professional student currently associated with Saravia and Griffith University. No? All BS? Okay.
Well, we've created a hallucinating model, but at least it um Where's that? Oh, Gary Park. So, So, remember, this is this is a tiny model, but this is
a perfect example of where a hallucination has occurred is because the fine-tuned model has actually learned the structure of what we wanted it to
output, but all the facts are wrong. Now, again, this is probably not a workflow that we would
actually want to do. Trying to get a small model to remember facts is probably not a good use case of fine-tuning. It was just one of the simplest
things that I could demo in two minutes. But, the structure here is exactly how we want it. So, it looks real. Now, how would we get this to be
factual? Well, this is probably where we'd want to build like a rag system of use everyone's names as a
database, but the Gemma 3 fine-tune model just takes those inputs and formats them how we like. So, that would be the next stage of this workflow.
But, for a V1, it's pretty good. Now, um let's go back to the keynote and see Oh, we need to finish with a haiku. So, oh, I said this we'd get this
model running on a phone. Now, this is running on uh an A10G. So, um I did actually get this running on a
phone. But, which is a sophisticated Nvidia GPU, we had to I can't do this one live cuz we have to convert it to MLX because MLX is like Apple's
version of PyTorch. So, we have the 8-bit MLX version here, um and that's 285 MB. So, this one is going to run on a phone, but then if we have the one
that we just trained is 535 36 MB. So, if we deploy it to our phone, I think this is going to play.
There we go. So, this is a screen recording. This app's on my phone. I can show it to you. Um but my phone's recording, of course. So, because it's
running locally, we get really fast tokens per second. So, this is really cool to see. So, if we try another name,
So, that's the beauty of local models. It can run This is I should have put it on airplane mode so you know that it's offline. There we go. We've got
that one correct. And now, we're going to finish with a haiku. Small model, big punch. Train your own or hire me and I'll do it for you. >> I think
it's a pun.
All right. Thank you, Dan. Um it would have been nice if something broke there and we could jump forward. All right. Well, we'll do I'm conscious
of time, so we might do sort of one or two questions and then let everyone get off. And then if I think we can have a little bit more dance time for
further questions, okay? >> All right. For these small ones, are they good for just text-to-text or
they can be multimodal as well, like speech-to-speech, speech-to-text? >> So, just tech So, can you say that again? >> For small ones, are they just
good for text-to-text or they can be multimodal as well, like speech-to-speech, speech-to-text? >> So, so I would say the text modality is definitely
the best by far. The vision modality is basically nearly on par. Um with Sunny, we had almost no
problems with the vision modality. Um but the speech-to-speech is probably the one I am least experienced with. However, that I think is seeing the
most growth in terms of uh models that have been recently published. Um so, would you What would be your use case? You want to generate synthetic
speech? Is that what you mean or transcribe from speech-to-text? Yeah. Uh I see. But um okay, so the
speech-to-text, you can there's several versions of these on Hugging Face right now that you can run state-of-the-art on your device live. The
text-to-speech, uh I don't have enough experience there with the yeah, the inverse of that. >> Uh thanks, man. That was a great talk. The I think
there was something on the fine-tuning data. I think there was 150 people and there was 7,000 seconds. So, I
think you had to twist >> Oh. So, I artificially increased the So, the original 1,000 was uh so, about 120 guests from the meet up page and the
original thousand was about five questions per person plus like random names like Elon Musk or Steve Irwin or something if you type in Steve Irwin
it'll say they're not here. But then I found that wasn't enough data to sort of saturate the model. So then I
just synthetically I guess data augmented the existing samples and gave them whole bunch of different variations of the same sort of question set. So
just upscaled the original base data set of basic QA pairs. And then also I found that lower case would really trip the model up. So I just over
indexed on lower case examples of if you typed in someone's name lower case or typos as well artificial
typos I put that in the data set. So that was V2 just basically upscaling all the original samples. >> All right yeah instead of just training on the
data that you used to train the model let's say for example you know of data that's on the internet which is similar to your image type data. If you
know that model has been trained on data that you didn't use there as fine tuning the model with that
additional data set does that is that is that worthwhile because the purpose I guess is to try and highlight the data that was being split up >> Yeah.
Is that a sensible approach? >> Yeah exactly. So even if a model that has been trained on the entire internet if you have a specific use case it's
generally worthwhile to fine tune on your specific use case cuz you will be quite shocked at how
quickly your base model can get really good at your specific use case if you have like a right data set up. Um so for example like the large models
that we're using like with the just the chat APIs chat interfaces um they are incredibly like large models. They've been trained on the internet. So,
they're operating in basically a zero-shot way every time. So, they're kind of using their large
parameter space to go, "Okay, I think this is what he wants." But, if your use case is specific, maybe you don't need that large generalist
capability. You're just like, "I just need this model to extract this certain data from this certain text the same way every time." >> You referred to
zero-shot models being better. You have you've tried using multiple shots, and they gave the AI to choose the
best of the multiple shots, like multiple shots. >> So, probably the best Yeah, I guess if I understand it correctly, probably the best technique that
I found for just if you're just prompting an API is to if you want the right response back is one it would be examples. So, just examples in the
prompt of like, "Hey, if I give you this input, output like this." So, in the NewsPies app that we
build, which just you take a photo of food, and it analyzes your food. In our prompt for Gemini, we just give it examples of like, "Hey, if you see a
dish like this, output it like that." So, we're giving it examples in the prompt, which is kind of like a mini fine-tuning. So, that's how I would
sort of explain how that works. Does that answer your question? >> Yeah, it answers it. >> Yeah. >>
What's your one more question, guys?
Yeah, very good example. And just wanted to know have you tried to put some evaluation against anything in your app? Like have a pipeline to when
you get more data, the model being more fine-tuned by itself. >> Yeah, so that's that's a great question. So, where does this loop go to next? So,
we've done one like a static fine-tuning here. So, if we were to deploy the this model, which is
terrible, right? We'd get a lot of responses from people going, "Hey, this model's terrible." But, the good news is we'd have those inputs and outputs
maybe tracked. And so, the V3 version of this model would be we would go, "Okay, let's track out if I type in Daniel Burken and it hallucinates, let's
track why did it do that?" And then we'll go, "In the next training run, instead of 8,000 samples,
maybe we'll use 25,000 samples or something like that to really overfit on a certain use case. Um for example, in the Nudge Phi app that we built, you
take photos of food and it just breaks it down. Um we have this exact pipeline in place. So, any photo that gets uploaded of food, um and we review
that in the back end. Mostly models review that. Um and then for the next training run, if we if we
need to deploy a new model, it'll review the sort of mismatches in the production pipeline, and then we'll push out a new version. And so, basically
our data sets and our models are versioned together so that we can track which model was trained on which data set. And so, that when we get
production data, um say model V3 trained on data set V3. Does that make sense? Yeah. >> Awesome. I think we'll
call it there. Dan, that has been an awesome talk and thank you for taking the time to prepare that as well. Um another plug, I don't think there are
enough plugs in there, Dan, to give Dan a call and uh get him to do some work for you. >> Thank you. >> Thank you, Dan. Awesome.