I'm thrilled to introduce this course on fast and efficient LLM inference, built in partnership with Red Hat, and taught by Sergey Kliger. Deploying
open source LLMs efficiently, so they can serve many users at once with low latency and reasonable cost, is challenging. In this course, you'll learn
how to take a large language model and serve it efficiently using vLLM, a widely adopted and very
popular open source serving system. A 70 billion parameter LLM might take about 140 gigs of memory just for the weights. Additionally, the KV cache
values, which store token information, also take up significant memory. So, a model like that might require multiple GPUs to serve a single request.
In this course, you'll learn techniques that help you manage that critical memory [music] efficiently.
For example, you apply quantization to shrink the memory footprint of the model, [music] which also speeds up how the data moves through memory. You
also see how vLLM's paged attention manages the model's memory, especially the KV cache at runtime, so your model can handle many concurrent requests.
Building on that, you'll learn how prefix caching vLLM uses previously computed values when requests
share the same system prompt, so the model doesn't redo work that's already done. >> To put all of this into practice, you'll run the optimized deploy
benchmark workflow on your own model. We'll start with quantization, where you'll learn how to compress the model without sacrificing accuracy. And
you'll learn exactly what a KV cache is and why it's important, and how to serve it with vLLM and
watch paged attention and prefix caching in action. And finally, you'll benchmark your deployment by simulating real-world traffic and measuring how
well your model performs with metrics like latency and throughput. Along the way, you'll learn how to navigate the tradeoffs between speed, cost, and
accuracy that come with every deployment decision. >> The techniques you learn in this course are
what power efficient LM serving in production today. And you also learn about VLMs, which is an important piece of AI infrastructure today. So, I hope you enjoy the course.