Science & Technology

How to Run LLMs Locally (Great For Learning and Privacy)

📚 Main topics

llama.cpp and GGUFllama.cpp runs models across CPUs, GPUs, and Apple silicon. Its GGUF format combines model weights, tokenizer, and metadata, and supports quantization to help models fit on consumer hardware. 0:00
Ollama for developmentOllama simplifies downloading and running models, and offers an OpenAI-compatible API. It’s a quick way for developers to go from choosing a model to using it in code. 1:02
LM Studio for browsing and chattingLM Studio provides a graphical interface for finding, downloading, comparing, and chatting with models, while showing hardware needs and quantization options. 1:33
Production serving with vLLM and SGLangvLLM uses paged attention and continuous batching to improve serving throughput. SGLang uses Radix Attention to reuse shared prompt prefixes, which can benefit RAG and multi-turn chat workloads. 2:36
MLX-LM on Apple siliconMLX-LM is designed for Macs with M-series chips, where shared CPU and GPU memory can let large models run without the separate GPU memory limits of typical PCs. 4:40

✨ Key takeaways

Choose a tool for your needsUse Ollama for quick developer workflows, LM Studio for an accessible desktop interface, vLLM or SGLang for production traffic, and MLX-LM for Apple silicon. 5:11
Local models are more accessibleLocal inference has become practical on personal hardware, with tools that support everything from experimentation to production serving. 5:42

🧠 Lessons learned

Match the runtime to the workloadLightweight runtimes suit constrained devices, desktop apps make model exploration easier, and serving engines are built to handle concurrent real-world requests. 1:02
Memory and scheduling affect performanceQuantization helps models fit in available memory, while paged attention and continuous batching can improve GPU utilization and throughput. 0:32

🏁 Conclusion/next steps

Start with the recommended toolPick the tool that fits your hardware and goal, then explore the linked AI engineering curriculum for project-based learning on LLMs, agents, RAG, and MCP. 5:42

Transcript excerpt

0:00 Open models like Qwen, Kimi, and the GLM family are now a strong enough that you don't always need a hosted API. You can run them on your own laptop, so no one sees your conversation or data. Here are five tools to run LLMs locally. llama.cpp is a C++ inference engine that runs on CPUs, GPUs, and Apple silicon. It started as a side project to run llama on a MacBook and grew into the foundation most other local tools are built on.

0:32 llama.cpp also introduced a standard file format for local models, GGUF. A GGUF file packs the weights, tokenizer, and metadata into one file, and supports quantization down to 4-bit and lower, which is what makes large models fit on consumer hardware. You download a GGUF from Hugging Face, run llama.cpp and provide the model and your prompt, and you get tokens back. Use llama.cpp when you want the lightest possible

1:02 runtime, or when you are deploying to constrained hardware, like an edge device or a laptop without a dedicated GPU. Ollama is a wrapper around llama.cpp that turns it into a developer tool. It handles model downloads, quantization choices, and starting a local server so you can chat with any LLM. You run Ollama run Gemma 4, it pulls the weights, it starts a local server, and gives you a chat prompt.

🔒 The full, searchable transcript is available with Pro.

Suggestions