Science & Technology

Building a home AI server from used parts

by Ice Tea Admin

📚 Main Topics

Project Overview
- Building a budget-friendly home AI server using used parts.
- Goal: Run local AI models for various tasks without subscriptions.
Hardware Selection
- Chose AMD's AM4 platform due to availability of used parts.
- Key components:
  - MotherboardROG Strix B550E Gaming
  - CPURyzen 9 3900X (12 cores, 24 threads)
  - RAM16 GB DDR4
  - Storage512 GB Samsung 960 EVO NVME SSD and a 120 GB SATA SSD
  - GPUsTwo Intel ARC A770 cards (16 GB VRAM each)
  - Power SupplyColink Continuum 1,200 W
  - CaseNCXT H710i
Operating System Installation
- Installed both Windows and Ubuntu to compare performance.
- Initial attempts to run AI models using LM Studio faced issues.
Software and Model Testing
- LM Studio was found to be unreliable; switched to Llama CPP for better control.
- Explored GPU backends: Vulcan and SLE (Intel's answer to CUDA).
- Successfully ran models using the SLE backend after initial failures.
Performance Benchmarking
- Conducted benchmarks using Llama Bench on both Windows and Linux.
- Results showed:
  - Linux was faster for prompt processing.
  - Windows had better token generation speeds.
  - Vulcan backend was faster in most cases but less stable.

✨ Key Takeaways

Building a home AI server can be done affordably with used parts.
Understanding hardware and software compatibility is crucial, especially with Intel GPUs.
Performance can vary significantly between different operating systems and software backends.
The experience provided valuable insights into the workings of AI models and server setups.

🧠 Lessons Learned

Don't rely solely on popular applications; sometimes, direct interaction with underlying software (like Llama CPP) is necessary for success.
Be prepared for troubleshooting and learning curves when working with less common hardware configurations.
Future upgrades may be necessary to run larger models effectively, highlighting the importance of scalability in hardware choices.

🏁 Conclusion

The project successfully demonstrated the feasibility of running local AI models on a budget, while also emphasizing the importance of understanding both hardware and software in achieving optimal performance. Future videos will explore further upgrades and enhancements to the system.

localai localllm homelab ai llm llamacpp intel intel arc intel arc a770 arc a770 a770

Transcript excerpt

0:00 Hello and welcome to my first YouTube video. In this video, I will show you how I built a home AI server with only used parts. The goal of this project was to build a budget friendly system that would still be powerful enough to run local AI models for coding, image generation, and other tasks. Like many of you, I had tried popular AI services like Chat GPT and Claude. But this was my first time trying to run AI on my own hardware. As you will see later on, some of the

0:32 choices I made caused a lot of headaches, but I learned a lot and in the end, I was able to run state-of-the-art local AI models at home without any subscriptions. I will show you the hardware and software that I used, what worked and what didn't, and some benchmarks at the end. So, without further ado, let's start by looking at the hardware. After browsing the used hardware market, I decided to build my system around AMD's AM4 platform. It has a large user base and because

1:04 people have started upgrading to AM5, they're now selling their old AM4 boards. I found a good deal on a bundle containing a motherboard, CPU, 16 GB of RAM, and an NVME SSD for around $400. The motherboard is the ROG Stricks B550E Gaming which features three PCIe X16 slots and supports PCIe4 which is important for modern GPUs.

🔒 The full, searchable transcript is available with Pro.

Suggestions