Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG
Run a private, high-performance internal AI coding assistant on your own hardware—no GPU required.
Keeping your code and documents on your own infrastructure is the gold standard for enterprise security. By combining Ollama, Qwen Coder, and Open WebUI, we can build a production-grade AI assistant that respects your privacy.
1. System Preparation
First, update your local environment and install necessary dependencies:
sudo apt update && sudo apt upgrade -y
sudo apt install -y curl git ca-certificates lscpu
2. Ollama & Model Tuning
Install Ollama and optimize the configuration for CPU-only inference by setting the thread count:
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl edit ollama
Add these environment variables to the service file (Adjust threads to your CPU cores):
[Service]
Environment="OLLAMA_NUM_THREADS=24"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_KEEP_ALIVE=60m"
Pull the balanced 14B Q4 model for the best speed-to-quality ratio:
ollama pull qwen2.5-coder:14b-q4_K_M
3. Open WebUI Deployment
Using Docker is the easiest way to get the UI and RAG features running:
docker run -d -p 3000:8080 \
--add-host=host.docker.internal:host-gateway \
-e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
-v open-webui:/app/backend/data \
--name open-webui --restart unless-stopped \
ghcr.io/open-webui/open-webui:main
4. Activating RAG (Your Codebase)
Upload your codebase to the Documents section. This allows the model to answer questions specifically about your project. Pro Tip: Exclude /node_modules and /target to keep the indexing fast.
5. Enterprise System Prompt
Define clear architectural rules to ensure the AI generates code that matches your company's standards:
FROM qwen2.5-coder:14b-q4_K_M
SYSTEM """
Expert in Spring Boot 3 & Angular 17.
Follow SOLID and Clean Code.
Use English naming conventions.
Prefer layered architecture (Controller/Service/Repository).
"""
Comments
Post a Comment