🚀 Nova-Rig

An 8× RTX 3090 local-AI inference server — the compute backbone that runs my own LLMs, agents, search, and voice. No cloud.

192 GB of GPU VRAM across eight RTX 3090s on a Threadripper PRO board, tuned to serve 200B-class models locally. It's the machine behind everything I build — an autonomous sales agent, a procurement analyst, an invoicing OS, a video factory, and a voice assistant all run their inference here, on hardware I own, with zero per-token cost.

This repo documents the build: the hardware, the software stack, the optimizations that actually moved the needle, and the failures I hit getting there.

Hardware · Software Stack · Engineering Wins · War Stories · What Runs On It

---

🖥 Hardware

Component	Spec	Notes
GPUs	8× NVIDIA RTX 3090 (192 GB VRAM)	PCIe bifurcation splitter, x8/x8 per slot
CPU	AMD Threadripper PRO	physical cores drive llama.cpp
Motherboard	ASUS Pro WS WRX80E-SAGE SE WIFI	7× PCIe x16 slots + bifurcation
Power	~3150 W PSU budget (3230 W peak)	280–300 W/GPU limit to stay in budget
Idle draw	~230 W (vs ~1100 W under inference load)	after vLLM 0.19 tuning
Networking	Tailscale	tailnet-only access, no public exposure

Warning

Never change PCIe bifurcation from x8/x8 to x16. It kills the splitter and drops both cards on that slot. Learned the documented way.

🧰 Software Stack

Layer	Choice	Detail
MoE inference	ik_llama.cpp	~1.9× faster MoE than mainline llama.cpp
LLM serving	vLLM 0.19.0	PP=6, TP=1; --enforce-eager removed
Agent model	GLM-4.7	served via vLLM for the agent fleet
Flagship model	Qwen 3.5 (122B) / Qwen3.5-REAP-212B	local, no external API
Web search	Vane + SearXNG	self-hosted meta-search + embeddings layer
Web scraping	Firecrawl	self-hosted Docker, structured page extraction
Voice (TTS)	Chatterbox TTS	drives the video pipeline
Monitoring	FastAPI + pynvml + M5Stack	live GPU stats to a hardware display, 1 s TTL cache
Ops	fan100.sh, power-limit + Docker log-limit scripts	keep it cool, quiet, and out of disk-full

Note

Companion node — Nova Jetson (Jetson Orin): Flux image generation runs on a separate Jetson Orin edge box, not on the 3090 rig. The rig handles LLM serving, search, scraping, and TTS.

⚙️ Engineering Wins

The optimizations that actually mattered:

• 1.9× MoE speedup — switched MoE inference to ik_llama.cpp, nearly doubling throughput over mainline llama.cpp.
• Pipeline parallelism over tensor parallelism — PP=6, TP=1 for Qwen3.5-REAP-212B. The model's KV-head count (2) limits TP options, so pipeline parallel wins.
• Idle power cut to ~230 W — vLLM 0.19.0 let me drop --enforce-eager; idle draw fell from prior baselines to ~230 W (vs ~1100 W under load).
• Power-limited to 280–300 W/GPU — keeps 8 cards inside a ~3150 W PSU budget with headroom, at negligible throughput cost.
• 1-second TTL GPU telemetry — the monitor caches pynvml reads so the M5Stack display and dashboards stay live without hammering the driver.

🔥 War Stories

Real failures, real fixes — the stuff that doesn't make it into tutorials:

Symptom	Root cause	Fix
Only 7 of 8 GPUs detected	card seated in bifurcation splitter port 2	reseated the splitter — all 8 came up
Disk suddenly full	a single Docker container wrote an 82 GB log	truncated + added log-size limits in daemon.json
vLLM first inference hung	GPU clock lock (nvidia-smi -lgc) caused a Triton FLA-kernel compilation timeout	removed the clock lock

🧩 What Runs On It

Nova-Rig is the inference layer under a stack of real projects:

• 🔭 ResearchOS — product research pipeline (Vane + Firecrawl + local Qwen)
• 🤖 eve.center — autonomous AI sales agent (GLM-4.7 via vLLM)
• 📊 AutoInvoice — invoicing OS
• 🎬 remotion-fireship — AI video pipeline (Chatterbox TTS here, Flux on the Jetson)
• 🛰️ Hermes — always-on assistant, falls back to local Qwen here when the cloud is down

🔐 Access

The rig is Tailscale-bound — reachable only inside the tailnet, never exposed to the public internet. No ports forwarded, no public IP.

📄 License

MIT — the docs, scripts, and configs here are free to learn from and adapt.