Builder & operator (personal lab) · 2026–present
Home Lab — a self-hosted AI compute fleet
A three-node local AI cluster unified behind one gateway — heterogeneous hardware (server GPU + laptops) turned into a single auto-routing inference pool, run with real reliability engineering — benchmarks, monitoring, incident postmortems, and read-only safety rails.
What it is
A self-hosted AI compute fleet I built to run large-language-model and diffusion workloads locally, on hardware I own, instead of renting cloud inference. The interesting part isn't that it runs models — it's that a pile of mismatched machines is operated like one reliable system: benchmarked, monitored, versioned, and fenced with the same discipline I'd expect in production.
The fleet
Three nodes, deliberately heterogeneous, each earning its keep:
- A server-class box with a 24 GB GPU (Tesla P40) and 128 GB RAM — the fast, reliable inference lane and coordinator host.
- A laptop with an Intel Arc iGPU — large-but-slower interactive overflow.
- A 2019 MacBook Pro (i9) — embeddings and tiny-model overflow.
Node roles weren't guessed. Each machine's real throughput was benchmarked (tokens/sec per model) and the results drive a measured "optimal loadout" per node rather than an assumption.
How it's unified
- One OpenAI-compatible gateway (LiteLLM) fronts the whole fleet with latency-based routing, per-tier retries/cooldowns, and explicit fallback chains. Any job is handed to the gateway by capability tier (quick / chat / coder / heavy / reason / vision / translation) and routed to whichever node serves it fastest — one auto-optimizing pool out of mismatched hardware.
- Self-hosted runtimes: Ollama, llama.cpp (
llama-serverrunninggpt-oss-20bas the GPU/CPU fast lane), IPEX-LLM for the Intel Arc iGPU, and an MLX sidecar on the Mac. - On-demand local diffusion on the GPU node — Qwen-Image, FLUX.1-schnell, and Wan2.1/2.2 video via ComfyUI — behind a GPU-arbitration script that stops the LLM stack and brings up ComfyUI (the 24 GB card can't hold both), then reverses it when I'm done. The Pascal-era card needed FP32 workarounds, which I found and documented.
Run like production, not a toy
This is where it stops being a hobby:
- Monitoring: Prometheus (node-exporter across all three nodes + a GPU exporter) into Grafana, plus an always-on health dashboard that tiles status by node and role.
- Self-healing & alerting: a node-down (
up==0) alert, a dead-man canary on the always-on pipeline, and a watchdog with auto-restart + backoff on the vector store — all added after a real 18-hour undetected node outage, whose postmortem (timeline, root cause, a graded prevent/detect/contain table) I wrote and then implemented. - Read-only safety rails: a dedicated observe-only review layer — a versioned inventory manifest, a drift sentinel that flags any unexpected new/stopped/unhealthy service, a re-verify harness that re-runs checks on a cadence, and an off-box dead-man heartbeat so failure of the alerting channel itself is caught. Cross-node access is limited mechanically (a forced-command SSH wrapper + scoped sudoers, not raw docker-group root) — a direct response to an incident where an automation agent with too much access ran a mutating command it shouldn't have.
- Config as code: every fleet config and decision is versioned in a self-hosted Gitea — diffable and reviewable, never live-only on a box or lost in chat history.
- Self-hosted services: Gitea for git, a Qdrant vector database (local
nomic-embed-textembeddings) as a RAG substrate, and a purpose-built read-only MCP server that exposes vetted fleet-health and knowledge-base tools to a chat agent.
Why it's here
It's the clearest proof of how I work when nobody's grading it: benchmark before deciding, monitor everything, write the postmortem, fence the sharp edges, and treat "runs on my hardware" as a reliability problem, not a bragging right. It's also where I understand hardware and software deeply enough to troubleshoot a misbehaving machine — and to know when to fix it myself versus escalate.