API bridging
Open WebUI speaks the Ollama API; llama.cpp's llama-server speaks an
OpenAI-compatible API in router mode. LlamaNexus presents both /ollama/api/* and
/openai/v1/* routes and translates between them, so you can point almost any
Ollama- or OpenAI-style client at it.
Distributed inference
Worker machines run the same Docker image in worker mode, exposing their GPU or
CPU to a primary serve instance over llama.cpp's RPC backend. Auto-discovery mode
finds workers over the LAN automatically (UDP heartbeat, restarts the router whenever a worker
appears or drops); manual mode connects to a fixed list instead. Either way, model layer
computation spreads across every connected machine, so inference isn't capped by one box's
memory.
Hugging Face model management
Downloads go through a small Python sidecar talking to huggingface_hub directly,
so real byte-level progress streams back to whatever client is watching - not the vague
all-or-nothing progress you get from letting llama-server auto-pull. Multi-shard
quantizations download and report progress as one batch. Models can be removed entirely
(file + any context-size override) from the WebUI or VS Code's model picker.
Context-size overrides
llama.cpp allocates its KV cache at model-load time, so changing context size normally means
restarting the server by hand. LlamaNexus writes the override into a per-model preset file,
restarts llama-server, and waits for it to come back before your request
proceeds - triggered from Open WebUI's own per-model context setting or the VS Code
extension's context command, no manual restart required.
Licensing and plans
A license key (from your account page) registers a proxy or client install against your plan - Trial, Basic, Pro, or Enterprise, each with its own proxy install limit and its own WebUI/VS Code install limits (tracked independently). A key is required to send chat messages at all; everything else (settings, past sessions, status checks) stays usable regardless. A brief connectivity outage to the license check is forgiving - it's tolerated silently for up to 7 days before anything is flagged, so a temporary network blip or an offline/LAN-only deployment never blocks you.