Agent Infrastructure Comparison
ArchivedDEPL_2025.04.05One ReAct agent, four ways to serve an LLM — home GPU, rented cloud GPU, API, or entirely in your browser. Archived; the in-browser agent still runs.
What This Was
The same LangGraph ReAct agent deployed four ways at once — a home RTX 4080, a rented Vast.ai GPU, OpenRouter's API, and the visitor's own browser — answering each query side-by-side while streaming live latency and cost metrics. The three server deployments are torn down now; the in-browser agent above still runs, free, entirely on your device.
Shared Design Decisions
One model across all three server deployments for an apples-to-apples comparison — runs on consumer GPUs, ships as a Q4_K_M GGUF for Ollama (~5.2 GB VRAM), and is available through OpenRouter.
Purpose-built for agents: clean LLM-ready text, 1,000 free requests a month. DuckDuckGo rate-limits scrapers, Brave returns raw JSON, Jina went auth-only.
Capped at 2 searches per query to stop runaway ReAct loops.
The demo capped at 3 concurrent users — Ollama handles that and runs natively on Windows. Production would swap in vLLM for continuous batching; both speak the OpenAI API, so only MODEL_BASE_URL changes.
50 requests per IP per day in front of all four proxies — protects the Tavily quota and GPU compute no matter which backend serves the query.
Cost Comparison
The Four Deployments
Ollama on the home RTX 4080, exposed over HTTPS by a free Cloudflare Tunnel — no port-forwarding, one command to start, $0 per query on hardware that was already there.
GitHub Actions booted the instance at 06:30 and shut it down at 17:00, Sunday–Thursday: near-zero idle cost without serverless cold starts, since no visitor waits 2–3 minutes for a model to load. A startup script rebuilt the whole stack on every boot — restart = redeploy.
No GPU at all — Railway's free tier proxies to OpenRouter and auto-deploys on push. Cost scales per query instead of per hour, and at real scale managed inference can beat paying the engineers who'd maintain self-hosted serving.
The whole ReAct loop runs in a Web Worker on Transformers.js + ONNX: the model emits a web_search action, the worker calls the Nitro → Tavily proxy, feeds the result back as an observation, and loops to a final answer. Desktop tries WebGPU with Qwen2.5-0.5B; mobile and low-RAM devices fall back to WASM with SmolLM2-135M. Nothing but the search query leaves the device.