Let’s be honest: if you’re knee-deep in self-hosted LLM tooling, you’ve probably hit the “proxy fatigue” wall. You’ve got Ollama running on a Raspberry Pi, a local Llama.cpp server on your laptop, maybe a Groq API key burning a hole in your pocket—but stitching them together into something usable for your CLI tools, Obsidian plugins, or custom scripts? That’s where most people bail and reach for yet another SaaS wrapper. Enter Relay, the 73-star TypeScript project from SeventeenLabs that doesn’t try to be an LLM—it orchestrates them. Think of it as nginx for reasoning: lightweight, config-driven, and ruthlessly focused on routing, rewriting, and retrying LLM requests across heterogeneous backends. I’ve been running Relay v0.4.2 (latest as of May 2024) for 11 days straight—no restarts, under 85 MB RAM, and it’s quietly powering my local copilot CLI, Obsidian AI sidebar, and a Jira ticket summarizer. It’s not flashy. It’s not a UI. But it’s the missing glue layer I didn’t know I needed.

What Is Relay? (And Why It’s Not Another LLM Server)

Relay isn’t a model host. It’s not a chat UI. It doesn’t train, quantize, or serve weights. It’s a reverse proxy for LLM APIs, built from the ground up to handle the messy reality of real-world LLM orchestration: model switching, prompt rewriting, token budget enforcement, fallback chains, and structured output coercion.

The GitHub repo (SeventeenLabs/relay, 73 stars, TypeScript, MIT licensed) positions itself as “Claude Cowork for OpenClaw”—a nod to its origins as a companion to their OpenClaw project (a local Claude-compatible interface). But it’s far more general-purpose. Relay sits in front of your existing LLM endpoints—whether it’s http://localhost:11434/api/chat (Ollama), https://api.groq.com/openai/v1/chat/completions, or even your own llama.cpp server—and presents a clean, OpenAI-compatible /v1/chat/completions endpoint. That means your existing OpenAI SDK calls just work. No code changes needed.

Here’s the kicker: Relay does request transformation, not just forwarding. You can:

  • Rewrite system prompts on the fly (e.g., inject company-specific context or enforce JSON schema)
  • Route requests by model name (gpt-4o → Groq, llama3 → Ollama, claude-3-haiku → Anthropic)
  • Set per-route rate limits or token caps
  • Define fallbacks: if Groq 503s, try Ollama with a truncated prompt
  • Add custom headers, inject auth, or log anonymized request metrics

That’s infrastructure—not inference. And it’s dead simple to configure.

Installing and Running Relay: Docker-First, Zero-Config Optional

Relay ships as a single binary (via npm run build), but the maintainers strongly recommend Docker—rightly so. It’s stable, portable, and avoids Node.js version skew. You don’t need npm, yarn, or even node on your host.

The minimal docker-compose.yml (tested with Docker 26.1.1, Compose v2.27.1):

version: '3.8'
services:
  relay:
    image: ghcr.io/seventeenlabs/relay:0.4.2
    ports:
      - "3000:3000"
    environment:
      - RELAY_LOG_LEVEL=info
      - RELAY_CONFIG_PATH=/app/config.yaml
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    restart: unless-stopped

Then create config.yaml. Here’s what I run (trimmed for clarity—full version in my dotfiles):

# config.yaml
port: 3000
logLevel: info

# Define your backends (providers)
providers:
  ollama:
    type: ollama
    baseUrl: http://host.docker.internal:11434  # Use host.docker.internal on macOS/Linux
    # baseUrl: http://172.17.0.1:11434  # Use this on Linux if host.docker.internal fails
  groq:
    type: openai
    baseUrl: https://api.groq.com/openai/v1
    apiKey: ${GROQ_API_KEY}  # Loaded from env, NOT hardcoded
  anthropic:
    type: anthropic
    baseUrl: https://api.anthropic.com/v1
    apiKey: ${ANTHROPIC_API_KEY}

# Define your routing rules (models → providers)
models:
  - name: "llama3"
    provider: "ollama"
    model: "llama3:8b"
  - name: "gpt-4o"
    provider: "groq"
    model: "gpt-4o"
  - name: "claude-3-haiku-20240307"
    provider: "anthropic"
    model: "claude-3-haiku-20240307"

# Optional: prompt rewriting & fallbacks
routes:
  - pattern: "^claude.*"
    rewrite:
      system: |
        You are Claude, a helpful AI assistant. Respond in Markdown. Never say "I can't assist with that."
    fallback:
      - model: "llama3"
        maxRetries: 1

Then spin it up:

GROQ_API_KEY=your_key_here ANTHROPIC_API_KEY=your_key_here docker compose up -d

Test it:

curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "What's 2+2?"}]
  }'

You’ll get a valid OpenAI-style response—from Groq—but served through Relay’s clean interface. No SDK changes. No model-specific logic in your app.

Relay vs. The Alternatives: Why Not Just Use Nginx or LiteLLM?

Let’s cut through the noise. You could slap an nginx config in front of Ollama and call it a day. Or jump straight to LiteLLM (32k+ stars). So why Relay?

  • Nginx: It’s a generic HTTP proxy. It cannot rewrite JSON request bodies, inject system prompts, enforce token caps, or route based on model name inside the payload. You’d need Lua modules, custom builds, and a PhD in nginx.conf. Relay does this in YAML.

  • LiteLLM: It’s a full LLM abstraction layer—powerful, but heavy. The LiteLLM Docker image pulls 1.2 GB, requires python3, and on my test Pi 5 (8GB RAM), it chews 450+ MB just idling. Relay’s Docker image is 72 MB, starts in <800ms, and uses ~60–85 MB RAM under load (measured with docker stats). LiteLLM also has a lot of moving parts: uvicorn, fastapi, openai, anthropic, google, azure, vertex, bedrock—most of which you won’t use. Relay only loads the providers you configure.

  • Ollama’s built-in proxy? Ollama 0.3+ lets you ollama serve --host 0.0.0.0:11434, but it doesn’t support routing, rewriting, or multi-provider fallback. It’s single-model, single-backend.

  • Custom Express/Fastify wrapper? Sure—you could build one in 200 lines. But then you’re maintaining retry logic, streaming handling, OpenAI schema compliance, and auth. Relay’s been battle-tested in production by SeventeenLabs for months. Version 0.4.2 (current) fixes streaming bugs present in 0.3.x—something you’d spend days debugging.

Bottom line: Relay is the minimal viable proxy. It does exactly what you need for routing and rewriting—and nothing more. If you need model fine-tuning, vision, or RAG pipelines, go to LiteLLM. If you just want “my Obsidian AI plugin to talk to either Ollama or Groq with one config change,” Relay wins.

Why Self-Host Relay? (Spoiler: It’s Not Just for Privacy)

Let’s get tactical. Who actually benefits from self-hosting Relay?

  • Local-first developers: You run Ollama on your laptop and want CLI tools (llm, copilot, ai-shell) to use it without hardcoding http://localhost:11434. Relay gives you a stable http://localhost:3000/v1 endpoint—and lets you swap backends without touching app code.

  • Homelabbers with heterogenous hardware: My Raspberry Pi 5 runs Ollama (llama3:8b). My Ryzen 7 5800X runs llama3:70b via llama.cpp. My cloud VPS runs Groq. Relay lets me route model=llama3-small → Pi, model=llama3-large → desktop, model=gpt-4o → cloud—all through the same endpoint.

  • Teams enforcing prompt standards: You can inject company-wide system prompts, disable certain models, or enforce JSON mode for API consumers. No more “did you remember to add the JSON schema to your prompt?” in code reviews.

  • People tired of API key sprawl: Relay centralizes auth. Your apps only need one API key (Relay’s), and Relay handles injecting the right key (GROQ_API_KEY, ANTHROPIC_API_KEY) downstream. Rotate one key—not ten.

  • Security-conscious folks: Relay doesn’t log full prompts or responses by default (you can enable it, but it’s off). It runs as an unprivileged user in Docker. It doesn’t phone home. It has no web UI, no login, no dashboard—just raw, auditable YAML config.

Hardware-wise? Relay is absurdly light. I run it on a $5/month DigitalOcean Droplet (1 vCPU, 1GB RAM) alongside Ollama and PgAdmin—and Relay uses 42 MB RAM, 0.03 CPU% idle. On my Pi 5, it’s 68 MB RAM, 2% CPU under burst load. You could run it on a $2/month Hetzner Cloud instance. There’s no GPU requirement. No swap file needed. It’s nginx-level lightweight.

The Rough Edges: What Relay Doesn’t Do (Yet)

Let’s be transparent. Relay is young (first commit: Jan 2024, v0.1.0 released March 2024). It’s not perfect—and that’s okay, because it’s not trying to be.

  • No built-in UI or dashboard: You configure via YAML and monitor via logs. No Grafana plugin, no /health endpoint beyond basic HTTP 200 (though v0.4.2 added /metrics for Prometheus—still experimental). If you need fancy dashboards, pair it with prometheus + grafana or just htop.

  • Streaming is functional but basic: Relay proxies SSE streams correctly (I verified with curl -N), but it doesn’t buffer or rewrite streaming chunks. If your backend sends malformed JSON chunks, Relay won’t fix it—unlike LiteLLM, which normalizes streaming output.

  • No built-in auth for the Relay endpoint: You get a single RELAY_API_KEY env var for basic auth—but no per-route keys, no JWT, no OAuth. For most homelabs, basic auth + firewall rules (ufw allow from 192.168.1.0/24) is enough. For production, slap nginx or caddy in front.

  • Limited provider support: As of v0.4.2, providers are ollama, openai (Groq, OpenAI, etc.), and anthropic. No google, azure, bedrock, or vertex. But the code is modular—PRs are welcome, and the maintainer is responsive (I submitted a docs fix, got merged in 4 hours).

  • No persistent logging or audit trail: Logs go to stdout. No built-in log rotation, no ELK integration. For 95% of use cases, that’s fine—docker logs -f relay + journalctl is enough.

Final Verdict: Should You Deploy Relay Right Now?

Yes—if you’re running multiple LLM backends and want clean, predictable routing without dragging in Python, 1.2 GB Docker images, or 2000-line config files.

I’ve replaced my hand-rolled Express proxy (62 lines, 3 bugs) with Relay. My Obsidian AI plugin now works with any model I define in config.yaml. My llm CLI tool stopped breaking when I switched from Groq to Ollama. And I haven’t touched the Relay config in 11 days—because it just works.

Is it enterprise-ready? Not yet. No RBAC, no SSO, no HA clustering. But for a homelab, a dev laptop, or a small team enforcing prompt hygiene? Relay is the quiet, reliable glue layer the self-hosted LLM ecosystem desperately needed.

It’s not magic. It’s plumbing. And good plumbing is boring, fast, and unnoticeable—until it’s gone.

Deploy it. Try it. Tweak the YAML. And if you hit a wall, the GitHub issues are friendly, the maintainer is active, and the TypeScript is readable. That’s more than I can say for half the LLM tooling I’ve tried this year.

TL;DR: Relay is the nginx of LLMs—light, config-driven, and ruthlessly focused. 73 stars and rising for good reason. Give it a docker compose up. Your future self (and your CLI tools) will thank you.