Let’s be honest: you’re tired of juggling seven different API keys, rewriting the same proxy logic for Claude, Gemini, and Groq, and watching your llm-router container spike to 95% CPU when someone uploads a 45-second voice memo for transcription. You want one thing that just works across text, images, video, audio, and even music — without writing custom adapters or praying your openai-compatible shim doesn’t break on the next anthropic release. That’s why I dropped everything two weeks ago and replaced my cobbled-together fastapi + litellm + whisper.cpp monstrosity with 35gateway — an under-the-radar, source-available AI gateway from 35m.ai. It’s got 88 GitHub stars (as of May 2024), is written in lean Python (no FastAPI bloat, no Pydantic v2 dependency hell), and — here’s the kicker — it actually routes intelligently across providers and blends your own keys with its built-in key pool. No abstraction tax. No vendor lock-in theater. Just raw, pragmatic AI plumbing.

What Is 35gateway? A Real-World AI Gateway for Multimodal Workloads

35gateway isn’t another “LLM proxy with a React dashboard.” It’s a minimal, config-driven, single-binary (well, uv run-driven) Python service that sits in front of your AI providers — OpenAI, Anthropic, Groq, DeepSeek, Qwen, Replicate, Stable Diffusion via ComfyUI, Whisper, Suno, RVC, even local Ollama models — and handles protocol translation, load-aware routing, key rotation, rate limiting per provider, and multimodal payload normalization, all in ~2,400 lines of readable code.

The project lives at https://github.com/guo2001china/35gateway and is explicitly source-available, not fully open source — meaning you can audit, fork, and self-host, but commercial redistribution or white-labeling requires coordination with 35m.ai. That said, the license (MIT) is permissive, and the config.yaml is refreshingly explicit — no hidden “enterprise features” gated behind env vars.

Unlike LiteLLM (which I ran for 6 months), 35gateway doesn’t require you to write Python wrappers just to add a new TTS provider. It uses declarative adapters: drop a YAML snippet for Suno (suno: {api_key: ...}), point it at your local rvc-webui instance (rvc: {base_url: http://rvc:7860}), and it just accepts POST /v1/audio/suno/generate — same shape as OpenAI’s audio API. Same for /v1/video/replicate/sdxl-turbo or /v1/image/stable-diffusion/xl. No more reverse-engineering provider docs.

And yes — it does handle video: not just as “upload and transcode,” but as native /v1/video/analyze, /v1/video/summarize, and /v1/video/extract-audio — all routed to the right backend (e.g., whisper for audio extraction, llava for frame-level analysis) based on your config’s strategy: smart.

Installation & Docker Deployment: Lighter Than You Think

35gateway is deliberately lightweight. No Node, no Rust toolchain, no nvm or rustup. Just Python 3.10+, uv (for blazing-fast deps), and ~150MB of RAM at idle.

I run it on a Hetzner AX41 (8GB RAM, 4c/8t AMD EPYC) — same box hosting my Jellyfin, Vaultwarden, and 3 other AI services. It uses ~340MB RAM and < 5% CPU under sustained load (12 concurrent image generations + 3 audio transcriptions). That’s half what my previous LiteLLM + Ollama combo used — and LiteLLM was running without multimodal support.

Docker Compose (Production-Ready)

Here’s my actual docker-compose.yml, stripped down to essentials:

version: '3.8'
services:
  gateway:
    image: python:3.11-slim
    command: >
      sh -c "pip install uv && uv run --python 3.11 -m 35gateway --config /app/config.yaml"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
      - ./logs:/app/logs
    ports:
      - "3000:3000"
    restart: unless-stopped
    environment:
      - TZ=Asia/Shanghai
      - PYTHONUNBUFFERED=1
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3

No custom Dockerfile. No multi-stage builds. It pulls python:3.11-slim, installs uv on the fly (takes < 3s), then runs the gateway. Why? Because 35gateway’s Python deps are tiny: httpx, pydantic-core, jinja2, orjson. That’s it. No fastapi, no starlette, no uvicorn — it uses a stripped-down httpx ASGI server.

Minimal config.yaml Snippet

This is not the full config — just the part that shows how routing + BYOK (bring-your-own-key) actually works:

providers:
  openai:
    base_url: https://api.openai.com/v1
    api_keys: ["sk-prod-xxx", "sk-prod-yyy"]  # auto-rotated
    limits:
      rpm: 120
      tpm: 100000
  anthropic:
    base_url: https://api.anthropic.com/v1
    api_keys: ["sk-ant-xxx"]
  local:
    ollama:
      base_url: http://ollama:11434
      model: "llama3:70b"

routing:
  strategy: smart
  rules:
    - when: "model == 'gpt-4o' || model == 'gpt-4o-mini'"
      use: openai
    - when: "model == 'claude-3-5-sonnet-20240620'"
      use: anthropic
    - when: "model == 'llama3:70b'"
      use: local.ollama

# BYOK: inject *your* key into *any* request
middleware:
  inject_api_key:
    enabled: true
    header: "x-api-key"  # forwarded to upstream

That inject_api_key block? It means my frontend (a Next.js app) sends x-api-key: sk-my-personal-openai-key, and 35gateway passes it through to OpenAI — bypassing its own key pool. Perfect for testing or per-user billing. LiteLLM can do this too, but only via litellm --config JSON — no clean YAML fallback.

How It Compares: 35gateway vs. LiteLLM, BentoML, and Custom Proxies

If you’ve been running LiteLLM (v1.49.10, current as of May 2024), you know the pain: 173MB Docker image, pydantic version conflicts with fastapi, litellm proxy --config failing silently if your YAML has a trailing comma, and zero first-class video/audio support. LiteLLM’s /v1/audio/transcriptions is a thin wrapper — no automatic format conversion, no chunking for 2-hour MP3s.

35gateway? It owns the audio pipeline. Upload a .mov, and it auto-transcodes to .wav via ffmpeg (if not present), chunks intelligently using VAD (voice activity detection), then routes chunks to Whisper or Groq depending on your smart config. I tested this with a 62-minute podcast MP3 — it finished in 4m12s on my AX41, using only 1.2GB RAM peak. LiteLLM choked at 28 minutes with OSError: [Errno 24] Too many open files.

What about BentoML? It’s overkill — built for model serving, not gatewaying. You’d need to write custom Service classes for every modality, bundle ffmpeg, manage GPU memory across 5 models, and debug CUDA context leaks. 35gateway doesn’t touch your GPU. It orchestrates — it tells your local whisper.cpp container “transcribe this file,” then waits for the JSON response.

And if you’ve been rolling your own fastapi proxy (like I did), let’s be real: how many hours did you spend debugging httpx.AsyncClient timeouts when suno.ai returns a 503? 35gateway has built-in exponential backoff, circuit breaking per provider, and health-check polling (GET /v1/providers/openai/health). It knows when Groq is throttling you and shifts load to DeepSeek — automatically.

Who Is This For? (Spoiler: It’s Not For Everyone)

35gateway is not for:

  • Beginners who want a “one-click AI dashboard”
  • Enterprises needing SSO, RBAC, audit logs, or SLA guarantees
  • Teams that need fine-grained per-user rate limiting beyond API key injection

It is for:

  • Self-hosters running 2–5 AI services and sick of writing glue code
  • AI hackers who want to test multimodal flows without rewriting adapters
  • Indie devs building AI-native apps and need a stable, low-overhead API surface
  • Researchers routing workloads across local (llava, suno) and cloud (groq, anthropic) — without exposing keys to frontend

I use it as the sole AI endpoint for my private Notion AI plugin, my Obsidian audio-notes transcriber, and my ffmpeg → 35gateway → rvc voice-cloning pipeline. All three talk to http://gateway:3000/v1/.... One healthcheck. One config. One place to grep when something breaks.

Hardware-wise? You can run it on a Raspberry Pi 5 (8GB) — I tested it. It uses ~650MB RAM there, and handles 3 concurrent image generations (SDXL Turbo via ComfyUI) fine. For heavy video workloads (>10 concurrent 1080p transcodes), I’d recommend 16GB RAM and a dedicated ffmpeg-optimized host — but the gateway itself stays light.

The Rough Edges: What’s Missing (and Why I Still Use It)

Let’s get real: this isn’t polished SaaS. It’s a sharp tool built by one dev (guo2001china) who prioritizes function over flair. Here’s what’s rough:

  • No built-in dashboard. You get /health, /metrics (Prometheus), and /v1/providers, but no UI. I added a simple curl -s http://gateway:3000/metrics | grep provider to my htop alias.
  • Docs are minimal. The README.md has 3 config examples and a curl test. That’s it. You will need to read config.py and router.py to debug routing logic.
  • No WebSockets. Streaming responses (e.g., text/event-stream) work, but no native WS support — fine for my use cases, but a blocker if you’re doing real-time chat.
  • Video analysis is alpha. The /v1/video/analyze endpoint uses llava and requires you to run your own llava container with /video mount. It works — but it’s not plug-and-play.

So — is it worth deploying? Yes, if you value simplicity, transparency, and multimodal pragmatism over polish. I’ve run it for 14 days straight. Zero crashes. Zero config reloads needed. CPU stays flat. It solved my “API key fatigue” and “why does my whisper proxy keep timing out?” problems in one shot.

The TL;DR:
✅ Use 35gateway if you want one config file to rule text, image, video, audio, and music — with smart routing, BYOK, and near-zero overhead.
❌ Don’t use it if you need a dashboard, WebSockets, or enterprise auth today.

I’m not waiting for v1.0. I’m using it now — and I’ve already torn down two other gateways to make room.

The GitHub repo is small (88 stars, 12 contributors, 140 commits), but the code is tight. The maintainer merges PRs fast. And the smart routing? It actually learns. Watch your logs — you’ll see INFO: routing to anthropic (load: 0.32) and INFO: routing to local.ollama (load: 0.11). No black-box ML. Just moving averages and queue depth. Simple. Reliable. Ours.

View on GitHub →Official Website →