Desktop automation MCP server β€” computer use for any AI agent. See the screen, control windows/mouse/keyboard, and drive Chrome over the Model Context Protocol (stdio). Desktop & browser automation need no API key; OCR runs locally; vision plugs into your own vision LLM (OpenAI-compatible or Anthropic native, BYOK).

nuphus-mcp is a lightweight, cross-platform desktop automation MCP server that exposes desktop + browser automation as standard MCP tools. It speaks JSON-RPC 2.0 over stdio β€” no daemon, no network service, one binary. Claude Desktop, Cursor, VS Code, Copilot, or any MCP client can connect and immediately control the screen, windows, keyboard/mouse, and Chrome β€” computer use for any AI agent β€” desktop & browser automation need no API key; local OCR is built in; vision works with your own vision LLM (OpenAI-compatible, BYOK).

πŸ‡¨πŸ‡³ Mainland China mirror: this repo is mirrored on Gitee for fast in-China access (Chinese docs served by default there). δΈ­ζ–‡ζ–‡ζ‘£

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   stdio JSON-RPC   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Any MCP Client  β”‚  ───────────────►  β”‚      nuphus-mcp      β”‚
β”‚  (Claude/Cursor/ β”‚  ◄───────────────  β”‚  desktop-api crate   │──► screen/window/mouse/keyboard
β”‚   Nuphus itself) β”‚  single-line JSON  β”‚  nuphus-browser crate│──► Chrome (CDP)
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                    β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Features

  • 36 MCP tools (15 desktop + 21 browser) β€” screenshots, window control, mouse/keyboard, Chrome CDP automation, and more β€” see TOOLS.md / TOOLS.zh-CN.md for the full reference.
  • Desktop automation: screen size, screenshot (PNG/base64), window list, window activate/screenshot/move/resize/info, mouse click/drag/scroll/position, keyboard input/hotkey, clipboard write/clean β€” implemented on the desktop-api crate (xcap + Win32, no Tauri dependency).
  • Computer vision pair: desktop_vision (BYOK β€” send a screenshot to your own vision model via an OpenAI-compatible or Anthropic native API) + desktop_perceive (local OCR with PaddleOCR, models auto-downloaded on first run; optional YOLO icon detection). Used together they give AI agents both semantic understanding and pixel-precise coordinates β€” the battle-tested visionβ†’perceive flow from the Nuphus desktop app. See TOOLS.md for BYOK env vars, model setup, and the recommended flow.
  • Browser automation: navigate, snapshot (accessibility tree with @N refs), click, type, exec, scroll, extract, screenshot, evaluate, back/forward, wait_for, cookies get/set/import, upload, tabs, downloads β€” implemented on nuphus-browser (chromiumoxide CDP).
  • Zero-cost stdio: no HTTP server, no daemon. The process reads single-line JSON from stdin and writes responses to stdout.
  • Safety-first: destructive tools are annotated per the MCP spec; optional strict-confirm mode; path validation for screenshot/upload.

Repository Layout

nuphus-mcp/
β”œβ”€β”€ Cargo.toml                  # workspace root
β”œβ”€β”€ TOOLS.md / TOOLS.zh-CN.md   # 36-tool reference
β”œβ”€β”€ crates/
β”‚   β”œβ”€β”€ nuphus-mcp/             # MCP Server (this repo's product)
β”‚   β”œβ”€β”€ nuphus-browser/         # Browser automation core (CDP)
β”‚   └── desktop-api/            # Desktop control core (vendored)
└── ...

Prerequisites

  • Rust toolchain (stable) β€” build from source with Cargo.
  • Chrome or Edge β€” required for browser tools. The server auto-detects an installed browser; if none is found, browser_* tools return a clear error.
  • Windows recommended for full desktop control β€” see Platform Support below.

Platform Support

Platform Browser tools Desktop tools
Windows Full Full (Win32 API)
macOS Full Desktop input requires Accessibility permission (System Settings β†’ Privacy & Security β†’ Accessibility)
Linux Available Partial β€” window/input capabilities are limited

API Keys & Local Models

Vision β€” BYOK, OpenAI-compatible or Anthropic native

desktop_vision uses your own vision model. It speaks two protocols:

  • OpenAI-compatible Chat Completions (default) β€” works with OpenAI, MiniMax, Qwen, Ollama, vLLM, …
  • Anthropic native Messages API β€” point NUPHUS_MCP_VISION_BASE_URL at https://api.anthropic.com/v1 and the protocol is auto-detected from the host; or force it with NUPHUS_MCP_VISION_PROVIDER=anthropic.

Nothing is required unless you call this tool β€” and when it is not configured the tool returns a clear error instead of silently failing.

Environment variable Required Default Description
NUPHUS_MCP_VISION_API_KEY βœ… β€” API key for your vision model
NUPHUS_MCP_VISION_BASE_URL β€” https://api.openai.com/v1 Base URL (https://api.anthropic.com/v1 for Claude)
NUPHUS_MCP_VISION_MODEL βœ… β€” Model id, e.g. gpt-4o-mini, qwen-vl-max, claude-sonnet-4-5
NUPHUS_MCP_VISION_PROVIDER β€” auto auto | openai | anthropic; auto infers from the base URL host
NUPHUS_MCP_VISION_MAX_TOKENS β€” 1024 Max output tokens (Zhipu GLM-4V-Flash caps at 1024; raise for text-heavy screenshots)

External browser (anti-detect / fingerprint browsers)

By default browser_* tools launch and manage their own Chrome instance. To drive an external browser instead β€” e.g. an anti-detect / fingerprint browser β€” start it with a debugging port and point the server at it:

Environment variable Required Default Description
NUPHUS_MCP_BROWSER_CDP_URL β€” β€” External CDP endpoint, e.g. http://127.0.0.1:9222 
# Example: start your fingerprint browser with a debugging port
chrome --remote-debugging-port=9222 --user-data-dir=...

// MCP client config
"env": { "NUPHUS_MCP_BROWSER_CDP_URL": "http://127.0.0.1:9222" }

When set, browser_* tools attach to that endpoint and never launch a managed Chrome; attach failures are hard errors (no silent fallback into the wrong browser). The external browser belongs to you β€” the server never kills it on exit.

Self-healing on window reopen (recommended)

Fingerprint browsers typically get a new random debug port every time a window is reopened, which would leave a fixed ..._CDP_URL pointing at a dead port. Provide the browser identity and the server re-resolves the live port automatically on the next tool call:

Environment variable Required Default Description
NUPHUS_BROWSER_EXE_PATH for self-healing β€” Absolute path to the browser exe β€” the identity key used to locate the running window process
NUPHUS_BROWSER_NAME β€” ζŒ‡ηΊΉζ΅θ§ˆε™¨ Display name used in error guidance
NUPHUS_BROWSER_USER_DATA_DIR β€” β€” Profile dir, used to read DevToolsActivePort when the window was started with a random port (--remote-debugging-port=0) and the process cmdline does not reveal it 
// MCP client config β€” attach + self-heal
"env": {
  "NUPHUS_MCP_BROWSER_CDP_URL": "http://127.0.0.1:9222",
  "NUPHUS_BROWSER_EXE_PATH": "C:\\path\\to\\fingerprint-browser.exe",
  "NUPHUS_BROWSER_NAME": "AdsPower",
  "NUPHUS_BROWSER_USER_DATA_DIR": "C:\\path\\to\\profile"
}

With the identity set, if the configured endpoint stops answering the server locates the window process by exe path, re-resolves its actual port (literal cmdline port, or DevToolsActivePort in the profile dir for random-port launches), verifies the candidate with a proxy-bypassing CDP probe and retries once β€” close and reopen the window and the next tool call just works. Without an identity, attach failures stay hard errors asking you to update the configured URL. Either way there is no fallback to a managed Chrome.

Perceive models (local, auto-downloaded)

desktop_perceive runs PaddleOCR and YOLO icon detection locally with ONNX Runtime. The first call downloads the OCR models and icon_detect.onnx together automatically into %APPDATA%\Nuphus\models (or NUPHUS_MODELS_DIR). Download failures return a clear error with manual instructions. YOLO is optional at runtime: if its download fails, perceive still returns OCR elements and reports yolo_available: false (use NUPHUS_MCP_YOLO_MODEL_URL for a custom source). See TOOLS.md β†’ Vision & Local Models.

All other tools need no API key.

Install & Run

Install via npm (recommended β€” all platforms, prebuilt binaries):

npm install -g @nuphus/nuphus-mcp

The nuphus-mcp meta package installs the prebuilt binary for your platform automatically (Windows x64/arm64, macOS arm64, Linux x64/arm64) and puts the nuphus-mcp command on your PATH. No Rust toolchain needed:

nuphus-mcp   # stdio MCP server

Build from source (requires the Rust toolchain):

cargo build --release -p nuphus-mcp
# binary at target/release/nuphus-mcp(.exe)

The server reads newline-delimited JSON from stdin and writes JSON-RPC responses to stdout. Logs go to stderr.

# quick smoke test
echo '{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test"}}}' | nuphus-mcp

This server can physically control the machine it runs on. By default write tools run without confirmation; we strongly recommend enabling strict confirmation so destructive operations require an explicit "confirm": true argument from the client (otherwise the tool is rejected with isError).

Any one of the following:

# CLI flag
nuphus-mcp --confirm-write

# Environment variable (recommended β€” survives across clients, keeps config simple)
export NUPHUS_MCP_CONFIRM_WRITE=1      # macOS / Linux
setx NUPHUS_MCP_CONFIRM_WRITE 1        # Windows (persistent for new shells)

# MCP client args
"args": ["--confirm-write"]

Claude Desktop β€” recommended claude_desktop_config.json:

{
  "mcpServers": {
    "nuphus-mcp": {
      "command": "nuphus-mcp",
      "args": ["--confirm-write"]
    }
  }
}

Prefer the environment variable: one setting applies to every MCP client on the machine. See SECURITY.md and the Safety Annotations section of TOOLS.md for the full threat model.

MCP Client Configuration

Point any MCP client at nuphus-mcp. After npm install -g @nuphus/nuphus-mcp the command is on your PATH; otherwise use the absolute path to the binary (nuphus-mcp / nuphus-mcp.exe).

Claude Desktop β€” claude_desktop_config.json:

{
  "mcpServers": {
    "nuphus-mcp": {
      "command": "nuphus-mcp",
      "args": []
    }
  }
}

Any MCP client (generic mcpServers JSON):

{
  "mcpServers": {
    "nuphus-mcp": {
      "command": "nuphus-mcp",
      "args": [],
      "env": {}
    }
  }
}

Supported MCP methods: initialize, notifications/initialized, ping, tools/list, tools/call.

Demo

A self-contained stdio client that walks through initialize β†’ tools/list β†’ tools/call:

cargo build -p nuphus-mcp
cargo run -p nuphus-mcp --example demo

Tests

cargo check --workspace
cargo test -p nuphus-mcp          # protocol + security + vision + models tests (60)

Safety

This server can physically control the machine it runs on. Read SECURITY.md and the Safety Annotations section of TOOLS.md before deploying. Recommended: run with --confirm-write (or NUPHUS_MCP_CONFIRM_WRITE=1) so write tools require an explicit "confirm": true argument.