CAP-X: The Robot Manipulation Benchmark for Coding Agents

Let’s be honest: if you’re building or testing coding agents that control real robots—whether it’s a Franka Emika Panda arm, a UR5e, or even a simulated ALOHA—you’ve probably hit this wall: “My agent writes perfect Python… but fails catastrophically when asked to pick up a red block.” That gap between code generation and embodied action is where CAP-X punches through. It’s not another LLM eval suite for trivia or coding katas. CAP-X is a robot manipulation-specific benchmarking framework, built from the ground up to test whether your coding agent actually understands physics, sensor feedback, tool use, and sequential planning—not just syntax. With 165 GitHub stars (as of May 2024), active commits from CapGym Labs, and a lean Python-first stack, CAP-X is already being quietly adopted by robotics labs at ETH Zurich and CMU for agent stress-testing. And yes—it’s fully self-hostable. No cloud APIs, no vendor lock-in. Just Docker, a GPU (optional but recommended), and ~15 minutes of your time.

What Is CAP-X? A Practical Framework for Robot Coding Agents

CAP-X stands for Coding Agent for Physical eXecution—and the name tells you exactly what it does. Unlike general-purpose agent benchmarks (e.g., SWE-bench or AgentBench), CAP-X is narrow, deep, and hardware-aware. It defines manipulation tasks as code-grounded environments: each task is a Python function signature (e.g., def stack_blocks(red: Object, blue: Object, target_pose: Pose) -> List[str]) paired with a simulation or real-robot runtime that executes the generated code step-by-step, validates preconditions, checks physics constraints, and scores success via pose error, grasp stability, and time-to-completion.

The magic is in the execution sandbox. CAP-X doesn’t just run your agent’s code—it injects real-time sensor feedback (/camera/rgb, /gripper/force, /robot/joint_states) into the Python namespace during execution, forces explicit state transitions, and aborts on unsafe motions (e.g., excessive joint torque or collision predictions). I ran it on my local Franka Panda + RealSense D435i setup and watched my GPT-4o-powered agent fail repeatedly on “rotate mug 90° without slipping” — not because the code was wrong, but because it ignored friction coefficients. That’s the kind of failure CAP-X surfaces—and logs—brutally well.

Under the hood, CAP-X uses PyBullet for fast simulation (default), but ships prebuilt ROS 2 bridges for real-robot integration (tested with URDF + MoveIt 2 configs for Franka, UR, and Kinova arms). It also supports Isaac Gym for high-fidelity contact simulation, though that demands an NVIDIA GPU with at least 8GB VRAM.

Installation and Local Setup (No Cloud Required)

CAP-X is Python 3.9+ only—a deliberate choice to avoid PyTorch 2.0+ CUDA versioning hell. I installed it on Ubuntu 22.04 with Python 3.10.12, and the process took <3 minutes. Here’s what actually works (not just what the README says):

# Clone + install core (no GPU needed for sim-only)
git clone https://github.com/capgym/cap-x.git
cd cap-x
pip install -e ".[dev,sim]"  # installs pybullet, numpy, pytest, etc.

# Optional: add ROS 2 support (if targeting real hardware)
pip install -e ".[ros2]"  # requires ROS 2 Humble or Foxy sourced

You’ll need pybullet (v4.1.10, pinned in setup.py) — and if you skip the sim extras, CAP-X won’t even start. Don’t try to pip install pybullet separately; the bundled version patches collision detection for multi-object stacking. Trust me—I learned this the hard way after 45 minutes of “ERROR: Contact manifold empty” spam.

For real-robot testing, you’ll need ROS 2 Humble (not Galactic or Rolling). CAP-X’s ROS bridge assumes /tf, /joint_states, and /camera/color/image_raw topics are live—and validates their timestamps. If your RealSense driver publishes image_raw/compressed, CAP-X will silently hang. Fix: remap in your launch file.

Docker and Docker Compose for Reproducible Environments

CAP-X ships with docker-compose.yml—and unlike most robotics repos, it actually works. The compose file defines three services: sim-server (PyBullet headless), agent-runner (your LLM agent container), and logger (structured JSONL logging with Prometheus metrics).

Here’s the minimal docker-compose.yml I use (updated for CAP-X v0.3.2, released 2024-04-18):

version: '3.8'
services:
  sim-server:
    image: capgym/cap-x:sim-v0.3.2
    ports:
      - "50051:50051"  # gRPC port for agent-server comm
    environment:
      - CAPX_ENV=franka_stack_v2
      - CAPX_HEADLESS=1
    volumes:
      - ./logs:/app/logs

  agent-runner:
    build: ./agents/gpt4o-demo  # your custom agent dir
    depends_on:
      - sim-server
    environment:
      - CAPX_GRPC_HOST=sim-server
      - CAPX_GRPC_PORT=50051
      - OPENAI_API_KEY=sk-...  # or use local Ollama
    restart: "no"

  logger:
    image: capgym/cap-x:logger-v0.3.2
    volumes:
      - ./logs:/app/logs
    command: ["--log-dir", "/app/logs", "--prom-port", "9091"]

Key details:

• The sim-v0.3.2 image is ~1.2GB and includes PyBullet + patched collision libs.
• CAPX_ENV=franka_stack_v2 loads the 2-block stacking environment (others: ur5e_drawer, kinova_cup_pick).
• You must share ./logs across containers—CAP-X uses file-based log aggregation, not stdout streaming.
• No GPU passthrough needed for sim-only runs, but add runtime: nvidia and NVIDIA_VISIBLE_DEVICES=all if you enable Isaac Gym mode.

I ran this on a 32GB RAM, Ryzen 7 5800X system with no GPU. CPU usage spiked to 95% during dense physics simulation—so avoid running this on a Pi or low-end laptop.

CAP-X vs. Alternatives: Why Not Just Use RLBench or RoboSuite?

If you’ve been using RLBench, RoboSuite, or even ManiSkill, here’s why CAP-X is different—and why you might want to switch for agent evaluation:

The TL;DR: RLBench and RoboSuite are reinforcement learning environments. CAP-X is a coding agent test harness. They solve different problems. If your pipeline ends with “call env.step(action)”, stick with RoboSuite. But if your pipeline is “generate Python → validate → execute → observe → revise”, CAP-X is the only framework that validates all four phases with hardware-aware rigor.

I tested the same Llama-3-70B agent on CAP-X’s franka_stack_v2 vs. ManiSkill2’s PickCube task. Success rate dropped from 78% (ManiSkill) to 31% (CAP-X)—because CAP-X required the agent to explicitly call gripper.close(force=20.0) before move_to(pose, max_velocity=0.1) and validate contact force >15N. ManiSkill just let it “teleport-grasp”. CAP-X doesn’t forgive.

Who Is CAP-X For? (Spoiler: Not Your Weekend Raspberry Pi Project)

CAP-X isn’t for beginners. It’s for:

• Robotics researchers validating coding agents (e.g., “Does my CodeLLaMA fine-tune handle multi-step tool use?”)
• Robot OEMs stress-testing SDKs against LLM-generated code (UR, Franka, and Kinova all have CAP-X compatibility PRs open)
• AI safety labs auditing whether coding agents hallucinate unsafe motions (e.g., “rotate wrist 360° at 2 rad/s” → joint limit violation)
• Self-hosters who want full control over robot-agent telemetry—no sending /joint_states to a cloud API

It is not for:

• Teaching robotics to undergrads (use PyBullet tutorials instead)
• Running on ARM64 (no ARM Docker images yet—build fails on M2 Macs)
• Low-latency teleoperation (gRPC overhead adds ~80ms median delay)
• Pure vision-language tasks (no built-in CLIP/ViT hooks—bring your own encoder)

Hardware-wise:

• Minimum: 16GB RAM, 4-core CPU, Ubuntu 22.04. Sim runs, but expect 0.3x real-time.
• Recommended: 32GB RAM, 8-core CPU, NVIDIA RTX 3080 (for Isaac Gym mode).
• Real robot: Requires ROS 2 Humble + real-time kernel (tested on PREEMPT_RT patch). My Franka setup needed sudo systemctl set-default multi-user.target to avoid GUI interference.

The Honest Take: Is CAP-X Worth Deploying Right Now?

Yes—but with caveats. I’ve run CAP-X daily for 3 weeks across 4 environments (franka_stack_v2, ur5e_drawer, kinova_cup_pick, and custom aloha_flip_switch). Here’s my unfiltered verdict:

✅ What works brilliantly:

• The gRPC agent-server contract is rock-solid. My custom Ollama + Llama-3 agent connects, executes, and logs without flaking.
• Physics validation is scarily precise. It caught my agent trying to “stack blocks upside-down” by checking convex hull intersections—not just bounding boxes.
• Logging is structured, searchable, and includes per-step sensor snapshots (/gripper/force at t=0.42s, joint_positions at t=0.45s).

⚠️ Rough edges you’ll hit:

• No web UI. Everything is CLI + JSONL logs. You will write a jq script to extract success rates.
• ROS 2 bridge is Humble-only—no Foxy or Rolling support. If your lab uses ROS 2 Galactic, you’re out of luck.
• No built-in agent zoo. You bring your own LLM. CAP-X ships with a GPT-4o demo, but no fine-tuned CodeLlama weights or config.
• Installation fails silently on macOS (missing libgl1-mesa-glx). The docs don’t mention it. Fix: brew install mesa-glu + symlink /opt/homebrew/lib/libGL.dylib to /usr/lib/libGL.dylib.

The biggest win? Transparency. CAP-X doesn’t hide physics behind reward functions. It shows you exactly where your agent’s mental model breaks: “You told the robot to move at 0.5 m/s, but max_velocity in the URDF is 0.2 m/s. Aborting.” That kind of feedback is gold—if you’re serious about building agents that don’t break your $80k robot arm.

So—deploy it? If you’re evaluating coding agents for real manipulation: absolutely. Start with the sim, validate your agent’s physics awareness, then move to hardware. If you just need a coding benchmark for a blog post? Use HumanEval. But if your agent needs to touch the world, CAP-X is the first tool that treats code not as output—but as executable, auditable, safety-checked intent. And that’s rare.

(Version used: CAP-X v0.3.2, commit a5c2d8f, 165 stars as of 2024-05-22.)