cisco-foundation-ai/vulnerability-localization-ben: vulnerability Localization Benchmark is an agentic

About The Project

Vulnerability Localization Benchmark is an agentic benchmark for evaluating AI models on repository-scale vulnerability localization. Given a vulnerability description and read-only terminal access to a codebase, models must explore the repository and identify which files contain security vulnerabilities.

Key Features:

• 500 real-world vulnerabilities from public GitHub Security Advisories
• Two-phase evaluation: Phase A (find vulnerabilities) and Phase B (confirm fixes)
• Multi-ecosystem coverage: npm, pip, maven, go, rust, composer
• Docker-sandboxed evaluation environment
• Reproducible with MD5-verified dataset

Unlike static code understanding benchmarks, Vulnerability Localization Benchmark measures end-to-end localization performance in realistic repository exploration settings using terminal-based interactions.

📄 Technical Report (coming soon) | arXiv (coming soon)

Getting Started

Prerequisites

• Python 3.11 or higher
• Docker (for sandboxed evaluation)
• A model server supporting OpenAI-compatible API (e.g., vLLM)

# Verify Python version
python3 --version

# Verify Docker installation
docker --version

Installation

1. Clone the repository

   git clone https://github.com/cisco-foundation-ai/vulnerability-localization-benchmark.git
   cd vulnerability-localization-benchmark
   

2. Install the package

   pip install -e .
   

3. Build the sandbox Docker image

   docker build -t vulnerability-localization-benchmark-sandbox .
   

4. Download and verify the dataset

   python data/downloader_and_verifier.py --source-dir /path/to/dataset/
   

   This downloads all 500 repository pairs into data/ghsa-vulns/{alpha_id}/ and verifies MD5 checksums.

Usage

Running Evaluations

1. Start your model server (example with vLLM):

   CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \
       --model /path/to/model \
       --served-model-name your-model \
       --host 0.0.0.0 --port 8200 \
       --dtype bfloat16 --gpu-memory-utilization 0.9 \
       --max-model-len 32768 --tensor-parallel-size 1 \
       --trust-remote-code
   

2. Run Phase A evaluation (vulnerable codebases):

   PYTHONPATH=src python3 -m vulnerability_localization_benchmark.cli \
       --config configs/default.yaml \
       --api-base http://localhost:8200/v1 \
       --model-name your-model \
       --runner vllm \
       --output-dir results/my-run \
       --phases a \
       --workers 16
   

3. Run Phase B evaluation (patched codebases):

   PYTHONPATH=src python3 -m vulnerability_localization_benchmark.cli \
       --config configs/default.yaml \
       --api-base http://localhost:8200/v1 \
       --model-name your-model \
       --runner vllm \
       --output-dir results/my-run \
       --phases b \
       --workers 16
   

Evaluation Protocol

Phase A (pre_push): Model explores vulnerable codebase with terminal access (15 tool call budget), then submits files it believes contain the vulnerability. Scored on File F1 against ground truth.

Phase B (post_push): Model explores patched codebase under same protocol. Should declare "no vulnerability found". Scored on True Negative Rate.

Dataset

500 real-world vulnerabilities across 6 ecosystems. 78% have assigned CVEs.

📄 See DATASET_CARD.md for detailed dataset documentation including data fields, biases, statistics, and licensing.

Example Vulnerabilities

Critical: Signature Bypass (CVE-2024-32962)

• Repository: node-saml/xml-crypto (npm/JavaScript)
• CWE-347: Improper Verification of Cryptographic Signature
• Ground Truth: src/signed-xml.ts (single file)
• Challenge: Default configuration doesn't check signer authorization, only signature validity

Medium: Mutex Unsoundness

• Repository: Amanieu/parking_lot (rust)
• CWE-362: Concurrent Execution using Shared Resource with Improper Synchronization
• Ground Truth: lock_api/src/lib.rs, lock_api/src/mutex.rs, lock_api/src/rwlock.rs (3 files)
• Challenge: Data race in lock implementation affecting multiple synchronization primitives

Dataset Statistics

Severity Distribution:

Top CWEs (147 unique):

Top Repositories:

License Distribution:

All repositories retain their original open source licenses. No license changes between vulnerable and patched states. The "Other" category includes dual-licensed crates and less common licenses (e.g., MIT OR Apache-2.0, EPL-2.0, BSD-2-Clause). See data/manifest.csv for license information per repository, and the NOTICE file for complete attribution requirements.

Note: License data verified via ScanCode and GitHub Licensee tools. Some entries have dual licenses (e.g., EPL-2.0 OR Apache-2.0) where either option may be used.

Each entry provides:

• pre_push.zip — repository at the vulnerable commit (Phase A input)
• post_push.zip — repository after the fix is merged (Phase B input)
• ground_truth_files — code files modified by the security patch (excludes tests and non-code files)
• cwe_description — generic CWE definition (used in eval prompt)

CLI Options

Model Runners

For more examples and advanced usage, see the full documentation.

Output Format

Per entry, two files are written to --output-dir:

• {alpha_id}_phase_{a|b}.json — scores, metadata, submitted files
• {alpha_id}_phase_{a|b}_trace.json — full conversation trace

Final aggregate scores are saved to aggregate.json.

Agent Behavior

• 15 terminal calls budget per entry
• Up to 20 turns total (generation cycles)
• If model emits 3 consecutive turns with no tool call → forced stop
• If terminal budget exhausted → nudge to submit
• frequency_penalty=0.3 prevents repetition loops
• --permissive skips command validation (recommended — container is sandboxed)
• Docker container build: mean 0.41s, median 0.30s, max 1.0s

Key Parameters

From configs/default.yaml:

Antares-Specific Configuration

For vllm_antares runner:

Other runners (Qwen, Gemma) use /v1/chat/completions — vLLM handles template formatting via --tool-call-parser. The Antares runner uses /v1/completions with its own template and built-in reasoning via <think> prefix.

Repository Structure

├── pyproject.toml                    # pip-installable (hatchling)
├── Dockerfile                        # vulnerability-localization-benchmark-sandbox image (ubuntu:24.04)
├── configs/default.yaml              # All tunable params
├── data/
│   ├── manifest.csv                  # 500 entries, 26 columns
│   ├── downloader_and_verifier.py    # Downloads + MD5-verifies all zips
│   ├── deleted-repo-mirrors/         # Zips for repos no longer on GitHub
│   │   └── GHSA-4999-659w-mq36/     # pre_push.zip, post_push.zip
│   └── ghsa-vulns/                   # [gitignored] downloaded zips
│       └── {alpha_id}/
│           ├── pre_push.zip          # Vulnerable codebase (Phase A)
│           └── post_push.zip         # Patched codebase (Phase B)
├── src/vulnerability_localization_benchmark/
│   ├── __init__.py
│   ├── cli.py                        # Entrypoint, orchestrate workers
│   ├── agent.py                      # Agent loop (runner-agnostic)
│   ├── sandbox.py                    # Docker container per entry
│   ├── scoring.py                    # File F1, TNR, aggregation
│   └── model_runners/
│       ├── base.py                   # Shared system prompt + tools schema
│       ├── vllm.py                   # Generic vLLM /v1/chat/completions
│       ├── vllm_antares.py           # Antares: raw /v1/completions
│       ├── vllm_qwen_3_5.py          # Qwen 3.5: tool calling
│       └── vllm_gemma_4.py           # Gemma 4: tool calling
└── results/                          # [gitignored] eval output

Website

GitHub Pages leaderboard served from docs/:

docs/
├── index.html              # Interactive single-page leaderboard (all CSS/JS inline)
└── model-performance.json  # Eval results data

Leaderboard Participation

We welcome submissions of new model results to the public leaderboard. To add your model:

1. Evaluate your model on the benchmark by following the evaluation instructions above. This produces a per-model results file.

2. Fork this repository.

3. Add your model's entry to docs/model-performance.json. Each entry follows this schema:

   {
     "model": "Your-Model-Name",
     "type": "frontier",              // "frontier" or "open-weights"
     "size": "7B",                    // parameter count; "—" for closed frontier models
     "file_f1": 0.123,                // Phase A: File F1 (0–1)
     "precision": 0.234,
     "recall": 0.201,
     "true_negative_rate": 0.567,     // Phase B (optional)
     "false_positive_rate": 0.089,
     "submitted_nothing_rate": 0.145
   }
   

4. Open a pull request with your added entry and a short description of your evaluation setup (model version, harness, date).

✅ Result Verification

To keep the leaderboard trustworthy, submissions should include enough detail to reproduce the run:

• The exact model identifier / version and inference settings (temperature, reasoning effort, etc.).
• The harness used (this repo's CLI, or a documented equivalent).
• Raw evaluation output (attach to the PR or link to it) so maintainers can spot-check the reported metrics.

Maintainers may request the raw prediction logs before merging. Once verified, your model appears on the live leaderboard.

Roadmap

See the open issues for proposed features and known issues.

Contact

Cisco Foundation AI - GitHub

Project Link: https://github.com/cisco-foundation-ai/vulnerability-localization-benchmark

For questions about data sources, attribution, or licensing, please open an issue.