vignesh2027/LLM-Evaluation-Framework: measure accuracy, latency, cost, hallucination

The most complete open-source LLM evaluation suite.
Measure accuracy, latency, cost, hallucination, and reasoning quality across any LLM — side by side.

🌐 Live Docs	🤗 HuggingFace	🐙 GitHub	⭐ Star the Repo

Evaluation+Framework+%E2%80%94+Demo+Coming+Soon&font=raleway" alt="Demo Placeholder">

📋 Table of Contents

• ✨ Why This Framework?
• 🎯 Key Features
• 🏗 Architecture
• 🚀 Quick Start
• 📦 Installation
• 🔑 API Keys Setup
• 💻 CLI Reference
• 🐍 Python API
• 🌐 REST API Reference
• 📊 Streamlit Dashboard
• 📏 Evaluation Metrics
• 🏆 Supported Benchmarks
• 🤖 Supported Models & Pricing
• 🗄 Database & Storage
• 📄 PDF Report Generation
• 🐳 Docker Deployment
• 🧪 Testing
• 🤗 HuggingFace Dataset
• 📁 Project Structure
• 🔧 Configuration Reference
• 🤝 Contributing
• 📜 License
• ⭐ Star History

✨ Why This Framework?

"You can't improve what you can't measure." — Peter Drucker

The LLM landscape is evolving at breakneck speed. New models appear every week, each claiming to be state-of-the-art. But how do you actually know which model is best for your use case?

Most existing benchmarking tools:

• ❌ Evaluate only a single model at a time
• ❌ Ignore latency and real-world cost
• ❌ Don't detect hallucinations
• ❌ Require complex setup
• ❌ Lack a usable dashboard

This framework solves all of that. It's the only open-source tool that evaluates GPT-4, Claude, Gemini, Mistral, and Llama side by side — on the same prompts — with 5 production-relevant metrics, a beautiful Streamlit dashboard, a REST API, and a CLI.

🎯 Key Features

🏗 Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                       LLM EVALUATION FRAMEWORK                          │
│                                                                         │
│   ┌──────────┐  ┌──────────────┐  ┌───────────────┐  ┌─────────────┐  │
│   │  Click   │  │   FastAPI    │  │   Streamlit   │  │  ReportLab  │  │
│   │   CLI    │  │  REST API    │  │   Dashboard   │  │PDF Generator│  │
│   │ 7 cmds   │  │ 12 endpoints │  │   5 pages     │  │             │  │
│   └────┬─────┘  └──────┬───────┘  └───────┬───────┘  └──────┬──────┘  │
│        └───────────────┴──────────────────┴─────────────────┘          │
│                                    │                                    │
│                        ┌───────────▼──────────┐                        │
│                        │    Core Evaluator     │                        │
│                        │   (Async Engine)      │                        │
│                        │  asyncio.Semaphore    │                        │
│                        │  configurable timeout │                        │
│                        │  progress callbacks   │                        │
│                        └───────────┬──────────┘                        │
│                                    │                                    │
│        ┌────────────────────┬──────┴──────┬───────────��────────┐       │
│        │                    │             │                    │       │
│  ┌─────▼──────┐    ┌────────▼──────┐  ┌──▼──────────┐  ┌──────▼───┐  │
│  │  Metrics   │    │  Benchmarks   │  │  Database   │  │ LiteLLM  │  │
│  │            │    │               │  │  (SQLite)   │  │          │  │
│  │ accuracy   │    │ MMLU (14K)    │  │             │  │ OpenAI   │  │
│  │ hallucin.  │    │ TruthfulQA    │  │ save_result │  │ Anthropic│  │
│  │ latency    │    │ Custom CSV    │  │ list_results│  │ Google   │  │
│  │ cost       │    │ Custom JSON   │  │ export_csv  │  │ Mistral  │  │
│  │ reasoning  │    │ HF Hub cache  │  │ export_json │  │ Together │  │
│  └────────────┘    └───────────────┘  └─────────────┘  └──────────┘  │
└──────────────────────────────────────────────────────────────────────��──┘

Data Flow

User Request → CLI/API/Dashboard
       ↓
EvaluationConfig (model, benchmark, num_samples, temperature, concurrency)
       ↓
LLMEvaluator.evaluate(config, samples)
       ↓
asyncio.Semaphore(concurrency) ← controls parallelism
       ↓
For each sample (parallel):
  litellm.acompletion() → response
  AccuracyMetric.score(response, expected)
  HallucinationMetric.score(prompt, response)
  HallucinationMetric.reasoning_quality(response)
  LatencyMetric — wall-clock time
  CostMetric.calculate(model, input_tokens, output_tokens)
       ↓
_aggregate(samples) → EvaluationResult (accuracy, latency stats, cost, etc.)
       ↓
Database.save_result() → SQLite
       ↓
Return EvaluationResult to caller

🚀 Quick Start

3-Minute Setup

# 1. Install
pip install llm-evaluation-framework

# 2. Set your API key
export OPENAI_API_KEY="sk-..."

# 3. Run your first evaluation
llm-eval run --model gpt-4o-mini --benchmark mmlu --samples 50

Expected output:

╭──────────────────────────────────────╮
│  Evaluation: gpt-4o-mini             │
├──────────────────┬───────────────────┤
│ Accuracy         │ 78.00%            │
│ Avg Latency      │ 432 ms            │
│ P95 Latency      │ 1240 ms           │
│ Total Cost       │ $0.0012           │
│ Cost / 1K Tokens │ $0.0015           │
│ Hallucination    │ 2.40%             │
│ Reasoning Score  │ 7.2 / 10          │
│ Samples          │ 50                │
│ Run ID           │ a3f92c1b          │
╰──────────────────┴───────────────────╯

📦 Installation

Option 1 — pip (Recommended)

pip install llm-evaluation-framework

Option 2 — With Extras

# Dashboard (Streamlit + Plotly + Pandas)
pip install "llm-evaluation-framework[dashboard]"

# PDF Reports (ReportLab)
pip install "llm-evaluation-framework[reports]"

# Development (pytest, mypy, ruff)
pip install "llm-evaluation-framework[dev]"

# Everything
pip install "llm-evaluation-framework[dashboard,reports,dev]"

Option 3 — From Source

git clone https://github.com/vignesh2027/LLM-Evaluation-Framework.git
cd LLM-Evaluation-Framework

# Create virtual environment
python -m venv .venv
source .venv/bin/activate        # Linux/Mac
# .venv\Scripts\activate         # Windows

# Install with all extras
pip install -e ".[dashboard,reports,dev]"

Option 4 — Docker

git clone https://github.com/vignesh2027/LLM-Evaluation-Framework.git
cd LLM-Evaluation-Framework
cp .env.example .env       # fill in your API keys
docker-compose up -d
# API:       http://localhost:8000/docs
# Dashboard: http://localhost:8501

Requirements

🔑 API Keys Setup

Copy .env.example to .env and fill in your keys:

cp .env.example .env

# .env

# ── OpenAI ──────────────────────────────────
OPENAI_API_KEY=sk-...

# ── Anthropic (Claude) ───────────────────────
ANTHROPIC_API_KEY=sk-ant-...

# ── Google (Gemini) ──────────────────────────
GEMINI_API_KEY=AI...
GOOGLE_API_KEY=AI...

# ── Mistral ───────────────────────���──────────
MISTRAL_API_KEY=...

# ── Together AI (Llama) ──────────────────────
TOGETHERAI_API_KEY=...

# ── HuggingFace (for dataset loading) ────────
HUGGINGFACE_TOKEN=hf_...

# ── App Settings ─────────────────────────────
LLM_EVAL_DB_PATH=llm_eval.db
LLM_EVAL_CACHE_DIR=~/.cache/llm_eval
PORT=8000
DASHBOARD_PORT=8501

Note: You only need the keys for the models you want to evaluate. The framework works with any subset of providers.

💻 CLI Reference

The CLI is installed as llm-eval and provides 7 subcommands:

llm-eval run — Evaluate a single model

llm-eval run [OPTIONS]

Options:
  -m, --model TEXT         LiteLLM model name         [required]
  -b, --benchmark TEXT     mmlu | truthfulqa | custom  [default: mmlu]
  -n, --samples INTEGER    Number of samples           [default: 20]
  --temperature FLOAT      Sampling temperature        [default: 0.0]
  --max-tokens INTEGER     Max output tokens           [default: 512]
  --concurrency INTEGER    Parallel API calls          [default: 5]
  -o, --output PATH        Save JSON result to file
  -v, --verbose            Enable debug logging

Examples:

# Quick 20-sample smoke test
llm-eval run --model gpt-4o-mini --benchmark mmlu

# Full 100-sample MMLU evaluation
llm-eval run --model gpt-4o --benchmark mmlu --samples 100 --concurrency 10

# TruthfulQA evaluation
llm-eval run --model claude-3-5-haiku-20241022 --benchmark truthfulqa --samples 50

# Save output to JSON
llm-eval run --model gpt-4o-mini --benchmark mmlu --samples 50 -o result.json

# Use a custom temperature
llm-eval run --model gpt-3.5-turbo --benchmark mmlu --temperature 0.3

llm-eval compare — Compare multiple models

llm-eval compare [OPTIONS]

Options:
  -m, --models TEXT        Model names (repeat flag)   [required, min 2]
  -b, --benchmark TEXT     Benchmark name              [default: mmlu]
  -n, --samples INTEGER    Samples per model           [default: 20]
  -o, --output PATH        Save JSON results to file

Examples:

# Compare 3 providers
llm-eval compare \
  --models gpt-4o-mini \
  --models claude-3-haiku-20240307 \
  --models gemini/gemini-1.5-flash \
  --benchmark mmlu --samples 50

# Compare on TruthfulQA
llm-eval compare \
  --models gpt-4o \
  --models claude-3-5-sonnet-20241022 \
  --benchmark truthfulqa --samples 100

# Save comparison results
llm-eval compare --models gpt-4o-mini --models claude-3-haiku-20240307 \
  --benchmark mmlu --output comparison.json

llm-eval results — View stored results

llm-eval results [OPTIONS]

Options:
  --model TEXT             Filter by model name
  --benchmark TEXT         Filter by benchmark
  --limit INTEGER          Max results to show     [default: 20]

Examples:

# Show all results
llm-eval results

# Filter by model
llm-eval results --model gpt-4o-mini

# Filter by benchmark
llm-eval results --benchmark mmlu --limit 50

llm-eval export — Export results

llm-eval export --format [csv|json] --output OUTPUT_PATH

# Export to CSV
llm-eval export --format csv --output results.csv

# Export to JSON
llm-eval export --format json --output results.json

llm-eval report — Generate PDF report

llm-eval report --run-ids RUN_ID [--run-ids RUN_ID ...] --output OUTPUT_DIR

# Single run
llm-eval report --run-ids a3f92c1b --output ./reports/

# Multiple runs in one report
llm-eval report --run-ids a3f92c1b --run-ids b4d03e2c --output ./reports/

llm-eval serve — Start FastAPI server

llm-eval serve [--host HOST] [--port PORT] [--reload]

llm-eval serve --port 8000 --reload
# → http://localhost:8000/docs

llm-eval dashboard — Launch Streamlit dashboard

llm-eval dashboard [--port PORT]

llm-eval dashboard --port 8501
# → http://localhost:8501

🐍 Python API

Basic Evaluation

import asyncio
from llm_eval.core.evaluator import LLMEvaluator, EvaluationConfig
from llm_eval.benchmarks.mmlu import MMLUBenchmark

async def main():
    evaluator = LLMEvaluator()                       # uses default db path
    samples = MMLUBenchmark().load(num_samples=100)  # loads from HF Hub or cache

    config = EvaluationConfig(
        model="gpt-4o-mini",
        benchmark="mmlu",
        num_samples=100,
        temperature=0.0,
        max_tokens=512,
        concurrency=10,    # parallel API calls
        timeout=30.0,      # seconds per request
    )

    result = await evaluator.evaluate(config, samples)

    print(f"Accuracy:          {result.accuracy:.2%}")
    print(f"Avg Latency:       {result.avg_latency_ms:.0f} ms")
    print(f"P95 Latency:       {result.p95_latency_ms:.0f} ms")
    print(f"P99 Latency:       {result.p99_latency_ms:.0f} ms")
    print(f"Total Cost:        ${result.total_cost_usd:.4f}")
    print(f"Cost per 1K:       ${result.cost_per_1k_tokens:.4f}")
    print(f"Hallucination:     {result.hallucination_rate:.2%}")
    print(f"Reasoning Score:   {result.avg_reasoning_score:.1f} / 10")
    print(f"Run ID:            {result.run_id}")

asyncio.run(main())

Side-by-Side Comparison

import asyncio
from llm_eval.core.evaluator import LLMEvaluator, EvaluationConfig
from llm_eval.benchmarks.mmlu import MMLUBenchmark

async def compare():
    evaluator = LLMEvaluator()
    # Load samples ONCE — all models see the same prompts
    samples = MMLUBenchmark().load(num_samples=50)

    configs = [
        EvaluationConfig(model="gpt-4o",                      benchmark="mmlu", num_samples=50),
        EvaluationConfig(model="gpt-4o-mini",                 benchmark="mmlu", num_samples=50),
        EvaluationConfig(model="claude-3-5-sonnet-20241022",  benchmark="mmlu", num_samples=50),
        EvaluationConfig(model="claude-3-haiku-20240307",     benchmark="mmlu", num_samples=50),
        EvaluationConfig(model="gemini/gemini-1.5-flash",     benchmark="mmlu", num_samples=50),
    ]

    # All 5 evaluations run in parallel
    results = await evaluator.evaluate_multiple(configs, samples)

    # Print ranked leaderboard
    header = f"{'Rank':<5} {'Model':<40} {'Acc':>8} {'Lat':>8} {'Cost/1K':>10} {'Score':>8}"
    print(header)
    print("─" * len(header))
    for i, r in enumerate(sorted(results, key=lambda x: x.accuracy, reverse=True), 1):
        print(
            f"{i:<5} {r.model:<40} "
            f"{r.accuracy:>7.1%} "
            f"{r.avg_latency_ms:>6.0f}ms "
            f"${r.cost_per_1k_tokens:>8.4f} "
            f"{r.avg_reasoning_score:>6.1f}/10"
        )

asyncio.run(compare())

Custom Benchmark

import asyncio
from llm_eval.core.evaluator import LLMEvaluator, EvaluationConfig
from llm_eval.benchmarks.custom import CustomBenchmark

async def custom_eval():
    # From a CSV file
    bench = CustomBenchmark.from_file("my_benchmark.csv")

    # Or from a string
    csv_content = """prompt,expected
"What is the capital of Python packaging?","PyPI"
"What does ACID stand for in databases?","Atomicity Consistency Isolation Durability"
"""
    bench = CustomBenchmark.from_string(csv_content, format="csv")

    evaluator = LLMEvaluator()
    config = EvaluationConfig(model="gpt-4o-mini", benchmark="custom", num_samples=50)
    result = await evaluator.evaluate(config, bench.load(50))
    print(f"Custom eval accuracy: {result.accuracy:.2%}")

asyncio.run(custom_eval())

Progress Tracking

import asyncio
from llm_eval.core.evaluator import LLMEvaluator, EvaluationConfig
from llm_eval.benchmarks.mmlu import MMLUBenchmark

async def with_progress():
    evaluator = LLMEvaluator()
    samples = MMLUBenchmark().load(100)
    config = EvaluationConfig(model="gpt-4o-mini", benchmark="mmlu", num_samples=100)

    async def on_progress(done: int, total: int):
        pct = done / total * 100
        bar = "█" * int(pct / 5) + "░" * (20 - int(pct / 5))
        print(f"\r[{bar}] {done}/{total} ({pct:.0f}%)", end="", flush=True)

    result = await evaluator.evaluate(config, samples, progress_callback=on_progress)
    print(f"\n✅ Done! Accuracy: {result.accuracy:.1%}")

asyncio.run(with_progress())

Database Queries

from llm_eval.database.models import Database

db = Database()  # or Database("custom_path.db")

# List all results
records = db.list_results(limit=50)

# Filter by model
records = db.list_results(model="gpt-4o-mini", limit=20)

# Filter by benchmark
records = db.list_results(benchmark="mmlu", limit=100)

# Get a specific run
record = db.get_result("a3f92c1b")
print(record.accuracy, record.avg_latency_ms)

# Compare models on a benchmark (latest run per model)
comparison = db.get_model_comparison("mmlu")
for row in comparison:
    print(row["model"], row["accuracy"])

# Export
db.export_csv("all_results.csv")
db.export_json("all_results.json")

# Delete a run
db.delete_result("a3f92c1b")

🌐 REST API Reference

Start the API server:

uvicorn llm_eval.api.main:app --reload --port 8000
# Interactive docs: http://localhost:8000/docs
# ReDoc:            http://localhost:8000/redoc

Endpoints

Request/Response Examples

POST /evaluate

// Request
{
  "model": "gpt-4o-mini",
  "benchmark": "mmlu",
  "num_samples": 50,
  "temperature": 0.0,
  "max_tokens": 512,
  "concurrency": 10
}

// Response
{
  "run_id": "a3f92c1b",
  "model": "gpt-4o-mini",
  "benchmark": "mmlu",
  "num_samples": 50,
  "accuracy": 0.78,
  "avg_latency_ms": 432.1,
  "p50_latency_ms": 380.0,
  "p95_latency_ms": 1100.0,
  "p99_latency_ms": 1840.0,
  "total_cost_usd": 0.0012,
  "cost_per_1k_tokens": 0.0015,
  "hallucination_rate": 0.024,
  "avg_reasoning_score": 7.2,
  "created_at": "2025-01-20T14:32:01"
}

POST /compare

// Request
{
  "models": ["gpt-4o-mini", "claude-3-haiku-20240307", "gemini/gemini-1.5-flash"],
  "benchmark": "mmlu",
  "num_samples": 30
}

// Response
{
  "results": [
    {"model": "gpt-4o-mini", "accuracy": 0.78, ...},
    {"model": "claude-3-haiku-20240307", "accuracy": 0.74, ...},
    {"model": "gemini/gemini-1.5-flash", "accuracy": 0.76, ...}
  ]
}

POST /evaluate/custom (file upload)

curl -X POST http://localhost:8000/evaluate/custom \
  -F "file=@my_benchmark.csv" \
  -F "model=gpt-4o-mini" \
  -F "num_samples=50"

📊 Streamlit Dashboard

Launch the dashboard:

streamlit run llm_eval/dashboard/app.py
# → http://localhost:8501

# Or via CLI
llm-eval dashboard --port 8501

Dashboard Pages

Dashboard Charts

• Radar Chart — 5-axis model comparison (accuracy, speed, cost efficiency, truthfulness, reasoning)
• Latency Histogram — distribution of response times per model
• Cost vs Quality Scatter — bubble chart (bubble size = sample count)
• Accuracy Bar Chart — ranked model comparison
• 3D Scatter — cost vs quality vs latency

📏 Evaluation Metrics

Accuracy Metric

Uses a cascade of matching strategies, applied in order:

1. Exact match — after lowercasing and stripping punctuation
2. Prefix normalization — removes "The answer is", "Answer:", etc.
3. Multiple-choice detection — extracts the first A/B/C/D letter from the prediction
4. Fuzzy match — Levenshtein ratio ≥ 0.85 (configurable)

from llm_eval.metrics.accuracy import AccuracyMetric

metric = AccuracyMetric(fuzzy_threshold=0.85)

# Multiple choice
metric.score("The answer is A", "A")          # True
metric.score("I think B is correct here", "B") # True

# Free-form
metric.score("mitochondria", "mitochondrion")  # True (fuzzy)
metric.score("Paris", "Paris")                  # True (exact)

# Batch scoring
results, accuracy = metric.batch_score(
    ["A", "B", "A", "D"],
    ["A", "B", "C", "D"]
)
# results = [True, True, False, True], accuracy = 0.75

Latency Metric

from llm_eval.metrics.latency import LatencyMetric

metric = LatencyMetric()

latencies = [120, 200, 350, 450, 800, 1200, 2000, 5500]  # ms
stats = metric.compute_stats(latencies)

print(f"Mean:  {stats.mean_ms:.0f} ms")
print(f"P50:   {stats.p50_ms:.0f} ms")
print(f"P95:   {stats.p95_ms:.0f} ms")
print(f"P99:   {stats.p99_ms:.0f} ms")

# SLA violations
rate = metric.sla_violation_rate(latencies, threshold_ms=5000.0)
print(f"SLA violations: {rate:.1%}")  # 12.5%

# Classification
print(metric.classify(200))   # "excellent"
print(metric.classify(1000))  # "good"
print(metric.classify(4000))  # "slow"

Cost Metric

from llm_eval.metrics.cost import CostMetric

metric = CostMetric()

# Calculate exact cost
cost = metric.calculate("gpt-4o-mini", input_tokens=150, output_tokens=200)
print(f"Per-sample cost: ${cost:.6f}")

# Pre-run estimate
estimate = metric.estimate_run_cost("gpt-4o-mini", num_samples=100)
print(f"Estimated run cost: ${estimate:.4f}")

# All pricing
for name, pricing in metric.get_all_pricing().items():
    print(f"{name}: ${pricing.input_per_1m}/M in, ${pricing.output_per_1m}/M out")

Hallucination + Reasoning Metric

from llm_eval.metrics.hallucination import HallucinationMetric

metric = HallucinationMetric()

# Hallucination score (0 = grounded, 1 = likely hallucinating)
score = metric.score(
    prompt="What is the capital of France?",
    response="I believe it was supposedly Paris, though I could be wrong."
)
print(f"Hallucination score: {score:.2f}")  # ~0.3 (hedging signals)

# Reasoning quality (1-10)
quality = metric.reasoning_quality(
    "First, we analyze the data. Based on the evidence specifically, "
    "therefore we can conclude that X. For example, this means..."
)
print(f"Reasoning quality: {quality:.1f}/10")  # ~7.0

🏆 Supported Benchmarks

MMLU (Massive Multitask Language Understanding)

• Size: ~14,000 test questions
• Format: 4-choice multiple choice
• Subjects: 57 academic subjects including:
  • STEM: abstract algebra, anatomy, astronomy, biology, chemistry, computer science, physics, mathematics
  • Humanities: history, law, philosophy, world religions
  • Social sciences: economics, geography, political science, psychology, sociology
  • Professional: medical, legal, accounting, marketing, nursing

from llm_eval.benchmarks.mmlu import MMLUBenchmark

bench = MMLUBenchmark(subject="all")  # or a specific subject
samples = bench.load(num_samples=100, seed=42)
# [{"prompt": "Question?\nA) ...\nB) ...\nAnswer:", "expected": "A"}, ...]

TruthfulQA

• Size: 817 questions
• Format: 4-choice multiple choice (MC1 format)
• Purpose: Tests whether models give truthful answers; questions are designed to elicit common misconceptions

from llm_eval.benchmarks.truthfulqa import TruthfulQABenchmark

bench = TruthfulQABenchmark()
samples = bench.load(num_samples=100, seed=42)

Custom Benchmark

• Format: CSV or JSON
• Required columns: prompt, expected
• Sources: File upload, string, or programmatic

from llm_eval.benchmarks.custom import CustomBenchmark

# From file
bench = CustomBenchmark.from_file("my_data.csv")

# From string
bench = CustomBenchmark.from_string("""
prompt,expected
"What is 2+2?",4
"Capital of Germany?",Berlin
""", format="csv")

print(f"Dataset size: {len(bench)} samples")
samples = bench.load(num_samples=50)

🤖 Supported Models & Pricing

OpenAI

Anthropic

Google

Mistral

Meta Llama (via Together AI)

Custom / Local Models

Any LiteLLM-compatible provider works:

# Ollama (local)
llm-eval run --model ollama/llama3 --benchmark mmlu --samples 20

# vLLM
llm-eval run --model hosted_vllm/meta-llama/Llama-3-8b --benchmark mmlu

# HuggingFace TGI
llm-eval run --model huggingface/mistralai/Mistral-7B-Instruct-v0.2 --benchmark mmlu

🗄 Database & Storage

All evaluation results are automatically saved to SQLite (llm_eval.db by default):

from llm_eval.database.models import Database

db = Database("llm_eval.db")   # default path
# or
db = Database("/data/eval_results.db")

# Query
records = db.list_results(model="gpt-4o-mini", benchmark="mmlu", limit=100)
record = db.get_result("a3f92c1b")

# Compare latest results per model for a benchmark
comparison = db.get_model_comparison("mmlu")

# Export
db.export_csv("results.csv")
db.export_json("results.json")

# Delete
db.delete_result("a3f92c1b")

Database Schema

CREATE TABLE evaluations (
    id                  INTEGER PRIMARY KEY AUTOINCREMENT,
    run_id              TEXT NOT NULL UNIQUE,
    model               TEXT NOT NULL,
    benchmark           TEXT NOT NULL,
    num_samples         INTEGER NOT NULL,
    accuracy            REAL NOT NULL,
    avg_latency_ms      REAL NOT NULL,
    p95_latency_ms      REAL NOT NULL,
    total_cost_usd      REAL NOT NULL,
    cost_per_1k_tokens  REAL NOT NULL,
    hallucination_rate  REAL NOT NULL,
    avg_reasoning_score REAL NOT NULL,
    created_at          TEXT NOT NULL,
    metadata            TEXT NOT NULL DEFAULT '{}'
);

📄 PDF Report Generation

from llm_eval.reports.generator import ReportGenerator
from llm_eval.database.models import Database

db = Database()
records = db.list_results(benchmark="mmlu", limit=5)

gen = ReportGenerator()
pdf_path = gen.generate(records, output_dir="./reports/")
print(f"Report saved to: {pdf_path}")

The generated PDF includes:

• Cover page — title, timestamp, model count
• Executive summary table — all models side by side
• Per-model detail page — all metrics, run config

Or via CLI:

llm-eval report --run-ids a3f92c1b b4d03e2c --output ./reports/

🐳 Docker Deployment

docker-compose.yml deploys both services:

# Start everything
docker-compose up -d

# View logs
docker-compose logs -f

# Stop
docker-compose down

# Rebuild after code changes
docker-compose up -d --build

Build individual images:

# API only
docker build --target api -t llm-eval-api:latest .
docker run -p 8000:8000 --env-file .env llm-eval-api:latest

# Dashboard only
docker build --target dashboard -t llm-eval-dashboard:latest .
docker run -p 8501:8501 --env-file .env llm-eval-dashboard:latest

Environment variables in Docker:

docker run -p 8000:8000 \
  -e OPENAI_API_KEY=sk-... \
  -e ANTHROPIC_API_KEY=sk-ant-... \
  -v ./data:/data \
  -e LLM_EVAL_DB_PATH=/data/llm_eval.db \
  llm-eval-api:latest

🧪 Testing

Run the full test suite

# All tests
pytest tests/ -v

# With coverage report
pytest tests/ -v --cov=llm_eval --cov-report=html --cov-report=term-missing

# Run a specific test class
pytest tests/test_evaluator.py::TestAccuracyMetric -v

# Run a specific test
pytest tests/test_evaluator.py::TestAccuracyMetric::test_mc_correct -v

No API keys required! All tests use mocked LiteLLM responses.

Test Coverage

Linting & Type Checking

# Ruff linter
ruff check llm_eval/ tests/

# Type checking
mypy llm_eval/ --ignore-missing-imports

# Format check
ruff format --check llm_eval/

🤗 HuggingFace Dataset

The evaluation benchmark dataset is published on HuggingFace:

from datasets import load_dataset

# Load all splits
ds = load_dataset("vigneshwar234/llm-eval-benchmark")
print(ds)
# DatasetDict({
#     train: Dataset({features: ['id', 'prompt', 'expected', 'subject', 'difficulty', 'source', 'choices'], num_rows: 500}),
#     validation: Dataset({features: ..., num_rows: 200}),
#     test: Dataset({features: ..., num_rows: 500})
# })

# Use as a custom benchmark
import pandas as pd
from llm_eval.benchmarks.custom import CustomBenchmark

df = pd.DataFrame(ds["test"])
samples = df[["prompt", "expected"]].to_dict("records")

Dataset Statistics

Re-generate and push:

python huggingface/create_dataset.py --push

📁 Project Structure

LLM-Evaluation-Framework/
│
├── llm_eval/                        ← Main package
│   ├── __init__.py                  ← Version: 1.0.0
│   │
│   ├── core/
│   │   ├── __init__.py
│   │   └── evaluator.py             ← LLMEvaluator, EvaluationConfig, EvaluationResult
│   │
│   ├── metrics/
│   │   ├── __init__.py
│   │   ├── accuracy.py              ← AccuracyMetric (exact/normalized/MC/fuzzy)
│   │   ├── hallucination.py         ← HallucinationMetric + reasoning_quality
│   │   ├── latency.py               ← LatencyMetric (percentile stats, SLA)
│   │   └── cost.py                  ← CostMetric (15+ provider pricing)
│   │
│   ├── benchmarks/
│   │   ├── __init__.py
│   │   ├── mmlu.py                  ← MMLUBenchmark (HF Hub + local cache + builtin)
│   │   ├── truthfulqa.py            ← TruthfulQABenchmark
│   │   └── custom.py                ← CustomBenchmark (CSV/JSON)
│   │
│   ├── dashboard/
│   │   ├── __init__.py
│   │   └── app.py                   ← 5-page Streamlit dashboard
│   │
│   ├── api/
│   │   ├── __init__.py
│   │   └── main.py                  ← FastAPI (12 endpoints)
│   │
│   ├── cli/
│   │   ├── __init__.py
│   │   └── main.py                  ← Click CLI (7 subcommands)
│   │
│   ├── reports/
│   │   ├── __init__.py
│   │   └── generator.py             ← ReportLab PDF generator
│   │
│   └── database/
│       ├── __init__.py
│       └── models.py                ← SQLite persistence layer
│
├── tests/
│   ├── __init__.py
│   └── test_evaluator.py            ← 40+ pytest tests (no API keys needed)
│
├── .github/
│   └── workflows/
│       └── ci.yml                   ← CI: test × 3 Python versions + lint + Docker + Pages
│
├── docs/
│   └── index.html                   ← GitHub Pages (green/yellow theme)
│
├── huggingface/
│   ├── README.md                    ← HuggingFace dataset card
│   ���── create_dataset.py            ← Dataset builder + HF Hub uploader
│   └── data/                        ← Generated: train/validation/test JSON
│
├── requirements.txt                 ← All pinned dependencies
├── .env.example                     ← All env var templates
├── Dockerfile                       ← Multi-stage (api + dashboard targets)
├── docker-compose.yml               ← API + Dashboard stack
├── setup.py                         ← Package setup
├── pyproject.toml                   ← Ruff, mypy, pytest config
├── LICENSE                          ← MIT
└── README.md                        ← This file

🔧 Configuration Reference

EvaluationConfig

@dataclass
class EvaluationConfig:
    model: str              # LiteLLM model string (required)
    benchmark: str          # "mmlu" | "truthfulqa" | "custom" (required)
    num_samples: int = 100  # Number of samples to evaluate
    temperature: float = 0.0  # Sampling temperature (0.0 = deterministic)
    max_tokens: int = 512   # Max response tokens
    timeout: float = 30.0   # Per-request timeout in seconds
    concurrency: int = 5    # Max parallel API calls
    run_id: str = auto      # Auto-generated 8-char hex ID
    tags: dict = {}         # Custom metadata tags

Environment Variables

🤝 Contributing

We welcome contributions! Here's how to get started:

Development Setup

git clone https://github.com/vignesh2027/LLM-Evaluation-Framework.git
cd LLM-Evaluation-Framework
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dashboard,reports,dev]"
pre-commit install  # optional but recommended

Contribution Guide

1. Fork the repository
2. Create a feature branch: git checkout -b feature/my-awesome-feature
3. Make your changes with tests
4. Verify: pytest tests/ -v && ruff check llm_eval/
5. Commit: git commit -m "feat: add my awesome feature"
6. Push and open a Pull Request

Good First Issues

• Add a new benchmark loader (e.g., HellaSwag, ARC)
• Add a new metric (e.g., BLEU, ROUGE, BERTScore)
• Improve hallucination detection with an NLI model
• Add a new chart to the dashboard
• Add support for a new provider

📜 License

This project is licensed under the MIT License — see the LICENSE file for details.

MIT License — Copyright (c) 2025 vignesh2027

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction...

⭐ Star History

If this project helped you, please star it — it helps others find it!

Made with 💚 by vignesh2027

⭐ Star · 🐛 Report Bug · 💡 Request Feature · 🤗 HuggingFace · 🌐 Live Docs