Dicklesworthstone/skillranker: for libraries with more than 254 eligible skills

sr demo --case useful     # Inspect an offline fixture before connecting a session
sr rank --allow-network   # Rank skills for the selected session
sr hook claude           # Run the Claude Code prompt-hook integration
sr tui                   # Inspect rankings in an inline terminal display

TL;DR

The problem. A large skill library gives an agent plenty of procedures to choose from, but choosing is itself a task. Similar descriptions obscure useful distinctions. A skill that helped at the start of a conversation can be irrelevant three turns later. Loading a plausible but unsuitable skill consumes context and can redirect otherwise sensible work.

The solution. SkillRanker (sr) combines the recent conversation, current request, workspace signals, and the selected harness's visible skill inventory. Jev from TypeSafe.ai is the key enabler of the system. It first compares the candidates broadly, then reads richer excerpts from a shortlist and evaluates whether each one fits. Both comparisons include a real “none of these” option. The result is advisory: the agent follows the user's instructions and decides what to consult.

For libraries with more than 254 eligible skills, Quill from FrankenSearch narrows the candidates locally before Jev evaluates them. Smaller rosters reach Jev in full. Explicit skill requests resolve locally before either stage.

SkillRanker does not include a local model or a substitute inference provider. The ranking workflow requires your own TypeSafe account and API key. Local retrieval prepares the candidates; Jev supplies the evaluations that make the recommendations possible.

Why sr?

The approach builds on the TypeSafe skill-suggestion recipe. SkillRanker adds session identity, harness visibility, bounded execution, and a local evaluation loop. The comprehensive plan explains the full design and acceptance criteria.

Quick Example

# See a labeled fixture result without a key, network, or private session.
sr demo --case useful

# Inspect local configuration and the available adapters.
sr doctor --json
sr doctor --config
sr capabilities --json

# Inspect visible, shadowed, and excluded skill records.
sr roster --json

# Preview the redacted wide-pass request without network or persistence effects.
sr rank --context scratch/context.json --dry-run

# Evaluate an explicitly selected conversation.
sr rank --context scratch/context.json --allow-network --json

# Find where an expected candidate was excluded; this adds no inference calls.
sr rank --context scratch/context.json --allow-network --why-not SKILL_ID --explain

# Preview the Claude hook settings change, then apply it.
sr install-hook claude
sr install-hook claude --apply

# Review observations without treating adoption as proof of usefulness.
sr stats --since 7d --by-skill

# Replay a labeled evaluation artifact without making network requests.
sr eval --dataset scratch/evaluation.json --explain

# Inspect description quality and suspected coverage gaps locally.
sr doctor --descriptions
sr gaps

Design Philosophy

1. Choose for the next action. The current request matters, as do the recent failure, the task context, and the instructions already loaded.
2. Resolve authority locally. The user decides what is required or excluded. The harness determines what can be loaded. A model answer cannot change either.
3. Separate preference from applicability. Winning a comparison is not enough. A recommendation must survive fit, visibility, loaded-state, and none-option checks.
4. Keep evidence inspectable. Preserve provider estimates and local arithmetic. --explain exposes computations without inventing model-generated reasons.
5. Separate adoption from usefulness. Observing a load is useful telemetry. Learning a better policy requires independently judged examples and a holdout.
6. Bound each evaluation. Input, discovery, subprocesses, networking, retries, persistence, and cleanup consume one ranking deadline. Batch evaluation also has explicit request and total-runtime limits.
7. Stay standalone. Discovery, parsing, redaction, retrieval, and feedback live in sr. No skill-manager service or private database is required.

How It Compares

These are workflow choices, not benchmark rankings.

SkillRanker recommends procedures. It does not execute skills, grant permissions, or override the agent's governing instructions.

Installation

From source

Build the sr binary with the repository's pinned Rust toolchain and lockfile:

git clone https://github.com/Dicklesworthstone/skillranker.git
cd skillranker
cargo install --locked --path . --bin sr

For a checkout-local binary:

cargo build --locked --release --bin sr
./target/release/sr --help

Runtime setup

Sign up for TypeSafe.ai, then create your own API key in the console. A TypeSafe API key is required to use SkillRanker's ranking system. Jev is the evaluation engine for the entire ranking workflow. Every user supplies their own credential; SkillRanker does not distribute a shared key.

Set TYPESAFE_API_KEY through your shell or secret manager. The environment example lists the service settings. For a local checkout, create .env from the example only if it does not already exist and restrict access with chmod 600 .env before entering your own API key. The .env file is ignored by Git. Export its values into the process environment before running sr or starting an agent whose hooks need the key:

# Run from your checkout, after filling in your own trusted .env file.
set +x
set -a
. ./.env
set +a

Treat .env as a local shell configuration file and source only content you trust. Keep the key out of shell history, logs, and tracked files. Credentials alone do not enable remote transmission.

sr does not require ms, a local inference server, or an embedding model. The primary local platform scope is Linux and macOS. Consult sr capabilities --json for the adapters, events, and optional features in a build.

Quick Start

1. Try an offline fixture. Run sr demo --case useful, then try none, explicit, or unavailable. These labeled examples exercise the local pipeline without reading a private session or contacting Jev. Their output is non-actionable and does not establish live provider health.
2. Sign up and configure your own TypeSafe API key. Create an account and key in the TypeSafe console, then export TYPESAFE_API_KEY using the runtime setup instructions. SkillRanker relies on Jev for its ranking evaluations.
3. Check the environment and roster. Run sr doctor --json, sr capabilities --json, and sr roster --json in the agent's workspace. Confirm that candidates are loadable, not merely present somewhere on disk.
4. Choose the session. Supply --context FILE, --transcript FILE --harness claude_code, or --session PATH for cass. Automatic discovery must resolve one unambiguous session.
5. Preview and rank. Use --dry-run to inspect the redacted wide payload and disclosure receipt, then --allow-network --json for a fresh evaluation.
6. Prepare a recorded shadow trial. Initialize optional history with sr ledger init, then explicitly set network.enabled = true in trusted user configuration if you want live hook evaluations. The earlier --allow-network flag authorized only that CLI run. Preview and apply sr install-hook claude; shadow mode evaluates without injecting advice. Without a ready ledger, the hook can rank but cannot promise recorded trial evidence; without network consent, it cannot obtain fresh Jev answers.
7. Enable advisory output deliberately. Set hook.mode = "advisory" in trusted user configuration after reviewing the integration and its behavior.

Readiness checks

sr doctor reports each prerequisite separately, names the next concrete step for a failed check, and lists local commands that remain useful:

Doctor runs locally by default and does not implicitly send a test request, install hooks, migrate storage, or change configuration. Demo uses bundled synthetic contexts and labeled synthetic or recorded responses without touching user configuration or state. Neither command supplies a replacement for Jev.

A previous transport check includes its time, scope, and runtime/endpoint/configuration identity. Incompatible changes invalidate it; a stored check does not establish current provider availability.

sr capabilities --json separates implemented adapters from tested harness versions/features and unverified versions. Adapter conformance covers prompt timing, branch identity, visibility, restrictions, compaction, load evidence, and hook output, deadlines, and delivery. Native advice requires passing real-harness evidence for every dimension on the installed version; protocol fixtures and official schema documentation alone cannot authorize it. Cass remains an explicit archive source. Native Codex, omp/pi, and Grok integrations each need their own support record. Incompatible identity or visibility semantics disable advice. The adapter contract defines these checks.

Command Reference

Bare sr is equivalent to sr rank: a table on a TTY, JSON otherwise. It does not start a TUI. Source flags are mutually exclusive, and piped stdin is consumed only by an explicit input mode. sr capabilities --json lists the commands, schemas, and optional features available in the installed build.

Ranking and inspection

Hooks, feedback, and analysis

--apply performs a previewed hook, snooze, budget, calibration, or ledger mutation. Calibration consumes a labeled evaluation artifact; it does not silently change project settings after a number of observed loads. Description audits use the network only with an explicit online request and network authorization.

Evaluation controls

sr eval defaults to replay with zero network requests. A live run needs --online, trusted network authorization, and an explicit --max-requests cap. That cap counts HTTP attempts across the entire batch, including retries. Each case also has its own ranking deadline; the batch stops scheduling work when a request or runtime limit is reached and reports unfinished cases.

# Reproduce a diagnostic selection; seed 42 alone supports no sampling guarantee.
sr eval --dataset scratch/evaluation.json --sample-size 100 --seed 42 --explain

# Draw and record a random sample, then authorize a bounded live evaluation.
sr eval --dataset scratch/evaluation.json --sample-size 100 \
  --online --allow-network --max-requests 400 --max-runtime-ms 600000

Sampling does not grant network access or enlarge the request budget. Missing stage responses remain unevaluated in replay; they are never replaced with invented scores. See evaluation and sampling for the report's denominators and uncertainty rules.

Ranking controls

Sizes satisfy 1 ≤ K ≤ M ≤ 32; fewer available candidates is normal. Parsing is strict, with documented aliases only. Invalid or conflicting privacy flags produce an error rather than being silently corrected.

An explicit chunk-overflow experiment uses bounded groups and reduction rounds. It has separate request limits and availability in the capabilities contract; the normal overflow policy uses local prefiltering.

JSON output

The versioned output contract specifies decision, error, quality, trace, and non-actionable report schemas.

This illustrative result has two eligible candidates. The score arithmetic uses w_fit = 1, with priors and phase weighting disabled; timing and usage are examples.

{
  "schema_version": 1,
  "event_id": "example-event-001",
  "decision": "ranked",
  "reason": "eligible-candidates",
  "harness": "claude_code",
  "context_quality": "complete",
  "quality": {
    "prompt_complete": true,
    "task_anchor_known": true,
    "history_windowed": true,
    "attachments_omitted": false,
    "source_gaps": false
  },
  "roster": {
    "total": 2,
    "eligible": 2,
    "wide_candidates": 2,
    "shortlist": 2,
    "partial": false,
    "retrieval": "full",
    "provenance": {
      "snapshot_id": "000000000000000000000000000000000000000000000000000000000000000a",
      "policy_version": "ranking-v1",
      "wide_set_id": "000000000000000000000000000000000000000000000000000000000000000b",
      "rerank_set_id": "000000000000000000000000000000000000000000000000000000000000000c"
    }
  },
  "needs_skill": 0.74,
  "choice_confidence": 0.81,
  "none_probability": 0.10,
  "phase": "debugging",
  "skills": [
    {
      "rank": 1,
      "skill_id": "s_01",
      "name": "rust-test-triage",
      "invocation_name": "rust-test-triage",
      "rank_score": 0.888889,
      "rerank_probability": 0.60,
      "wide_probability": 0.55,
      "fits": 0.80,
      "path": ".claude/skills/rust-test-triage/SKILL.md",
      "content_hash": "0000000000000000000000000000000000000000000000000000000000000001"
    },
    {
      "rank": 2,
      "skill_id": "s_02",
      "name": "rust-code-review",
      "invocation_name": "rust-code-review",
      "rank_score": 0.111111,
      "rerank_probability": 0.30,
      "wide_probability": 0.35,
      "fits": 0.50,
      "path": ".claude/skills/rust-code-review/SKILL.md",
      "content_hash": "0000000000000000000000000000000000000000000000000000000000000002"
    }
  ],
  "omitted_rank_mass": 0.0,
  "cache": {
    "hit": false,
    "wide_hit": false,
    "rerank_hit": false,
    "age_ms": null,
    "stale": false
  },
  "model": {
    "requested": "jev-latest",
    "wide_returned": "jev-latest",
    "rerank_returned": "jev-latest",
    "immutable_revision": null
  },
  "usage": {
    "requests": 2,
    "http_attempts": 2,
    "input_tokens": 6400,
    "output_tokens": 480,
    "unknown_usage_attempts": 0
  },
  "persistence": "recorded",
  "warnings": [],
  "warnings_omitted": 0,
  "elapsed_ms": 720
}

Demo and replay use separately versioned, non-actionable envelopes around their synthetic or historical decisions. A demo's context is synthetic; any recorded provider response retains its own provenance. It remains demonstration evidence, not a live evaluation or a quality-gate result. Neither artifact uses the live hook output channel.

choice_confidence describes the rerank distribution. fits is a model estimate of suitability. rank_score is a relative local score over eligible candidates. They are different quantities. Top-K truncation preserves the original eligible normalization and reports omitted mass. Fields from an unexecuted stage are null, not fabricated zeros.

Cache hits and returned model identities are recorded separately for each stage. When reranking is required, a wide-stage hit alone is not a complete offline result. An alias such as jev-latest does not identify an immutable model revision. persistence: "recorded" means the ranking metadata was committed before output; it does not mean the harness acknowledged or consumed the recommendation.

Quality metadata includes prompt_complete, task_anchor_known, history_windowed, attachments_omitted, and source_gaps. These describe the admitted input; context_quality: "complete" does not claim that the entire conversation history was read. Warnings are bounded to 32 details plus an omitted count. Live decision JSON and individual demo/replay/report summary envelopes are capped at 2 MiB. The separate 256 MiB evaluation-artifact limit bounds a streamed dataset or report with at most 10,000 case records; it does not enlarge an individual output envelope. Roster listings and full-wide explanations paginate against a fixed snapshot; a changed snapshot requires restarting. Trace pages contain at most 128 entries and bind both snapshot and query identity. Unevaluated entries keep operands and reasons null instead of inventing evidence.

Exit codes

JSON errors include schema_version, decision: "unavailable", and an error object with code, kebab-case kind, message, hint, and retryable. Ordinary ranking can succeed with a storage warning; an explicit feedback write cannot claim success when its required write failed. Incomplete essential context and output-limit failures use exit 7, unresolved explicit requests and Quill retrieval failures use 5, and superseded input uses 3. Overall deadline exhaustion uses 6. retryable means a fresh invocation with the same intended inputs may succeed; it does not grant network access or relax a deadline.

Evaluation and replay reports distinguish execution from quality: run_status is complete or partial; gate_status is passed, failed, not-established, or not-applicable. Exit zero means the report was produced, not that every comparison was replayable or a policy passed. Promotion requires the intended complete cohort, compatible evidence, and explicit passed gates. Fatal errors retain their nonzero exit code and available partial-work/usage data. Each summary accounts for requested/completed cases and required/completed stages. Partial, incompatible, empty, or synthetic evidence cannot carry a passed gate. A completed report can faithfully describe a failed historical request without treating that historical error as a failure to generate the report.

The dedicated hook maps recommendation failures to quiet exit-zero behavior so it never blocks the agent. CLI failures retain their meaningful exit codes.

Configuration

Ordinary settings resolve from lowest to highest priority:

built-in defaults
  -> trusted user configuration
  -> allowlisted workspace configuration
  -> recognized SR_* environment variables
  -> command-line flags

On Linux, user configuration falls back to ~/.config/sr/config.toml; project configuration is .sr/config.toml at the workspace root. Other platforms use native configuration directories.

Trusted user settings for an advisory hook include:

[network]
enabled = true

[hook]
mode = "advisory"

Without those choices, remote transmission is disabled and the hook runs in shadow mode. --allow-network can authorize a single CLI evaluation.

Workspace configuration may tune bounded ranking values and exclusions. It cannot authorize networking, change endpoints/proxies, supply credentials, expand transcript access, disable redaction, or enable raw retention. Unknown or duplicate keys, forbidden project settings, and invalid values are errors after bounded configuration reads and before discovery, networking, or mutation. Unrecognized SR_* environment names are errors too. Project context settings may reduce message/character budgets, select minimal, or remove tool content; they cannot widen a trusted user's disclosure settings. Exclusions and skill roots merge as unions. Project roots must remain relative to the workspace; only trusted user configuration can authorize absolute roots. Readers also check symlink containment when opening files.

Credentials are environment-only and kept outside serializable configuration. The v1 endpoint override is environment-only; model overrides use trusted user configuration or SR_MODEL. Proxy, redaction-disable, and raw-retention settings are reserved and rejected. The configuration contract lists every key, bound, source layer, and restriction.

The endpoint accepts an empty or root path and rejects URL credentials, query strings, fragments, and non-root paths. Credentials, cache identity, and request allowance scope use the canonical origin. A development-only loopback HTTP exception cannot carry production credentials.

sr doctor --config shows non-secret effective values and their winning sources. Disallowed overrides are configuration errors; no valid policy or fingerprint is reported for them. Credential presence is reported without its value, and endpoint overrides expose only their presence. --offline and --allow-network are accepted here but conflict; this inspection command never sends a request.

The library's cli::ConfigFiles shares bounded initial configuration reads with file refreshes. Refresh keeps the invocation's validated environment and CLI layers, returning the current configuration and a boundary-specific comparison against its typed policy receipt. Malformed, unreadable, or late reads fail rather than authorize output. Rank orchestration and its HTTP/publication call sites are not implemented yet; these library checks do not establish live ranking support. Shared request allowances and snoozes are explicit trusted-user controls. Project configuration cannot enable, raise, or disable the allowance.

--offline and --dry-run conflict with --allow-network. Case capture conflicts with --dry-run and --no-persist. --no-cache and --no-ledger disable their respective stores independently; --no-persist also disables persistent runtime coordination. Ordinary configuration reads remain allowed.

Each provider attempt rechecks the effective disclosure and admission policy. Publication separately rechecks the fields governing eligibility and hook mode. A revoked permission or changed governing value withholds the affected action; it never authorizes a replacement request. An edit hidden by an unchanged CLI or environment override does not change the effective policy.

How Ranking Works

1. Establish the exact context

The Claude hook uses the incoming prompt as the current request, even when the transcript has not yet recorded it. Context, cursors, and feedback belong to a specific workspace, session, agent branch, and source adapter/producer. An explicit source that fails does not silently fall through to another conversation.

Native JSONL reads process complete records within a bounded tail. Replacement, truncation, compaction, and incomplete final lines are handled explicitly. An empty first transcript can still yield prompt-only context; malformed existing history is a different condition.

Windowing drops reasoning blocks, binary/media payloads, and prior sr advice. Tool summaries preserve invocation/result association and useful failure lines. The latest request gets budget priority, with explicit head/tail truncation for oversized input. Redaction runs on complete bounded fields before truncation, then on the assembled provider payload.

Explicit directives are resolved from the full bounded local request before redaction or truncation. The normalized input envelope carries local identity and events; it is validated and reduced to a separate provider schema, never sent wholesale to Jev.

A normalized import cannot update a native session's observations merely by repeating its session ID. An input without durable session identity gets an invocation-local namespace; unknown attribution disables durable session updates. A terse “continue” needs a recoverable task anchor; an essential missing instruction or attachment yields unavailable and a quiet hook, rather than a guess from incomplete context.

Project signals use language/framework filenames, allowlisted tools on a trusted PATH, and bounded repository-relative dirty paths. Absolute workspace paths and branch names remain local by default. Git inspection disables filesystem-monitor hooks, optional index locks, and submodule traversal; an unsupported safe invocation omits the optional signal instead of executing project helpers.

The library boundary context::signals::collect implements these optional local signals with an explicit authorized workspace and trusted executable roots. Its 250 ms stage intersects the invocation budget; status output is capped at 64 KiB and 100 paths. Non-UTF-8/unsafe paths and partial inventories are reported rather than treated as complete absence. Results are not serializable provider payloads: callers must still redact them. sr rank collects them for every evaluation, using fixed system directories (/usr/local/bin, /usr/bin, /bin) as its trusted executable roots; the minimal context profile omits dirty paths. Synchronous filesystem/spawn calls retain the subprocess boundary's documented uninterruptible-kernel-I/O limitation.

2. Resolve what is loadable

An explicit --roster FILE replaces discovery. Otherwise, a harness inventory or its visibility adapter determines roots, overrides, plugins, and load targets. The presence of a directory does not mean the selected harness loads its skills. Generic file mode uses explicitly configured roots and exposes uncertain visibility.

Each skill has an opaque stable ID, its actual invocation name, display name, source, content hash, load target, metadata, and visibility. Same-name skills remain distinct where the harness permits; shadowed or ambiguously invocable entries are excluded from hook suggestions.

For Claude, sr rank resolves same-name skills by Claude's documented precedence (project skills over personal ones). That precedence is not yet backed by conformance evidence, so every rank result carries "visibility": "unverified", and the decision's first warning is unverified-visibility: confirm that a suggested skill loads before relying on it. sr roster and sr doctor claim no precedence at all. Names whose authority is withheld (for example, a malformed file that could claim the same name), ambiguous names and shadowed entries are never suggested.

Invocation restrictions are part of eligibility. Claude skills with disable-model-invocation: true or an effective user-only restriction are excluded from automatic advice; user-invocable: false alone does not exclude agent use. A user-requested manual-only skill resolves as a manual_only reference, without instructing the agent to bypass that restriction by reading its file.

Reject duplicate keys and duplicate record definitions within each schema's collection/namespace in normalized inputs, rosters, configuration, replay/evaluation artifacts, frontmatter, and provider responses. The same skill can still be referenced across wide and rerank stages. Ambiguous skill metadata excludes that record and marks coverage partial. Configuration cannot execute interpolation or recursive includes. Evaluation imports use bounded streaming and enforce per-case limits as well as the total cap.

Source snapshots supply both hashes and excerpts. Before publishing a live advisory decision or no-match claim, sr checks membership, precedence, and indexed/wide content that conditioned the decision, plus the entire shortlist's content and restrictions. A changed candidate outside the shortlist or a new overflow match can invalidate the result too. Explicit resolution checks every target and its name's precedence. Trusted adapter generations can avoid a rescan only when they cover all required dependencies. Otherwise bounded re-enumeration/content checks use the same deadline; missing required validation withholds output. A runner-up cannot replace an answer conditioned on stale alternatives. These are last-validation observations, not a freeze of the filesystem; the harness still checks its later load. A supplied roster replaces discovery but grants no new path access or invocation permissions.

sr roster and the roster check in sr doctor also inspect directories from roster.roots in trusted-user and project configuration. Each directory uses the <skill-name>/SKILL.md layout. Project roots stay inside the workspace, including when symlinks are involved; trusted-user configuration can name absolute roots. Configured roots extend the default Claude directories, and aliases of an already opened directory are deduplicated. Reading a custom root does not prove that the harness can invoke its skills: those entries remain unverified.

sr roster --snapshot FILE explicitly exports an owner-only manifest, bounded to 32 MiB and 10,000 records, without implicitly overwriting an existing file. Manifests can contain private skill names. sr roster --diff FILE compares the snapshot with fresh authorized discovery in the same workspace, adapter, and source namespace. It reports additions/removals, content and restriction changes, shadowing, and invocation-name changes. Incompatible manifests are identified; incomplete source coverage remains unknown rather than becoming a confirmed deletion. Saved paths grant no new read access and cannot restore a removed skill.

3. Retrieve, then compare

Explicit requirements are resolved from the complete visible roster first. They bypass probabilistic retrieval and cannot be vetoed by a low gate. Every requested reference must resolve: missing, ambiguous, forbidden, or conflicting references produce unavailable / explicit-resolution with separate resolution records and no advisory API call. Successful explicit lists are not truncated to top-K; the input limit is 32 explicit references.

For advisory ranking, Quill, the native lexical engine in FrankenSearch, supplies BM25 retrieval when the eligible roster exceeds 254 real skills. Every Choice also includes __none__, for at most 255 total options. Retrieval uses the latest request plus bounded task and error context; a terse “continue” retains useful prior evidence.

Configured sizes first satisfy 1 ≤ K ≤ M ≤ 32. The effective rerank size is min(M, admitted_wide_count), and the output cap is min(K, effective_M). For example, three Quill matches produce a wide Choice with three skills plus none, a rerank of at most three skills, and at most three returned suggestions. A single match is valid and still competes against none. An initially empty roster is a roster failure; a valid roster reduced to zero by explicit exclusions or proven available references yields a local abstention.

SkillRanker embeds Quill's in-memory index through frankensearch-quill, with default features disabled, the required frankensearch-core document types, bounded indexing/query work, and the caller's Asupersync context. Names and aliases are searchable titles; descriptions and tags are searchable content, rather than stored-only metadata. Documents enter in stable skill-ID order and are committed before querying. Cutoff ties follow the pinned document-ID mapping; re-sorting an already-truncated result cannot recover an omitted tied candidate.

Queries are a deduplicated OR of escaped literal terms. Only the adapter adds the OR separators; conversation text cannot introduce Boolean operators, wildcards, ranges, or field syntax. The limit of 128 distinct terms and 4,096 Unicode scalar values applies after escaping and separators; analysis input and work are bounded too. No analyzed terms means retrieval-empty. Parser diagnostics and truncation are reported. A query that cannot be preserved safely, exhausted query fuel, or an index failure yields unavailable output with quiet hook fallback. Retrieval failures use exit 5; exhaustion of the overall invocation deadline uses 6. Partial work never becomes a complete candidate set, and there is no silent fallback to another engine.

Overflow results identify retrieval: "quill-bm25", the admitted count, and engine/schema provenance. Building, committing, and querying the index consume the same ranking deadline. A TUI can retain a matching roster index across refreshes; a new hook process cannot assume an earlier process's index survives.

Quill is the only lexical search engine used by this project. Tantivy is not used for runtime search, fallbacks, tests, benchmarks, or reference code. The hybrid search facade, legacy lexical engine, Quill gauntlet, and optional oracle/compatibility features are excluded. Dependency checks cover normal, build, and development feature graphs. Quill verification uses native tests and independent expected-result fixtures. SkillRanker does not need the fsfs command, an embedding model, a search service, or an imported foreign index.

The wide pass combines a Choice, phase distribution, and three oriented gates:

needs_skill = mean(
    specialized_method,
    material_help,
    1 - context_suffices
)

needs_skill is a heuristic score. Below the default 0.30 threshold, sr abstains without a rerank. The wording includes planning, analysis, writing, and explanation skills; acting on files is not a prerequisite for needing a method.

When the gate passes, up to eight real candidates proceed to a detailed Choice with another none option and one fit Noul per candidate. If none wins the wide comparison, the detailed comparison still runs when the need gate passes: richer skill excerpts can resolve ambiguity left by short descriptions. The client uses the TypeSafe HTTP API, preserving typed answers and validating every requested option before scoring.

4. Apply eligibility and rank survivors

A candidate is removed if it is excluded, below the fit threshold, or a reusable reference whose relevant content is proven present in the current context epoch. Workflows and unknown usage kinds remain eligible for repeat invocation. A changed shortlist invalidates the in-flight result.

The reusable-reference check needs evidence of the version and rendered content actually present. A source file's current hash cannot establish what a past read consumed. Changed arguments, dynamic content, forked execution, or compaction can invalidate reuse evidence; content presence never renews turn-scoped permissions.

Every remaining candidate must individually beat the none option's raw rerank probability. Ties are excluded. If no candidates survive, sr abstains; local priors and fit blending cannot re-admit a candidate that failed this check.

For each eligible candidate:

eps = 1e-6
clip(x) = min(1 - eps, max(eps, x))
log_odds(x) = ln(clip(x) / (1 - clip(x)))

utility_i = ln(clip(p_rerank_i))
          + w_fit   * log_odds(fits_i)
          + w_prior * prior_delta_i
          + w_phase * phase_match_i

rank_score_i = softmax(utility)_i

Defaults are w_fit = 1.0, w_prior = 0.0, and w_phase = 0.0. Priors and phase weighting are optional evaluated policy choices. A skill is not penalized just because a previous suggestion went unobserved. After compaction, uncertain loaded-state evidence cannot suppress a skill indefinitely.

Explain And Replay A Result

Find where a candidate was lost

--why-not SKILL_ID --explain follows a candidate through discovery, visibility and restrictions, local policy, Quill admission, the wide shortlist, fit/none eligibility, final ordering, and publication. It reports the first decisive exclusion and any later stages actually evaluated.

sr rank --context scratch/context.json --allow-network \
  --why-not SKILL_ID --explain --json

An unevaluated stage reports not-evaluated; an unknown ID reports not-in-snapshot. Neither receives a fabricated zero fit. Explanations include threshold operands, tie handling, content/policy versions, and bounded recovery hints. They do not expand discovery, insert the target into a shortlist, change the ranked result or provider request bytes, or add a provider call. Hints identify actions and arguments for review; they do not execute commands or relax policy.

Save a case and compare local policies offline

Explicit capture turns a surprising result into a reproducible case:

# Opt in to retaining this run's bounded redacted inputs and recorded answers.
# Use a new output path in an existing private directory.
sr rank --context scratch/context.json --allow-network \
  --save-case scratch/ranking-case.json --explain --json

# Reproduce the decision without accessing the source session or Jev.
sr replay scratch/ranking-case.json

# Compare compatible local thresholds/weights against the same recorded answers.
sr replay scratch/ranking-case.json --policy scratch/baseline.toml \
  --compare-policy scratch/candidate.toml

A case binds the actual redacted request inputs, exact option maps and content digests, validated recorded responses, eligibility evidence, policy, and model/adapter provenance. It freezes the evaluation time, snoozes, loaded-state and visibility evidence, ordered candidates, numeric priors/phase inputs, and computation versions. Replay never consults today's clock, configuration, or ledger priors. Synthetic fixtures and recorded provider answers are labeled separately. Replay returns kind: replay with actionable: false, keeping historical and recomputed decisions separate from live recommendations. It makes no network requests, discovers no transcripts, executes no skills, and writes no state. Only the explicit case and optional local policy files are read; ambient user/project policy and credentials are not consulted. Embedded source paths remain inert.

Local policy comparison requires the compatible inputs and responses needed by that comparison. A completeness manifest identifies each captured stage. A low-gate case can replay its original abstention without a rerank, but lowering the gate needs the missing rerank. Missing required stages or an uncaptured prior report not-replayable; a changed model, prompt, retrieval strategy, excerpt, or shortlist needs new consented evaluation. Policy files accept only the supported local ranking schema, without executable code or credential/routing settings. A changed score is not evidence of better task outcomes. Ordinary metadata-only history cannot reconstruct a case.

Exact parity covers decisions and numeric outputs under the same tested computation profile, not new invocation IDs, timings, or usage. Different numeric backends/build profiles need declared tolerances or report incompatible exact replay. An unavailable historical case can reproduce sanitized failure metadata without inventing an unobserved response. Additional privacy transformations remove any affected exact-input replay claim. Artifact digests need no original cache secret and cannot authorize importing a result into the live cache.

Capture is opt-in because redacted prose can remain confidential. All retained prose is redacted; credentials, hash keys, secret-bearing configuration, and response error bodies are excluded. Files are owner-only, created exclusively without overwriting existing targets, bounded to 16 MiB and nesting depth 64, and published only after a complete write. Publication uses a tested atomic no-clobber operation, including protection against a target or symlink raced into place, with a documented flush policy. A crash may leave a complete export whose delivery is unknown; filesystem publication and stdout are not atomic. Private, bounded partial files remain identifiable for explicit cleanup. These publication rules also apply to roster snapshots. Per-field limits still apply. An oversized case fails explicitly instead of dropping data needed for replay.

Capture consumes the invocation deadline; a failed requested write reports a storage or timeout failure, preserving any already-incurred attempts and usage. --save-case conflicts with --dry-run, --no-persist, and hook mode. Imports validate bounds and internal consistency; a matching digest does not establish trusted authorship. Imported labels and responses remain untrusted evaluation data.

Local Feedback And Calibration

SkillRanker keeps observations and judgments separate.

sr stats puts usefulness, interruptions, and cost in one local report:

Cost per judged-useful suggestion uses only that cohort's matched attempts and labels. With no useful labels, the ratio is not estimable. Unknown usage or missing/inapplicable pricing also prevents an exact monetary ratio; report known attempt and token counts instead. Unlabeled traffic does not inherit measured usefulness, adoption is not task success, and token savings are not estimated labor savings. Reports contain no raw examples by default and do not enable adaptation or advisory mode.

Shadow, advisory-hook, CLI, and TUI records have separate denominators. Writing zero bytes in shadow mode does not count as delivered advice. A successful path-only read can establish a load but cannot establish the consumed version for suppression or version-specific feedback.

Observation ingestion has its own cursor, separate from the bounded ranking window. Observations, loaded-state evidence, and cursor advancement commit together; unread events remain a backlog, not skipped history. Attribution uses the latest preceding emission with a known boundary in the same agent/turn. Ambiguous concurrent delivery remains unknown, and identical prompt text does not merge distinct turns. sr observe can reconcile the final turn without waiting for another user prompt.

Observation requires an explicit source, with the same meaning as ranking: --transcript FILE --harness NAME is native, --session PATH selects cass, and --context FILE selects a normalized producer. Those namespaces remain distinct; matching a session ID or path cannot authorize updating another adapter's cursor. Unknown durable identity is an error. sr observe never calls Jev and requires a ready ledger; --no-ledger and --no-persist conflict with its promised update.

sr observe --transcript scratch/session.jsonl --harness claude_code
sr stats --since 7d --by-skill
sr feedback EVENT_ID --skill SKILL_ID --verdict useful
sr eval --dataset scratch/labeled-cases.json
sr calibrate --evaluation scratch/evaluation-report.json
sr calibrate --evaluation scratch/evaluation-report.json --apply

When a different skill would have helped, record the correction directly:

sr feedback EVENT_ID --skill ORIGINAL_ID --instead BETTER_ID

The two distinct candidates resolve against historical membership, versions, and advisory eligibility. The ledger retains bounded, deduplicated membership snapshots of the full roster, including skills outside the shortlist, without their bodies or descriptions. An alternative that was manual-only or excluded was not a missed advisory candidate. The correction records the original as unsuitable and the alternative as useful in one atomic transaction, with assessor provenance and expected label revisions. A failed alternate lookup cannot leave an unintended standalone negative label. --instead and --verdict are mutually exclusive. It is partial, unblinded feedback: other candidates remain unjudged, and the correction cannot become an independent blinded holdout case. A known-absent alternative becomes a separate prospective proposal. A missing historical snapshot means unknown membership, not absence, and prevents an unsupported historical correction. Changed or unknown versions need separately identified evidence. Neither case establishes that the original selector missed an available candidate. For “not now,” use snooze instead of a negative usefulness label.

Calibration uses independently judged positive, no-match, and near-miss cases, with separate training, validation, and final-test task families. Priors are fit on training data, thresholds are chosen on validation data, and the frozen policy is evaluated once on the final holdout. Future labels cannot enter an earlier case's prior snapshot. The default tuning loss is:

Compare policies on the same predeclared judged cases and denominator. An operational failure keeps its unavailable status and separate failure count; its loss penalty prevents dropping hard cases from making a policy look better. On equal loss, prefer fewer failures, then the frozen baseline. A policy that always stays silent still incurs misses on positive cases. Bounded-loss sampling uses this loss divided by two; reports identify the scale and keep it distinct from a binary error rate.

Missing replay evidence is different from an observed failed attempt. A policy whose required responses are absent is not estimable; do not invent a result or remove those cases to complete its comparison. Lowering a gate needs rerank evidence, and changing retrieval, shortlist size, prompts, or model can require new evaluations. A batch stopped before finishing the declared cohort remains partial and cannot promote a policy.

Experimental priors use centered, shrunk Beta(1, 4) estimates from judged usefulness. They are disabled by default and can only reorder eligible candidates. A fixed observation count alone never enables learning.

Policy rollback follows the same preview/apply workflow:

sr calibrate --rollback POLICY_REVISION
sr calibrate --rollback POLICY_REVISION --apply

Rollback restores only managed ranking-policy fields after compatibility, digest, and conflict checks. It preserves current credentials, networking authorization, hook installation, feedback, and unrelated settings; it does not replace the entire configuration with an old file.

Description diagnostics flag missing metadata, duplicate visible prefixes, and rerank disagreements. Gap reports identify suspected missing coverage while showing retrieval and context quality. Text clustering requires separately enabled retention of redacted excerpts; metadata cannot reconstruct a private request. Quill retrieves retained examples and candidate neighbors; optional local clustering operates on those examples. Neither command edits skills or invokes a skill manager.

Evaluation, Sampling, And Risk Monitoring

SkillRanker evaluates whether a recommendation helps under the user's current constraints and the harness's permissions. Labels come from independent review of the full visible roster. Each case can have several acceptable additional invocations, or none. Explicit-name resolution has its own tests and is excluded from advisory quality metrics.

Measure the whole selection pipeline

Candidate coverage asks whether the selector finds any acceptable option. Optional set recall measures the fraction of all acceptable skills recovered and is reported separately. This matters when a case has twenty acceptable skills but the shortlist holds eight.

Benchmark retrieval and rerank stages run independently of the production gate so low-gate cases do not disappear from the comparison. Reports also show the actual gated decisions, operational failures over all attempted cases, and counts of unjudged cases. Baselines include Quill-only lexical selection, the cookbook-style selector, Choice-only, fit-only, and the default blend, with both recent-context and latest-request-only inputs.

Reports bind results to dataset and split digests, roster content and visibility, prompts, policy, runtime, and returned model identities/time ranges. Related sessions and task variants stay in one split. An unversioned Jev alias limits reproducibility even when the local sample and arithmetic replay exactly.

Robustness evaluation includes option-order permutations, opaque-ID renaming, equivalent whitespace, duplicate-looking descriptions, irrelevant decoys, long distracting text, and hostile ranking instructions. Local transformations with unchanged semantics have exact parsing/scoring invariants. Jev comparisons report decision and coverage changes rather than demanding identical probabilities from a stochastic, candidate-dependent model. New provider inputs require new responses and consume the live budget. Related variants retain their original task family and split.

Spend the evaluation budget deliberately

--sample-size freezes a sampling manifest before selected cases run. Unless replaying a recorded draw, omit --seed for a fresh seed from trusted OS randomness. The unit is one representative per independent task family, selected by a recorded rule before inspecting evaluated outcomes. The manifest records:

• The consented frame, split digests, and observable strata, such as normal versus overflow retrieval and complete versus degraded input.
• Stratum population and sample sizes, selected IDs, and justified inclusion probabilities for random selection without replacement within a stratum; diagnostic selections leave these probabilities unknown.
• The RNG algorithm, version, seed, randomization provenance, design status, policy/model identities, and label/request budgets needed to replay selection.

A manually fixed seed supplies reproducibility, not evidence for the claimed inclusion probabilities or design-based uncertainty. Such a run is diagnostic unless its input carries a matching prior randomization manifest; replaying that manifest is the same draw, not an additional independent sample. Imported provenance remains supplied/unverified. Full-frame evaluation needs no sampling randomness. Guarantees assume uniform selection within strata; the recorded PRNG and bounded-draw checks support the implementation without proving exact uniformity over every possible seed. Never redraw to obtain favorable results.

Every represented stratum receives a positive allocation. Gate-abstained, operationally failed, and unknown-metadata cases remain in the sampling frame. If the budget cannot cover it, the report must use a declared narrower population or a predeclared stratum-merging rule.

Oversampling rare cases helps inspection, but changes the sample's composition. For error indicators or another declared loss in [0,1], the report weights each stratum by its share of the original frame:

weight_h = population_h / total_population
estimated_loss = sum_h(weight_h * sampled_mean_loss_h)

For example, take a frame of 900 routine cases and 100 overflow cases, with 50 sampled from each. If the observed error rates are 2% and 20%, respectively:

The design-weighted mean is a Horvitz–Thompson estimate for that declared frame. It does not establish risk on unseen projects. Weighted precision is a ratio estimate, with its own uncertainty requirements; weighted rows cannot be treated as ordinary binomial counts. Sampling estimator reference

Fixed-sample bounded-loss reports use a conservative per-stratum sampling bound with a shared error budget; a fully enumerated stratum contributes its exact mean. Missing labels receive lower/upper loss assignments rather than silently shrinking the denominator. Unknown inclusion probabilities or a changed frame invalidate the corresponding estimation claim. These reports supplement the separate promotion cohorts below.

Near-threshold decisions, wide/rerank disagreements, overflow misses, and sparse categories also enter a diagnostic review queue. That queue helps choose examples to inspect; it is kept out of representative holdout denominators. Optional allocation based on pilot variance and labeling cost is compared at equal label, token, and request budgets before adoption. Sampling never changes which skill the live agent is instructed to execute or adds work to the hook path.

Require evidence before promoting a policy

The acceptance policy fixes these requirements before tuning. They are evaluation thresholds, not claimed benchmark results:

Primary relevance rates use two-sided 95% Wilson intervals, with one preselected case per independent task family. Additional variants do not inflate that denominator. Deliberately oversampled benchmark categories do not establish production prevalence or production precision.

Independent families alone do not establish a binomial sampling model or remove selection bias. Freeze the population, sampling design, endpoint model, and interval method before inspecting results. Wilson intervals are nominal model-based intervals, not exact coverage guarantees for an arbitrary fixed or quota-selected benchmark. Stratified probability samples use design-aware bounds and separate ratio methods; weighted rows are not binomial counts. Diagnostic rates without a justified inferential model cannot pass promotion gates.

The harm comparison pairs advice-enabled and baseline runs from equivalent isolated snapshots, with identical permissions and budgets, randomized arm order, and blinded outcome review. A task family counts as new harm if any planned paired run is harmful with advice and non-harmful without it. Missing or unjudgeable outcomes are also flagged for this conservative harm-or-unresolved gate. Count each family once and report judged harm and unresolved-only outcomes separately. Improvements on other tasks do not cancel flagged units; the net harm difference is reported separately.

The gate uses a one-sided 95% Clopper–Pearson upper bound when the prespecified binomial model is justified; otherwise it needs a prespecified design-valid bound. With zero flagged events in n independent identically distributed task-family units, the binomial upper bound is 1 - 0.05^(1/n): about 2.95% for 100 units and 1.98% for 150 units. Repeating one task 150 times does not provide 150 independent units, and zero observed events does not justify a zero uncertainty interval. Exact binomial interval reference

Insufficient independent cases, no emitted suggestions, or missing subgroup coverage means the requirement is not established. The harm cohort is separate from the relevance holdout, and its result applies to the declared population and experiment.

Monitor repeated evaluations without resetting the evidence

Optional sequential monitoring tracks a new-harm-or-unresolved composite from prospectively ordered, controlled task-family pairs. A finalized unit is flagged for judged new harm or an outcome still unjudgeable at its frozen deadline. Each family counts once; reports separate judged harm, unresolved-only, and unflagged units. The null bounds the conditional probability of this composite at 2%, using a fixed mixture of alternatives at 5%, 10%, and 20%. Under that conditional-risk assumption, its evidence threshold controls the chance of ever raising a false alarm across repeated looks. Time-uniform evidence reference

This is not a false-alarm guarantee for true harm alone under arbitrary missing labels. With zero actual harm but 5% unresolved units, the composite already exceeds the 2% null. An alarm can identify inadequate evidence without demonstrating harmful advice.

The mixture starts at one and alarms at 1 / alpha_monitor; a monitor allocated alpha_monitor = 0.05 has threshold 20. Each finalized unit contributes once, in the predeclared order. Pending labels wait until their adjudication deadline. Labels can change before finalization; the analyzed label version is then frozen. A later correction invalidates the inferential epoch: preserve its original trace, alarms, and spent alpha. Corrected historical recomputation is descriptive only. Resume inference with fresh prospective units and a fresh allocation unless a separately proven revision-aware method is available.

The total false-alarm budget is allocated across monitors and restarts in advance. A restart does not replenish it. State is bound to the rubric, baseline, policy/model cohort, and ordering; missing or corrupt state or an unresolved label revision reports unmonitored. An alarm blocks further policy promotion and recommends the frozen baseline or shadow mode. Configuration changes still require the explicit apply path.

The absence of an alarm does not establish that the risk ceiling is met, and an evidence value is not a posterior probability. The monitor cannot be applied to an arbitrary fixed sample merely because its sampling estimate is valid, or infer harm from skill adoption. It runs on evaluation evidence without launching production experiments or making extra network calls.

sr eval --explain includes mathematical explanation cards with equations, substituted values, assumptions, and what further evidence would change the conclusion. These belong in reports; hook advice stays short.

Opt-in retrieval and description experiments

The evaluation tools support three bounded experiments. They use held-out cases and the existing request, disclosure, and latency budgets; they do not silently replace the default selector.

Passage retrieval reads only already authorized, bounded skill bytes, scans them for secrets before excerpting, and retains heading, position, and version. A successful query with no passage match uses the declared opening excerpt; parser, fuel, and index errors retain typed failure behavior. It loads no referenced files, scripts, or additional candidates.

Multiple query views share aggregate term, character, fuel, memory, and deadline budgets. Each returns at most 254 matches. A pinned reciprocal-rank rule such as sum_v 1/(60 + rank_v) uses ranks starting at one, zero contribution for absent hits, equal view weights, and stable skill-ID ties. Identical normalized views are deduplicated. An attempted view failure invalidates the result; an empty union remains retrieval-empty. Fusion scores are retrieval heuristics, not Jev probabilities, and rosters of 254 or fewer still bypass the prefilter.

Description overlays bind to source content digests and are unavailable to hooks. A changed description requires a new request fingerprint and responses; compatible unchanged inputs may use recorded answers. Reports compare coverage, false suggestions, truncation, unknown cases, and attempts. Stale or ambiguous sources are rejected. The experiment exports a reviewable suggestion without rewriting skills or changing loadability. Untouched task families supply final validation; a better prose rubric score alone cannot justify promotion.

Control Requests And Interruptions

Share an HTTP-attempt allowance across sessions

An optional trusted-user allowance bounds HTTP attempts across local sr processes in a configured user/endpoint scope and named time window. It includes retries and live evaluation, and intersects each invocation and batch limit. Setup explicitly initializes the allowance and preflight reports the maximum attempts. It is neither a cross-machine billing limit nor a hard monetary cap; cost estimates use separately versioned prices and retain unknown usage.

sr budget                                      # Inspect local allowance and health
sr budget --max-attempts 100 --window 1h         # Preview scope and window semantics
sr budget --max-attempts 100 --window 1h --apply # Explicitly configure this limit

The allowance accepts 1–10,000 admissions in fixed one-hour UTC windows. The preview names the user/endpoint scope, exact window boundaries, charged attempts, and remaining allowance. This is not a rolling-hour cap: adjacent windows can each consume their allowance close to the boundary. Configuration changes preserve still-applicable charges. Accounting shares the 64 MiB cache/coordinator budget, but unexpired charges cannot be evicted; exhausted storage withholds new requests. Under the trusted configuration lock, setup first publishes a durable activation intent and new guard generation, then prepares accounting while preserving charges, then publishes the matching ready generation. Missing or mismatched state after the intent blocks admission. Retrying --apply resumes that intent; before it is visible, the old policy remains in force. Each attempt checks the active generation at admission, including processes started earlier. Already admitted requests cannot be recalled. Inspection itself makes no network requests.

Persistent attempts hold the same bounded lock as setup while rereading guard configuration, validating the accounting generation, and debiting. They release it before HTTP; lock contention cannot turn into an unbounded hook wait. With --no-persist, the final trusted-config read is the local admission boundary only when no guard is active. An intent or enabled guard refuses the request; a later activation cannot revoke a stateless request already admitted. Retries must recheck, not reuse startup-time settings.

Each attempt is atomically reserved/debited before transmission. Its single-use permit binds an attempt ID, request/endpoint, guard generation, window, and remaining monotonic dead