Martyx00/VulnFanatic-NG: LLM-Assisted Vulnerability Research for Binary Ninja

LLM-assisted vulnerability research for Binary Ninja.

VulnFanatic-NG adds a sidebar panel that scans the current binary and asks an LLM — a locally hosted OpenAI-compatible model by default, or Anthropic Claude, Google Gemini, or Azure OpenAI (see LLM backends) — to judge whether suspicious code is actually vulnerable. It works primarily from Binary Ninja's decompiler output (HLIL), falling back to assembly when needed, and reports only confirmed issues with clickable references back to the code.

How it works

A scan runs in up to three phases (Phase 3 is opt-in and online-only):

Phase 1 — dangerous function calls

Finds call sites of dangerous functions defined in rules/phase1_rules.json — strcpy, memcpy, sprintf/format strings, system, alloca, scanf, command/exec APIs, weak RNG, the free/delete family (use-after-free / double-free), reads of untrusted input into fixed buffers (recv/read/fread/ReadFile), SQL injection (sqlite3_exec/mysql_query/PQexec), disabled TLS certificate verification (SSL_CTX_set_verify/curl), SSRF, and improper privilege management (setuid/setresgid), the memset/bzero family, and comparisons with an attacker-controlled length (memcmp/strncmp → authentication bypass), across C/C++, Win32, and (best-effort) Rust FFI. Coverage includes fortified _chk (FORTIFY) and Annex-K _s variants. Bounded formatted-output functions (snprintf and variants) have their own default-safe rule so a correct size argument is not reported as an overflow. Call sites are found three ways: direct calls to the named symbols; calls routed through forwarding thunks / PLT stubs (the real callers are recovered, so an import reached only through a stub is not missed); and — unless vulnfanatic.scanIndirectCalls is off — indirect calls dispatched through a function pointer or vtable that Binary Ninja resolved to a dangerous function. For each call site it builds an interprocedural, decompiler-centric context, budgeted to a token limit (default 100k):

• the call expression and its arguments,
• the declared prototype of the called function (from Binary Ninja's type info, else a built-in table) so the model maps arguments to parameters correctly — fortified __*_chk and bounds-checked *_s variants take extra leading arguments, shifting the format/size/destination position,
• the type and byte size of each call argument (buffer capacities), derived from the argument's HLIL expression type so that a struct field like s->buf resolves to the field's real array size rather than the pointer size of s; struct definitions in the types section also carry per-field byte sizes,
• the concrete value / range of each argument resolved by Binary Ninja's constant-propagation and value-set analysis (e.g. a length proven constant 0x40 or bounded to [0, 0xff]), which the model uses as ground truth when comparing a size against a buffer capacity instead of guessing,
• the stack frame layout of the calling function (variable offsets and byte sizes) when it contains a fixed-size buffer, so a stack overflow can be judged against the adjacent variables and the saved return address (vulnfanatic.includeStackLayout),
• the path constraints (the if/loop/switch conditions guarding the call),
• an argument data-flow summary — where each call argument is defined and used within the function,
• parameter resolution across callers — when a dangerous argument is a parameter of the calling function, the context reports what every caller actually passes for it (e.g. "all callers pass a string literal"), so a format/ size parameter that is always constant isn't mistaken for attacker-controlled,
• the full decompiled body of the calling function,
• data-type definitions (struct/union/enum) for the types referenced across the call chain and the argument variables, so the model knows real buffer/field sizes and integer widths,
• the decompiled bodies of functions that produce or consume the call's argument variables (traced via HLIL def/use), which is what makes use-after-free / double-free and tainted-size reasoning possible,
• call paths from entry points / exported functions down to the call,
• the decompiled body of every function along those call paths (nearest the dangerous call first), each annotated with the call site and the conditions guarding the next hop,
• the bodies of other functions those path functions call (e.g. for MAIN→ABCD→strcpy, also the functions MAIN and ABCD call elsewhere), since they may hold the bounds/validation checks that gate the dangerous value (vulnfanatic.includeCallPathSiblings, filled while budget allows), and
• tainted-source hints (input functions like recv/read/getenv called in the same function).

That context plus a rule-specific prompt is sent to the model, which returns a structured verdict. Non-issues are dropped. The prompts are tuned for a strong local code model (e.g. Qwen2.5-Coder) and instruct it to analyze the entire flow and emit JSON only.

Reasoning, scratchpad, and confidence

The model is told to favor recall — report plausible, security-relevant issues and express uncertainty through a Confidence rather than dropping anything it can't fully prove. It shows its work in a scratchpad that quotes the verbatim code snippets it relied on (the input source, each guard, the size/length, the relevant type, and the sink), which is stored on the finding so you can audit the reasoning.

Each finding carries a Confidence (high/medium/low): high = the whole chain is shown in the context; medium = likely, with a link or two inferred; low = a lead worth manual review. This is the headline metric (the model's severity estimate is a secondary field). Set vulnfanatic.minConfidence to drop anything below a threshold.

Tuning precision vs. recall

By default VulnFanatic-NG favors recall (catching real issues). If you get too many false positives, tighten with any of:

• vulnfanatic.validationPass (default off) — runs a second LLM pass that double-checks each flagged issue against the same context (verifying the scratchpad snippets and re-tracing the flow) and can correct the verdict or confidence. Doubles LLM calls for flagged candidates.
  • Separate validator model (recommended when you use the validation pass). Set vulnfanatic.validatorModel (plus validatorProvider / validatorBaseUrl / validatorApiKey) to run the second pass on a different model. A second opinion is far more useful from an independent model — it shares fewer blind spots and is much less likely to rubber-stamp the first verdict (models tend to prefer their own answers). A good pattern is a cascade: a fast model as the analyst (broad recall) and your strongest model as the validator, which only runs on flagged candidates. Leave the validator model blank to validate with the analyst model. The validator should be at least as capable as the analyst — a weaker one mostly adds false rejections. Everything except provider/base URL/key/ model is inherited from the analyst connection settings; a blank validator key reuses the analyst key; and if the validator endpoint is unreachable the first verdict is kept (the finding is never lost to a validator outage).
• vulnfanatic.minConfidence (default low) — raise to medium/high to report only stronger findings.
• vulnfanatic.skipConstantArgCalls (default off) — skip overflow-class call sites whose arguments are all compile-time constants.

Speed. Most per-call latency is the written reasoning, so vulnfanatic.verdictReasoning controls how much the model writes:

• concise (default) — a brief 1–3 sentence rationale, no verbatim code. Much faster than full with little accuracy loss; you can also lower vulnfanatic.maxResponseTokens.
• full — the detailed scratchpad with quoted snippets (most auditable, slowest).
• none — verdict only. Fastest; pair it with a reasoning-capable backend (vulnfanatic.reasoningEffort) so the model's internal thinking does the work. On a plain local model none loses accuracy (no chain-of-thought at all).

Supporting precision features that are always on (they inform the model without suppressing findings):

• Argument provenance — the context tells the model, per argument, whether it is a compile-time constant, a function parameter, or derived from a tainted source, which the model uses to set confidence.
• Bounds-checked-variant awareness — the prompts treat _s (Annex K) and _chk (FORTIFY) variants and length-bounded APIs as safe unless the size argument itself is wrong.
• Explicit buffer sizes — argument and struct-field byte capacities are provided so the model compares capacity against bytes written instead of guessing.

Scan Offline (no LLM)

The Scan Offline button runs Phase 1 with no model — purely programmatic heuristics declared in the offline block of each rule in phase1_rules.json. It flags dangerous call sites and eliminates the obviously-safe ones, assigning a heuristic Confidence:

• Eliminated (not reported): a memcpy/memmove with a constant length, a strcpy from a constant string, a printf with a constant format, a system with a constant command, etc. — calls whose governing argument is a compile-time constant and so can't be attacker-controlled. "Constant" includes values that Binary Ninja's value-set analysis pinned to a fixed number upstream, not just literal arguments.
• High confidence: flagged with no mitigating check found anywhere on the execution flow.
• Downgraded (→ medium): the size/length argument was either resolved to a constant/bounded range by Binary Ninja's value-set analysis, or a real bounds/validation check comparing the relevant argument (e.g. a strlen/size comparison, if (len < …)) was found somewhere on the flow — including in the functions called along the path — so it may already be handled. (A branch that merely mentions the variable without comparing it no longer counts, removing a source of spurious downgrades.)

The heuristics use a small declarative vocabulary in the rules (constant_safe_args, eliminate_if_all_args_constant, format_arg_lookup, length_guard_vars, base_confidence, skip) evaluated by Python predicates — no embedded code to exec. Most rules have an offline definition (overflow, format-string, command-exec, scanf, path handling, weak-RNG, weak numeric parsing, privilege changes, allocation size, …). Only the two categories that genuinely need semantic analysis are skipped offline and left for the LLM: the free/delete family (use-after-free / double-free, which needs pointer-lifetime tracking) and TLS verification (the bug is a specific constant value like SSL_VERIFY_NONE). The offline summary reports how many sites were flagged / eliminated / skipped (need the LLM) / failed, so the counts add up. This is a fast triage; for real judgment — and for the skipped categories — run the full LLM scan.

Offline findings still build the same full interprocedural context an online scan would send (only for the flagged sites) and store it, so once you triage them they can be exported as fine-tuning data just like online findings. Disable with vulnfanatic.offlineBuildContext if you want maximum offline speed.

Phase 2 — security-sensitive code (symbol-gated)

Only runs when the binary appears to have real symbols / variable names. Locates security-sensitive functions defined in rules/phase2_rules.json — authentication, cryptography (incl. weak algorithms), signature/certificate verification, session/token handling, access control, secret/key handling, input validation, non-constant-time secret comparison, and insecure deserialization — matched by function name and referenced strings, then audited by the model.

Phase 3 — hardware-attack hardening audit (online, opt-in)

A firmware hardening audit against fault-injection (voltage/clock/EM glitching) and side-channel (timing/power) attacks, based on hardware-attack mitigation guidance. Unlike Phases 1–2 (which find bugs), Phase 3 reports a missing or violated hardening control on a security-critical function — for example: default-fail branches, double-checked security decisions, post-loop counter validation, high-Hamming-distance state constants (vs plain 0/1), constant-time full-length secret comparison, randomized-offset secret access/clearing, encrypt-then-verify (anti-DFA), control-flow integrity counters, avoiding user-land crypto, and not handling raw key material directly (rules/phase3_rules.json).

Because compiler optimizations can strip source-level protections, these controls are best verified on the compiled binary — exactly what this checks. Phase 3 is LLM-only (online), symbol-gated, and disabled by default; enable it per scan with the Phase 3 checkbox on the New Scan tab (it never runs in offline mode).

Findings are listed in a table (status, confidence, phase, CWE, function, address, title) with a detail pane that shows the explanation, the analysis scratchpad, and the validation notes. Double-click a row to navigate the binary view to the code.

Triage workflow

Every finding starts Untriaged. Right-click a row to set its status — Mark as Real Issue, Mark as False Positive, or Mark as Untriaged. Each status change pops up a "Provide reason:" text box (the reason is stored with the finding). The table makes status obvious: Real Issues are green/bold and sort to the top, False Positives are grey/struck-through and sort to the bottom, Untriaged sit in between with their confidence colour. A summary line shows the counts.

Each result tab has an Export triaged (fine-tuning)… button that exports only the triaged findings (Real Issue + False Positive) as OpenAI chat-format JSONL for fine-tuning: each example pairs the original system+user prompt with the human-corrected verdict as the assistant target (a False Positive teaches is_vulnerable=false with your reason; a Real Issue reinforces is_vulnerable=true), so you can iteratively improve the model's accuracy on your binaries.

The per-finding context shown in the detail pane (and used to reconstruct the fine-tuning prompts) is, by default, kept in full — controlled by vulnfanatic.storedContextChars (0 = unlimited; set a positive cap, e.g. 4000, to limit BNDB growth at the cost of context fidelity).

Multiple scans (tabs)

The panel is tabbed. The first tab is always New Scan, where you set:

• an optional Scan name (blank → <timestamp> <mode>, e.g. 2026-06-15 14:03:50 offline),
• an optional Phase 1 / Phase 2 / Phase 3 custom rules path (blank → the bundled defaults), so you can run an alternative rule set,
• a Phase 3 checkbox (off by default) to additionally run the online-only hardware-attack hardening audit,

then press Start Scan or Scan Offline. Each run opens its own result tab and findings stream into it live. All scans are stored in the BNDB, so you can e.g. keep an offline scan and later add an online scan, or compare runs with different rule sets, side by side — they reappear as tabs when you reopen the database. Closing a tab permanently deletes that scan from the BNDB — to prevent accidents it pops up a confirmation that requires ticking "I confirm that I will lose the results from forever." before the Delete results forever button activates. Export current scan… writes the selected tab to Markdown/JSON.

Each open binary has its own independent panel state — its own scan tabs and running scan. Starting a scan in one binary and switching to another shows the second binary's results (and lets you scan it separately); the first binary's scan keeps running in the background and is intact when you switch back.

Installation

This plugin's package folder is named vulnfanatic_ng (a valid Python identifier — Binary Ninja imports the plugin folder name as a module, so a hyphenated name like VulnFanatic-NG would not load).

1. (Optional) Install accurate token counting into Binary Ninja's Python:

   pip install tiktoken
   

2. Symlink or copy the vulnfanatic_ng folder into your Binary Ninja user plugins directory:

   • macOS: ~/Library/Application Support/Binary Ninja/plugins/
   • Linux: ~/.binaryninja/plugins/
   • Windows: %APPDATA%\Binary Ninja\plugins\

   For example, on macOS:

   ln -s "$(pwd)/vulnfanatic_ng" "$HOME/Library/Application Support/Binary Ninja/plugins/vulnfanatic_ng"
   

3. Restart Binary Ninja (or run Reload Plugins). A VF icon appears in the right sidebar.

Configuration

Open Settings (the gear / Edit ▸ Preferences ▸ Settings) and search for vulnfanatic. Set at minimum:

LLM backends

vulnfanatic.apiProvider selects how requests are formed and authenticated. The verdict contract (and all rule prompts) are identical across providers.

AWS Bedrock can be used through the openai provider via its OpenAI-compatible endpoint, so it does not need a dedicated backend.

Other useful settings: vulnfanatic.maxContextTokens (default 100000), vulnfanatic.maxResponseTokens, vulnfanatic.temperature, vulnfanatic.reasoningEffort (off/low/medium/high; default high — asks the model to think before answering where supported, mapped per provider: openai/azure reasoning_effort, anthropic adaptive thinking + output_config.effort, google dynamic thinkingConfig; auto-stripped and retried if a model rejects it), vulnfanatic.requestTimeoutSec, vulnfanatic.callPathMaxDepth / vulnfanatic.callPathMaxPaths, vulnfanatic.callPathIncludeBodies (include the decompiled bodies of functions along the call path; default on) / vulnfanatic.callPathMaxBodies (cap, default 12), vulnfanatic.includeCallPathSiblings (also include other functions called along the path, which may hold the bounds/validation checks; default on) / vulnfanatic.callPathSiblingMaxBodies (cap, default 12), vulnfanatic.includeDataTypes (include struct/union/enum definitions; default on) / vulnfanatic.maxTypeDefs (cap, default 24), vulnfanatic.includeVariableDataflow (trace call arguments back through their producers /consumers and include those bodies; default on) / vulnfanatic.dataflowMaxFunctions (cap, default 8), vulnfanatic.includeStackLayout (include the calling function's stack-variable layout when it has a fixed-size buffer; default on), vulnfanatic.scanIndirectCalls (also match dangerous calls dispatched through a resolved function pointer/vtable; default on — turn off for a faster scan on very large binaries), vulnfanatic.validationPass (run the second double-check pass; default off) / vulnfanatic.validatorModel / vulnfanatic.validatorProvider / vulnfanatic.validatorBaseUrl / vulnfanatic.validatorApiKey (run the validation pass on a separate, independent model — blank = same model as the analyst) / vulnfanatic.minConfidence (low/medium/high; drop findings below this; default low), vulnfanatic.flagUnparseableResponses (report sites the model couldn't score as UNKNOWN-confidence "Unscored" leads instead of dropping them; default on), vulnfanatic.skipConstantArgCalls (skip all-constant overflow call sites; default off), vulnfanatic.verdictReasoning (concise/full/none; how much reasoning the model writes per verdict — the main speed lever; default concise), vulnfanatic.runPhase1 / vulnfanatic.runPhase2 / vulnfanatic.runPhase3 (enable each phase; Phase 3 is online-only and usually toggled per-scan via the New Scan checkbox rather than here), vulnfanatic.phase2RequireSymbols / vulnfanatic.phase2ForceEnable, vulnfanatic.tokenizerEncoding (tiktoken encoding for token estimates; falls back to a character heuristic if tiktoken is not installed), vulnfanatic.offlineBuildContext (build full context for offline findings so they can be exported for fine-tuning; default on), vulnfanatic.debugLogging (verbose pipeline trace to the console; default off) / vulnfanatic.debugAnonymous (redact all binary-identifying details so the log can be shared — see Debug logging below), vulnfanatic.sendJsonResponseFormat, vulnfanatic.tlsVerify (verify HTTPS certificates; default on) / vulnfanatic.caBundlePath (CA bundle for HTTPS — see Troubleshooting if you hit CERTIFICATE_VERIFY_FAILED), and vulnfanatic.rulesPhase1Path / vulnfanatic.rulesPhase2Path / vulnfanatic.rulesPhase3Path (point these at your own rule files to customize detections and prompts).

Security note: the API key is stored in Binary Ninja's settings in plaintext. Prefer the environment-variable override for sensitive keys.

Test mode (dry run, no LLM)

Set vulnfanatic.apiBaseUrl to the literal value TEST to run without any LLM:

• The model is never called (no network, no API key/model needed).
• Every candidate (each dangerous call site in Phase 1, each security-sensitive function in Phase 2) is flagged.
• The full prompt — system prompt and the complete generated context — for each candidate is written to its own file under /tmp/vulnfanatic_ng/<binary>-<timestamp>/.
• The findings table shows a Prompt File column (hover for the full path), and the detail pane and exports include the path.

Use this to inspect and validate exactly what VulnFanatic-NG would send to the model, and to iterate on the rule prompts/context without spending model time.

Debug logging

Turn on vulnfanatic.debugLogging to print a verbose, step-by-step trace of the scan pipeline (both online and offline) to the Binary Ninja log/console: each call site, every skip/elimination decision, context build (size only), each LLM request (provider/model/endpoint, retries, fallbacks), every verdict, and each reported finding. API keys are never logged.

While debug logging is on, an online scan keeps every candidate in the results table instead of dropping the ones that don't become confirmed issues, each tagged with a debug-only status (dimmed, sorted to the bottom):

• REJECTED — the LLM returned a verdict of not an issue.
• SKIPPED — eliminated before the LLM by a provably-safe rule (a constant format string, or all-constant arguments); the row explains which.
• ERROR — the candidate could not be analyzed (context build failed, or the LLM response was unparseable / the connection failed); the row carries the error.

So a debug scan shows one row per candidate in the /N total, and the summary reports issues vs. rejected/skipped/error counts separately. You can right-click any of these rows to re-triage it as a Real Issue or False Positive (which makes it eligible for fine-tuning export). (Offline scans are unaffected — they never call the LLM.)

Independently of debug mode, when a model returns an unparseable response — a stray token like Gemma's <unused…>, prose instead of JSON, or an empty message (only a role, no content) — the client makes one corrective retry, re-asking for JSON only with the structured-output format disabled; if that succeeds it keeps the format off for the rest of the scan. The client also reads the reasoning channel (reasoning_content / reasoning) when content is empty, so reasoning models that put their answer there still work.

The empty-message case is common with reasoning models such as GPT-OSS / o1 served over an OpenAI-compatible API (e.g. mlx-community/gpt-oss-20b): with response_format=json_object set, the harmony "final" answer channel is often suppressed and the server returns {"role": "assistant"} with no content. These models can also burn their whole output budget on the reasoning channel and get truncated mid-thought, returning prose with no JSON at all. The auto-retry recovers the format-related cases; if it persists, turn off vulnfanatic.sendJsonResponseFormat, lower vulnfanatic.reasoningEffort (so less budget goes to thinking), and/or raise vulnfanatic.maxResponseTokens. A persistent <unused…>/garbage reply instead usually means the prompt exceeds the model's context window (set vulnfanatic.modelContextWindow and/or raise the server's context length), or that the model is a poor fit for strict JSON output (a code model such as Qwen2.5-Coder behaves far better than Gemma here).

Recall-preserving fallback. When a candidate still can't be scored after the retry, vulnfanatic.flagUnparseableResponses (default on) reports it anyway as an "Unscored" finding with UNKNOWN confidence — a value distinct from low (the model never produced a verdict, so it isn't a low-confidence judgement) that sorts to the bottom — keeping the model's partial output as the explanation, so you don't lose the site, you just review it manually. Turn it off to drop such sites instead (they then surface only as analysis errors, or debug ERROR rows).

Also enable vulnfanatic.debugAnonymous to make the log safe to share: it redacts everything that could identify the analyzed file — symbol/variable names and addresses become per-run salted hashes (still consistent within a run so the flow is followable), the file name is hidden, finding text is replaced with <redacted>, the LLM endpoint host is hashed, and decompiled code / prompts / context are logged as sizes only (never the content). So you can send a debug log to report an issue without disclosing anything about your binary.

Usage

1. Open a binary and let analysis finish.
2. Click the VF sidebar icon to open VulnFanatic-NG.
3. Press Start Scan (full LLM scan) or Scan Offline (fast, no-LLM programmatic Phase 1 — see above). Progress shows in the panel and the Binary Ninja status bar; findings appear live and can be cancelled.
4. Click a finding to read the explanation; double-click to jump to the code.
5. Right-click a finding to Mark as false positive — it moves to the bottom of the table, greyed out and struck through, and the menu entry flips to Mark as real issue to undo it. (The right-click menu also has Go to code.)
6. Export to Markdown or JSON, or Clear to discard saved findings.

Findings — including their false-positive status — are stored in the Binary Ninja database. They are written into the .bndb when you save the database (and flushed immediately if a .bndb already exists), so they survive reopening.

The scan analyzes every matched call site (no cap), which is appropriate for local models. For a hosted/paid endpoint, be mindful of volume on large binaries.

Customizing rules

Both rule files share an envelope with a shared system_prompt and output_schema, plus a list of rules. Copy a bundled file, edit the functions/keywords/prompts, and point vulnfanatic.rulesPhase1Path / vulnfanatic.rulesPhase2Path at your copy. Phase 1 rules match by functions (exact) and name_regex; Phase 2 rules match by name_keywords, name_regex, and string_keywords. Each rule's prompt may use the {function} placeholder.

Fine-tuning the model (MLX, Apple Silicon)

The triaged exports are designed to be fed straight back into the model. After triaging findings across several binaries and clicking Export triaged (fine-tuning)… on each (collecting the .jsonl files into one folder), scripts/finetune_mlx.py runs an MLX LoRA fine-tune on them.

pip install mlx-lm     # Apple Silicon / macOS

# Fine-tune a local 4-bit model on every *.jsonl under ./exports
python scripts/finetune_mlx.py ./exports \
    --model mlx-community/Qwen2.5-Coder-7B-Instruct-4bit \
    --adapter-path ./vf-adapters --iters 800

# ...then fuse the adapters into a standalone model
python scripts/finetune_mlx.py ./exports --model <base> \
    --fuse --fused-path ./vf-qwen-coder-vuln

The script takes the training-data folder as its positional argument and the base --model (local path or MLX/HF repo id); other parameters are optional: --adapter-path, --valid-split (0.1), --iters, --batch-size (auto-clamped to fit a tiny split), --num-layers, --learning-rate, --max-seq-length (0 = auto-fit to the longest example, capped at 16384; set a positive value to force it), --fine-tune-type (lora/dora/full), --seed, --fuse/--fused-path, and --dry-run (prepare data + print the command without training). Anything after a literal -- is forwarded verbatim to mlx_lm lora. It recursively merges every *.jsonl in the folder, validates and de-duplicates the chat examples, makes the train.jsonl/valid.jsonl split MLX expects, then launches python -m mlx_lm lora (and mlx_lm fuse with --fuse).

Serve the result with an OpenAI-compatible server (mlx_lm.server --model <path>) and point vulnfanatic.apiBaseUrl back at it to scan with your tuned model.

VulnFanatic-NG contexts are large, so by default the script auto-fits --max-seq-length to your longest example (rounded up, capped at 16384 tokens). Long sequences dominate training memory, so a large model near this cap can run a smaller Mac out of memory. If your examples exceed the cap they're truncated — pass a higher --max-seq-length (more memory) or lower vulnfanatic.storedContextChars before exporting. If training is killed by a signal (e.g. exit -10 / SIGBUS), that's an out-of-memory crash: lower --max-seq-length, add -- --grad-checkpoint, or use a smaller model.

Development & testing

The plugin has zero required third-party dependencies. Pure modules (rules, tokens, llm, findings, settings, prototypes) are covered by an offline test suite that needs neither Binary Ninja nor a network. The tests/ suite lives in the project's source repository (it is not shipped inside the published plugin); run it from there. From the package directory you can still syntax-check every module:

python3 -m py_compile *.py ui/*.py
python3 -m unittest discover -s tests   # from the source repository

The Binary Ninja-facing modules (context_builder, phase1, phase2) import cleanly without Binary Ninja (their API access is guarded) but require a running Binary Ninja to exercise.

Manual in-Binary-Ninja test

1. Build a small vulnerable C program (e.g. one that strcpys argv[1] into a fixed stack buffer and calls system() on input). Compile with symbols to also exercise Phase 2.
2. Start your local OpenAI-compatible server and set vulnfanatic.apiBaseUrl, vulnfanatic.apiKey, and vulnfanatic.model.
3. Open the binary, run Start Scan, and confirm the dangerous calls are reported and double-clicking navigates to the call sites.

Troubleshooting

SSL: CERTIFICATE_VERIFY_FAILED ... unable to get local issuer certificate — the HTTPS endpoint's certificate is fine, but Binary Ninja's bundled Python has no CA bundle to verify it against (common on macOS and in embedded Pythons; you'll see this with hosted endpoints like AWS Bedrock, Anthropic, Google, Azure). Fix with one of, in order of preference:

• Install certifi into the Python Binary Ninja uses: pip install certifi. VulnFanatic-NG picks it up automatically.
• Point at a CA bundle: set vulnfanatic.caBundlePath to a bundle file (or directory) — e.g. the path printed by python3 -m certifi, or /etc/ssl/cert.pem.
• Last resort: turn off vulnfanatic.tlsVerify (only for a trusted/internal endpoint or a self-signed local server — this disables certificate checking).

HTTP 400 ... tokenizer.chat_template is not set — the model you are serving has no chat template, so the /chat/completions endpoint cannot format the messages. VulnFanatic-NG automatically falls back to the /completions endpoint for the rest of the scan when it sees this error, so scanning continues. To avoid the first failed request entirely, set vulnfanatic.apiMode to completions. Alternatively, fix it server-side by serving a model that ships a chat template, or pass one to your server — e.g. for vLLM: --chat-template <template.jinja> (or use an -Instruct/-Chat model variant). The dedicated chat template usually gives better results than the flattened completions prompt.

No JSON object found ... response looks truncated — the model's reply was cut off before the JSON finished. Two causes:

• The scratchpad exceeded the response budget → raise vulnfanatic.maxResponseTokens.
• More commonly: the prompt fills the model's context window, leaving no room to generate, so the reply stops after a few tokens no matter how high maxResponseTokens is. Local servers often have a small window (ollama defaults to num_ctx=2048!). Fix it by setting vulnfanatic.modelContextWindow to your server's window (e.g. ollama num_ctx, llama.cpp -c, vLLM --max-model-len) — VulnFanatic-NG then automatically caps the context it sends so prompt + response fit. Also keep vulnfanatic.maxResponseTokens reasonable (≈8192, not 65535) and/or raise the server's window. Tiny windows (≤8k) cannot hold the full interprocedural context; use a model/server configured for 32k+.

IncompleteRead / Could not complete request ... after N attempt(s) — the server accepted the request but closed the connection before sending the full response. This almost always means the model server died or stalled mid-generation: out-of-memory (large context + long output), an internal/worker timeout, or a proxy resetting the connection. VulnFanatic-NG retries once automatically and then skips that site. Check the model server's own logs for the real cause; reducing vulnfanatic.maxContextTokens and/or vulnfanatic.maxResponseTokens, or giving the server more memory / a larger context window, usually resolves it.

Limitations

• Phase 2 is symbol-aware. On a stripped binary, name-based matches are unreliable, so by default only the string-evidence rules run (functions that reference tell-tale string constants are still auditable). Set vulnfanatic.phase2ForceEnable to also audit name-based matches, or vulnfanatic.phase2RequireSymbols=off. The symbol gate is a heuristic.
• HLIL ↔ call-site mapping can fail; VulnFanatic-NG falls back to MLIL/assembly and notes the representation used per finding.
• Verdicts are only as good as the model. Treat findings as leads for manual review, not ground truth.
• Some local servers ignore or reject response_format=json_object; the client tolerates that and still extracts JSON. Disable vulnfanatic.sendJsonResponseFormat if your server rejects the parameter outright.