Hallucination Guard (verify-ai-output)

verify-ai-output flags claims in an AI answer that don't match your real repository — fabricated files, imports, symbols, test paths, and npm scripts. It is deterministic and fully offline: no network, no second LLM. It reuses SigMap's symbol index, file map, and import resolver, so every check is grounded in your actual code.

sigmap verify-ai-output answer.md

Exit code is 1 when any issue is found, 0 when the answer is clean — drop it straight into CI.

What it detects

Type	Meaning	Confidence
fake-file	A referenced path is not on disk	High
fake-test-file	A referenced test path is not on disk	High
fake-import	A relative import doesn't resolve, or a bare package isn't in package.json	High
fake-symbol	A called function/class isn't in the repo index or the installed libraries	Medium
fake-npm-script	npm run X where X isn't a package.json script	High

Node/Python builtins, scoped packages, and language globals are allow-listed to keep precision high. Python bare imports are intentionally not flagged (stdlib is unbounded offline).

Installed-library grounding (v8.1.0, v9.0 G5/D5 — the moat)

The fake-symbol check no longer knows only your repo's own symbols. It also grounds against the libraries actually installed in node_modules: for each direct dependency in package.json, SigMap reads that package's TypeScript declaration entry (types/typings, else index.d.ts) and unions its exported symbols into the known-symbol universe. So a genuine call like `Router()` or `debounce(fn)` from a real installed library is no longer false-flagged, while a call to a symbol that exists in neither the repo nor an installed library still surfaces. This is grounding no public-doc tool can do — it verifies against the real installed tree, with each library's version pinned in the summary (name@version). It's deterministic (byte-stable given a fixed installed tree), zero-dependency, cached, and runs automatically.

Ecosystems: JS/TS from node_modules (.d.ts exports). Since v8.3.0, Python too — SigMap reads direct deps from requirements.txt/pyproject.toml, locates the project's venv site-packages (.venv|venv|env → lib/python*/ site-packages, or Lib/site-packages on Windows) without spawning Python, and extracts each installed package's exports from its __init__.py/.pyi (__all__, top-level def/class, public assignments, and from … import re-exports). A genuine call into an installed Python library (e.g. `Session()` from requests) is no longer false-flagged.

Closest-match suggestions

When a flagged name is a near miss for something real, the report adds a heuristic suggestion (labeled as such — it's a Levenshtein guess, not a fact):

  L12  [Fake symbol]  Symbol not found in repo index: loadConfg()
         ↳ Did you mean `loadConfig()` in src/config/loader.js:42?
  L14  [Fake npm script]  npm script not in package.json: buidl
         ↳ Did you mean `build`?

Output modes

Terminal (default)

Grouped counts plus one line per issue, with suggestions inline.

JSON — for CI

sigmap verify-ai-output answer.md --json

{
  "file": "answer.md",
  "issues": [
    {
      "type": "fake-symbol",
      "value": "loadConfg",
      "line": 12,
      "location": "L12",
      "message": "Symbol not found in repo index: loadConfg()",
      "confidence": "medium",
      "suggestion": "Did you mean `loadConfig()` in src/config/loader.js:42?"
    }
  ],
  "summary": {
    "total": 1,
    "byType": { "fake-file": 0, "fake-test-file": 0, "fake-import": 0, "fake-symbol": 1, "fake-npm-script": 0 },
    "clean": false,
    "symbolsIndexed": 1842,
    "withSuggestion": 1,
    "librariesIndexed": 12,
    "libraries": [{ "name": "express", "version": "4.19.2", "symbols": 41, "typed": true }]
  }
}

HTML report

sigmap verify-ai-output answer.md --report report.html

Writes a standalone, self-contained HTML report (red/amber/green per issue, suggestions inline). It has no external assets or scripts, so it's safe to publish or paste as a screenshot into a PR. --report can be combined with --json (the report is written, JSON goes to stdout).

Use in CI

- name: Verify AI answer against the repo
  run: |
    npx sigmap verify-ai-output answer.md --json > verify.json
    npx sigmap verify-ai-output answer.md --report verify-report.html

A non-zero exit fails the job when the answer contains fabricated references.

Precision

Detector precision is validated by a proof harness (npm run benchmark:verify) that scores each detector group against labeled cases and enforces targets (file ≥ 95%, import ≥ 85%, symbol ≥ 75%, script ≥ 95%). Point it at your own repos with a manifest:

node scripts/run-verify-benchmark.mjs --manifest cases.json

It emits a per-detector precision/recall CSV.