Generalization

Why this matters

SigMap was not tuned for one repo. This benchmark matters because it shows the same workflow transfers across different languages, repo sizes, and architectures without manual tuning.

Official v8.85.0 benchmark snapshot

Benchmark ID: sigmap-v8.85-main  ·  Date: 2026-10-27 (with R language)

Metric	Value
Hit@5 (retrieval corpus, 105 tasks / 18 repos)	80.4%
Honest grep comparison (125 tasks / 19 repos)	92.8% vs 39.2% single-shot grep — 2.37× lift
Honest lift (vs grep agent)	2.37×
Prompt reduction	44.4% (2.84 → 1.58)
Task success proxy	61.9%
Overall token reduction	95.2%
GPT-4o overflow (without → with)	14/21 → 0/21

The important part of SigMap's benchmark story is not just the topline score. It is that the same retrieval approach works across a mixed set of repos rather than one curated demo project.

What "generalization" means here

SigMap's signature extractors are hand-written regex patterns, not ML models. Generalization means: do the patterns hold up on codebases the authors never inspected? The answer across these 105 tasks is yes — 80.4% hit@5 in the latest saved v8.85.0 run — with one caveat that matters: the benchmark pins each repo's srcDirs in benchmarks/config-overrides.json (all 21 repos). The zero-config, third-party measurement is the xrepo corpus: 55.3% hit@5 over 208 tasks in 35 repos, indexed the way a first-time user's sigmap run would.

• 21 repos (including 3 R language repos)
• 38 languages (added R, GDScript, and CI/pipeline definitions)
• multiple domains
• 80.4% overall hit@5
• source directories pinned per repo (benchmarks/config-overrides.json, 21 of 21 repos) — the only per-repo setting; no extractor or ranking parameter differs between repos

That snapshot is shared with the retrieval benchmark and the task benchmark, so the public docs now use one release number set instead of mixing older runs.

The repo mix, task selection, and per-language breakdown rules are in benchmark methodology.

Why this matters

SigMap uses hand-written extractors and lightweight ranking rather than a hosted retrieval stack. The strongest proof of generalization is therefore breadth:

• frameworks and application repos
• libraries and dev tools
• small, medium, and large codebases
• languages with very different syntax shapes

Representative coverage

Category	Example repos
Web frameworks	express, flask, gin, rails, laravel, fastapi, fastify, vapor
Libraries / tooling	axios, okhttp, serilog, riverpod, rust-analyzer, abseil-cpp, akka
UI frameworks	vue-core, svelte

Practical takeaway

If you want one number to carry into launch messaging, use the shared v8.85.0 snapshot rather than an older per-page variant:

Domain	Repos	Hit@5	Example repo
Dev tools	1	100%	rust-analyzer
Systems lib	1	100%	abseil-cpp
State management	1	80%	riverpod
Concurrency	1	100%	akka
UI framework	2	90%	vue-core, svelte
Web app	1	80%	spring-petclinic
HTTP client	2	81.3%	axios, okhttp
Web framework	8	78.1%	express, rails, gin, laravel, flask, vapor, fastify, fastapi
Logging	1	20%	serilog

Three domains land at 100% and every other domain except logging is at 75% or above. The low outliers — serilog (logging, 20%) and vapor (web framework, 20%) — are files whose signatures genuinely lack the query vocabulary; those need semantic retrieval and are the known residual misses. The variation is explained by repo structure (fragmented vs modular signatures) rather than language or domain category.

---

By repo size — small to 1,533 files

Size	File count	Repos	Avg hit@5
Small	≤25 files	5	81%
Medium	26–200 files	5	56.5%
Large	>200 files	8	95%

Large repos benefit most. Without SigMap, the random baseline for a 1,000-file repo is effectively 0% (5/1000 = 0.5%). SigMap's ranked retrieval closes that gap, scoring 100% hit@5 on rails (1,179 files) and laravel (1,533 files).

---

Anti-overfitting evidence

SigMap's extractors use hand-written regex patterns per language — not ML models, not embeddings. They were written against a small set of internal fixtures. The 18 benchmark repos were never inspected during development.

Key signals that the results are not overfit:

• One tool, one ranker — the same gen-context.js and the same ranking ran on every repo; the only per-repo setting is the pinned srcDirs in benchmarks/config-overrides.json, which exists because default source-root detection is not good enough for some layouts (rails, akka, laravel). That is tuning to these repos, which is why the zero-config xrepo corpus is reported beside the headline
• Selection — repos were chosen by GitHub star count and language diversity (methodology); the pins were added afterwards, where detection failed
• Failure modes are honest — C#/serilog 20%, Swift/vapor 20%, TypeScript/axios 62.5%, JS/fastify 62.5%, Python/flask 62.5% — genuine weak spots, not massaged away
• Large repos score higher — if the extractor patterns were memorized, they'd degrade on unseen large codebases; instead they improve (95% vs 81% for small repos)

---

Repo inventory

Repo	Language	Domain	Files	Hit@5
express	JavaScript	Web framework	6	100%
flask	Python	Web framework	19	62.5%
gin	Go	Web framework	107	100%
spring-petclinic	Java	Web app	13	80%
rails	Ruby	Web framework	1,179	100%
axios	TypeScript	HTTP client	25	62.5%
rust-analyzer	Rust	Dev tools	635	100%
abseil-cpp	C++	Systems lib	700	100%
serilog	C#	Logging	99	20%
riverpod	Dart	State management	446	80%
okhttp	Kotlin	HTTP client	18	100%
laravel	PHP	Web framework	1,533	100%
akka	Scala	Concurrency	211	100%
vapor	Swift	Web framework	131	20%
vue-core	Vue	UI framework	232	100%
svelte	Svelte	UI framework	370	80%
fastify	JavaScript	Web framework	31	62.5%
fastapi	Python	Web framework	48	80%

---

Made in Amsterdam, Netherlands 🇳🇱