Task benchmark
Official v8.85.0 benchmark snapshot
Benchmark ID: sigmap-v8.85-main · Date: 2026-10-27 (with R language)
| Metric | Value |
|---|---|
| Hit@5 (retrieval corpus, 105 tasks / 18 repos) | 80.4% |
| Honest grep comparison (125 tasks / 19 repos) | 92.8% vs 39.2% single-shot grep — 2.37× lift |
| Graph-boosted hit@5 | 80.4% |
| Honest lift (vs grep agent) | 2.37× |
| Prompt reduction | 44.4% (2.84 → 1.58) |
| Task success proxy | 61.9% |
| Token reduction (21 repos) | 95.2% |
| GPT-4o overflow (without → with) | 14/21 → 0/21 |
Latest saved run: 2026-10-27 (v8.85.0) — includes R language support (ggplot2, dplyr, shiny)
The success/proxy definitions and the task corpus are described in benchmark methodology.
This page answers the question people care about most:
does SigMap help the developer finish the task with fewer retries?
Headline result
| Metric | Without SigMap | With SigMap |
|---|---|---|
| Task success proxy | 10% | 61.9% |
| Prompts per task | 2.84 | 1.6 |
| Prompt reduction | — | 44.4% |
| Retrieval hit@5 | 13.6% | 80.4% |
| Token reduction | — | 95.2% |
Why the task benchmark exists
Retrieval is a prerequisite, but not the whole story. Developers feel the difference as:
- fewer prompt retries
- fewer "can you share more files?" loops
- fewer answers grounded in the wrong module
The task benchmark models that outcome from the ranked file quality tiers:
- rank 1 hit → likely one prompt
- rank 2–5 hit → likely follow-up prompt
- miss → likely multiple retries
Current saved score card
| Tier | Meaning | Tasks | Share |
|---|---|---|---|
| Correct | Right file was ranked first | 65 | 61.9% |
| Partial | Right file was present but not first | 19 | 18.1% |
| Wrong | Right file never surfaced in top 5 | 21 | 20.0% |
Prompt model summary
| Metric | Value |
|---|---|
| Average prompts without SigMap | 2.84 |
| Average prompts with SigMap | 1.6 |
| Reduction | 44.4% |
| Honest hit@5 lift | 2.37x — 92.8% vs 39.2% on the honest corpus (125 tasks / 19 repos); per-repo random lifts remain in the report as data |
What changed in the v5 story
The earlier SigMap story was mostly "smaller context." The v5 story is more useful:
- use ask to build the focused context
- use validate to make sure coverage is healthy
- use judge to check whether the answer was actually grounded
- use learning when the same files repeatedly help or hurt
That makes the benchmark more than a marketing claim. It maps onto the actual daily workflow.
Benchmark snapshot by repo
| Repo | Prompt reduction | Correct / Partial / Wrong |
|---|---|---|
| express | 43.8% | 7 / 1 / 0 |
| flask | 34.8% | 5 / 0 / 3 |
| gin | 61.8% | 7 / 1 / 0 |
| spring-petclinic | 44.8% | 4 / 0 / 1 |
| rails | 53.3% | 3 / 2 / 0 |
| axios | 32.1% | 4 / 1 / 3 |
| rust-analyzer | 59.9% | 4 / 1 / 0 |
| abseil-cpp | 53.2% | 3 / 2 / 0 |
| serilog | 4.7% | 0 / 1 / 4 |
| riverpod | 53.1% | 4 / 0 / 1 |
| okhttp | 55.0% | 4 / 1 / 0 |
| laravel | 60.0% | 4 / 1 / 0 |
| akka | 46.2% | 2 / 3 / 0 |
| vapor | 5.2% | 0 / 1 / 4 |
| vue-core | 59.6% | 4 / 1 / 0 |
| svelte | 39.7% | 2 / 2 / 1 |
| fastify | 37.7% | 5 / 0 / 3 |
| fastapi | 44.4% | 3 / 1 / 1 |
These rows show why the task benchmark matters. Some repos have great retrieval lift but still need workflow help around validation and judge-based trust.
Reproduce
node scripts/run-task-benchmark.mjs --save
node scripts/run-task-benchmark.mjs --jsonFor the full multi-benchmark dashboard:
node scripts/run-benchmark-matrix.mjs --save --skip-clone
open benchmarks/reports/benchmark-report.html