Call-graph accuracy
--callers, --callees and get_method_impact answer from SigMap's method-level call graph (src/graph/call-graph.js). The docs called its answer a lower bound and named the scope searched; they did not say how much of the truth that bound is. This page measures it against an independent reference.
The reference is a SCIP index of the same code, written by the language's own indexer (the TypeScript checker for TypeScript and JavaScript, Pyright for Python, the Go type checker, javac for Java). Every reference occurrence of a function, method or class is mapped to the function that contains it, which gives caller-to-callee edges that the graph's regular-expression-and-brace matching never touched. Call it compiler-derived, not ground truth: a compiler cannot resolve a call through an untyped value, so the reference misses some real edges, and the precision below is a lower bound.
The method, the repositories and the pass/fail threshold were written down and committed before the scorer ran on any of them (benchmarks/CALLGRAPH_ACCURACY_PROTOCOL.md). Everything after the first run is logged in that file's deviation log, with the first run's figures kept. The full numbers, and the first misses of every repository, are in benchmarks/reports/callgraph-accuracy.json.
Results
Measured on SigMap 8.82.0 (commit 5ec27b7), protocol 85268ae, indexers scip-go@0.2.9, scip-java@0.12.3, scip-python@0.6.8, scip-typescript@0.5.2; Node v25.6.0; Go 1.27.3, scip-go 0.2.9; JDK 21.0.13+9, scip-java 0.12.3, Apache Maven 3.9.16 (2bdd9fddda4b155ebf8000e807eb73fd829a51d5).
Headline
Recall and precision are over edges between the files both the graph and the compiler index cover. Brackets are 95% cluster-bootstrap intervals over caller files. "Function edges" are calls to a free function; "method edges through a receiver" are this.m(), self.m(), x.m(); a cell with fewer than 100 edges is not quoted.
| Language | Repositories | Reference edges | Recall, all edges | Recall, function edges | Recall, method edges through a receiver | Precision | Precision, judgeable edges |
|---|---|---|---|---|---|---|---|
| TypeScript | zod, excalidraw | 1,472 | 5.6% (3-9) | 13.0% (8-22) | 0.0% (0-0) | 6.0% (3-11) | n=87, too few |
| JavaScript | express, fastify | 140 | 95.7% (88-99) | 99.3% (97-100) | n=2, too few | 45.1% (32-60) | 74.0% (62-86) |
| Python | flask, fastapi, django | 52,161 | 10.8% (9-12) | 48.1% (43-53) | 0.1% (0-0) | 69.3% (64-74) | 69.4% (64-74) |
| Go | gin, gorm | 3,232 | 36.3% (29-44) | 89.1% (83-93) | 0.0% (0-0) | 98.9% (98-100) | 98.9% (98-100) |
| Java | spring-petclinic | 230 | 45.2% (38-54) | - | 43.8% (33-55) | 98.1% (93-100) | 100.0% (100-100) |
By repository
| Language | Repository | Reference edges | Graph edges | Recall | Function recall | Precision | Judgeable precision |
|---|---|---|---|---|---|---|---|
| TypeScript | zod | 711 | 71 | 8.7% | 13.6% | 87.3% | 96.9% |
| TypeScript | excalidraw | 761 | 1,294 | 2.6% | 11.6% | 1.5% | 87.0% |
| JavaScript | express | 3 | 33 | 100.0% | 100.0% | 9.1% | 37.5% |
| JavaScript | fastify | 137 | 264 | 95.6% | 99.2% | 49.6% | 75.7% |
| Python | flask | 311 | 604 | 14.8% | 66.2% | 7.6% | 7.6% |
| Python | fastapi | 3,233 | 608 | 11.5% | 29.1% | 61.0% | 61.0% |
| Python | django | 48,617 | 6,912 | 10.7% | 50.5% | 75.4% | 75.6% |
| Go | gin | 2,212 | 1,018 | 45.7% | 97.9% | 99.2% | 99.2% |
| Go | gorm | 1,020 | 169 | 16.1% | 57.5% | 97.0% | 97.0% |
| Java | gson | unmeasured | the indexer exited 1: [ERROR] COMPILATION ERROR : | ||||
| Java | spring-petclinic | 230 | 106 | 45.2% | - | 98.1% | 100.0% |
Recall by edge kind and call site
| Edge kind / call site | TypeScript | JavaScript | Python | Go | Java |
|---|---|---|---|---|---|
function/bare | 26.5% (n=309) | 99.3% (n=135) | 55.7% (n=9,447) | 99.0% (n=1,186) | - |
function/qualified | 0.0% (n=321) | - | 0.0% (n=1,492) | 0.0% (n=131) | - |
method/bare | - | - | 97.1% (n=345) | n=1, too few | n=25, too few |
method/self | 0.0% (n=235) | n=2, too few | 0.0% (n=13,216) | - | n=5, too few |
method/receiver | 0.0% (n=429) | - | 0.3% (n=10,647) | 0.0% (n=1,883) | 45.1% (n=173) |
constructor/bare | 0.0% (n=153) | n=3, too few | 0.0% (n=14,099) | n=27, too few | n=27, too few |
constructor/qualified | n=25, too few | - | 0.0% (n=2,915) | n=4, too few | - |
Why calls written as f() were missed
Why the graph missed a call written as f() | TypeScript | JavaScript | Python | Go | Java |
|---|---|---|---|---|---|
| the graph has no node for the caller | 82 | 0 | 0 | 0 | 0 |
| the graph has no node for the callee | 105 | 0 | 0 | 1 | 0 |
| every call site is inside a string, template literal or comment the graph masks | 5 | 0 | 3 | 0 | 0 |
| both nodes exist, same file, call not seen | 5 | 0 | 262 | 2 | 0 |
| both nodes exist, other file, call not linked | 30 | 1 | 3,920 | 9 | 0 |
missed function/bare edges | 227 | 1 | 4,185 | 12 | 0 |
Graph edges the reference does not have
| Graph edges the reference does not have | TypeScript | JavaScript | Python | Go | Java |
|---|---|---|---|---|---|
| callee is a function-local constant the compiler index cannot name (unjudged) | 1,040 | 42 | 16 | 0 | 2 |
| caller is one it cannot name (unjudged) | 238 | 74 | 0 | 0 | 0 |
callee is declared inside the caller: a nested def name( read as a call | 1 | 40 | 2,026 | 3 | 0 |
| both named by the index, and the compiler found no such call | 4 | 7 | 454 | 10 | 0 |
| graph edges the compiler index lacks | 1,283 | 163 | 2,496 | 13 | 2 |
| graph edges between files both sides cover | 1,365 | 297 | 8,124 | 1,187 | 106 |
Scope
| Language | Files both sides cover | Files the graph walked | Files indexed | Reference edges | of them in covered files | Recall, scope-inclusive | File-pair precision | File-pair recall |
|---|---|---|---|---|---|---|---|---|
| TypeScript | 578 | 609 | 1,029 | 1,551 | 1,472 | 5.3% | 0.7% | 1.6% |
| JavaScript | 162 | 163 | 438 | 203 | 140 | 66.0% | 69.7% | 92.0% |
| Python | 2,652 | 2,669 | 4,135 | 52,456 | 52,161 | 10.7% | 98.5% | 17.6% |
| Go | 186 | 249 | 210 | 3,240 | 3,232 | 36.2% | 97.4% | 38.4% |
| Java | 39 | 39 | 47 | 240 | 230 | 43.3% | 100.0% | 43.1% |
The registered threshold and predictions
| Language | Receiver-only method recall | Reference edges | Registered threshold of 70% |
|---|---|---|---|
| TypeScript | 0.0% | 664 | spike-warranted |
| JavaScript | 0.0% | 2 | insufficient-sample |
| Python | 0.1% | 23,863 | spike-warranted |
| Go | 0.0% | 1,883 | spike-warranted |
| Java | 43.8% | 178 | spike-warranted |
| Registered prediction | Result | Measured | |
|---|---|---|---|
| P1 | receiver-only method recall is below 15% in typescript, javascript, python and go, because buildCallGraph resolves .name( calls through a receiver type only for Java (typeMap is built only for JAVA_EXTS) | held | TypeScript 0.0%, Python 0.1%, Go 0.0% |
| P2 | recall of function edges with a bare site is between 50% and 90% in every measured group | missed | TypeScript 26.5%, JavaScript 99.3%, Python 55.7%, Go 99.0% |
| P3 | overall precision is between 50% and 90% in every measured group, driven by calls that match a same-named definition that is not the one the compiler resolved | missed | TypeScript 6.0%, JavaScript 45.1%, Python 69.3%, Go 98.9%, Java 98.1% |
| P4 | constructor recall is below 10% in typescript, javascript and python, because classes are not nodes of the graph | held | TypeScript 0.0%, Python 0.0% |
| P5 | in java, receiver-only method recall is higher than in any other group but still below 70% | held | Java 43.8% |
What the figures say
- A call through a receiver is not in the graph, except in Java.
this.save(),self.save()andrepo.save()have no edge: 0.0% of 664 reference edges in TypeScript, 0.1% of 23,863 in Python, 0.0% of 1,883 in Go. The graph resolves.name(calls through a declared type only for Java (buildTypeMapis built only for.javafiles, andcallsInRangeskips a.name(access). In Java, on the one repository that built, it finds 43.8% of 178. The threshold registered before the run was 70%, so by the protocol's own rule a receiver-type pass is warranted in TypeScript, Python, Go and Java. JavaScript is not testable: its two repositories give only 2 such edges, because express assigns most of its functions to properties (app.use = function use() {}), which the compiler index does not name. - Plain calls
f()are found well where the graph has the nodes and can follow the import. Go: 99.0% of 1,186. JavaScript: 99.3% of 135, almost all fastify's. Python is lower at 55.7% of 9,447, and 3,920 of its 4,185 misses are a call to a function in another file that the graph did not link to its definition. TypeScript is lowest at 26.5% of 309, and 187 of its 227 misses involve a function the graph has no node for. A one-off check of the 111 distinct callee functions without a node in zod and excalidraw found 106 are generic declarations,function pointFrom<P extends Point>(…): the graph's function pattern requires(right after the name. The other 5 are method signatures declared without a body. Calls hidden inside a template literal (`${f()}`) are a real blind spot too, but they account for only 5 of those 227. - Classes are not nodes.
new Repo()andRepo()are not edges: constructor recall is 0.0% in TypeScript (178 edges) and Python (17,014). Go has 31 such edges and Java 27, too few to quote (Go found none, Java 7.4%). - When the graph does draw an edge, it is right in Go and Java and mostly right in Python. Precision is 98.9% in Go and 98.1% in Java (one repository). In Python it is 69.3%, and 2,026 of its 2,496 wrong edges (81%) are an edge to a function declared inside the caller: the nested
def decorator(f):line is read as a call todecorator. Without those, Python precision would be 92.3%. flask's 7.6% is its test suite for the same reason: 555 of its 558 wrong edges are the localdef index():views its tests define. - For TypeScript and JavaScript, most of the graph's edges cannot be judged. 1,278 of the 1,365 TypeScript graph edges have a caller or callee that is a function-local constant (
const onClick = () => …inside a React component), which scip-typescript gives no callable identity, so they are counted as wrong by the registered definition and are not shown to be. The 87 edges both sides can name are right 94.3% of the time, a cell too small to quote. The registered precision stays the headline; the judgeable column says how much of it is the reference's blind spot. - The graph reads fewer files than the index covers. The scope table has the counts, and recall counted over every reference edge, including files the graph never walked, is lower still: 5.3% in TypeScript, 10.7% in Python, 36.2% in Go.
What this does not say
- Nothing about PHP, C#, C/C++, Swift, Dart, Lua, GDScript, Vue, Svelte or Astro: the graph reads none of them (#979). Nor about Rust, Kotlin, Scala or Ruby, where no SCIP indexer was run.
- Java is one repository (spring-petclinic). gson did not build under scip-java: its
pom.xmlpasses Error Prone tojavac, scip-java's compiler wrapper rejects it, and removing it led to further build checks. It is reported unmeasured with the error, not replaced. - Two or three repositories per language is a small sample. The brackets are over caller files within those repositories, not over repositories, and a monorepo or a generated-code-heavy repository may score differently. express contributes 3 reference edges to the JavaScript row.
- Dependencies are not installed in the measured copy, so a call whose receiver type comes from an installed package is unresolved by the indexer. JSX element references count as non-calls on both sides.
- The reference cannot see function-local constants, class fields assigned a function, or calls through an untyped value; it is a lower bound on the real edges as well as on the graph's precision.
- A git co-change reference (files that change together) was considered and is not included: the pinned clones are single-commit checkouts.
Reproduce
node scripts/fetch-xrepo-repos.mjs --only zod,excalidraw,express,fastify,flask,fastapi,django,gin,gorm,gson
node scripts/run-callgraph-accuracy.mjs --setup-indexers <dir> --with go,java
node scripts/run-callgraph-accuracy.mjs --indexers <dir> # prints the tables
node scripts/run-callgraph-accuracy.mjs --indexers <dir> --save # also writes the report
node scripts/run-callgraph-accuracy.mjs --write-doc # re-renders the tables above from the reportEach repository is exported at its pinned commit with git archive into a scratch directory, so the clones under benchmarks/repos are only read. Indexers and toolchains are installed into <dir>, never system-wide, and every download is checked against a published checksum. --save refuses a partial run, a protocol that differs from the committed one, and a dirty scored tree. The harness is dev-only: nothing here ships in the npm package, and nothing at runtime depends on an indexer.