Import-graph accuracy
--impact, get_impact, the blast radius in plan and the counts in sigmap doctor answer from SigMap's import graph (src/graph/builder.js). Until #980 it read eight language families; it now reads fifteen (C/C++/Objective-C, PHP, C#, Dart, Vue/Svelte/Astro, Lua and GDScript were added) and gives a module-level answer for Swift. This page measures how far the edges it draws agree with what each language's own tooling resolves.
The reference is never SigMap's own resolver. It is the compiler or the framework's parser: clang -E -H for C and C++, TypeScript's module resolution behind the Vue, Svelte and Astro parsers for components, PHP's tokenizer and Composer's autoloader for PHP, and the Swift compiler with SwiftPM for Swift. Call it compiler-derived, not ground truth: a reference reports the configuration it ran, and an import that depends on another configuration is not in it.
The method, the repositories, the predictions and the thresholds were written down and committed before the scorer ran on any of them (benchmarks/IMPORT_RESOLVER_PROTOCOL.md). The deviation log in that file records everything that happened after the first run, with the first run's figures kept. The full numbers, and the first disagreements of every repository, are in benchmarks/reports/import-accuracy.json.
Results
Measured on SigMap 8.82.0 (commit 7287d5a), protocol 975d281; Node v25.6.0 on darwin-arm64. References: Apple clang version 21.0.0 (clang-2100.1.1.101); typescript 6.0.3, @vue/compiler-sfc 3.5.44, svelte 5.57.2, @astrojs/compiler 4.0.0; PHP 8.4.23, Composer 2.10.3; Apple Swift version 6.3.2 (swiftlang-6.3.2.1.108 clang-2100.1.1.101).
Headline
Precision and recall are over edges whose importing file both the graph and the reference read. Brackets are 95% cluster-bootstrap intervals over importing files; a cell with fewer than 100 edges is not quoted.
| Group | Repositories | Reference edges | Recall | Graph edges | Precision | F1 |
|---|---|---|---|---|---|---|
| C / C++ / Objective-C | libuv, abseil-cpp | 4,611 | 99.7% (99-100) | 4,609 | 99.7% (99-100) | 99.7% |
| Vue / Svelte / Astro | vue-core, svelte, astro | 1,941 | 79.1% (77-81) | 1,535 | 100.0% (100-100) | 88.3% |
| PHP | laravel, composer | 4,975 | 100.0% (100-100) | 4,976 | 100.0% (100-100) | 100.0% |
| Swift (module level) | alamofire, vapor | 151 | 99.3% (98-100) | 150 | 100.0% (100-100) | 99.7% |
By repository
| Group | Repository | Commit | Importing files scored | Reference edges | Graph edges | Precision | Recall |
|---|---|---|---|---|---|---|---|
| C / C++ / Objective-C | libuv | 20b0834 | 330 | 677 | 683 | 99.1% | 100.0% |
| C / C++ / Objective-C | abseil-cpp | ea64b6e | 860 | 3,934 | 3,926 | 99.8% | 99.6% |
| Vue / Svelte / Astro | vue-core | 7df0edd | 11 | 14 | 14 | 100.0% | 100.0% |
| Vue / Svelte / Astro | svelte | 4a50e8e | 4,238 | 1,316 | 926 | 100.0% | 70.4% |
| Vue / Svelte / Astro | astro | 4c1470a | 1,098 | 611 | 595 | 100.0% | 97.4% |
| PHP | laravel | 635129b | 1,619 | 3,625 | 3,625 | 100.0% | 100.0% |
| PHP | composer | 28145ca | 314 | 1,350 | 1,351 | 99.9% | 100.0% |
| Swift (module level) | alamofire | 903c53c | 98 | 43 | 43 | 100.0% | 100.0% |
| Swift (module level) | vapor | cfd8f43 | 294 | 108 | 107 | 100.0% | 99.1% |
Scope
| Group | Repository | Files the graph walked | Files the reference could analyse | Scored (both) | Graph edges outside | Reference edges outside |
|---|---|---|---|---|---|---|
| C / C++ / Objective-C | libuv | 330 | 330 | 330 | 0 | 0 |
| C / C++ / Objective-C | abseil-cpp | 860 | 860 | 860 | 0 | 0 |
| Vue / Svelte / Astro | vue-core | 11 | 11 | 11 | 0 | 0 |
| Vue / Svelte / Astro | svelte | 4,339 | 4,238 | 4,238 | 0 | 0 |
| Vue / Svelte / Astro | astro | 1,101 | 1,098 | 1,098 | 0 | 0 |
| PHP | laravel | 1,619 | 1,619 | 1,619 | 0 | 0 |
| PHP | composer | 314 | 314 | 314 | 0 | 0 |
| Swift (module level) | alamofire | 98 | 98 | 98 | 0 | 0 |
| Swift (module level) | vapor | 294 | 294 | 294 | 0 | 0 |
Why edges were missed
| Reference edges the graph lacks, by how the import reaches its file | C / C++ / Objective-C | Vue / Svelte / Astro | PHP | Swift (module level) |
|---|---|---|---|---|
| a bare specifier that resolves inside the repository | 0 | 389 | 0 | 0 |
| reaches a .inc file | 16 | 0 | 0 | 0 |
| written .js/.jsx, the file is .ts/.tsx | 0 | 12 | 0 | 0 |
| a JSON module | 0 | 3 | 0 | 0 |
| a "#name" import of the package.json imports map | 0 | 1 | 0 | 0 |
| a module with no Swift file (a C-language target) | 0 | 0 | 0 | 1 |
| a relative path | 0 | 1 | 0 | 0 |
| all | 16 | 406 | 0 | 1 |
Where the missed edges are
| Group | Repository | How the missed import reaches its file | Reference edges missed |
|---|---|---|---|
| C / C++ / Objective-C | abseil-cpp | reaches a .inc file | 16 |
| Vue / Svelte / Astro | svelte | a bare specifier that resolves inside the repository | 389 |
| Vue / Svelte / Astro | svelte | a relative path | 1 |
| Vue / Svelte / Astro | astro | written .js/.jsx, the file is .ts/.tsx | 12 |
| Vue / Svelte / Astro | astro | a JSON module | 3 |
| Vue / Svelte / Astro | astro | a "#name" import of the package.json imports map | 1 |
| Swift (module level) | vapor | a module with no Swift file (a C-language target) | 1 |
Graph edges the reference does not have
| Graph edges the reference lacks, by what they reach | C / C++ / Objective-C | Vue / Svelte / Astro | PHP | Swift (module level) |
|---|---|---|---|---|
| reaches a .h file | 14 | 0 | 0 | 0 |
| reaches a .php file | 0 | 0 | 1 | 0 |
| all | 14 | 0 | 1 | 0 |
Not measured
| What | Kind | Why there is no figure |
|---|---|---|
| C# | language | no compiler-grade reference was run |
| Dart | language | no compiler-grade reference was run |
| Lua | language | no reference exists to run |
| GDScript | language | no reference exists to run |
The registered predictions
| Registered prediction | Result | Measured | |
|---|---|---|---|
| P1 | C/C++ recall is at least 90%: the resolver searches the way the compiler does, and the compiler's include dirs are the ones registered | held | 99.7% |
| P2 | C/C++ precision is at least 80%: a platform branch the three configurations do not enable is an include the graph counts and the reference does not | held | 99.7% |
| P3 | component precision is at least 95%: a specifier either resolves to a file TypeScript also resolves it to, or not at all | held | 100.0% |
| P4 | component recall is at least 90%; the misses are the bundler aliases and .js-to-.ts mappings the JS resolver does not read | missed | 79.1% |
| P5 | PHP precision is at least 95% | held | 100.0% |
| P6 | PHP recall is at least 85% | held | 100.0% |
| P7 | Swift module-level precision is at least 99% | held | 100.0% |
| P8 | Swift module-level recall is at least 99% | held | 99.3% |
How to read them
Precision is the share of the graph's edges that the reference also has. Recall is the share of the reference's edges that the graph has. Both are over the importing files that the graph walked and the reference could analyse, so a file the reference could not parse is out of both counts and shows in the scope table as walked but not scored (svelte's deliberately invalid compiler-error samples are of this kind). The graph's walk is the zero-config one a user gets.
C, C++ and Objective-C. The two agree on almost every include. The extra edges are includes behind a platform switch (unix.h including aix.h, os390.h, sunos.h): the reference ran the host, Linux and Windows, and an include behind another switch is not in it, so the precision is a lower bound. The missed edges are all .inc files, which the graph does not treat as source files (abseil uses them for per-platform fragments).
PHP. The two agree on every use of a class in laravel and composer. The one extra edge is a literal require of a file, which the reference does not read, so PHP precision is also a lower bound.
Swift is module level. A Swift file imports a module, not a file, so the graph draws no file-to-file edge for Swift and the reference edge is "this file imports that target of the package". --impact on a Swift file names its target and lists the files in other targets that import it. The one miss is a C-language target of vapor, which has no Swift file to ask about.
Components, and the prediction that was missed. Every edge the graph draws between components is one the reference has (precision above), but component recall is the lowest of the four groups. The registered prediction was 90% recall; it was missed. None of the remaining causes is in the component reader (two defects there were found by this benchmark and fixed, and are in the deviation log): the script found in a .vue, .svelte or .astro file goes through the same specifier resolution as a .ts file, and that resolver does not read
- the package's own name, resolved through the
exportsfield of its ownpackage.json(every file of thesveltepackage that importssveltereachestypes/index.d.ts; all of the "bare specifier" misses are in that one repository); - a specifier written
./x.jsfor a filex.ts, or./x.jsxforx.tsx, which is how TypeScript projects write imports under Node's rules; - a JSON module, and a
#nameimport of theimportsmap; - an extension-less specifier that reaches a
.d.tsfile.
The registered guess named bundler aliases and .js-to-.ts mappings; the second is real (a dozen edges, all in astro), but the larger cause is the package naming itself, in svelte. These are real gaps for every .js and .ts file too, which this benchmark does not measure. They are not fixed here: that resolver also builds the graph the ranker reads, so widening it moves retrieval (#934) and needs its own review against the retrieval gate. A list of imports that spans several lines is read in a component's script and, for the same reason, is not read in a plain .js or .ts file.
Not measured. C#, Dart, Lua and GDScript have resolvers and test fixtures with known answers, but no figure. A C# or Dart reference is a compiler front end that needs an SDK, a restore and a build of the repository; Lua finds a module through a path the project sets at run time; and GDScript is resolved by the engine. A number from a weaker oracle would be a number about the oracle, so there is none.
What the figures do and do not say
- They say how often the graph's edges agree with what each language's tooling resolves, for these repositories, over the files both read. They are not a statement about every repository, and a repository's own build configuration (a
-Dswitch, a bundler alias table) can change what its tooling resolves. - A cell with fewer than 100 edges is shown and not quoted; the intervals are 95% cluster-bootstrap intervals over importing files.
- Precision says nothing about completeness and recall says nothing about the extra edges; read the two together, and read each against the reference's own limits above.
- The ranker's neighbour boost still reads only the original eight families. These figures are about the blast-radius surfaces, not about retrieval.
Reproduce
node scripts/fetch-xrepo-repos.mjs --only libuv,abseil-cpp,vue-core,svelte,astro,laravel,composer,alamofire,vapor
node scripts/run-import-accuracy.mjs --setup-references /path/to/refs # pinned TypeScript, the framework parsers, PHP and Composer
node scripts/run-import-accuracy.mjs --references /path/to/refs # prints the tables
node scripts/run-import-accuracy.mjs --references /path/to/refs --save # also writes benchmarks/reports/import-accuracy.json
node scripts/run-import-accuracy.mjs --write-doc # re-renders the tables above from the saved reportC and Swift need clang and swift on the path. Each repository is exported at its pinned commit into a scratch directory, so the shared clones are only read. --save refuses a partial run, a protocol that differs from the committed one, and a scored tree with uncommitted changes.