How a scan works

The graph walk, the questions, and how probabilities turn into a ranking.

perch scan scores every method and ranks them worst first. Every answer is a probability, and the ranking is arithmetic over all of them. The only cutoff is the floor on what gets listed.

1. tree-sitter#

Every file tracked at HEAD with a known extension is parsed. Vendored and built files are excluded, as are two languages whose methods cannot be spliced back as one region: Dart, whose signature splits from its body, and Elixir, whose functions are macro calls.

JavaScript, TypeScript, TSX, Python, Rust, Go, Java, Kotlin, Scala, Groovy, C, C++, C#, Ruby, PHP, Lua, Swift, Zig, Solidity, Bash.

For each named method the analyzer records its line range, a hash of its source, the calls it makes, the identifiers it passes as values, and five metrics:

2. The call graph#

Nodes are named methods. An edge is drawn when a call resolves:

• to a method in the same file, by name;
• through an import, to a method in another scanned file;
• for Go, to a method in the same directory;
• as a handover, when a function is passed as a value rather than called — a tool handler, a callback, an event listener. These resolve only when the name is unique across the repository, and the model is told the edge is a handover rather than a call site.

Handover edges are how perch sees its own tool handlers. Without them the graph had 190 edges and 62 methods with no resolvable caller; with them, 588 and 46.

3. The walk#

A stack, seeded with the riskiest method by risk_score:

1. Pop the riskiest unvisited method and ask about it.
2. Push its unvisited callees and callers, riskiest on top.
3. Push the neighbor the model said to follow above all of them.
4. When the stack empties, take the next riskiest method overall.

Test methods are analyzed, so they can appear as callers, but never questioned. Up to --parallel methods are in flight at once.

Every method in scope is read. What scope is comes from --paths or --since, and nothing else: a cap on how many methods a run reads leaves a report that looks complete and is not.

Every reading goes into scan.jsonl, rewritten whole each run, so what it holds is what this run says about this commit and nothing older. A reading carries forward when the request that produced it would go out word for word the same: the method's source, the neighbours in the state, and the wording of every question including your rules. That is a hash on each reading, compared before anything is sent. A rescan of untouched code costs nothing and reads the same to the percentage, which matters as much as the cost — an issue you looked at yesterday should not have moved because a model was asked twice.

4. The questions#

The questions are declared in scan.yaml, which perch ships and perch.yaml can reword or add to. Your own rules about a method are asked in that method's request, beside perch's: every question in a request is scored against the state by itself, and the state is what the request is mostly made of, so a method covered by five rules is one reading rather than six.

One HTTP request per method. The state carries the method with its lines tagged L0042|, the comment above it, its metrics, its file's imports and module scope, the source of up to 8 callees with the names of their own callees, up to 8 callers with the line where each calls it, and the call edges among all of them — trimmed to stay under 48KB.

A method too long for one request is read in overlapping passes, each sized to what the budget actually holds. Only lines a pass can see are offered to its where question. The answers merge: the worst defect found anywhere is the method's defect, a vulnerability is the likeliest reading from any pass, and the first pass — the one carrying the callers and callees — speaks for the method's shape and documentation. Eight passes is the cap; a method longer than that is read in part and says so, on the record and in perch issues <id>.

A method that still cannot be read is recorded against itself and the walk carries on; perch doctor lists them with the error each one failed on. If nothing can be read at all — a bad key, a service that is down — two full batches of failures in a row ends the run.

Each question uses the System One primitive that fits it. A noul is a probability that something is true. A choice picks one option and returns the distribution over all of them. A score grades against a rubric and returns the distribution over its levels.

The names in that table are the ids written in scan.yaml. What a row prints is the label they map to, so boundary reads as off_by_one and missing_null_handling reads as unhandled_null. perch issues --types lists the labels, which are what --filter kind= takes.

System One does not bill output tokens, so asking thirty questions of a method costs what asking one costs. That is why the set is wide rather than staged.

Why these primitives#

kind was once eight independent yes/no questions. It is a pick-one — a defect has a kind — so it is a choice, and the distribution it returns is what the row prints.

severity was once three yes/no questions, and before that a single rating. It is a rubric, so it is a score:

The security classes stay independent yes/no questions, because they are: a method can be both injectable and leaking a secret.

5. Issues#

Each answer becomes at most one issue, carrying the probability that it is real.

A defect is has_bug, labelled with the likeliest kind. The kind does not discount it: multiplying the two would say a method certainly broken but of uncertain kind is only half broken.

A vulnerability is the likeliest security class. Eight of the sixteen need something from outside to reach the method, so those are a joint probability:

P(vulnerable) = P(class) × P(exposed)

The other eight — memory safety, concurrency, type confusion, weak crypto — are wrong whoever the caller is, and are not gated. Gating them once hid a planted double-free scored use_after_free 95% behind exposed 46%.

Design issues are the refactor the refactor choice picked, 1 − P(does what it claims), and docs.

The floor#

An issue is listed when its probability is over 0.5. That is not a number picked to make the list a nice length. A noul is the probability that something is true, so above a half is the model saying yes and below it is the model saying no; printing everything means printing every method in the repository, because no answer ever comes back at zero. On perch itself the floor takes 364 listed methods down to 240, and --filter type=defect from 354 to 25.

It is a floor on what is claimed, not on the arithmetic:

• Ranking counts the whole distribution. An issue at 49% still weighs 0.49 in where its method sorts, so there is no cliff at the boundary — only a line below which perch stops saying it found something.
• perch check reports the whole distribution too. Halving a 40% defect is visible as that, even though neither the before nor the after is listed.

--min P moves the line, in percent. --min 0 prints everything the scan answered.

6. Ranking#

Issues split in two. Correctness is the defect and the vulnerability; design is the rest. Each side is the sum of its probabilities, so two issues at 50% weigh what one at 100% weighs and nothing has to cross a line to count.

A method is ranked by what its problems would cost, not how many it has:

weight = correctness × mean + design

mean is the severity distribution's mean, worked out below. This is the only weighting in the ranking, and it is measured rather than chosen: the number it replaced was a × 2 somebody made up.

Design problems weigh as themselves. They are the ones the rubric's own bottom level describes: no caller would notice.

The severity formula#

The score comes back as a distribution over the four levels, not a level. Level 0 is "no caller would notice" and level 3 is "data lost, corrupted, or exposed, or a check that should stop someone bypassed". The bands run the other way, so level 0 is P3 and level 3 is P0.

Three numbers come out of that distribution:

mean  = Σ level × p(level)                  0 to 3, worst at 3
band  = [P3, P2, P1, P0][round(mean)]       the label
shown = 3 − mean                            the same number on the P scale

mean is what the ranking multiplies by. band is the label a filter matches. shown is the number in brackets, on the scale the bands are named in, where 0 is worst and 3 is harmless.

Worked, on the buildGraph distribution from reading the issues, which perch prints as P1 (1.1):

Severity   P1 83%   P2 12%   P0 4%   P3 1%
             ↓        ↓        ↓       ↓
  level      2        1        3       0

mean  = 2(0.83) + 1(0.12) + 3(0.04) + 0(0.01) = 1.90
band  = [P3, P2, P1, P0][round(1.90)] = [P3, P2, P1, P0][2] = P1
shown = 3 − 1.90 = 1.1

Which is why P1 (0.9) is nearly a P0 and P1 (1.2) is settling toward P2.

The rubric is about how much a caller would feel whatever is wrong, not about defects alone: its top level is a vulnerability in so many words. A defect and a vulnerability are both weighed by it, and both carry the band.

Naming a method by the band holding the most probability throws the rest away. A spread of P0 33% / P1 31% / P2 30% / P3 6% is called P0 on the strength of a third of the mass, and reads as worse than a method with 54% on P1 and 29% on P0 that is in fact expected to do more damage. The mean says so; the label does not. perch issues <id> prints the whole distribution.

7. Filtering#

--filter takes type=, kind= and severity=, which are the three columns that name a problem. Clauses on the same key are alternatives; clauses on different keys must all hold.

A filtered list is also ranked by what was filtered for, since ranking it by overall weight would bury the strongest match. Within a key the likeliest matching issue speaks for it; across keys they multiply:

strength = max(matching issues in key A) × max(matching issues in key B) × …

So --filter type=security leads with the likeliest vulnerability in the repository rather than with whichever method is heaviest overall. Each row also reorders to lead with the match, so a row selected for a vulnerability does not print refactor in its Type column.

A filter reads the listed issues, so it inherits the floor: type=security is the methods probably carrying a vulnerability, not every method that scored nonzero on one. --min 80 narrows it further; --min 0 widens it to everything the scan answered.