Benchmarks
What we measure, and where we lose
Every number on this page comes from an artifact committed to the repository, and the file is named next to the number. We publish the losses in the same table as the wins, because a security tool that only shows its wins is a marketing page.
Hunter scorecard against the static engine
Seven specialist roles ran over 143 labelled contracts: 1001 model calls. Strict recall counts a hit only when the quoted code sits inside the cited lines. A role with no target of its own class is not measured, never zero.
| Role | Targets | Answered | Strict recall | Precision | Static engine | Delta vs engine | False alarms |
|---|---|---|---|---|---|---|---|
| hunt_reentrancy | 31 | 31 | 100% | 67% | 6% | +94% | 15 |
| hunt_arithmetic_rounding | 15 | 15 | 100% | 50% | 7% | +93% | 15 |
| hunt_delegatecall_proxy_storage | 1 | 1 | 100% | 25% | 0% | +100% | 3 |
| hunt_unchecked_external_call | 52 | 52 | 98% | 86% | 4% | +94% | 8 |
| hunt_access_control | 18 | 18 | 94% | 41% | 11% | +83% | 24 |
| hunt_dos_griefing | 6 | 5 | 60% | 11% | 0% | +60% | 25 |
| hunt_flash_loan_economic | 2 | 1 | 0% | 0% | 0% | 0% | 8 |
Correction: This scorecard replaced an earlier run measured with a response budget half the size of the library's. Truncated calls failed, and the tail of each answer, where the weakest findings sit, was cut: 55 false alarms became 98. The precision published above is the corrected, lower one.
Same tokens, three platforms
13 tokens discovered on the open market through DexScreener, no address invented, queried on VegaSec, GoPlus and Honeypot.is.
Answered: VegaSec 13 of 13, GoPlus 10, Honeypot.is 7.
None of the three platforms flagged a honeypot or a hidden owner on any of the 13 tokens. That is a fact about the sample, not a clean bill of health for the tokens.
BNBO · Binancians · STREET · zc · CFO · MDP · SNOWMOON · LONGBNB · USDC · WETH · UNI · LINK · SHIB
Judgement calibration
The judgment layer writes prose about findings. Two independent readers scored 37 published findings against the contract source with the same rubric, without seeing each other.
34 correct, 3 partial, 0 wrong. Raw agreement between the two readers: 35/37 = 94,6% (antes de aplicar a regra; divergentes #19 e #31).
Correct-explanation rate: 91.9%, with a 95% confidence interval from 78.7% to 97.2%.
The rate comes from 37 judged explanations, which is why the interval is wide, and we print the range instead of the number alone.
Adversarial verification
Before a finding reaches a report, an adversary role receives the contract source and the finding, and is told to refute it.
It caught 3/3 of the findings known to overstate their case, confirmed 19 outright, and flagged 14/34 of the findings a human reader had passed. Flagged means review, not wrong.
Declared limits
Read the table above with these in hand.
- The corpus is SmartBugs, a public labelled dataset. It is not a client contract, and it is a convenience sample, not a random one.
- Three roles carry one or two targets. Numbers there are not decisions, and we say so instead of cutting them on noise.
- Classes with no hunter role: timestamp-dependence, oracle-manipulation, other, front-running. Declared, not omitted.
- Transport failures: 4. Read failures: 1. Every call, its tokens and its reason are in the raw artifact.
Reproduce it
The summary on this page is generated from the raw artifacts by one script. Point it at the files and compare.
# partindo do repositorio, em apps/web
npx tsx scripts/benchmarks/emit-summary.ts
# saida: src/app/benchmarks/data/summary.jsonRaw artifacts
docs/benchmarks/vps-scorecard/2026-09-27-swarm-hunters-base-completa.jsondocs/benchmarks/2026-09-27-cross-platform/amostra-mercado.jsondocs/benchmarks/2026-09-27-judgment-adjudicacao-v1.jsondocs/benchmarks/2026-09-25-judgment-calibracao.jsondocs/benchmarks/2026-09-27-verificador-adversarial.json
Measured on smartbugs, runs from 2026-09-27T18:21:23.824Z to 2026-09-27T19:19:06.773Z. The page is generated, not typed: if a number here disagreed with the artifact, the build would be lying in the filename.