Evaluating Mythos 5.1 on Smart Contract Security

Summary

Following the expansion of Anthropic's Cyber Verification Program, we obtained access to Mythos 5.1 and began routine internal evaluations of frontier models for cybersecurity tasks. In this post, we share our experiments with this model on a smart contract project preparing for a production launch.

We ran Mythos 5.1 using Claude Code in ultracode mode with four different configurations: one open-ended security review prompt (security-review), one focused on property specification (scribble), one focused on rounding errors (roundme), and one focused on hunting for risks based on a pre-existing threat model of the protocol (threat-hunt). The objective was to measure how tools, prompts, and dynamic workflows affect results and cost. This research can inform the development of more complex workflows in which a stronger orchestrator can direct lower-cost open-weight models to achieve better overall cost-performance results.

Together, the runs found 26 distinct production issues: 1 High, 10 Medium, and 15 Low. Each of the four runs found at least one issue the others missed. The open-ended review led in precision and F1-score. The threat-hunt review found the most issues, led in relative recall, and earned the most severity points. The Scribble review earned the most points per dollar. The rounding review had the lowest relative recall and F1-score but found one issue that all other runs missed.

Methodology

All four runs used Mythos 5.1 in Claude Code (ultracode) against the same pinned code revision. These are the task prompts:

security-review: “Perform a security review of this project.”
scribble: “Perform a security review of this project by first instrumenting it with scribble and then focusing on what it unlocks.”
roundme: “Perform a security review of this project by focusing on rounding errors and using roundme to methodically analyze the rounding direction of mathematical operations.”
threat-hunt: “Perform a security review of this project with the help of its threat model by using one hunt agent per threat.”

We collected 196 findings from the four runs. Using GPT-6 Sol Daybreak Blue (Ultra), we triaged and grouped overlapping claims. We then produced a unified report documenting each finding’s provenance and whether it matched a final accepted issue exactly or partially. Human review determined the validity and severity of each finding.

Alongside production issues, we classified defensive improvements, behavior dependent on product decisions, threat-model corrections, and documentation differences as Informational findings. The comparison covers validated production issues rated High, Medium, or Low. For precision, relative recall, and F1-score, eligible claims are those originally reported as High, Medium, or Low and classified as comparable code-defect claims. Claims about the same underlying mechanism are counted once per run. Threat-model corrections and documentation differences are excluded from this metric set.

Eligible issues found are validated High, Medium, or Low issues found through claims originally reported at those severity levels. The overlap charts and severity points also include findings first rated Informational if they match a validated production issue.

We tracked the following metrics:

Precision = the number of eligible claims that matched a validated issue divided by the total number of eligible claims.
Relative recall = the number of eligible claims that matched a validated issue divided by the total number of distinct issues found by any of the four runs.
F1-score = the harmonic mean of precision and relative recall.
Score share (%) = a run’s severity points divided by the total severity points awarded to all four runs, multiplied by 100. The default weights are 10 for High, 3 for Medium, 1 for Low, and 0 for Informational. Partial matches receive half credit.
Score share per USD 100 = a run’s score share divided by its model spend in USD, multiplied by 100. The unit is percentage points (pp) per USD 100.
Wall-clock time = elapsed time from the start of a run to its completion. Concurrent agent runtimes are not added together.
Spend = reported model usage cost for a run in USD at list prices, excluding report assembly, human adjudication, and unpriced local compute.

These metrics describe one run of each prompt against one target. The codebase had undergone security reviews, but we have not yet completed a comparison of the 26 findings against earlier reports.

Results

Detection quality

Review	Eligible issues found	Precision	Relative recall	F1-score
security-review	13	0.813	0.500	0.619
scribble	5	0.625	0.192	0.294
roundme	3	0.750	0.115	0.200
threat-hunt	15	0.385	0.577	0.462

Cost, time, and score share

Review	Score share	Wall-clock time	Model spend (USD)	Score share per USD 100
security-review	34.20%	10h 30m	1,324.76	2.58 pp
scribble	9.84%	2h 17m	204.85	4.81 pp
roundme	7.25%	6h 24m	1,214.70	0.60 pp
threat-hunt	48.70%	6h 33m	2,984.39	1.63 pp

security-review

Eligible issues found: 13
Precision: 0.813
Relative recall: 0.500
F1-score: 0.619
Score share: 34.20%
Wall-clock time: 10h 30m
Model spend (USD): 1,324.76
Score share per USD 100: 2.58 pp

scribble

Eligible issues found: 5
Precision: 0.625
Relative recall: 0.192
F1-score: 0.294
Score share: 9.84%
Wall-clock time: 2h 17m
Model spend (USD): 204.85
Score share per USD 100: 4.81 pp

roundme

Eligible issues found: 3
Precision: 0.750
Relative recall: 0.115
F1-score: 0.200
Score share: 7.25%
Wall-clock time: 6h 24m
Model spend (USD): 1,214.70
Score share per USD 100: 0.60 pp

threat-hunt

Eligible issues found: 15
Precision: 0.385
Relative recall: 0.577
F1-score: 0.462
Score share: 48.70%
Wall-clock time: 6h 33m
Model spend (USD): 2,984.39
Score share per USD 100: 1.63 pp

Score share is each run’s percentage of the total severity points. “pp” means percentage points.

We assigned each issue one primary type from the OWASP Smart Contract Weakness Enumeration (SCWE) where a close match existed. The most common mapped weakness type is edge-case handling, which includes several different state and boundary errors.

OWASP SCWE type	High	Medium	Low	Total
SCWE-083: Failure to Handle Edge Cases	0	2	7	9
SCWE-017: Privileged Role Mismanagement	0	4	1	5
SCWE-124: Inconsistent Rounding Direction in Financial Math	0	1	1	2
SCWE-090: Missing Slippage Protection in Automated Token Swaps	1	0	0	1
SCWE-082: Lack of Proper Gas Management	0	1	0	1
SCWE-036: Inadequate Gas Limit Handling	0	1	0	1
SCWE-046: Reentrancy Attacks	0	1	0	1
SCWE-014: Lack of Emergency Stop Mechanism	0	0	1	1
SCWE-016: Insufficient Authorization Checks	0	0	1	1
SCWE-047: Integer Overflows and Underflows	0	0	1	1
SCWE-086: Missing Validation of Oracle Response Fields	0	0	1	1
SCWE-149: Transfers to Addresses That Cannot Receive Funds	0	0	1	1
No close SCWE match	0	0	1	1
Total	1	10	15	26

Across the 25 issues with a close SCWE match, the runs covered 12 weakness types. The security-review and threat-hunt runs each covered 10 types, Scribble covered seven, and Roundme covered two. Three types appeared in only one run: SCWE-047 (integer overflows and underflows) in security-review, SCWE-014 (lack of an emergency stop mechanism) in Scribble, and SCWE-046 (reentrancy) in threat-hunt. Roundme found a unique issue (no close SCWE match), but both of its weakness types also appeared in other runs.

UpSet plot of all 26 issues, labeled by SCWE type. Threat-hunt detected 20 issues, security-review 14, Scribble seven, and Roundme three. Roundme’s exclusive intersection contains one issue with no close SCWE match. Vertical labels give the weakness types and issue counts within each intersection. — Shared and exclusive issues, labeled by weakness type. Bar heights show the number of issues. Colored bands identify which runs found them.

UpSet plot of 26 validated issues across four Mythos reviews. Threat-hunt detected 20 issues, including ten exclusive issues. The other reviews detected 14, seven, and three. Purple marks threat-hunt, berry security-review, blue Scribble, and teal Roundme. — Shared and exclusive issues across the four runs. Bar heights show the number of issues. Colored bands identify which runs found them.

Grouped bars show precision, relative recall, and F1-score for each of four Mythos reviews. Each review has its own color; patterns distinguish the metrics. The open-ended review has the highest F1-score; threat-hunt has the highest relative recall. — Precision, relative recall, and F1-score use eligible defect claims. Relative recall uses the issues found by these four runs as its denominator.

Cost analysis

The four runs spent USD 5,728.70 in reported model usage. This excludes report assembly, manual adjudication, and unpriced local compute. Threat-hunt cost USD 2,984.39, more than the other three runs combined. It found the most issues and earned the most severity points. Its dynamic workflow used over 700 agents to investigate and verify threats from a pre-existing threat model generated with Opus 5.5 (ultracode). The general security-review prompt had the best F1-score at less than half that cost, despite being lower in relative recall.

Cost versus F1-score scatter plot for four Mythos reviews. The open-ended review has an F1-score of 0.619 at USD 1,324.76; threat-hunt has an F1-score of 0.462 at USD 2,984.39. — Model spend and F1-score for each run.

F1-score treats all validated issues equally. We also scored each issue by final severity, using weights inspired by Code4rena audit competitions. In our benchmark, each High-severity issue is worth 10 points, each Medium-severity issue 3 points, each Low-severity issue 1 point, and each Informational issue 0 points. Partial findings (reports that identify a mechanism but do not explain its full validated impact) receive 50% of the issue’s points. Each run receives its own credit for shared issues, with no duplicate penalty. These are benchmark scoring rules, not a reproduction of competition payouts.

The chart below lets readers change the weights, partial credit, duplicate multiplier, and point-pool split to explore how sensitive the results are to these scoring choices. It also has comparison presets for Code4rena, Sherlock, Immunefi, and Cantina.

Cost and score share

Each run’s score share is its points divided by the total points for all four runs. Score share per USD 100 is measured in percentage points.

Review	Model spend (USD)	Score share	Score share per USD 100
security-review	1,324.76	34.2%	2.58 pp
scribble	204.85	9.84%	4.81 pp
roundme	1,214.70	7.25%	0.60 pp
threat-hunt	2,984.39	48.7%	1.63 pp

Edit scoring rules

High pointsMedium pointsLow pointsInformational pointsDuplicate multiplierPartial credit fractionSplit issue points

Each additional detecting review multiplies an issue's point pool by the duplicate multiplier. A value of 1 leaves the pool unchanged. With splitting off, each review gets the pool times its credit fraction. With splitting on, reviews share the pool in proportion to their credit. Each review counts once per issue.

The Default preset uses H = 10, M = 3, L = 1, I = 0; no duplicate decay or split. The Code4rena preset models its High and Medium main pool. Sherlock refers to the now deprecated audit competition ruleset. Immunefi shows only the shared part of its competition award. Cantina follows its worked duplicate examples.

Observed dynamic workflows

Claude Code’s ultracode setting lets the model decide when to use dynamic workflows. In these workflows, Claude writes orchestration scripts that break a task into subtasks, run agents in parallel, and verify their results before combining them. We also examined how Mythos’s plan varied across prompts. The run records show these stages:

Review	Observed plan
`security-review`	Four Find → Dedup → Verify rounds, with Critic feedback after the first three. The first round covered 17 subsystem lenses.
`scribble`	Eight parallel Understand passes, then 242 Scribble properties and a stateful test campaign.
`roundme`	Six parallel Inventory passes mapped 215 rounding sites, then three Hunt → Verify → Critic rounds.
`threat-hunt`	Two Hunt streams covered 146 modeled threats. Two independent reviewers checked qualifying claims; a judge resolved disagreements. Root-cause grouping followed, and 42 cases received a third review.

These are observed plans. The prompts, tools, scopes, and budgets changed together, so these runs cannot show which change caused a result.

Limitations

These runs used one target and one code revision. The issues raised provide a relative benchmark, as unknown defects remain outside it. Severity and partial-credit decisions reflect the current adjudication and may change after further review.

This post is for informational and research purposes only and does not constitute security advice. These results alone do not establish that a codebase is secure or that one model or workflow is generally superior. They do not replace independent security reviews.

Next steps

Our next step is to design more complex dynamic workflows in Ultrafuzz, using these findings to improve cost-performance across different audit profiles. If you’re working on something similar, reach out.