Open sourcing Ultrafuzz

Introduction

Today we are open sourcing Ultrafuzz, an agentic smart-contract fuzzing and threat-hunting orchestrator.

Ultrafuzz coordinates hundreds of specialized agents that configure a target repository, define actors and user flows, extract properties, threat-model the system, and launch parallel strategies to investigate bugs, build stateful and stateless fuzz tests, deduplicate findings, and generate a report.

Our original post introduced the workflow and results from the first pilot. Since then, more than 20 runs on Monad ecosystem projects have shaped the current version: a modular system built around public and private evals, with support for both Vyper and Solidity, frontier and open-weight models, new providers, better observability, and better workflows. This post covers what changed, what we learned, and how to run or contribute to Ultrafuzz.

Get started

Prompt your agent with something like:

Your task is to launch an Ultrafuzz campaign on this project and
make sure it succeeds from start to finish.
Show me the available audit profiles and explain the tradeoffs.
Recommend the best fit for this target.
Before installing any dependencies, ask for my confirmation.
After the campaign starts, use the CLI to monitor its progress.
Keep monitoring until it finishes.
If a node fails, for example because of a cyber-safety refusal,
resume the campaign from where it stopped.
Ultrafuzz: https://github.com/monad-developers/ultrafuzz

What changed since our last post

Private beta feedback

Since the original announcement, the private beta focused on making Ultrafuzz easier to extend, monitor, and evaluate. We invited security researchers from across the broader EVM and non-EVM communities to review the codebase and provide feedback. We also presented the project in several venues: CypherTalk, ERC55, and technical talks with Runtime Verification and Category Labs.

Rewrite and stabilization

The project was rewritten from the ground up with a focus on modularity, maintainability, and evaluation-driven development. We improved the local dashboard and added monitor commands for following campaigns as they run. Support for Vyper was also added alongside Solidity. Before the open-source release, we focused heavily on reliability and stability, improving machine- and human-readable artifacts, errors, and logs.

Easier onboarding

Zero-configuration audit profiles were added to facilitate onboarding, so that users would not need to configure every individual depth or breadth knob by hand, although this is still possible for advanced use cases.

Evals, evals, evals

After the private release, we needed a way to reliably keep improving the tool with confidence. This led us to build an evaluation suite around three relatively small projects covering common EVM development frameworks, so we established UltrafuzzBench to make sure prompt changes did not cause regressions against the benchmarks. On every release, a "smoke" run executes with cheap-but-smart models (defaulting to gpt-5.6-luna) and tracks precision, recall, and F1 score, as well as total time, cost, and token usage.

Once the eval was good enough to tell us that the tool worked and would not break across releases, we wanted to understand how it could get better. We added support for EVMBench and SCFuzzBench targets, and kept a few private datasets from Monad ecosystem projects to test against bugs that were not publicly known.

And then we began experimenting and building up our thesis.

Throughout this effort, we developed three variants to distinguish the historical experiments from the system we are releasing today:

• Original refers to the fuzz-test-oriented version evaluated after our first post. Its agents were generally instructed to turn properties into executable fuzz tests.
• NoFuzz used the same topology and property specialists as Original, but instructed downstream agents to investigate those properties directly instead of requiring them to author fuzz tests.
• NoProperties kept the NoFuzz approach but removed all property-specialist prompts.

These experiments led us to improve the prompts and the overall topology, which we present today as Ultrafuzz v0.1.0, the open-source release.

Our investigation started with the simple question behind the Ultrafuzz announcement: what would explain our tool's advantage over other methodologies?

Hypothesis 1: Fuzzing finds different bugs than manual reviews

One assumption at the end of the previous blog post was that the AI audits we compared against would focus on "static analysis"-type approaches, similar to a traditional human security review, as opposed to a fuzzing engagement. So we asked ourselves:

Does requiring executable fuzz tests improve bug finding?

We kept the original target as an eval and compared Original against NoFuzz. NoFuzz used the same topology, but its prompts instructed agents to, for example, "Find a bug that breaks property X" instead of "Author fuzz tests for property X."

For example, consider a strategy that tries to find bugs related to encoding and decoding. There are many different ways security researchers could catch such a bug. One way to do it is to simply review the implementation and notice asymmetries. Another way to do it is to explore the round-trip property $f^{-1}(f(x)) = x$, where $f = \operatorname{encode}$ and $f^{-1} = \operatorname{decode}$ (Original).

So we proposed alternative prompts for NoFuzz:

Similar changes were introduced in all strategies, and this alternative implementation was benchmarked against the original version.

We ran each condition five times using gpt-5.6-luna xhigh and the same benchmark target. Each run used a separate machine (ten machines in total), so Original and NoFuzz did not share local caches or other machine state.

The results were surprising. NoFuzz appeared to find far more bugs than Original.

Why was that?

We were expecting the complete opposite.

After preliminary investigation, we realized benchmark conditions did not match the original blog post. What if the difference came from the model? After all, many benchmarks show that gpt-5.6-luna is less capable than gpt-5.5. Thus this was not an apples-to-apples comparison.

We also checked whether the model could have accidentally cheated the eval by looking up the solutions in a subsequent commit. No evidence of this was found, but instead we noticed a different problem: some early agents started on the pre-patch revision, while later agents continued from origin/main due to prompts that tried to enforce independence from previous nodes. This explained why some bugs were not found by either run: the agents were essentially targeting the post-fix commit hash.

Despite both versions having run under the same conditions, we considered the results poisoned and reran the experiment with a correct fork, no Git history, and the same gpt-5.5 model with xhigh reasoning used in the original post.

Unexpectedly, once again, the conclusion did not change.

To our surprise, NoFuzz used fewer tokens and found more bugs than Original. Not only that, but it also found all the bugs that Original did.

Had our whole thesis been wrong?

These results did not show that fuzzing is generally ineffective. They showed that requiring every strategy to author fuzz tests does not appear to be the most efficient strategy for this target. Direct property-guided investigation produced better recall at lower cost, so we adopted that prompt style in the current Ultrafuzz workflow.

We could make the benchmark better, increase the number of runs, individually analyse all prompts and logs, exclude nodes with partial failures, and do many more things. But we decided to move on.

For Hypothesis 1, we concluded that it had not been proven for this specific workflow. From the results at hand, we believe that frontier models are able to find simple, stateless bugs by simply "reading the code," without necessarily "testing the code." An alternative explanation is that test harness authoring is already expensive enough in terms of context management and might not be optimal as a bug-finding strategy for this specific target.

So we continued the investigation with a different working hypothesis.

Hypothesis 2: Stateful fuzzing is the differentiator of Ultrafuzz

We began investigating which bugs actually benefit from stateful execution.

Manually reviewing the agent prompts supported the fact that some test-authoring-focused strategies made little sense from a cost-benefit perspective.

For example, one strategy would try to author fuzz tests around "admin actions," even though those properties are not generally meaningfully fuzzed.

For historical context, before AI-assisted vulnerability research, the short duration of manual fuzzing engagements forced researchers to prioritize aggressively when specifying properties and implementing fuzz tests. Generally speaking, they would focus heavily on high-level invariants and global properties that would usually cause critical issues if broken. If time permitted, lower-priority properties could also be implemented, but only a small portion of protocols proceeded with this follow-up. Developers would usually weigh the cost-benefit tradeoff and prefer to conduct more manual reviews rather than implement fuzz tests for bugs that could easily be found by simply reading the code.

Could the same be true for AI?

For that, we launched a new eval against a SCFuzzBench target. This benchmark suite contains human-authored harnesses from fuzzing specialists, alongside human-derived properties that could be compared against machine-generated code.

This experiment supported our instinct.

The target was different, so a direct comparison with the other experiment is not possible. The fuzzing treatment surfaced a different finding that NoFuzz did not produce. The broken property depended on a transaction, the passage of time, and another transaction. This strengthened our motivation for keeping fuzzing as a core component of the tool. We do not claim that a "static analysis"-type prompt would have missed this, but the fact that the bug depends on multiple interactions or a specific sequence of interactions suggests that finding it requires deeper reasoning or combining more information.

Still, this was a useful signal.

But what about stateless tests?

Hypothesis 3: Properties are the differentiator of Ultrafuzz

Another question still bugged us. Assuming Hypothesis 2 is correct and stateful fuzzing is useful, we still cannot explain where the previous results came from.

So we looked at how we were generating properties.

Here, a property is a concrete statement about how the system should behave, derived from its code, tests, documentation, and intended behavior. For example:

• depositing x assets should increase totalAssets() by no more than x;
• the sum of all token balances should equal totalSupply();
• accumulated fees should only increase or stay unchanged; and
• a transfer should decrease the sender's balance and increase the recipient's balance by the same amount.

Some properties apply to one operation, while invariants must hold across every reachable state. They are not a feature list or raw documentation: they are testable claims that give agents a specific correctness condition to investigate.

Was property specification the reason we were finding more bugs than open- and closed-source alternatives?

This started the "NoProperties" experiment: what if we keep NoFuzz but remove all the property-specification lenses? If we remove the explicit bias toward deriving system properties and invariants, surely NoFuzz will perform worse?

Across five runs, this is what we observed:

As expected, NoProperties did perform worse, but by less than we expected.

The union fell from 26 bugs to 22, which strengthened our confidence that property specification is an important step in AI-assisted vulnerability research. Nevertheless, this still does not clearly explain why Ultrafuzz outperformed the alternatives in the announcement evaluation.

Hypothesis 4: Exploration breadth is the differentiator of Ultrafuzz

Recent research on the same topic, including our own analysis of Ultrafuzz runs, has made it clear that compute plays a major role in the success of AI-assisted vulnerability discovery.

Running the same prompt multiple times yields different results, which means that one simple yet effective lever available to users is to burn more tokens on the problem.

Therefore, we consider the number of distinct strategies another likely explanation for Ultrafuzz's success. In the first blog post, recall grew consistently from one iteration to the next as we added strategies. We have since expanded the workflow from 5 strategies to more than 20, and we believe that continuing to add well-targeted strategies is an easy step toward expanding the tool's bug-finding capabilities.

This conclusion led us to adopt ideas from other publications whenever they strengthened our workflow. The goal-hunting lane is a new addition that draws directly from Trail of Bits' research, which describes how you can give agents a /goal to find bugs after a threat-modeling pass. We build on this approach by adding a web3 vulnerability database on top of the threat-modeling exercise.

Web3 Vulnerability Database

Ultrafuzz currently uses the OWASP Smart Contract Security Project as its vulnerability database, although we believe this is an area with a lot of room for improvement. Preliminary experiments show that giving agents a structured starting point for investigating known bug classes improves recall.

We propose to create a graph of bug categories, subcategories, sub-subcategories, and so on. Related findings can be connected into a taxonomy of smart contract bug classes that agents can use to guide their investigations.

For example, "denial of service" is a bug category that can have multiple subcategories, such as "liquidation denial of service." Within that subcategory, "creating dust positions that prevent liquidation" and "fixed-term positions that cannot be liquidated before they are overdue" could be sub-subcategories describing ways this liquidation DoS can happen. The first bug is generic to collateralized lending protocols, while the second is specific to fixed-rate lending. Individual bugs from audit reports are concrete instances that show how these failure modes occur in practice. The graph's "depth" is a parameter that agents can use to spend more or fewer tokens on their hunts, and coverage of the taxonomy can be measured by how much of the graph they explore.

We foresee a common, shared, extensible taxonomy of bug types that can help with automated vulnerability research. We believe that organizing this shared knowledge into specific, reusable failure modes can help agents investigate protocols more systematically and improve vulnerability detection. If you are interested in collaborating, reach out.

It is easy to see how this all comes down to a time-and-money tradeoff, so it becomes another optimization problem that we should tackle next: how can we keep recall consistent while minimizing spend?

Our contribution

Our experiments changed the design: use direct, property-guided investigation broadly; reserve executable stateful fuzzing for bugs that benefit from sequences and changing state; and improve recall through diverse, specialized strategies.

At the end of these four experiments, we propose the Ultrafuzz workflow: a graph of specialized agents connected by artifact handoffs.

The prompts can be extended, adapted, or ported to another orchestration engine or framework. The project is open source under the MIT License, and we believe the workflow itself is the main contribution to the security industry. We welcome community contributions that extend or adapt it or focus on performance or cost improvements.

Next steps

With a growing number of vulnerability discovery harnesses coming to market, the question is how teams should evolve their security approach. Our evals showed again that different tools tend to surface different bugs, so we expect users to combine as many useful approaches as possible.

We want Ultrafuzz to be community-driven, open to everyone, extensible, and research-driven.

You are free to adapt and modify it, and we encourage teams to give back by submitting issues and PRs that make the tool better for builders across the EVM ecosystem.

Reusing the prompts in the workflow orchestrator of your choice is also valid. We believe the biggest value lies in the prompts, topology, and tools—and all of it is built on the great work of other open-source projects.

Thank you

We would especially like to thank thank_you for contributing and reporting issues that helped harden early versions of Ultrafuzz. We also thank StErMi for thoughtful discussions about nondeterminism, evaluation design, and the tradeoffs between deterministic checks and adaptable agentic workflows. We also thank JoohhnnnChase for interesting discussions about the reliability of multi-agent harnesses.

Finally, we thank all the Monad builders who gave us access to their repositories and trusted us to help make their codebases more secure.

If you have any questions, please reach out or submit a GitHub issue.