Cutting AI-Driven Security Testing Noise: A Framework for Precision Over Hype

The security field is drowning in two kinds of noise.  

The first is technical: AI agents spewing false positives, hallucinated exploit chains, and confidently wrong vulnerability reports. This has been demonstrated in the bug bounty space, where programs have seen submissions rising by 76% in just one year, hitting an all-time high in March 2026. But the percentage of submissions that turned out to be valid, exploitable vulnerabilities held flat at roughly 25%. More noise, than signal.  

The second is cultural: the deafening debate about whether AI will replace us or destroy our human ingenuity. 

Both are distractions that miss the point. At the most recent DEF CON, where much of the buzz was centered on AI, I heard two talks that changed the game for me. The first was James Kettle's "Can AI Do Novel Security Research?", and the second was Jason Haddix's "Building Hackbots."  

These talks crystallized something I'd been sensing: when you ground your pen testing methodology deep enough in logical baselines, the noise stops being a distraction, and the signal becomes clear. 

Two Kinds of Security Testing Noise: False Positives and Panic 

Let’s start with technical noise. We all know and love traditional scripts, but they're fragile. They break when the app changes, the auth flow shifts, or the API version alters. If we wrap these scripts in AI without changing the foundational approach, we just import a smarter-sounding version of the same problem.  

Kettle found something telling in his research. Even frontier models kept mistaking ordinary HTTP pipelining, completely benign traffic, for a smuggling vulnerability. This is an easy mistake to make, but one indicative of a novice's approach. When he tried to prompt his way out of the issue, it didn't work. 

Then there’s cultural noise. One camp fears that AI will automate and replace us out of a job. The other fears a drastically diminishing quality of results with AI flooding the market with shallow, generic findings. On the surface, these feel like opposites. Is AI too capable, or is it only capable of producing bunk results? 

But they share a single root cause: how we ask a question. Ask broadly, and AI hallucinates garbage. Expect it to architect a full pen test autonomously, and it fails to replace experienced judgment. The resolution to each fear lies in the instruction set underneath the tool.  

Haddix’s hackbots and Kettle’s research overcome these issues for the same reason. They reduce scope to simple truths. Haddix doesn’t ask an AI to "hack this app." Rather, he asks it to execute one markdown-defined skill at a time.  

Kettle doesn’t ask for it to “produce desync vulnerabilities.” He feeds the AI 1 to 3 sentence RFC fragments and asks for a small set of hypotheses rather than a sprawling one. The noise, whether from false positives or cultural panic, comes from asking the wrong questions at the wrong scale. Precision kills both the garbage and the dependency on the AI to figure it out alone. 

From Noise to Signal: Logical Baselines as the Foundation 

The alternative to noise is simple, but profound: a logical baseline. It’s the smallest, irreducible condition for a vulnerability to exist. This is not a payload or a script but a single condition. 

• For IDOR: User A’s identifier is accepted in User B’s session without re-authorization. 
• For XSS: Untrusted input reaches a dynamic execution context without encoding. 
• For SSRF: User-supplied URLs are fetched without validating the target domain. 

Notice anything missing? There are no tool names, parameters, or framework syntax here. A baseline is tool-agnostic, meaning it describes the flaw, not the discovery method. 

Haddix’s 200+ hackbot skills are exactly this, methodologies listed in markdown, not Python or Bash. The AI reads the baseline, picks tools (curl, browser, interpreter), and executes. Kettle’s micro-inspiration works the same way in miniature. Give 15,000 RFC fragments, each an atomic protocol behavior, small enough that the AI can't drift into hallucination but has to reason from the fragment itself. 

The insight is clear. We don’t need AI to be a better hacker. As a team, we need to be better at extracting baselines from our own expertise. Once a baseline is in place, the AI can execute it across any target without generating noise. The hard part isn’t building the AI. It’s articulating what you actually know. 

Cutting Technical Noise: Code Gates and Minimal Prompts 

Now let's talk about how we actually kill the false positives, the garbage that can make AI-assisted testing unusable at scale. 

Haddix and Kettle arrived at the same solution from different directions: The AI can never be the final judge. 

To combat AI "laziness," the tendency for models to skip large workloads or truncate output, Haddix implements multi-model orchestration. That is, one agent generates while another critiques and verifies completion, resulting in a peer review system for machines. He also builds in checkpoints in the form of hard stops that cap how much an agent can chew through in one pass, and force it to prove it actually finished before moving on. 

Kettle takes a more deterministic approach. When he realized that even frontier models produced false positives and repeated rookie mistakes, he gradually moved responsibility for critical checks to deterministic code. The exploitation and evidence-gathering process was split into separate stages protected by code-validation gates, with each stage receiving fresh context to prevent flawed reasoning from propagating to the next. Eventually, the system was able to produce zero false positives. 

He frames this as "AI vs Code vs Human." That is, start with an AI-heavy approach for exploration and hypothesis generation, then progressively shift responsibility to deterministic code for validation and speed. The AI proposes, and the code decides. 

Cutting Cultural Noise: Precision and Earned Craft 

Let's address skepticism directly, because I count myself among the skeptics on some days. 

The fear surrounding AI and its capabilities is valid. A flood of generic, LLM-generated pen test reports does dilute the expected technical quality of the entire field. If every junior tester can generate a plausible-looking report with a single prompt, the signal-to-noise ratio in our industry collapses. Clients will get buried in false positives. Real vulnerabilities will get lost in the noise. The craft degrades as trust is lost. 

But here's the thing: generic AI results from generic prompts. A shallow ask, "find vulnerabilities in this app," gets you shallow, hallucinated output. A deeply decomposed baseline, one that names the exact trust boundary and the exact condition, gets you precision instead. The AI isn't guessing anymore. It's running a defined experiment with a clear pass/fail criterion. 

Haddix makes the distinction that hackbots should be "human-in-the-loop systems" that can help us elevate every part of our craft. Rather than replacing our judgment, they're scaling it. 

This idea should be a warning, not a comfort, to anyone who hopes to lazily skate by on a prompt-addled shortcut to succeed in the field. AI isn't making high-quality work worse. It exposes teams that don't have high-quality mental models to begin with. If your "expertise" is just a memorized checklist of payloads and common misconfigurations, AI will replicate that checklist and cheapen your work. Your checklist becomes commodity and your value erodes as a result. 

But if your expertise is the ability to decompose a sprawling microservices architecture into its fundamental trust assumptions, if you can look at a complex system and see the numerous logical baselines that matter, AI becomes a powerful accelerant to reach that depth, honing your aim rather than replacing you. 

This is what I refer to as "growing the human roots." The root isn't the code  or the prompt. It's the human-driven cognitive framework the expert brings, including the ability to see patterns, extract baselines, and reason about trust in complex systems. AI is just the soil that lets those roots spread faster and reach further. 

Encoding Expertise: Using AI to Scale Human Knowledge Without Losing It 

This is where the conversation shifts from technical to strategic. 

Every senior penetration tester carries a mental model of how applications break. This is a hard-won intuition about where to look, what to test, and what to ignore. That intuition is invaluable, and it often walks out the door when they leave. 

Haddix's skills framework offers a solution in the form of encoding over 200 offensive-security methodologies as markdown files. Each skill captures years of accumulated knowledge, not as code, but as methodology. A junior tester, or an AI agent, can read that skill and execute it without needing the senior tester in the room. 

But here's the crucial nuance: the skills aren't replacing the senior tester's intuition. They're forcing that intuition to be articulated. 

Writing a skill requires answering, "What exactly am I testing for, and how do I know if I found it?" That question forces you to decompose your mental model into explicit, testable conditions. It exposes the assumptions you've been making unconsciously. 

Kettle's work reinforces this from a different angle. He built the HTTP Terminator by encoding his own research methodology, four years of expertise on HTTP desync attacks, into an autonomous system. But he's clear the AI didn't replace him. Rather, he calls the human role a "power amplifier," not a bystander. He's now open-sourcing the system along with a blueprint for other researchers to encode their own methods and areas of expertise. 

The institutional parallel is that when you articulate your methodology as skills or baselines, you're preserving and scaling your team's intellectual property in a form that outlasts any single individual. 

My personal takeaway and immediate action is this: before wiring anything to an LLM, we need to wire our own methodology first. I'm starting with a handful of vulnerability classes: IDOR, XSS, and SSRF. For each, I'm writing a one-paragraph logical baseline. There's no code, no tool names, just the condition. If a junior tester can't execute it, the baseline is too vague. Refine until it's bulletproof. Only then does it become a skill. 

Even without the AI, this exercise has already surfaced something valuable. Some of my "standard" procedures were just habits. I could exploit the vulnerability, but through writing baselines, I've been forced to examine my own methodology and discard the noise. 

Kettle's experience validates this incremental approach. He started AI-heavy and progressively moved responsibility to deterministic code as he discovered what worked. The key is to start small, validate rigorously, and scale what works. 

We're not trying to boil the ocean. We're planting roots. 

Planting Roots Before Algorithms: Your Blueprint for AI-Driven Security Work 

Let's circle back to where we started. 

The never-ending noise about AI’s effect on our work isn’t wrong, but it is reductive. The technical noise will be filtered by the discipline both speakers converged on: AI proposes, Code decides, Human directs. The cultural noise will be filtered by clarity of purpose and a stubborn commitment to depth over speed. 

Haddix showed that hackbots can scale human expertise across increasingly complex environments. Kettle showed that AI can discover genuinely novel attack techniques when fed the right atomic inputs. Both succeeded because they reduced complex problems to singular logical units. 

But neither of them claimed the AI did it alone. Even Kettle's "autonomous" system was a human-built loop, and his single greatest discovery still needed him in it. 

Haddix's hackbots are systems with human feedback loops baked in. Kettle's HTTP Terminator was built around his own research methodology. The AI didn't replace the researcher. It amplified them. 

Kettle framed this perfectly in his research objectives. He wasn't trying to prove AI could replace him. He was trying to find where it couldn't, so that he could focus his own efforts there. 

That's the mindset we're adopting. It isn't "what can AI do for us?" but "what can we do that AI can't, and how do we record what we know in a way that allows AI to handle the rest?" 

The teams that thrive in this era won't be the ones who spend the biggest budgets on GPU clusters or the most expensive API credits. They'll be the ones with the deepest knowledge who are willing to take the time to articulate why a vulnerability exists, not just how to look for it. They'll be the ones who treat AI as a forcing function for clarity, not a shortcut around thinking. 

What I'm left with is this. The AI hype cycle is loud. It crashes like thunder while the winds of change continue to blow. What remains is the clarity of thought developed while everyone else was chasing the noise. We can either grow our roots, or get blown down in the weather.