AI in Penetration Testing What's Actually Working and What Isn't

On nearly all sales calls, we receive questions around how Schellman is using AI on the pen test team, how AI is going to change our processes, and how fast these changes will be implemented.

And these are all fair questions given the increasing presence of AI in the broader penetration testing space lately. Models from Anthropic and OpenAI are starting to find real vulnerabilities in open-source tools on their own, and the autonomous testing vendors are getting louder.

In this post, we’re going to explore the use of autonomous AI pen testing, including where AI actually helps, where we’re seeing limitations, and how we at Schellman are implementing AI in our pen testing processes.

What To Know About Autonomous AI Pen Testing

No major compliance framework currently accepts a fully automated AI pen test at the time of this writing. Not FedRAMP, PCI DSS, or any of the other standards our clients are regularly assessed against. Compliance bodies that set the standards move slowly by design, and right now there are no accepted standards for how AI should conduct security testing. This means there are currently no requirements for what model was used, what prompts drove it, or how results were checked.

For example, if you have two firms both saying they have AI-assisted testing on a web application, and they follow the OWASP Web Security Testing Guide as methodology. One uses a fine-tuned open-source model that is hosted out of AWS Bedrock and the other runs calls through a commercial API with no post training and only custom prompts. How would you know which approach is right for you?

You wouldn't! So, the next best thing is to do what you've always done, look at the firm to track record, experience, and people. The important social proof is the part that hasn't changed.

The Judgment Problem: AI Doesn’t Know When to Stop

A key component of pen testing where AI still struggles, but good testers handle naturally, is knowing when to stop.

Say that a model is running through a web app assessment and starts testing POST requests. It finds a form, checks for injection issues, and keeps going. A tester looks at that same form and recognizes it's part of the user deletion workflow, so they know to back off. The model might not make that connection, so it likely would keep testing, and now all the users in that tenant are gone.

In a recent Doyensec comparison of two AI testing platforms, one platform deleted the testing account mid-engagement, changed the application's authentication mechanism which locked testers out, and caused resource exhaustion that took the target server down twice. The test generated over 4,800 one-time token emails and created more than 4,000 database entries.

What should have been a straightforward assessment took over a week and more than 22 support emails to complete. One caveat worth knowing, the study was funded by Aikido, a direct competitor to the platform involved (XBOW). Doyensec published the full dataset for anyone to verify. I encourage you to check it out and draw your own conclusions.

While that test was focused on web applications, there is also the question on internal network pen testing. As of the date of this writing, I, personally, wouldn't let a model run unsupervised on an internal network. A human can read the situation and make a call. A model, at least today, just can’t be fully trusted.

There's also a problem with limited memory and context. Complex environments require holding a lot in mind at once, including how authentication flows between services and which systems are connected. Experienced testers carry that context naturally, whereas large language models have hard limits on how context much they can hold (and understand) at one time.

These limitations together suggest AI still isn’t ready to operate unsupervised in high-risk environments. Until AI can reliably read context the way a human tester does, it needs a human in the loop as a safeguard. In the meantime, AI is best treated as a powerful assistant for the mechanical parts of testing.

The Importance of Knowing Where Your Data Goes Before Hiring Anyone Using AI

One key consideration remains, where does your data go once it leaves your environment?

Your scope details, your findings, application responses from your systems, all of it could be used to train the next version of whatever model your pen test team is running. That's a valid concern, and you need to ask these questions before starting an engagement.

Before you engage anyone using AI in their testing process, you should get specific answers. Are they running an enterprise solution that blocks third-party model training on your data? What happens to that data after the engagement closes? Who has access to it then?

At Schellman, we monitor outbound network traffic on the AI-assisted workflow, paying close attention to log what goes in and the traffic that goes out. We follow least privilege principles before any client data touches a model, which is the baseline before we even start. Data is only ever sent to one of three places during a pen test, Anthropic’s API covered under Schellman’s Enterprise Agreement (prohibits Anthropic from training on our data), AWS Bedrock within our own VPC (virtual private cloud), or a self-hosted model running on our own hardware locally. Everything stays within our control and internal compliance requirements.

You should be aware if a testing provider can't clearly answer where your data goes, who can access it, and whether it's used for model training. AI-assisted testing should tighten your security posture, not create a new, unmonitored channel for your most sensitive data to leak out.

Where AI Actually Helps with Pen Testing

That said, I'm genuinely excited about what AI can do when it's working alongside a tester, which we’ve seen in action firsthand.

Models like Opus and Mythos are finding real vulnerabilities in open-source software. They read source code directly, spot security flaws, and in some cases develop working exploits from what they find. On actual client engagements that are restricted by time and scope, we've discovered things we would not have found manually in the time we had, and not because our testers aren't thorough, but because the model covered ground that we simply couldn't get to within the same engagement window.

The day-to-day support work adds up too, most notably, drafting a finding from raw notes is faster and cleaner and flagging a methodology gap mid-engagement is useful. Surfacing a connection between two systems that a tester noted but didn't fully dig into can turn a solid test into a great one. These efficiencies allow us to provide an even higher quality deliverable than before.

We’re not focused on removing human pen testers from the process. Our goal is to make the time a tester spends on an engagement more productive, more valuable to our clients, and more thorough by leveraging AI, not by outsourcing judgment to AI.

How Schellman Has Built AI Into Our Pen Testing Process

By actively bringing AI into our report writing QA process, we're building specialized agent harnesses for external network and web application assessments. We're evaluating agentic frameworks, building our own tooling, and keeping a close eye on new research from Anthropic's blog (and many others) as soon as they are published, staying on the bleeding edge of innovation within this space.

These initiatives have led to new workflows, better methods, and more effective ways to optimize results. What works today may look completely different in 90 days, but what hasn’t changed is the structure of our pen tests, involving a tester who drives, and the agent that assists. AI can help coverage consistency, flags gaps, and produce a cleaner deliverable. The tester makes the calls that require judgment surrounding what to chase, what to leave alone, and what a finding actually means for a specific environment. We’re able to spend more time on higher value activities for our clients as a result.

At the end of every engagement, an experienced human tester signs off on the findings, and that person is accountable for them. If you call with a question six months later, there will be someone on the other end who is informed and ready to help. For clients in regulated environments, that accountability is impactful.

Why An AI-Powered Pen Test Is Only as Useful as The Team Behind It

Most of our clients aren't running pen tests in isolation as they are typically part of broader compliance initiatives spanning various frameworks and certifications. They're often working toward a FedRAMP certification, maintaining PCI compliance, going through SOC 2 assessments, or pursuing ISO certifications. A pen test feeds directly into all these programs and is often a baseline requirement. When the team doing your test is already working with the people handling your audit, findings get contextualized faster, remediation guidance fits your actual compliance requirements, and nothing falls through the gap between two vendors who never talk to each other.

Working with a single firm across multiple services means you're not re-explaining your environment to a new team for every engagement. You have people who already know your history, understand your program, and can seamlessly connect dots across assessments. When something new comes up, like ISO 42001, the AI governance standard gaining global traction, there's already someone in your corner who knows your environment and can tell you what it means for your specific situation.

That's a different relationship and overall experience from hiring a vendor to run a test and never hearing from them again to hiring a team of subject matter experts across multiple disciplines, all working toward the same goal. When questions come up, when requirements change, when something in a new assessment connects back to work done two years ago, those conversations happen naturally because everyone is already in the room.

An automated tool can run a scan, but it can't do any of that.

What To Know About the Autonomous Options

XBOW or Aikido are worth evaluating if you want to see what fully automated testing looks like right now. Other firms are building in this space, and some of the tools are genuinely impressive.

But the claim that AI can run an entire engagement without human involvement is getting ahead of where the technology actually is. There are no compliance standards for AI-driven pen testing, as well as no requirements for model version, prompt design, or result validation. You're largely trusting the vendor's word on methodology and for some use cases, that may be fine. For the environments our clients ask us to test, most of them aren't willing to make that tradeoff.

The firms that figure this out over the next few years will be the ones using AI to make their testers better, not replace them. This results in better coverage, easier to reproduce findings, and someone accountable for the work. That's what we're building toward.