Not All Red Team Reports Are Created Equal: How to Spot a Penetration Test in Disguise

If you've received a report labeled "Red Team Assessment" and can’t help but notice it reads more like a penetration test report, you're not alone. We've seen this pattern repeatedly. Organizations invest in what they believe is a Red Team engagement, only to receive a penetration test with a different label. This deception can be more damaging than helpful as it is fundamental to your security posture that you understand the depth of assessment your organization actually received.  

In this article, we’ll cover the core misconception that leads to this confusion, red flags that signal your deliverable is just a pen test, and what a real red team assessment should include. You’ll come away with a better understanding of the difference between a penetration test and an authentic red team assessment so you can be more confident about your assessment deliverables. 

The Core Misconception 

Red teaming is often confused with penetration testing because of the simple but flawed logic that if you're not on the blue team, then you must be on the red team. Therefore, a penetration test assessment can be assumed to be a red team assessment, right? Wrong. 
 
While both disciplines are involved in the “offensive security” strategy, they differ dramatically in scope, methodology, objectives, and execution. Red Team assessments are more in-depth, adversarial simulations that shouldn't be watered down or confused with point-in-time penetration testing. Understanding this difference will come in handy when evaluating whether the report sitting on your desk meets the standard for a Red Team assessment or if you need to push back on your vendor. 

Red Flags That Signal Your Assessment Is Just a Pen Test 

1. Limited or Pre-Negotiated Scope

You should first consider if the assessment began with the testing team being given a specific IP range or application URL. A true Red Team assessment doesn't start with you handing over the keys. It begins with reconnaissance against your entire organization, not just a specific set of hosts or IP addresses. If your report lacks OSINT (Open Source Intelligence) gathering from the external perimeter and only focuses on the in-scope hosts provided instead, this is much more in alignment with a penetration test. 

2. Provided Credentials

The level of credentials provided is perhaps the most glaring indicator of the type of assessment you received. If the testing team was provided with credentials to the web application environment, they skipped the "breach" phase. Red Team assessments focus heavily on breaching the network from the Internet and are never focused on authenticated web app pen testing.  

There is the concept of an “assumed breach” phase, where the tester is positioned inside the network. This is an acceptable option during a Red Team assessment if you’re seeking to fast-forward the process and evaluate what would happen once someone does gain internal access.  

However, the overall stealthiness of the test decreases in these scenarios as someone will have to provide the tester with internal network access, running a provided payload, or onboarding them as an employee. This expands the circle of people who know this assessment is taking place and impacts the realistic nature of the test.  

3. Phishing That Bypassed Controls

You should also consider if phishing was included in the scope and if the email traversed your real security controls, or if it was simply allowed inbound past technical controls by IT. Legitimate spear phishing exercises should be targeted, advanced, and representative of real threat actor behavior.  

These campaigns should focus on gaining remote access or achieving account compromise and not just testing whether users click links. It’s most important to consider what happened after the phish was successful: did the testers stop and report, or did they pivot internally, escalate privileges, and pursue their objectives? If the answer is no, then this is not a red team engagement.  

4. Timeframe: One Week or Less

If your Red Team assessment was completed in a week or less, alarm bells should be ringing. Red Team engagements are measured, methodical operations that simulate advanced, persistent threats. They require time for reconnaissance, initial access attempts, establishing persistence, lateral movement, and working toward their objectives.  

A one-week engagement suggests point-in-time vulnerability testing took place, not adversarial simulation. Our red team assessments are typically 4 weeks on average, with that time increasing based on origination size and objectives. 

5. The Report Says "Penetration Test"

You should always read your report carefully. Does the executive summary reference "penetration testing methodologies?" Does it talk about "vulnerabilities discovered" rather than "objectives pursued" or "attack paths identified?" This language matters. Red Team reports should read like intelligence briefings documenting an adversarial campaign, not vulnerability assessment findings. 

Understanding Assessment Objectives vs. Boundaries 

For organizations pursuing FedRAMP authorization, noting that formal Red Team requirements are currently in draft guidance, it’s important to understand the relationship between assessment scope and reporting. 

A Red Team assessment targets your entire organization, with the goal being to breach a specific boundary, such as a FedRAMP authorization boundary. The testers should not limit their focus to just the systems within that boundary, and should instead assess your complete attack surface, just as a real adversary would. 

It’s important to note that the assessment is not a failure if it doesn't achieve access to the target boundary. Red Team assessments document the adversarial path, attempts made, defenses encountered, and the security posture observed, providing beneficial insights. It doesn't matter if the ultimate objective was achieved. 

The Importance of Assessment Reporting Nuances 

Here's where understanding the difference becomes essential for compliance and risk management: 

• The Red Team Report captures all findings from the entire assessment, including every technique attempted, every vulnerability discovered across your organization, all lessons learned, and insightful recommendations for improvement. 

• For FedRAMP, the Security Assessment Report (SAR) or Risk Exposure Table (RET) includes only findings that directly relate to the target environment, or that could lead to access to that boundary. 

This distinction is powerful. Organizations can take action on all findings to improve their overall security posture, whether those vulnerabilities exist in internal corporate IT policies, an internet-facing host, or physical security. However, findings that don't create a direct path to the specific authorization boundary remain actionable intelligence without impacting that boundary's compliance documentation. 

What a Real Red Team Assessment Includes 

A legitimate Red Team assessment must:

• Cast a Wide Net: Assessment should cover your entire organization's attack surface, domains, subsidiaries, partners, employees, physical locations, and anything else an adversary might leverage. 

• Include Deep OSINT: The engagement should begin with reconnaissance from the external perimeter, domain analysis, exposed services, employee enumeration, supply chain research, and social media profiling. 

• Demonstrate Realistic Breach Attempts: There should be a strong focus on breaching your network from the Internet using realistic attack vectors, without provided access or credentials. 

• Execute Advanced Spear Phishing: Targeted campaigns should be launched against high-value personnel, designed to achieve remote access or account compromise, rather than bulk phishing awareness tests.
  

• Conduct Advanced Internal Operations: After initial access, the team should demonstrate lateral movement, privilege escalation, persistence mechanisms, and pursuit of defined objectives against your most important assets. 

• Document the Journey: The report should tell the story of the adversarial campaign. Include what was attempted, what worked, what didn't work, why defenses succeeded or failed, and how to improve in the future. 

Don’t Settle for a Pen Test When You Hired a Red Team 

When reviewing your next report labeled as Red Team assessment findings, ask yourself: did this test simulate a real adversary working toward a specific objective, or did it identify vulnerabilities in a pre-scoped environment? Both have value, but only one is truly a Red Team assessment. 

Your organization deserves the assessment you paid for so you shouldn’t accept penetration test deliverables when you've contracted for Red Team services. Red Team assessments are more in-depth, more comprehensive, and provide more realistic security validation than traditional penetration testing.  

To learn more about the differences between penetration testing and Red Team assessments, contact us today. And for even more information on Red Team methodologies and evolving requirements, refer to these helpful resources:  

• What is Red Teaming? 
• Red Team Methodology: Understanding the Stages 
• FedRAMP Red Team Assessments FAQ