Penetration Testing a CI/CD Pipeline: How to Use a Holistic Approach

Penetration Testing a CI/CD Pipeline: How to Use a Holistic Approach

When a software production company requests a security assessment of its Continuous Integration (CI) and Continuous Delivery (CD) pipeline, they usually want an evaluation of the strength of its existing security measures and identification of potential security risks associated with the different components involved in storing, updating, building, and deploying their application.

As very experienced penetration testers, we get requests for this sort of thing a lot, and as such, we’ve built up a methodology that examines the entire range of elements within a CI/CD pipeline that enables a variety of broad attack paths that can be employed to compromise its integrity.

Our holistic methodology takes into account this heterogeneous landscape, and in this article, we’re going to detail how you too can take different perspectives during a penetration test of a CI/CD pipeline to ensure a more comprehensive security assessment.

What are the Components of a CI/CD Pipeline?

But before we get into the process, let’s ensure we understand the aforementioned “range of elements” that make up a CI/CD pipeline, as this understanding is a first step in devising an offensive plan.

Continuous Integration and Continuous Delivery are broad terms used to describe systems that automate the process of software development, build, and delivery to customers. While implementations vary, a CI/CD pipeline usually features the following:

Source Code Repository

Details: Because this is a crucial target and a significant attack surface available to the penetration tester, the repository's hosting environment—whether on-premise, cloud-based, or Software-as-a-Service (SaaS)—can significantly influence the attack techniques employed during the test.
(You can also assess the version control tool itself for known or unknown vulnerabilities.)
Examples: GitHub, Gitlab, Bitbucket, etc.

CI Platforms

Details: As another critical component of the pipeline, the CI platform, build server, and its underlying build software, are together responsible for managing and executing the build stage of the final product, but the dependencies used to do so introduce another layer that could be compromised through supply-chain attacks.
(The attack vectors for this category will vary depending on the type of hosting and technology stack in use.)
Examples: Travis CI, CircleCI, Jenkins, and other cloud-based CI solutions.

Container Repositories and Registries

Details: While these play a more static role in the delivery phase of the pipeline, they’re nonetheless a major component as they’re responsible for storing and managing the container images that make up the final product.
(This stage of the pipeline may be vulnerable to tampering if safeguards are not in place.)
Examples: Docker Hub, AWS Elastic Container Registry (ECR).

Infrastructure as a Service (IaaS) Provider

Details: Responsible for the underlying infrastructure on which the pipeline runs, these are leveraged by the pipeline's various other components and include the following within the IaaS provider:

Virtual machines
Storage
Networking

Examples: Amazon Web Services, Google Cloud Platform, or a self-hosted solution like OpenStack.

Questions to Answer During a Penetration Test of a CI/CD Pipeline

As we said, this diverse range of components used in the CI/CD pipeline, as well as their interactions with one another, creates a heterogeneous landscape enabling many possible attacks.

That’s why, when you’re engaged to assess these elements within an organization’s CI/CD pipeline implementation, you may be tasked with concretely answering the following security-related questions involving the aforementioned components and regarding the following elements:

Intellectual Property: Are measures in place to safeguard source code from unauthorized access or unintentional code leaks?
Sensitive Environment Variables and Secrets (e.g., API Keys, Private Keys, Account Credentials): Are they securely stored in source code repository configurations, Jenkins instances, cloud storage, and vaults?
Unit Test Suites: Can the development credentials, or any other developmental or QA stage processes they’re using be leveraged to execute unexpected attacks?
Version-Controlled Code and Repository Branches: Can these be tampered with, leading to the propagated compromise of the corresponding application builds?
Upstream Packages, Dependencies, and Other Trusted Sources: Can the pipeline be affected by a software supply chain attack initiated from these angles?
Roles and Permissions for Users with Varying Levels of Access to Code Infrastructure: Have the CI platform, container repository, and cloud infrastructure, been designed with adequate controls to avoid any gaps or authorization vulnerabilities?

CI/CD Pipeline Penetration Test Methodology

To successfully answer and address these, the methodology we use at Schellman in conducting a CI/CD pipeline penetration test involves elements and areas of expertise inherited from other attack vectors, such as our assessments of:

To best conduct a comprehensive penetration test of a CI/CD pipeline, multiple levels of access should ideally be granted within the environment. Such levels could be designed with the following scenarios:

Perspective	Goals
As an attacker outside the company	Apply external network penetration testing techniques with goals such as: Determining if any of the CI/CD pipeline components are public-facing and bypassing authentication required for any of them Gaining unauthorized access to the internal environment hosting the CI/CD pipeline Identifying vulnerable elements within either the CI/CD pipeline components or the supply chain and software ecosystem to gain a foothold into the infrastructure
As a malicious internal employee or contractor without access to the pipeline	Apply internal network penetration testing techniques apply with goals such as: Gaining unauthorized access to any of the components of the CI/CD pipeline using only general in-house credentials Stealing sensitive information or source code from the pipeline by abusing internal company access privileges Disrupting the pipeline's availability by intentionally causing errors or delays in the build and delivery process
As an attacker from within the development team	Apply web application penetration testing techniques with goals such as: Escalating privileges from those of a developer with read-only permissions on the main production branch to those of a repository owner Tricking the system or other employees into approving and merging pull requests to the main production branches and injecting malicious code into the application Tampering with critical repository settings and configurations through authorization bypass or logic flaws Identifying known vulnerabilities or zero-day bugs within the user interfaces and systems available to the developer and attempting to compromise the pipeline through privilege escalation or propagated attacks
As an attacker from within the DevOps team	Apply cloud security penetration testing and security review techniques with goals such as: Bypassing elevated security controls (e.g., multi-factor authentication (MFA)) Forging, disabling, or reducing the effectiveness of security configurations, controls, and monitoring processes, including audit logs and alerting systems designed to detect such attacks Tampering with formal change management processes in place, or with business continuity and disaster recovery processes

Interested in Learning More?

CI/CD pipelines are often critical to the success of software developers, and keeping every element secure is of paramount importance. A penetration test of a pipeline is a good start, but to ensure every gap is found, the tester(s) must take a holistic approach to the evaluation—like the one we’ve just laid out.

By examining the elements from multiple levels of privileges, a penetration test can identify weaknesses in a company's ability to protect the CI/CD pipeline from both outsider and insider threats, as well as provide remediation suggestions to better protect this critical business asset.

For more insight into helpful penetration testing techniques, check out our other articles detailing several different methods to different concepts that our team has developed and put to use:

About Loic Duros

Loic Duros is a Manager with Schellman. Prior to joining Schellman in 2020, Loic worked as a Senior Security Engineer specialized in application security. As a Senior Associate with Schellman, Loic Duros is focused on Penetration Testing.