Defending CI/CD Environments - The NSA/CISA Way

A look at the recently published Cybersecurity Information Sheet from the NSA/CISA on Defending CI/CD Environments

If you’re working in any modern software development environment, you’ve inevitably run into the terms Continuous Integration and Continuous Delivery/Deployment (CI/CD). This often includes the use of a CI/CD pipeline to facilitate the automated testing, delivery and deployment of software, and while not always, often utilizes cloud-native environments and tooling as well.

CI/CD pipeline and environments have undoubtedly revolutionized the way we produce and deliver modern software. It’s enabled methodologies such as DevOps and DevSecOps and helps with the push to “shift security left”.

That said, despite its tremendous value, CI/CD environments also have their own risks and represent part of your attack surface. When not properly secured they can lead to compromises and the ability of malicious actors to poison downstream environments and software consumers and are a key aspect of securing the modern software supply chain.

The National Security Agency (NSA) and Cybersecurity Infrastructure & Security Agency (CISA) recently released a publication titled “Defending Continuous Integration/Continuous Delivery (CI/CD) Environments”, which we will take a look at in this article.

At a high level, the guidance explains how organizations can implement security best practices into typical DevSecOps and CI/CD environments, and it does so in a tool and vendor agnostic fashion. The guidance also emphasizes that CI/CD environments are increasingly the target malicious actors looking to compromise organizations, and often, their downstream software consumers.

One obvious example nearly everyone is aware of is SolarWinds, which impacted many organizations, including U.S. Federal agencies.

It was in fact the SolarWinds incident that helped contribute to and shape the Cybersecurity Executive Order (EO) 14028, particularly Section 4, which focused heavily on securing the software supply chain and has led to various efforts such as a revision of NIST Secure Software Development Framework, OMB Memos 22-18 and 23-16 and CISA’s Secure Software Attestation Form - I’ve covered all of these items extensively and I’ve hyperlinked their text to take you to those respective articles.

The NSA/CISA guidance opens with a high-level representation of “How Malicious Cyber Actors Threaten the CI/CD Pipeline”

You’ll notice the threats exist from the Developers writing/integrating the code, all the way through Deployments to production environments. If you’ve been following this topic and space, you’ll also cite obvious parallels with the diagram to other sources, such as the Supply Chain Levels for Software Artifacts (SLSA), CNCF Software Supply Chain Best Practices, or CIS Software Supply Chain Security Guide.

That is one of the great things about the guidance in my opinion, is that it doesn’t entirely attempt to reinvent the wheel, and leverages a variety of existing sources of guidance. As I often say, we are best practices rich and implementation poor - this means we need more of a focus on implementing existing best-practices rather than publishing new ones.

To see some of the diagram parallels, take a look at this SLSA diagram below, contrasted with the NSA/CISA diagram.

NSA/CISA published the guidance to provide best practices and hardening guidance for CI/CD pipelines and environments and it utilizes the MITRE ATT&CK framework to help enumerate the most significant vulnerabilities against known threats. It also maps the tactics, techniques and procedures (TTP)’s to the ATT&CK and D3FEND frameworks to bolster the guidance and environments from malicious actors attacks.

Terminology

The guidance opens with key terms and I am going to copy those definitions verbatim here to 1) ensure a common lexicon and 2) familiarize any reads whom are unfamiliar with the terms

• Continuous delivery (CD) is the stage after continuous integration where code changes are deployed to a testing and/or staging environment after the build stage. Continuous deployment is similar to continuous delivery except that the releases happen automatically, and changes to code are available to customers immediately after they are made. The automatic release process may in many instances include A/B testing to facilitate slow rollout of new features, thereby mitigating the impact of failure resulting from a bug or error.

• Continuous integration (CI) involves developers frequently merging code changes into a central repository where automated builds and tests run. Build is the process of converting the source code to executable code for the platform on which it is intended to run. In the CI/CD pipeline software, the developer’s changes are validated by creating a build and running automated tests against the build. This process avoids the integration challenges that can happen when waiting for release day to merge changes into the release branch.

• CI/CD pipeline is a component of a broader toolchain that entails continuous integration, version control, automated testing, delivery, and deployment. It automates the integration and delivery of applications and enables organizations to deploy applications quickly and efficiently.

• Development Operations (DevOps) is a set of practices that combines software development and information technology (IT) operations. It aims to shorten the systems development lifecycle and provide continuous delivery with high software quality.

• DevSecOps (DevSecOps) is an approach that integrates development (Dev), security (Sec), and delivery/operations (Ops) of software systems to reduce the time from a recognized need to capability availability and provide continuous integration and continuous delivery (CI/CD) with high software quality.

• A software supply chain is composed of the components, libraries, tools, and processes used to develop, build, and publish a software artifact. Software vendors often create products by assembling open source and commercial software components. Software supply chains are made up of software components, such as open source packages and infrastructure as code (IaC) templates, as well as underlying delivery pipelines, such as version control systems and CI/CD pipelines.

• Software Composition Analysis (SCA) is an automated process that identifies the software in a code base. This analysis is performed to evaluate security, license compliance, and code quality.

CI/CD Security Threats

The guidance goes on to cover relevant CI/CD Security Threats. To do so it uses examples leveraging the OWASP Top 10 CI/CD Security Risks. This is a project that was originally created by Cider Security and then donated to OWASP. I interviewed one of the Cider Security founders Daniel Krivelevich on the Resilient Cyber Podcast, and you can find that episode here, where we discuss CI/CD security.

The threats captured include:

• Insecure first-party code

• Insecure third-party code

• Poisoned pipeline execution

• Insufficient pipeline access controls

• Insecure system configuration

• Usage of insecure third-party services

• Exposure of secrets

Attack Surface

Moving on from threats, the guidance begins discussing the CI/CD attack surface. In my opinion this section draws a lot of parallels from the CIS and CNCF guidance I cited above.

The guidance points out that an insecure CI/CD pipeline leads to an insecure application and cites some of the following as part of the attack surface:

• Insecure code

• Poisoned pipeline execution

• Insufficient pipeline access controls

• Insecure system configuration

• Usage of third-party services

• Exposure of secrets

The guidance lays out three different threat scenarios and corresponding mitigations. These scenarios include examples such as compromised developer credentials, compromised application libraries, tools, or application images and malicious modifications of CI/CD or IaC configurations.

For further details on the scenarios and mitigations I recommend reading the guidance directly, as well as the other resources I’ve cited. However to quickly demonstrate examples, I will share a diagram they show of the different attack vectors in a hypothetical AWS CI/CD pipeline below:

Active Hardening

It should come as no surprise that in addition to the specific mitigations the guidance mentions, they also recommend organizations transitioning to Zero Trust, and utilizing CISA’s Zero Trust Maturity Model (which recently had a v2 released) and the NSA”s Advancing Zero Trust Maturity Throughout the User Pillar guidance.

The guidance covers specific mitigations for areas, which we will list below.

Authentication and Access Mitigations

• Use NSA-recommended cryptography

• Minimize the use of long-term credentials

• Add signature to CI/CD configuration and verify it

• Utilize two-person rules (2PR) for all code updates

• Implement least-privilege policies for CI/CD access

• Secure user accounts

• Secure secrets

• Implement network segmentation and traffic filtering

Developer Environment Mitigations

• Maintain up-to-date software and operating systems

• Keep CI/CD tools up-to-date

• Remove unnecessary applications

• Implement endpoint detection and response (EDR) tools

Development Process Mitigations

• Implement security scanning as part of the CI/CD pipeline

• Restrict untrusted libraries and tools

• Analyze committed code

• Remove any temporary resources

• Keep audit logs

• Implement Software Bill of Materials (SBOM) and Software Composition Analysis (SCA)

• Plan, build, and test for resiliency

Conclusion

This NSA/CISA CI/CD guidance provides yet another excellent resource for the community on not just bolstering CI/CD security, but more broadly, software supply chain security.

Malicious actors can wreak havoc on organizations directly as well as their partners, customers and clients by compromising a CI/CD environment and its associated artifacts, and we’ve already seen several of these attacks occur in the wild.

The NSA/CISA guide links to a robust set of additional guidance as well citing existing work from organizations such as OWASP, NIST, the DoD and more - so if you really want to dig in, I definitely recommend checking out the additional guidance and works cited sections of this NSA/CISA guide.

As evident from the mitigations we cited above from the guidance, this problem space is very complex, with a variety of moving parts, pieces and processes and a high probability for missteps and vulnerabilities, all of which malicious actors are anxiously waiting to exploit.

The key here is to assess your CI/CD and DevSecOps implementations, architectures and processes, determine where you have gaps, document them and then iteratively and methodically remedy them - to mitigate risk to your organization as well as anyone consuming software and artifacts from your pipelines.