Software Supply Chain Security in DevSecOps & CI/CD

A look at NIST's 800-204 "Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines"

Software supply chain (SSC) attacks continue to be one of the most discussed topics in the cybersecurity industry as of late, and for good reason, with some sources showing these attacks rising over 742% in the latest three years.

Source: Sonatype 8th Annual State of Software Supply Chain

With the continued growth of these attacks, organizations continue to look for ways to mitigate the risks of SSC attacks, and various industry organizations continue to publish guidance to help organizations. We’ve seen publications from organizations such as CNCF, CIS, NSA and of course NIST, to name a few.

The latest release comes from the National Institute of Standards and Technology (NIST), in their special publication (SP) 800-204, also known as “Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines”. 

In this article we will take a look at this document, some of the key recommendations and why they are relevant to the modern organizations developing and delivering software. 

Context

The modern organization developing and delivering software is increasingly doing so leveraging cloud-native environments and technologies, such as CI/CD pipelines and utilizing DevSecOps methodologies. While a deep dive of DevSecOps is beyond the scope of this article, I recommend resources such as The DevOps Handbook, Agile Application Security and the Application Security Program Handbook.

This combination of technologies and practices allow organizations to streamline the software development process, automate tasks and integrate security tooling and practices through the software development lifecycle (SDLC) to mitigate risk. That said, when these environments and technologies aren’t properly secured, exploitation can occur, having a downstream impact on those consuming the software. 

This guidance strives to arm organizations with recommendations to mitigate those risks and bolster software supply chain security. Protecting software in the CI/CD context means ensuring that the software is not compromised through the various stages of build, test, package and deploy. 

A good visualization of this flow is available from the Supply Chain Level for Software Artifacts (aka SLSA) framework.

As NIST points out, the SSC includes not just the output of specific activities but also assurances that the various activities that were carried out to create and deliver software were not compromised and have supporting artifacts and attestations to prove it.

This is why we’re seeing efforts such as “in-toto”, which aims to make it transparent to users what steps were performed throughout the software supply chain activities, by whom and in what order. in-toto helps provide verifiable claims about any aspect of how a piece of software is produced and we’ve seen innovative software supply chain security companies rally around this effort, as well as it garner support from industry leading organizations such as CNCF. Attestations have also gotten increased attention due to emerging Federal software supply chain requirements, such as OMB 22-16 and 23-18, which deal with software suppliers selling to the Federal government, and we’ve covered these emerging requirements in articles here and here.

We have, and will continue to see SSC attacks rise because of the economies of scale it offers malicious actors. Rather than target a single organization for exploitation, attackers can target a widely used vendor, service or OSS component/project and have a massive downstream impact across a large range of victims. 

Part of the challenge on the defensive side of the SSC is that the modern SSC involves many different stakeholders, organizations, industries, technologies and communities, making it incredibly difficult to have a single unified approach to governance and security. 

NIST defines a SSC attack as a three phased scenario:

Artifact, Step or Actor Compromise > Propagation > Exploitation  

Risk Factors of the SSC

When dealing with modern software development environments, there are various entities and risk factors at play. These include the development environment, threat actors, attack vectors and targets and the types of exploitation that are being carried out. These factors may look different across organizations and industries, but ultimately play a part in any SSC compromise scenario. 

Developer environments represent a primary attack vector for malicious actors looking to carry out SSC attacks. This is particularly challenging due to the nature of the modern distributed workforce, bring your own device (BYoD) scenarios and the combination of SaaS and self-hosted source code management systems. Organizations must have measures in place to mitigate risk from the compromise of a development environment, developer account, and their associated endpoints. 

Threat actors include a combination of external and internal attackers. The external attackers may vary depending on the reason for the malicious activity from everything from a cyber activity to a full on nation state looking to impact foreign adversaries and their national institutions and critical infrastructure.

It could also include malicious insiders driven by factors such as dissatisfaction or extortion and bribery. These attacks often utilize a variety of attack techniques such to compromise software the supply chain, such as phishing, malware and social engineering. 

The attack vectors threat actors can take ranges as well depending on the target environment, organization and scenario. It can include malware, social engineering, or network and physical-based attacks. Each of these attack vectors warrants a corresponding appropriate mitigation control/technique, making it difficult for large complex organizations in particular to mitigate every attack vector at all times. 

Attack targets include source code, credentials and sensitive data. Types exploits vary from activities such as the injection of vulnerabilities and malware, stolen credentials and injecting malicious code into repositories. NIST lists a robust set of mitigation measures from fundamental activities such as patch management and access control to auditing and monitoring as well as aligning with applicable regulatory frameworks. 

For an excellent resource on the various types of software supply chain attacks, I cover the CNCF Catalogue of Software Supply Chain Compromises here, which breaks down the various types of attacks we’ve seen into categories.

CI/CD Pipelines - Trust, Goals and Secure Outcomes

As we discussed above, modern DevSecOps environments are using CI/CD platforms and technologies to facilitate iterative software development and delivery. As software passes through the various phases of build, package, test and deployment, these phases produce metadata, which can facilitate assurance around the provenance of the software, and the steps and measures that went into producing it.

A compromise of any of the steps, as well as the underlying CI/CD environments and platforms can have a downstream impact on the integrity of the software artifacts that are produced and distributed. 

Organizations must take security measures for both internally developed (first party) code, as well as third-party components, such as open source software, which are increasingly making up the bulk of modern software, at least from a source code perspective, as documented in reports, such as Synopsys’ 2023 Open Source Security and Risk Analysis Report.

These OSS components have a various of risks as documented in the Synopsys’ report, such as licensing conflicts, out of date components and unmaintained components as well.

The above risks are just a subset of those facing OSS, as I covered in the article discussing Endor Labs “Top 10 OSS Risks Report”, risks include not just known vulnerabilities, but also other risks such as the compromise of legitimate packages, name confusion attacks, untracked dependencies and more.

That said, as I discussed in the article titled “False Dichotomies and Overemphasizing on OSS”, there are nuances to OSS risks when it comes to AppSec as well, such as reports and findings from organizations such as Contrast Security, who found that 78% of active code in your application is proprietary/first-party code, which saw a 21% YoY increase in serious vulnerabilities.

This is why capabilities such as reachability analysis to help highlight the code, components, libraries and functions that are actually invoked and reachable by malicious actors is key to helping aid in vulnerability prioritization and can eliminate up to 80% of false positives that drain development and engineering hours chasing.

Ultimately organizations are looking to ensure that attackers cannot tamper with the software production process, introduce malicious software updates or compromise the integrity of CI/CD pipeline artifacts and activities. NIST provides the below table demonstrating the artifacts that need to be trusted in typical CI/CD environments, as well as the repository the artifacts generally reside in/depend on:

SSC Security in CI/CD Pipelines

Now that we’ve discussed some of the background, security goals and entities involved in trusted CI/CD pipelines, let’s take a look at some of the specific SSC security activities that NIST emphasizes in their guidance. 

It should come as no surprise that NIST evangelizes Zero Trust principles here as well, given their publication of 800-207 “Zero Trust Architecture''. The recommendations cited include defining roles for system operators, mapped to specific permissions and implementing least-privileged access aligned with the concept of role-based access control (RBAC). Activities like these mitigate the risk should a specific actor's account or assets get compromised. 

NIST also recommends automating the use of SAST and DAST, as well as declaratively defining the development and deployment of application code and CI/CD activities through techniques such as Infrastructure-as-Code (IaC) and Policy/Configuration-as-Code, which can specify runtime settings for security and compliance purposes. 

This is increasingly gaining momentum through movements such as “GitOps”, which leverages practices from software development such as version control, collaboration, compliance and the use of CI/CD pipelines and applies them to infrastructure automation thanks to the emergence of IaC languages such as CloudFormation from CSP’s such as AWS, as well as CSP-agnostic languages such as Terraform, and more recently OSS options such as “OpenTofu”, the open source IaC tool, managed by the Linux Foundation.

The workflows of CI/CD pipelines must also be secure, including build, push/pull of artifacts from repositories and software updates or code commits. 

On the build front recommendations include key activities such as specifying build policies and the use of isolated build platforms as well as permissions for those performing build activities.

Organizations should also make use of policy enforcement engines and ensure that during the software build process evidence and attestations of secure build processes is produced. These may include attestations for the environment, process, materials and artifacts involved. NIST recommends the use of hashing to include the final build artifact, files, libraries as well as the events that produce the final artifacts. 

There is then a recommendation to sign the attestation and securely store it where it can be used to demonstrate policy compliance. Doing so can help demonstrate that software was built by authorized entities, tools and with alignment to defined policies and compliance requirements.

In addition to the need for secure build activities NIST also recommends securing pull-push operations on SCM repositories. This includes the pull of code from repositories by developers, its modification and then the push of code back to the repository, each of which presents an opportunity for tampering.

Recommendations include automated security checks on artifacts, ensuring confidence in the source code origin, and requiring explicit approval for all outside collaborators looking to push and pull from a repository. The below image from Francois Proulx demonstrates how a malicious actor can take various actions to gain unauthorized access to a GitHub repository and submit malicious code to a repository.

Other key recommendations NIST makes is maintaining the integrity of evidence generation during software updates, securing code commits and securing workflows in CD pipelines. Attackers may look to erase or tamper with software update trails to mitigate investigation and detective controls. In addition, to ensure code commits don’t introduce malicious code or vulnerable code, NIST recommends the use of SAST/DAST tooling in CI/CD pipelines with broad language coverage, and the use of SCA tooling to verify the security of OSS components and dependencies. 

Since CD pipelines revolve around workflows and many modern environments are making use of technologies such as containerization, NIST recommends ensuring that containers being deployed were actually generated by the defined build process and that they have been scanned for vulnerabilities in alignment with an organization's vulnerability management requirements. 

Lastly, given the myriad of high-profile secret exposures the industry has seen lately, NIST recommends organizations scan for the presence of secrets in code, such as keys or access tokens, which can be abused by malicious actors for nefarious purposes.

Conclusion

If you’re walking away from this article and coverage of the NIST publication thinking that seems complex, it’s because it is, and you’re right. The modern software supply chain, including its technologies, processes and attack surface is complex.

It’s often quipped that “defenders have to be right all of the time, attackers only need to be right once”. The unfortunate reality for us in this case with software supply chain attacks is that when they are right, especially for a heavily used vendor, product or OSS component, they are able to have an outsized impact due to the economies of scale that software supply chain attacks provide.

That said, all hope isn’t lost, and we have and will continue to see reputable organizations publish software supply chain security guidance as well as innovative startups raise capital and seek to help tackle this problem as well as the rich OSS community continue to come forward with promising projects and efforts that organizations can use to mitigate software supply chain threats.

Like anything in cybersecurity, the path to software supply chain security is a journey and there’s no better time to start than now, so get to it.