Managing Open Source and SBOM's
A look at the NSA's "Securing the Software Supply Chain: Recommended Practices for Managing OSS and SBOM's"
Software supply chain security continues to be a critical topic of the cybersecurity and software industry as a whole.
From continued attacks against large software vendors to malicious focus on the open source software ecosystem from attackers, it's a topic that is front and center for most CISO’s and security practitioners.
It’s also led to a thriving ecosystem of software supply chain startups, each focusing on unique aspects of the modern software supply chain attack surface.
Luckily, organizations continue to produce solid guidance to help practitioners mitigate software supply chain risks. The latest publication comes from the NSA, with a particular focus on open source software and SBOM’s.
It also builds on previous publications such as the White House Cybersecurity Executive Order (EO) and memos and forthcoming requirements for Federal agencies, such as the OMB’s memos 22-18 and 23-16, which require software suppliers selling to the federal government to self-attest to aligning with publications such as NIST’s Secure Software Development Framework (SSDF) and even provide artifacts such as SBOM’s in some cases.
I’ve written extensively about these previously OMB memos, such as 22-18 and SSDF which can be found here, as well as 23-16, which can be found here.
While the NSA guidance points to previous publications from The White House, NIST and OMB, this publication is relevant to all organizations producing and consuming software, leveraging OSS and looking to embrace artifacts such as SBOM’s.
So let’s dive in and take a look at some of the key guidance, recommendations and takeaways from the NSA’s “Securing the Software Supply Chain: Recommended Practices for Managing OSS and SBOM’s”.
Structure
The NSA guidance focuses on four keys areas, which are captured in the table below, and aligned with their respect SSDF Activities:
We will take a look at each of these sections, as well as some key recommendations and takeaways.
Open-Source Software Management
This section of the guidance defines key roles and responsibilities, such as Developers and Suppliers. It notes that Developers have responsibilities such as identifying potential OSS solutions to use, and integrating OSS solutions into product software, as well as tracking updates to those components. Suppliers are those producing a product or service and performing activities such as monitoring for license changes or vulnerabilities of OSS components included in products, due to the risk it could pass on to downstream consumers.
One distinction I wanted to make is that OSS maintainers/contributors are not suppliers in the traditional sense. OSS is most often provided “as-is” with no assurances on its quality, upkeep, security and so on. If you’re widely using OSS and assuming the maintainers owes you anything, you’re in for a bumpy ride. I discuss this in my article “Supplier Misnomer: Understanding who are, and aren’t your software suppliers”
The NSA lays out primary considerations for using OSS, such as evaluating OSS components for vulnerabilities in sources such as the NVD and other vulnerability databases and ensuring that vulnerable components aren’t being included in products. It also recommends organizations stay aware of licensing considerations such as license compliance, as well as export controls, such as the evolving EU regulations which may impact the incorporation of OSS into products.
The guidance recommends embracing SBOM’s as both a means of inventory management of OSS components, as well as transparency for downstream consumers, and visibility into the vulnerability posture of the components being included via development into products.
SBOM’s of course are a nested inventory of software components and the predominant formats are The Linux Foundation’s SPDX and OWASP’s CycloneDX and the NSA recommended generated SBOM’s meet the minimum element requirements documented from the NTIA’s “Minimum Elements for a SBOM”. In addition to NTIA’s guidance, there is also the OWASP’s Bill of Material Maturity Model, which I cover in an article titled “Understanding OWASP’s Bill of Material Maturity Model”.
I have previously written for CSO Online in an article titled “SBOM Formats SPDX and CycloneDX Compared” where I examine both, for those interested in learning more.
In addition, the NSA emphasizes understanding license compliance, such as having the legal rights to use OSS you select, ensuring it doesn’t encumber your proprietary code with code sharing obligations ads well as ensuring you agree and comply with associated license policies.
Creating and Maintaining a Company Internal Secure Open-Source Repository
Source: Synopsys Open Source Security Risk Analysis Report
Given how widely organizations are consuming and using OSS, with some studies citing figures from 70-90% of modern codebases being OSS and 90%~ of codebases containing some OSS components, another standout recommendation from the NSA guidance is creating and maintaining a company internal secure OSS repository. This repository helps vet OSS components to ensure they meet organizational risk and compliance requirements before being made available to product and development teams.
The NSA recommends using tools such as Software Composition Analysis (SCA) tools to identify vulnerabilities as well as licensing concerns before making the OSS components available to developers via the secure repository as demonstrated in their image below:
This of course means the organization needs to define processes to add packages for consumption and the security analysis and documentation needed to allow the addition of packages to the secured repository. It also notes that there may be different policies for environments such as development compared to production releases and recommends aligning with industry frameworks such as the Secure Supply Chain Consumption (S2C2F) which focuses on empowering developers with secure OSS consumption and use.
Organizations will undoubtedly wrestle with this recommendation, as it can be difficult to implement governance at this scale across many disparate development teams, each pulling in components and dependencies and likely resistant to a centralized governance scheme that may impede their development velocity. It can also be difficult to implement a centralized process such as this, as anyone who has tried to maintain “golden images” of virtual machines and AMI’s in a large enterprise can attest, now imagine doing this with hundreds/thousands of OSS components and dependencies.
The OSS adoption process should include activities such as Software Composition Analysis (SCA), virus scanning and fuzz testing, occurring in isolated secure environments. There’s also recommendations for developers to take other factors into consideration such as licensing, vulnerability history and the benefits of adopting the components in terms of time and decreased internal development costs.
The NSA also recommends using tools such as the OpenSSF Scorecard to analyze the component or The OpenSSF Scorecard of course looks beyond lagging indicators of risk such as known vulnerabilities and looks at aspects such as project code reviews, contributor density, maintenance upkeep and more. If you’re looking to learn more about OpenSSF Scorecards, I wrote an article titled “How OpenSSF Scorecards Can Help to Evaluate Open-Source Software Risks”.
It’s noted that small organizations may have a single secure repository and have Developers implementing the governance, where larger organizations may utilize Open-Source Review Boards (OSRB)’s to review adoption requests and involve multi-faceted stakeholders such as Development, Management, and Security. Large organizations are increasingly establishing Open Source Program Offices (OSPO), as they realize how critical OSS is to their development and product efforts. I published an article titled “The OSPO - the front line for secure open-source supply chain governance” which dives into the role of the OSPO and how it can help with OSS supply chain security.
Recommendations include monitoring repositories of authorized components for new vulnerabilities and maintaining awareness of what developer groups and product teams have adopted a component so that they can be part of any necessary incident response activities if a component is subsequently vulnerable or compromised.
In an effort to provide context and prioritization to downstream product and software consumers, the guidance recommends suppliers and developers adopt Vulnerability Exploitability eXchange (VEX) documents to help consumers and customers know which components are actually impacted by a vulnerability, which have been resolved and what should potentially be addressed via compensating controls.
The reality is that a presence of a vulnerability doesn’t automatically mean exploitable. In fact, less than 5% of CVE’s annually are ever exploited, and additional factors such as reachability and compensating controls can also influence exploitability. VEX helps provide clarity on the vulnerability of a product and its components. For those looking to learn more, I published an article titled “Vulnerability Exploitability eXchange Explained: How VEX Makes SBOM’s Actionable”
In the article I discuss the role of VEX as a companion to SBOM’s or as a standalone document, as well as the four primary status options of a VEX document, which are: Not Affected, Affected, Fixed and Under Investigation, which provide insight to customers and consumers on the status of vulnerabilities for components in products.
The NSA also recommends suppliers and vendors adopt attestation processes to demonstrate the secure development of a product throughout the building, scanning and packaging of product development and distribution.
This of course is being led by industry efforts such as in-toto and SSDF and self-attestations when machine readable artifacts are not generated and used. This helps provide assurance of not just the components of an end product but the secure development process as well.
in-toto provides a specification for generating verifiable claims about any spect of how a piece of software was produced.
Additionally, CycloneDX recently announced that CycloneDX v1.6 is introducing support for attestations of compliance with any standard (e.g. SSDF, OMB Memos et. al). This effort is being led by folks such as Steve Springett, who helps lead CycloneDX as well as Jeff Williams, longtime AppSec leader and Co-Founder of Contrast Security. Jeff shared the below image on LinkedIn this week, demonstrating a workflow of attesting to compliance frameworks via CycloneDX.
To address vulnerabilities the NSA recommends using not just CVE and NVD but also other vulnerability databases such as OSV as well as vulnerability intelligence sources such as the CISA Known Exploited Vulnerability (KEV) catalog and Exploit Prediction Scoring System (EPSS). This provides insight into not just vulnerability base scoring and severity but known or likely exploitation as well.
Looking at these factors is incredibly important as the scale of CVE’s discovered and reported only continues to grow with over 200,000 a year. That said, studies from sources such as Cyentia Institute demonstrate that organizations only have the ability to remediate between 5%-20% of their vulnerabilities per month, leading to infinitely growing vulnerability backlogs.
Further complicating the situation is the fact that less than 10% of all known vulnerabilities are ever exploited in the wild. Sources such as EPSS focus on the probability of a vulnerability being exploited in the next 30 days, and the image below demonstrates the immense effectiveness of focusing on known exploitation and exploitability per EPSS rather than the widely used approach of base CVSS scores and “fixing all highs and criticals”, which Cyentia and leaders like Wade Forbes have contested is as effective as just randomly picking and fixing vulnerabilities.
To learn more about EPSS and why it is poised to replace legacy vulnerability management practices like solely using CVSS scores, I published the article “A Look at the Exploit Prediction Scoring System (EPSS) 3.0”.
Thanks for reading Resilient Cyber! Subscribe for free and join nearly 4,000 readers to receive new posts and support my work.
Open-Source Software Maintenance, Support and Crisis Management
Another key activity emphasized is secure code signing requirements, such as performing code signing, using proven cryptography and securing the code signing infrastructure itself. NIST has an excellent document titled “Security Considerations for Code Signing”.
The NSA also recommends having a crisis management plan in place, building on guidance such as NIST’s Incident Handling Guide. It includes activities such as defining a crisis, structuring how your organization will respond and who will be involved. This is critical to have in place in the event of active exploitation in the wild of an OSS component. This also involves creating and maturing a Crisis Management Team (CMT) along with roles and responsibilities.
This team will investigate potential incidents tied to active exploitation, determine if a system or application is affected, if the component is within the execution path and determine if a vulnerability can be patched or if compensating controls are necessary.
SBOM Creation, Validation and Artifacts
This section of the guidance focuses on tools, processes and consideration for creating and utilizing SBOM’s. As mentioned previously, SBOM’s initially got traction with the NTIA, the Cybersecurity EO and then CISA. The two dominant formats are SPDX and CycloneDX.
To effectively use SBOM’s suppliers need to understand how products are built and what components they contain. The NSA guidance breaks SBOM tools into the four categories of source, binary, package and runtime extractors, given SBOM’s can be created at various phases of the SDLC.
The NSA recommends prior to signing an SBOM, that the organization verifies the SBOM contents for accuracy. It also again points out that SBOM’s can be accompanied by documents such as VEX to provide clarity around vulnerable components and what activities have been taken by suppliers to address vulnerabilities and what components potentially remain vulnerable or exploitable.
Since SBOM is an emerging and evolving topic, the NSA points to various resources from SPDX, CycloneDX, and The Linux Foundation, where organizations can learn more about SBOM’s, their role and how they can be used to mitigate organizational risk.
While SBOM’s are heavily emphasized in this document from the NSA and advocated for by sources such as CISA and several security startups and industry leaders focused on software supply chain, I did want to provide some balanced perspective.
Recent research by John Speed Meyers, Dan Geer and Santiago Torres Arias found that when looking at a large publicly available dataset of SBOM’s (3,000~), less than 1% met the NTIA defined minimum elements for an SBOM. They published this research in an excellent paper titled “A Viewpoint of Knowing SBOM Quality When You See It”.
Critics of the research have pointed out valid claims, such as it being skewed due to the sample data being publicly available SBOM’s, and not SBOM’s that organizations are allegedly using at scale which are more complete and robust, but of course without access to said robust SBOM’s, such claims may be hard for researchers to verify.
Similar concerns around SBOM’s were recently raised in a congressional testimony to the “Committee on Oversight and Accountability”, titled “Safeguarding the Federal Software Supply Chain” which included Jennifer Biscgelle, CEO of Interos, as well as Jamis Jaffer of the National Security Institute, James Lewis of the Center for Strategic & International Studies (CSIS) and Roger Walden of The Coalition for Government Procurement. The testimonies included concerns about the push for SBOM’s, state and maturity of tooling and the SBOM ecosystem and a lack of clear guidance from the Government to industry. You can listen to that testimony here.
Concerns aside, there does seem to be widespread consensus from leading security experts that Transparency is a good thing, and there is a need to address longstanding information asymmetries that exist between software suppliers and consumers. Many other industries such as automobiles, pharmaceuticals and food have valid examples of providing transparency of what goes into a product, while software remains opaque, despite powering nearly ever aspect of modern digital society.
The guidance stresses the speed of malicious exploits, occurring on average 5.5 days after a vulnerability is found and disclosed and recommends utilizing SBOM’s within automation pipelines such as CI/CD to automate SBOM generation. It also advises to view CISA’s resources for SBOM and VEX.
For a detailed visualization of how VEX documents and SBOM’s work together, the below image is provided below:
This demonstrates how an SBOM can provide transparency of software components and VEX can accompany that from software suppliers to communicate what components are affected or not by vulnerabilities and may require action and mitigating controls by customers and consumers.
SBOM’s and accompanying VEX documents can be provided to consumers to clearly communicate what OSS components are within a product or application and the VEX can help maximize the use of resources to focus on what aspects of the product are actually vulnerable and the details of potential actions taken by the supplier or vendor. This addresses a longstanding lack of transparency that has existed between software suppliers and consumers and offers an opportunity to bolster software supply chain security across the ecosystem.
For further details on SBOM processes for suppliers, the NSA guidance recommends the NTIA’s “Software Suppliers Playbook: SBOM Production and Provision”.
Conclusion
This publication from the NSA represents another excellent contribution to the broader discussion on software supply chain security. It provides detailed guidance to Developers, Engineers, Product Teams and organizations with regards to governing the secure use of OSS as well as insights into the role of artifacts such as SBOM’s and VEX documents.
We live in a complex modern software ecosystem and publications such as this one from the NSA help organizations think through how to threat model their current digital footprint, key considerations around OSS usage and how to provide transparency to downstream customers and consumers as well as meet emerging regulations and requirements from the largest single buyer of IT and software - the U.S. Federal Government.
Thanks for reading Resilient Cyber! Subscribe for free and join nearly 4,000 readers to receive new posts and support my work.