Security Throwing Toil Over the Fence

Shifting Smart over Shifting Left: The need for context to mitigate developer toil

I recently shared an article from Contrast Security’s CTO/Co-Founder and longtime Application Security (AppSec) leader, Jeff William’s that was shared in Forbes titled “Application Security: ‘Shifting Left’ or Shifting Smart?”

In the article, Jeff makes the case that we’ve seen a push in the past decade to “shift security left” rather than right before (or after) a production release. Jeff pointed out that the long cited statistic of remediating vulnerabilities earlier in the SDLC being “100 times” less expensive than fixing them in production could potentially be tied to a non-existent study, despite being regularly cited.

That said, Jeff did concede that there are benefits to shifting left from both a quality and security perspective. However, as Jeff notes, often security testing requires tooling and expertise that developers may not have. While this is true, I would also posit that many security practitioners lack the software development expertise to interpret the results of said tooling, at least with sufficient depth, and need to engage the developer(s) nonetheless. This is why we’re seeing some developers successfully make the transition to AppSec roles, given the context and domain specific knowledge they bring with them.

The overarching premise of Jeff’s stance is that shifting security left is more complicated than a catchphrase, and many types of vulnerabilities require context that you simply don’t have/can’t get too early in the SDLC, such as SQL Injection, Authentication/Authorization and Encryption strength.

Shifting Smart over Shifting Left

Jeff makes the case that rather than obsessively and blindly pushing security testing and activities left, we should instead follow a strategy of shifting smart.

This means performing security testing when you have enough context on how applications function to identify exploitable vulnerabilities that pose real organizational risk.

Reading Jeff’s headline of shifting smart over shifting left, I am reminded of the perils of an industry clinging to a catchphrase, only to realize reality isn’t quite so simple. Specifically we saw this play out with the U.S. Federal Government’s evolution of “Cloud First” to “Cloud Smart” when it came to their adoption of cloud as part of digital modernization efforts, as they realized cloud adoption isn’t as simple as a tagline and lifting and shifting everything into an IaaS provider and divesting of data centers. As it turns out, application modernization, cybersecurity and a competent workforce with relevant knowledge, skills and abilities actually need to be part of the picture too when it comes to cloud adoption, but I digress.

As we have seen the push for DevSecOps, we’ve seen a lot of industry dialogue around breaking down silos, and integrating security into Developer’s workflows.

This has manifest in CICD pipelines with robust security tooling such as SAST, DAST, Secrets Scanning, SCA, Container Vulnerability Scanning, Software Bill of Materials (SBOM) and more (SCA and SBOM bring particularly granular visibility of components and libraries and their potential vulnerabilities, but also the potential for significant noise, but that’s a story for another article).

While all of these potentially help to identify vulnerabilities, they also often lack context, are noisy, full of false positives, putting Developers into a guilty until proven innocent scenario, where they either need to spend non-trivial amounts of time addressing trivial vulnerabilities that aren’t known to be exploitable, aren’t likely to be exploited, or are unreachable in deployment runtime environments.

Another often overlooked impact of this reality is that it creates resentment between Developers and Security which erodes trust, rapport and a healthy culture, as Developers are understandably frustrated by needing to wrestle with the toil of addressing hundreds/thousands of vulnerabilities from these robust set of tools, often which lack context, and often to peers who lack software development expertise to understand the explanations and justifications they provide.

In a previous article I wrote titled “Vulnerability Management and Developer Toil” I discussed a 2022 report from Qualys that showed that of the 25,000+ known vulnerabilities in 2022, less than 1% were actually exploited by malicious actors.

This means in addition to addressing vulnerabilities that lack context, may not be reachable, or actually be relevant in a production environment, less than 1% of known vulnerabilities in the past year were actually exploited by malicious actors.

Statistics and industry-wide techniques to “shifting left” and vulnerability management like this are a prime reason why some surveyed OSS developers are cited as referring to security as a “soul withering chore” in a study by The Linux Foundation and Harvard.

Exploitation Windows

While Jeff makes tremendous points that as an industry we need to heed when it comes to mitigating the blind “shift left” obsession without considering context and the burden we impose on our Developer peers, there is also something to be said for exploitation windows.

As I discussed in the “Vulnerability Management and Developer Toil” article, citing research from Qualys, malicious actors continue to outpace cybersecurity defenders, with a time to weaponize vulnerabilities of 19.5 days while defenders have an average mean-time-to-remediate (MTTR) vulnerabilities at 30.6 days, not to mention research I also cited from Rezilion which showed that organizations admit to being able to patch and remediate less than 50% of vulnerabilities overall.

This means that while we want to avoid imposing unnecessary toil and burden on developers, we also need to be cognizant of the fact that vulnerabilities that do make it to runtime environments have less than a 50% chance of ever being remediated and malicious actors have shown an ability to outpace us in their weaponization of vulnerabilities contrasted with our ability to remediate them.

Business Enablement or Source of Frustration?

Another slogan we have seen catch on in our industry is “cybersecurity as a business enabler”.

In “The New Kingmakers: How Developers Conquered the World”, author Stephen O’Grady quips:

“The Developers can make a company great or break it apart. Developers can help a company conquer the world and make new kings”

In the business context, many organizations indeed do view Developers as being absolutely critical to organizational growth and value delivery, especially as organizations find themselves increasingly competing in the digital domain, delivering value to customers through applications and with applications and software being their direct conduit to revenue generating activities and customer/stakeholder interactions.

The same applies in the Federal and Defense communities, with software being core to delivering civic societal services and enabling national security in the Fifth Domain, as cyber is referred to by authors Richard Clarke and Robert Knake in their book by the same title “The Fifth Domain: Defending Our Country, Our Companies, and Ourselves in the Age of Cyber Threats”.

This means that Security practitioners and teams need to be cognizant of the unnecessary toil and burden they impose on their Development peers who are directly contributing to the revenue, business and mission of their respective organizations in the modern digital realm.

This comes down to things such as rational security “gates” in Development workflows that are grounded in context rich security testing and findings, and meaningly vulnerability metrics not tied to arbitrary criteria such as only using Common Vulnerability Scoring System (CVSS) severities when making decisions on breaking builds or blocking deployments, something that the stewards of CVSS themselves recommend against, but large communities such as the Federal and Defense sector do nonetheless.

As Jeff makes the case, security testing should be shifted to where it can provide meaningful, relevant, actionable insights to mitigate organizational risks versus inflating vulnerability metrics and the act of performing what is often referred to “security theater”, which throwing context-less lengthy vulnerability scan reports over the fence to Developers can qualify as.

Throwing It Over the Fence

In the discussion of DevOps, it is often said that Developers historically throw code over the fence to the operations team and basically tell them to figure out out, its their problem now. This of course frustrated Operations folks who are focused on stability and maintaining the organizations infrastructure and systems, not necessarily introducing new features, code and functionality.

In the age of “shift left” we’ve now seen a paradigm emerge where Security throws vulnerabilities with little to no context from their robust security tooling over the fence to Developers and tell them to figure it out and often block/break builds as part of the process.

Developers are often left with lengthy spreadsheets of vulnerabilities and findings or wrangling together findings from the slew of bespoke, poorly configured and inefficiently implemented security tooling to provide justifications and responses to security to be able to move forward with delivering code to production - aka delivering value for the digital organization.

In the old paradigm that led to DevOps, Developers throwing the code over the fence to Operations caused cultural rifts and tension between the two teams/groups and Developers weren’t incentivized to own the reliability of the product, since it was now an Ops problem.

Currently, we have a situation, particularly with the “security is everyone’s job” mantra, where Security isn’t necessarily incentivized to do the leg work of enriching vulnerability data with context, and instead throw it over the fence to their Developer peers and tell them to deal with it prior to being able to deploy to production.

This of course is creating cultural rifts, resentment and frustration between the teams and the is the exact sort of behavior that encourages “shadow IT”, where entities avoid engaging with security and compliance teams because it becomes such a laborious and dreadful experience.

One last thought, we’re also seeing people now use the phrase “shift everywhere” and I want to say I am not a fan. Its irrational and implies we can be everywhere or that everything is a priority. We all know the saying when everything is a priority, nothing is, and that is the last thing we want for security.

I’m a fan of the “shift smart” mantra because it implies applying critical thinking to our security testing strategies and methodologies and looking to provide value to our respective organizations and peers.

While being a business/mission enabler may look different for each security team, one thing is for sure, false positives, noise and toil are contradictory to business enablement and as an industry we need to mature and shift to providing real value to mitigate organizational risk and foster enablement.