"Shift Left" is Starting to Rust
A look into the origins of the "shift left" movement, and ways it has been implemented poorly
Interested in sponsoring an issue of Resilient Cyber?
This includes reaching over 7,000 subscribers, ranging from Developers, Engineers, Architects, CISO’s/Security Leaders and Business Executives
Reach out below!
Origins and Challenges
Anyone who has been working in and around cybersecurity and software development in the last 5+ years has inevitably heard the phrase “shift left”.
It coincides with the rise of other mantras, such as DevSecOps and focuses on integrating security throughout the SDLC, and in the case of shift left, moving security earlier in the SDLC to identify and fix defects before they reach production.
The common phrase supporting shift left is that it is “x” much cheaper to fix defects and bugs earlier in the SDLC than it is to do so in production.
The problem?
The claim and data to support it is built on a lie, or at least very questionable sources that have never been substantiated. At least that is the case being made in a report submitted by a subcommittee to CISA as part of the CISA Cybersecurity Advisory Committee (CSAC). See below:
The report from CISA points out the origins of the speculative cost savings of “shifting left” were founded on a group that hasn’t existed since the 1980’s and wasn’t even an official research body. Similarly, another claim, made by Barry Boehm saying fixing a software problem after delivery is “100 times more expensive” than finding and fixing it during the design phase was also unfounded and no actual empirical data to substantiate the claims.
The report also points out even more modern reports, such as IBM and Ponemon’s Cost of a Data Breach report provide no actual data on the cost if security vulnerabilities are found and fixed before a security breach occurs.
So while shifting left and fixing software defects prior to production where they can be exploited is logical and makes sense, the movement was built on speculative unofficial sources, some of which may have never even existed.
In a thread I posted on this topic on LinkedIn, one user even jumped in to share the below image, which apparently was the original source used to try and justify the cost savings of shifting left.
CISA goes on in the report to recommend that further studies should be performed to provide empirical data to substantiate whether fixing security vulnerabilities early in the SDLC is truly more cost effective.
They even say:
“If it is found there is no measurable financial impact, then it may make sense to move away from using an economic rationale as a security development incentive”
But if there is no financial impact or economic value to shifting left, how do we intend to motivate businesses, which are of course financially driven, to prioritize security early in the SDLC and mitigate risks before they are passed downstream to customers, consumers and society?
We can say from the security perspective, it is important to mitigate defects, flaws, misconfigurations, insecure design and vulnerabilities before production so they cannot be exploited, but of course that doesn’t have the same ring as “it saves money” like the previous claimed studies cited around 100x or more in terms of developer cost savings.
This is why CISA recommends formal studies to get empirical data to see if there is indeed documented cost savings from shifting left, and fixing defects earlier in the SDLC. It is important in security we present findings in a way that the business cares about, which is Dollars and Cents.
Talk of vulnerabilities, exploitability, and other cyber-specific language is unlikely to get the attention of our business peers. As the saying goes, show me the money.
Further complicating the challenge is that increasingly, we see that the correlation between security incidents and business metrics like share price are weak at best.
Recent studies show organizations that filed SEC 8K’s for example, only suffered a ~1.4% share price decline, which bottomed out 40 days or so after the incident, recovered in 53 days and continued to climb to pre-incident levels. The same study even showed that perplexingly breaches of sensitive data such as SSN’s had less of a negative impact on share prices than information like email addresses.
CISA even stated similar findings in their own report, when they discussed the below:
“Fact or Myth? - Security flaws make people walk away from solutions”
Their consensus?
“In general, it seems that quality failures don’t always affect customer loyalty”.
They point out the three below examples:
Using Target, Samsung and SolarWinds as examples, they demonstrate that widely impactful and visible security incidents often don’t negatively impact organizations in terms of customer base, revenue, market share and more.
As they state, there are several factors that impact the behavior of customers beyond an incident, including switching costs, the LOE for the customer to even dig into and understand the incident and how the impacted company handles PR coming out of the incident.
What’s peculiar about the CISA report is it even plainly states simply telling organizations that not fixing bugs will impact their business isn’t a sufficient incentive to drive action.
Yet, when we look around at things such as the Secure-by-Design/Default guidance, and the Secure-by-Default Pledge and other examples (not looking to pick on CISA because I am a big fan of their efforts), they are all just that, us as a security industry telling vendors and organizations they should fix bugs because.. well, because.
If they inherently know, and it is now being validated by data, that the cost savings of fixing bugs earlier in the SDLC is speculative, and that incidents, breaches and so on have marginal impacts on the businesses bottom line, do we really think they will just altruistically prioritize security?
Tool-Centric Focus
Another challenge that has hindered the adoption and success of Shift Left is that, as usual, the industry has taken a tool-centric approach to what is a process, not a product, much like security itself isn’t something you buy, but is something you do.
These are cyber anti-patterns and even called out in Mark Simos RSA talk “You’re Doing It Wrong! Common Security Anti-patterns”. (If you’re interested in a deeper dive, you can find my Resilient Cyber interview with Mark here).
For years, terms like shift left and DevSecOps have been bastardized to mean throwing a slew of acronym soup tools (e.g. SAST, SCA, IaC et. al) in a pipeline and saying you’ve shifted left and/or are doing DevSecOps.
Don’t get me wrong, these tools are indeed invaluable to identifying defects and vulnerabilities earlier in the SDLC and fixing them, but they are far from what the original concept of building security in, rather than bolting it on actually means.
Activities such as Threat Modeling and Secure Coding, involving security in the system design, architecture security reviews and more are equally as fundamental and critical to integrating security in the onset and early stages of the SDLC.
A decent representation of this can be seen in the DoD DevSecOps Ref. Design below:
There are various reasons for the tool-centric thinking, some of which ties to human nature, and looking for quick fixes, and thinking that buying something will immediately solve our problems. Tool-centric thinking is the Ozempic of the cybersecurity thinking.
Another factor at play, as my friend and fellow co-author of “Software Transparency” Tony Turner pointed out on LinkedIn, much of the tool-centric focus is driven by vendors due to the fact that tools are much easier to sell than some of the other activities, which are more so practices/activities and something we do, not something we can buy.
It is far easier for a startup to approach VC’s with a product pitch around a tool, total addressable market (TAM), propose integrations with other products, such as CI/CD pipelines and also develop a go to market (GTM) to sell into enterprise environments with a product that can be bought and deployed and/or integrated into environments than it is to do so with an activity that requires active participation, such as Threat Modeling, Secure Coding and Secure System Requirements and Design (although tools do exist and can help with these activities as well).
We have to remember that when a trend begins (e.g. Shift Left, Zero Trust, Secure-by-Design etc.), these trends are seen as business opportunities for startups, innovative practitioners and investors alike. They are looking to productize capabilities and commercialize them for consumption.
However, this tool-centric approach to security, including shift left, is starting to show cracks. As security leaders and programs continue on their game of security tool Pokemon (gotta catch them all!) and/or Gartner Buzzword Quandrant Bingo.
This SiliconAngle article I’ve discussed before “Cybersecurity tool sprawl is out of control - and its only going to get worse” covers some of the state of the issue.
They site several surveys showing the average CISO/Security org having 70-90 and even 130 security tools and growing, with 51% of respondents saying they expect the security stack to grow even further in the next 12 months. The irony is the same article goes on to say organizations are 10-20% of cyber technology is actually fully used (E.g. effectively deployed, implemented, configured, tuned, optimized etc.)
This tool sprawl has more impacts than just financial bloat too, as I’ve written in articles such as “Cybersecurity Tool Sprawl Can Lead to Team Overload and Lower Impact”, with tool sprawl also causing cognitive overload, fatigue and burnout for security teams, and actually making them less effective as they try and juggle the sprawl portfolio of security tools. The security tools are also products and software, with their own inherent misconfigurations, vulnerabilities and weaknesses, making them not only a burden administratively, but also a porous part of the organizations attack surface.
This tool sprawl is now being seen as a business opportunity, with industry leaders such as Palo Alto, Crowdstrike and others pushing for “platformization”, where they roll various disparate tools and functions into a comprehensive platform, promising more effective security oversight and operations as well as potential cost savings.
That said, not all platforms are created equal, as discussed in Ross’ article “Platform vs. Best of Breed is a Wrong Way of Looking at the Industry” where he lays out how point products mature into platforms, best in class products/platforms eventually become “good enough” and then get disrupted by new entrants, taking the place of incumbents and repeating this cycle indefinitely.
Lack of Context and Quality
Something else impeding the widespread adoption and effectiveness of shift left, especially given the hyper-focus on tools, and CI/CD pipelines is that in security we’ve taken legacy tools in the various categories (e.g. SAST, SCA etc.) and jammed them into Developer workflows and have buried our Developers in low fidelity, false positives and findings that lack any actual context.
The painful irony is we’ve done all of this while cyber practitioners take to the speaking circuit to claim “security is a business enabler”, while ultimately grinding development velocity to a crawl, building resentment from our peers and stifling the businesses ability to be agile and quickly respond to market demands, customer requests and feature needs.
While we’ve been busy anointing ourselves as business enablers, Developers and the business have been festering in resentment and frustration with what we now call “shift left” security.
While modern AppSec tooling is beginning to rally around valuable vulnerability intelligence such as CISA’s Known Exploited Vulnerability (KEV), the Exploit Prediction Scoring System (EPSS) and reachability analysis, as well as business-specific context (e.g. asset criticality, data sensitivity, Internet exposure, and compensating controls) we’ve still got a lot of work to do to improve the sad state of shift left as it stands now.