Deploying code into production is loaded with inherent risk. It doesn't matter how big or small your company or app is, nor how much network-level protection you're employing, nor your level of software development hygiene. 50-90% of software in workloads is unused—and can be exploited, even in high-security environments with golden base images and advanced policy enforcement tooling.
Our research and real-world experience show that 80% of software components can be removed from production workloads. The number is so surprising because a significant majority of deployed software is built on trusted open-source libraries that have large dependency trees. In fact, modern software development is based mainly on trust and open source software has earned it.
Where would we be today without free, open-source software like Apache Web Server, Linux, MySQL, or WordPress? It's unimaginable. Building on and extending freely-available code has enabled one of the greatest leaps forward in human productivity, but it's not without drawbacks.
The proliferation of microservice-based, containerized architecture has created a new risk that companies are only beginning to address: the process of containerization engages the entire software supply chain.
In this article, we'll define the software supply chain, discuss the two primary vectors of supply chain risk, and share how the industry (including governmental agencies) is responding to this rapidly-changing landscape.
What is the software supply chain and why is it risky?
Our working definition of "software supply chain risk" is:
A systemic risk that arises from using software components or applications not developed internally.
Most software used at any organization is developed externally (e.g. open-source, third party, APIs, COTS). Though the move towards smaller, more portable infrastructure like containers is a big step forward for software attack surface management, there is still plenty of room for improvement.
There is fundamental mathematics of integrated systems that make the challenge more difficult. For example, if you have 20 components, each with a 2% chance of breach within the next year, i.e., they're effectively 98% "safe for this year." When you chain them together, threats across the entire system yield a 33% chance of a breach within a year. So, the more components you have in a system, the greater the combined risk.
A second surprising systemic mathematical phenomenon is that your infrastructure is only as strong as the weakest link—or vulnerable component—and often, that weakest link is a human being. It only takes one bad employee with poor software hygiene to create an opportunity for malicious activity. Whether your team is 100% fit or not, supply chains are growing longer because the attack surface is increasing:
- More software is open source than ever, and that continues to grow.
- People work from home using various devices, SaaS, IaaS, PaaS, and API services.
- Greater adoption of cloud-based services means bad actors increasingly target them.
- 85% of breaches in 2020 involved a human element.
The linked nature of systems makes them very hard to defend.
What are the primary sources of software supply chain risk?
The primary source of modern, cloud-native software supply chain risk is the amount of trust put into open-source software. Let's start with a few key points:
- 90% of code in containers is comprised of OSS
- Containerized microservice-based architectures are quickly becoming the leading software deployment paradigm
- DevOps teams optimize for speed to market and delivery of business value, not to maximize security
All this leads to a large attack surface for production applications. There can be thousands of known vulnerabilities in a typical system—often, orders of magnitude more than that. Significant productivity losses can occur with remediation efforts, usually with mediocre results, which slows the process. So as you can see, modern software development has a high degree of "embedded trust."
What are the primary attack vectors for software supply chain risk?
The two primary vectors of software supply chain risk are: intentional and incidental. There are a number of intentional supply chain attacks, including:
Intentional software supply chain risk occurs when a bad actor intentionally inserts malicious functionality upstream of build so that it can be later exploited. Examples of this are:
- Dependency Confusion
- Delivering a malicious payload by leveraging the way software is updated (e.g. fake pull requests)
- Tricking CI/CD systems into pulling harmful code from untrusted repos
- Typo-squatting
- Malicious packages with similar names to popular packages can trigger execution if a typing mistake is made that runs the wrong package
- Malicious source code injection (this one creates the most angst for defenders)
- Stealing credentials from a project maintainer
- Releasing "new" versions of code to a public repository
- Tampering with open source dev tools to inject bad code (example: CodeDev)
- Hacking suppliers to inject malicious code into patches(example: SolarWinds)
Incidental software supply chain risk risks arise from the nature of the software itself. A malicious actor doesn't need to proactively insert a vulnerability into a workload because it's already there. When it's discovered, it's documented by organizations that report vulnerabilities. Combine that with a proof of concept demo (POC), and an attacker has the recipe for an exploit that can be added to a toolset and "weaponized."
Bad actors understand this process well and exploit even the process of remediation. The time window between a proof of concept release and in-the-wild attacks has shrunk to just a few hours. Following the Atlassian Confluence vulnerability in 2021 (CVE-2021-26084), there were more than 300 copycat apps within 72 hours and more than 10,000 in the first 30 days. Attacks often happened on weekends when they knew security expert availability was lower. Once a vulnerability and its exploit become known to the industry at large, bad actors can weaponize it.
How is the industry responding to software supply chain risk?
The popularity of containerized infrastructure has increased industry awareness of software supply chain risk. In 2019, 16% of companies used containers and that number is expected to jump to 65% by 2023, according to Gartner. It's an area of great concern for security teams, who are pushing technologists to get security certifications, especially as development teams are managing infrastructure as code.
The federal government is keenly aware of the increasing risk of the supply chain. In fact, the following organizations have issued formal bulletins and regulations mandating increased risk management:
- NSIA/CISA
- The White House
- FedRAMP
- DISA/DOD
- IST, ISO, CIS, NTIA
There is unanimous federal pressure to improve and enforce container scanning and hardening methodologies as these are the primary vehicles of measuring and managing software supply chain risk..
How is software supply chain risk measured and managed?
There are various techniques for reducing container OSS supply chain risk:
- Golden B base Images with repository integrations and component white-listing
- The idea is only to use pre-approved packages, but the software world moves so fast, it's very easy to get out of sync and build a huge backlog
- SCA Scanning with manual vulnerability removal
- This is the most common method with a fair amount of success
- Scan artifacts, to know what's in them, prioritize risk, and remediate
- Static, dynamic, interactive, and runtime application security testing (SAST, DAST, IAST, and RAST) are other ways of identifying vulnerabilities
- Policy Agents
- Enforces configurable security policies across a tech stack
- Instrumentation
- Monitor the tech stack with a dashboard and tooling
- Container slimming and attack surface reduction
- Eliminate what's unnecessary from the software container
There's rarely a single approach that solves the entire security problem and supply chain risk is no exception. The primary consideration factors are about where in the SDLC these tactics are employed and when they are used. Golden base images, for example, require a lot of work before software development can even begin. SCA scanning with manual vulnerability removal happens primarily at the end of the SDLC, shortly before code is deployed to a production environment. Policy agents and instrumentation can be used throughout the SDLC. But these all take time and drain developer resources.
Getting into more detail with software supply chain risk
Software supply chain is an intricate and layered concern for anyone running production infrastructure. Given the world's increasing dependence on open-source software for development velocity and functionality, there's more cause for concern than ever.
In our next article, we will discuss our approach to managing supply chain risk using SCA scanning with manual (and automatic) vulnerability removal. We believe there is a tremendous opportunity with this approach: increased velocity for business value delivery, less cognitive load for development teams, and superior security.
RapidFort's scanner is a powerful tool that solves most of software supply chain risks. Request a demo to learn more.