We’ve all heard Mark Zuckerberg’s engineering mantra “move fast and break things.” In the early days of Facebook, scaling for hypergrowth was pursued at all costs. But moving fast isn’t simply writing code really fast or running a script that provisions scalable infrastructure—it’s all about leverage.
Open Source Software (OSS) components help tech companies get to market faster, deliver more value, and improve the user experience. Developers can leverage a staggering amount of functionality with modern OSS packages. Pull together a few free code libraries, write some glue code, and have a high-functioning app online in just a few hours. Humans have never had such technological power available at our fingertips.
Even though it’s “free as in beer,” it’s not without costs.
Leverage and trade-offs of OSS components
There’s a difficult trade-off with OSS that no one wants to talk about. As fast as we move (and break things) with OSS, we’re building staggering levels of risk into our software and infrastructure. We also introduce excessive bloat into our apps by including entire libraries—even if we only use a bit of functionality from each.
In fact, before co-founding RapidFort, our CTO Rajeev Thakur couldn’t deliver an app to production because of the number of vulnerabilities that unknowingly came pre-packaged in OSS components. He and his team spent six months remediating the most important vulnerabilities and still ended up sitting on the product.
Is this moving fast and breaking things? Probably not what Mark Zuckerberg meant when he first uttered those words.
Rajeev’s story isn’t unique. Most companies use many OSS components to get to market quickly. Unfortunately, they do so without understanding their software attack surface risk; they don’t scan their software to understand the vulnerabilities it contains. In other words: they don’t want to know or have visibility into their security risks.
How companies manage OSS vulnerabilities and risk
Thankfully, more companies are becoming aware of their software attack risk. They use SCA scanners to learn how many known vulnerabilities are contained in their software. Once they have a list of issues and severity, they decide what to do next. A majority of companies use patching, but when we’re talking about thousands (yes, literally thousands) of vulnerabilities in a software workload, there’s too much to patch. It’s expensive, official patches are unavailable, and it quickly becomes an impossible task.
Ranking and remediating vulnerabilities by CVSS score
The next step is to prioritize vulnerabilities into critical- and high-severity issues to address. This is often done with a Common Vulnerability Scoring System (CVSS) ranking, which ends up becoming an issue of internal negotiation. Security, development, and operations leaders have to ask whether fixes are possible, reasonable, or cost-effective.
High and critical issues are usually fixed by waiting for patches. Teams face a difficult reality that they don’t have the expertise or time to patch software they didn’t write, so they are dependent upon vendors, many of which consist of teams of volunteer developers. This is a significant problem for issue remediation. Nobody wants to play the waiting game, especially when there are customers waiting for new features or product deploys.
Virtual patching
For critical and dangerous situations like Log4j, vendors can take months to roll out patches. Some companies use virtual patches, which are just a temporary fix. They have the smell and taste of a fix, but use network monitoring and diagnostics to offer the appearance of patched, invulnerable infrastructure.
Virtual patching tools have to meet certain requirements to be effective. Most importantly, they need to deeply inspect network traffic and “look around the corners” to see if someone is trying to perform an exploit. They need sophisticated network activity detection to determine the state of the exploit, appropriately notify any support personnel, and aid teams in putting an end to the attack.
Experienced companies can get very good at virtual patching and effectively work around the risks of virtual patch management. The downside of patching is that if someone makes a small mistake, they can inadvertently bring down the entire network. The network inspection and response tools used for virtual patches have the authority to make changes (sometimes autonomously), so virtual patches need to be thoroughly tested before deploying to production.
Shift left: let the devs worry about security
Shift left is an idea that sounds great on paper, but in practice becomes a significant challenge. Though security vulnerabilities can be addressed much earlier in the SDLC with a shift left strategy, it requires a level of rigor and sophistication that many companies can’t meet. Teams need to scan early, find issues, and address those issues while also implementing new features.
Companies that go all in on shift left are still spending an average of 6.5 hours per vulnerability. No matter the size of the dev team, that’s not scalable. It’s simply unrealistic to expect nearly one full developer day per vulnerability when OSS components literally have dozens—if not hundreds—of known vulnerabilities.
Application Security Testing
Several flavors of application security testing are available to dev teams today, but they present problems that are unrealistic for most companies to manage. Static application security testing (SAST) analyzes the source code to find vulnerabilities early in the coding phase. Its applications are fairly limited and can only identify a small scope of issues, like buffer overflows and injections.
Dynamic application security testing (DAST), better known as “black box testing,” is meant to identify vulnerabilities in web apps. When DAST tools identify vulnerabilities, there’s no easy way to understand where the vulnerabilities exist in the source code, so devs have to spend time reverse-engineering the cause.
Another issue with SAST and DAST is the high number of false positive and false negative results. But perhaps the biggest issue with these application security tests is that they don’t help devs with vulnerabilities in the OSS components they’re using behind the scenes. What is a dev team supposed to do with a vulnerability discovered in code they didn’t write?
About 50-90% of code in a typical production container is OSS, so what good will SAST and DAST really do? Assuming someone is willing to wade through OSS code they didn’t write, how much time and money should be spent on remediation? Even if the results aren’t false, it’s really hard to fix the issue. Some vendors are combining SAST, DAST, and SCA scanners to provide more verbose and thorough results, but they do nothing to reduce the amount of work an engineer needs to do.
Fixing vulnerabilities ain’t cheap!
Setting aside the costs of the tools and operational requirements of the processes we’ve just described, it’s incredibly expensive to fix software. A typical vulnerability costs anywhere from 6-9 hours to fix if it’s discovered post-launch. For issues discovered before launch, it’s about 8.9 hours. Vulnerabilities are often found in the final stages of application testing, making it even more expensive to fix than if discovered earlier (a la shift left).
The costs of vulnerability management and remediation can easily be millions of dollars using the most common methods. Let’s factor in issues like network downtime in the event of a bad virtual patch response. These types of issues can cause hours of downtime, which for some companies can be millions of dollars. Even if the company is extra careful in its virtual patch testing, those tests aren’t free either.
Move fast and fix things
RapidFort has a great alternative to all of these approaches. We have something brand new. In fact, it’s so revolutionary that there was literally nothing like it when we started the company. We watch a software workload and identify files that aren’t used during testing, then we delete whatever isn’t used. RapidFort is regularly reducing OSS component bloat by 80% without affecting any functionality of the container.
For example, a popular software container like nginx is very versatile and can be used in a variety of ways. It can be a web server, load balancer, reverse proxy, or something very narrow like providing a heartbeat for an app. These use cases might use 10-15% of the code that comes in a typical nginx container. That code sits there with vulnerabilities and keeps the software attack surface unnecessarily large.
RapidFort says, “You don’t need these components because you don’t use them.” Then we remove them—automatically. The resulting containers are much smaller, use fewer computing resources, and increase operational efficiencies. In fact, we have a library of popular open-source containers that we harden every day and make available for free to the public.
Want to get started with RapidFort? Sign up for free to take RapidFort for a test run.