What to do while Apache fixes Log4J

Companies across the globe are working around the clock to fix a newly-discovered security flaw in the popular open source software Apache Log4J. Bad actors are working hard to exploit this vulnerability while it’s still in the wild. According to CNN, “more than 100 hacking attempts [are] occurring per minute” and it could take “years” for the exploit to be completely eradicated.

The fact that a general-purpose news organization like CNN is reporting on this should tell you what a major problem this is. It’s very rare to hear about open source software in the mainstream media.

Open source software is a mainstay for companies of all sizes and it comes with a trade-off. When you don’t write the software you run, you (usually) can’t control the security concerns that come with it. The Log4J team attempted to fix the vulnerability (CVE-2021-44228), but the fix was incomplete, and now we have CVE-2021-45056. There’s no guarantee the next fix will be a complete fix, either.

While we all await the final patch(es), it’s tempting to feel helpless. This Log4J situation is just one of many examples of software supply chain hacks. We all have our personal data and livelihoods sitting in databases owned by vulnerable companies. Furthermore, who knows what other vulnerabilities remain to be discovered in other open source software we’re using every day?

Our approach at RapidFort is to reduce our infrastructure’s attack surface by removing open source code that isn’t executed in the apps we use every day. This article will show you how we are approaching the Log4J situation and how we’re fixing it altogether—before the Log4J team comes out with their final patch.

Scanning, Assessing, and Removing Vulnerable Log4J Code

Here at RapidFort, all of our infrastructure is containerized. We do this because we believe in containerization as an infrastructure strategy and because we built a product that hardens and secures containers. So, anything using Log4J runs inside a container, but that container is stripped down to its bare minimum components. More on that in a bit…

When we learned about the Log4J issue, we immediately scanned our workloads to size and scope the risk.

We found Log4j in a number of components within our infrastructure. In some containers, it posed lower risk and in others, it needed to be addressed. Our report showed us where it was and in what context it was running, so we could focus on the real risks—not the perceived risks—of this exploit.

Important note: any version of Log4J <= 2.14 is vulnerable to this exploit and any application using Log4J is affected. Immediate action must be taken to resolve the issue. To do so, affected versions of Log4J should be upgraded to the latest 2.175.0 release.

Combining the results of our scan with the known Log4J version numbers, we patch and harden our containers by removing whatever components are not necessary to run. This might sound like black magic, but it’s pretty simple.

Profiling container runtimes and removing unnecessary code

The first thing we do when hardening a container is to remove everything we know we won’t need in the container, like Bash and SSH. Then, we run our containers through a series of tests to build a DAG and see what code/files are touched. Files that aren’t touched can either be deleted right away or remain as part of an overall risk profile.

The DAG helps us to identify only the components required for that particular application to run. Once we assess the importance of those components, we can decide whether to remove the files that contain the components’ code. Removing components helps produce a hardened, optimized container that contains only the necessary components.

This entire process is executed as part of our CI/CD process.

Running the attack ourselves

To prove to ourselves that our containers were sufficiently hardened and secured, we exploited the vulnerability ourselves on our own infrastructure. The results were surprising.

We couldn't do anything from the minimal shells remaining in the workload. The most nefarious thing we could do was overwrite critical files and make the services unstable, only in one instance. We could not access or remove any sensitive information. Our monitoring tools quickly detected the failed service, which alerted us to the need for attention and investigation.

Conclusion

The recent Log4J exploit is a classic example of why containers should be optimized and hardened for workloads. Though hackers could still gain access to a Log4j container that has been optimized and hardened, they can't easily access additional tools to explore deeper attack opportunities.

Assuming that the hardened container still contains a shell program (as scripts within the container might use the shell program), removing all other unnecessary software makes it very difficult for the attacker to navigate and look for other opportunities. We call this "reducing the blast radius of the attack."

We don’t know how long it will be until Apache releases a final patch for Log4J, and we don’t know how long it will take for various software vendors to update their Log4J dependencies. In the meantime, we can at least harden and secure containers where it is running. At the very least, even if the vulnerability is still exploitable, our containers will not have enough code for the attackers to expand the depth or breadth of their access.

In light of this Log4J problem, we are providing our services for free to companies that need it. Please email us at info@rapidfort.com, and we will happily help you to harden and secure your Log4J containers.

‍