Log4J has been the talk of the town recently and everyone is focused on the technical details of the specific vulnerabilities found. In this episode, James talks about the overarching ideas around dealing with vulnerable components. Are you vulnerable? If so, what needs to be done?
Listen to the Episode:
For more info go to https://www.developsec.com or follow us on twitter (@developsec).
On this episode, James talks about the log4j event and how you can handle vulnerable third party components in your applications. Tackling the challenge to integrate security into the development process, looking for insights, answers and practical solutions to avoid getting overwhelmed. Welcome to the DevelopSec podcast where our focus is your success in securing and improving development processes.
And here is your host, James Jardine.
Hey, everyone, welcome to the show. If you’re like most of the internet, you’re probably still busy with dealing with the log4j event that just recently happened a week ago as of the recording of the show. And you’re probably struggling through that to try to deal with understanding things about are you vulnerable, and what your next steps might be to be able to deal with this incident.
Rather than focus on the details of what the log4j issue is, or the vulnerability is, instead, I want to actually focus at a higher level of going forward. What are the things that we can do as an organization, as a development team, as a security team to be able to be prepared for the next time something like this comes along?
I actually had the question the other day, wondering to myself, is this kind of like the 100-year storm? When we talk about third party components and vulnerabilities in there? So often, we think, oh, you know, there’s some vulnerabilities, but they’re usually not critical vulnerabilities. And often we wonder, is this even affected in my code base. The majority of the time, most of these aren’t really going to be a high or critical rated risk to your organization, then we have things like Heartbleed. Now we have the log4j event, these type of things that every five years or four years, we’re seeing something that comes out that is critical, and all of a sudden, we start scrambling for this.
We need to think about how do we handle when this type of thing comes up. So instead of diving into the details of the actual log for J incident, I’d rather talk at more of a high level, let’s first start by talking about where this sits from a vulnerability standpoint, if you follow the OWASP top 10, back in 2017, vulnerable components, or third party sources, was added to the list that really just basically defines that, hey, if you’re pulling in external components, or dependencies within your application, this is code that you’re not writing yourself, somebody else is creating, and you’re just using that component. If there’s vulnerabilities in that code, we should be able to identify that quickly, and then be able to respond to that right and patch appropriately, or take appropriate mitigations to be able to reduce the risk of whatever that issue may be.
Most of the time, when we see these come across, they’re probably lows or mediums. And then once in a while, we’ll get something that is a huge deal and everybody’s scrambling to try to fix that. So we want to make sure that we have a path in place a plan to be able to handle any type of situation that might come across. So that way when these critical times happen, we’re not running around all crazy, we actually have a plan in place to be able to handle it. We know who we need to talk to, we know where these components are.
So if you’ve been doing anything appsec related, and you follow OWASP, you should already have at least an introduction to the topic of third party components and vulnerable third party components. But I want to talk about that just for a moment to kind of discuss what that vulnerability classification is all about. When we think about vulnerable third party components, this isn’t the time where we’re scanning all of our components, and we’re looking for vulnerabilities and we’re doing static analysis on them and dynamic analysis on them.
What this classification really covers is when somebody has identified a vulnerability in a component, so we’ll use log4j as an example. If somebody identifies a vulnerability in log4j, A CVE, gets created, and now all of a sudden, you have some tools out there then there’s tons of tools that do this. You have tools that will then compare your app, your dependency list against this list of CVS to say, Do I have any current dependencies that have known vulnerabilities in them. And if I do, it’s going to flag it, it’ll tell you the information on what that vulnerability is what the risk is, obviously, you need to be able to analyze that risk yourself to determine what the risk is to your situation. But then you have something to start with. But again, this is just identifying known vulnerabilities. It has nothing to do with unknown vulnerabilities.
So when we think about something like log4j, it’s been out there for a long, long time, it’s used all over the place. And until somebody noticed a vulnerability, it didn’t technically have vulnerabilities from that standpoint. So when we think about known vulnerabilities, it wasn’t there, even though that library still may have been vulnerable. That’s what we’re looking for when we see this. So it’s not about trying to go in and find vulnerabilities in these code snippets or these components that you’re using. It’s about being able to quickly identify when a vulnerability has been shared, that you can then appropriately patch.
So now that we know what we’re thinking about, we talk about the vulnerable third party components. And then something like this happens. And if you’re running any type of scanning software, Black Duck, I think, Veracode does it Contrast does it, there’s tons of vendors out there, I’m not promoting any of those. But there’s tons of vendors that do this that scan OWASP dependency check. If you’re running tools like that, to be able to scan your dependencies, one thing you should notice pretty quickly is this happens all the time and you should be running those tools on a regular basis.
This isn’t a I scan it when I import it into my code base. And then I never scan it again, because we could have a vulnerability appear tomorrow. And if I scanned it today, and I say I got a clean bill of health, and then tomorrow, a vulnerability comes out. But I don’t scan again, then I don’t know that I now have a vulnerable component. So we want to be scanning this on a recurring basis to make sure that we’re staying up to date with the vulnerabilities that have come out to deal with.
So now that we’ve identified that we have this, and like I said, if you’re using these type of tools, you may see how often this really happens. But again, it doesn’t happen to the critical length that we’re seeing in this current instance, most of the time, it’s not giving full RCE capabilities to the system. So they’re not stopping everything, you’re not dropping everything to try to go and patch right away. But there are certain events that are going to trigger that type of thing.
When we’re scanning for this stuff, we’re going to start to identify what risks that we have. The next step is, how do we identify are we vulnerable? In this case, we had a situation where a zero day was basically dropped. And it wasn’t a Oh, my scanning tool, pick this up, let me go do the normal research. Let me see what’s going on. Let me identify how it works in our system. And you have time for that when something like this happens, and it’s a critical vulnerability, and it’s just dropped out there, and everybody knows about it. And within a few hours, you’ve already got masking or is already trying to take advantage of this. It changes that dynamic. But it doesn’t change the process that we would go through when something like this happens.
The big difference is when I’m running my scanners, I obviously know that I’m using those components to what length I’m using them, maybe I don’t know, right off the top of my head. But I know to what extent that I am using those components. In this case, it’s coming from an outside source, somebody posted a blog post or a Twitter post, hey, this is vulnerable. Here’s the component. Now we’re starting to figure Okay, well, do I use that? Where do I use that? What’s affected?
So the first step that we want to think about when we’re dealing with this type of event is, am I vulnerable? And, you know, if you’ve got one application, and it doesn’t have a lot of components in it, maybe off the top of your head, you could quickly rattle off like, yeah, I use that component. But maybe not. Here’s the thing with third party components and dependencies, is that they spider web down and they nest so deep, that you might pull in two components at a high level into your application.
And those two components may actually pull in 1000 transient dependencies that are following underneath that, that you don’t see. Right, I buy the car, but I don’t see all the electronics that sit inside the radio of that thing or all the components Are there I know there’s wheels on it. And I know there’s other pieces. But there’s a lot of stuff that I might not realize, even though I pulled something in, I put the radio in the radio has got a ton of components. So we have to be able to quickly identify, are we using these components? Where are those components? What are they doing? Does it make us vulnerable?
One of the ways that we can do this is if you are you already using a tool to be able to scan for third party dependencies, if that’s storing a history of all your dependencies, maybe it has search capability to be able to quickly say, hey, in my application, am I using this component? That’s a quick way to be able to find some components in your applications. But what happens when that expands out? And now you’re 1000 applications deep within your organization? Are you scanning all of those applications? Are you scanning all your repos? Do you know what the coverage is?
To be able to quickly say, Yes, I feel very certain, I’ve scanned everything, versus we scan the major apps, or we scan this small portion, or maybe even a bigger portion. But we didn’t scan at all. So even using these tools, we have to consider what is the coverage that we’re getting with this to make sure that I can quickly within my environment, identify all the locations that we might have this component running, so we can start to analyze how we’re going to deal with this. So that’s the first thing we want to consider. Are we vulnerable to this, are we using those components.
The other way that you could do this is by using an SBOM, which is your software bill of materials. This is something it’s been around for quite a long time. But it’s starting to gain a lot more traction, there was recently an executive order on improving the nation’s cybersecurity which is calling for SBOMs for software. And the basic premise behind this is similar to what we just talked about with the tooling, I want to have something where if I have an application, or I’m selling software, or even devices, right Internet of Things, devices, the components that are on there, some way to be able to identify every little piece that goes into that, you can think of it kind of like the ingredients list or the nutrition label on a can of soup. It lists out all the stuff that’s there.
That’s what the software bill of materials is for, it’s going to help list out. So you can quickly search. And do I have this component in any of this. And it’s going to list out what component what version of those components. The issue with SBOM is adoption rates are hit or miss according to different organizations. There’s a lot of different formats, you can put it in how do you get it from your organization to another organization, there’s still a lot of stuff that’s going on around SBOM to help kind of mature what they’re there to do. I just actually attended the CISA’s SBOM-Arama, virtual meeting that they had just the other day. And, you know, lots of great information of building this stuff up, what is the history of SBOM? And, you know, it’s got a lot of potential to be able to help with this type of scenario with log4j, I don’t know if enough people have it actually implement it to be able to make it useful today. But that, again, is the reason why this podcast is not focused on the details of log4j. It’s focused on how do we handle events like this, so that in the future when something like this happens, then we have a plan to be able to go through and handle it. And will our hair still be on fire when we do it? Probably. Because something critical like this that comes out is scary.
It’s causing us to think what are we going to do, we do not want to be breached? By something like this, the first thing that we’re trying to do is identify are we vulnerable. And we could do that using different vendor tools. Some of these vendor tools will release or export SBOM for you. But something like that to be able to have some sort of inventory.
So let’s talk about inventory for a second. Application inventory has been something that has been on the list for a lot of years now. Yet still kind of difficult to really dig into an implement in a lot of organizations. Everybody always asked the question, how can we do an application inventory? And the purpose of the application inventory is I just want to know what are the applications we have? I need to all the applications. So we’re like an asset inventory where I know all the assets in my organization. I want to know where all the applications are, what do they do, but that’s not always easy to do, and then go one level down from that and we have our component inventory, which is what the SBOM would basically do or what Some of these other tools help us do is really understand the coverage. And what do we have out there?
I don’t really know if I’m vulnerable if I don’t know what’s around, right, if I’ve never been in this house before, and I don’t know all the stuff that’s in here, if you were to say something is vulnerable to an issue, like this type of light because of fire in the house, but if I don’t know, I have those type of lights. I’m kind of stuck. Because Do I go searching the whole house and frantically trying to figure out? Do I have something like this? To determine if I’m vulnerable or not? Do I just say, now I’m not vulnerable? And that’s easy to do? It’s easy to say I’m not vulnerable. But that doesn’t mean you’re right. And we need to be able to understand with some certainty, are we vulnerable? Or are we not vulnerable? Because if we just say we’re not, and then we find out, we were, that could be a way bigger problem that we want to be able to handle. So we have to know what’s in our house, to be able to quickly respond to these type of things. So we find out we are vulnerable.
What’s our next step? Well, obviously, we want to figure out, okay, we’re vulnerable. How far does this reach out? What is the extent that this is going to cause issues within our frameworks within our organization? And how quickly do we need to recover from this? And apply patches? mitigations? What are those next steps?
So the first thing is containing the incident? Okay, we want to make sure that we fully understand where it is, and then how do we contain it, maybe just temporarily, so we can then put permanent fixes in with log4j, we’re seeing patches come out. So of course people are rushing to apply the patches and redeploy their software. Unfortunately, we’re still seeing that the patches have issues. And it’s just kind of repetitive cycle of patch patch patch, trying to stay one step ahead. I don’t think that’s indicative of every time we’re going to see some sort of third party component vulnerability. But something that is as mainstream as this is something that we would expect, potentially to see, because we’re trying so hard to fix the issue, that sometimes we overlook other issues that might be there. There’s now lots of eyes looking at this component looking for other issues.
So we want to make sure that we can contain, we want to look at ways that we could detect if this is happening. In the case of this, right. It’s making DNS queries out to other sites. So are there ways for us to filter and stop that type of egress out of our systems as a way to monitor to see if that’s happened? Do your systems log these events happening? So you go back through and look at your logs? To see, hey, this system is making DNS requests out to here? Why is that happening? That goes back to understanding what is the baseline for our network look like? We always want to have a baseline to know what does normal traffic look like. So we can see these outliers to be able to do that.
So we’ve identified how we can start mitigating this. So we patch it, we redeploy, we put in some egress filtering or whatever solution fix it, there’s none of these are solutions for log4j. They are things that have been mentioned. But I’m not supporting them as solutions or solutions. These are just things that we can do. And depending on the component that you have, most of the time is probably going to be we can patch if a patch is available, of course if a patch is not available that changes the game a whole bunch, right? Well, what are we going to do now? So we’re being able to identify how to do the technical fix, and resolve this and verify that we have fixed it.
The next thing we have to figure out is how do we communicate that out to not necessarily the rest of the world, but our customers, our clients, there’s going to be a lot of people that use your software or your applications that are going to want to know, hey, what are you doing?
Are we at risk, we have a lot of data with you. We want to make sure that we’re safe there too. So you have to figure out how that communication is going to work. And a lot of people overlook how that process should work. This is something that you should have laid out your plan, similar to an incident response not to say that this is a breach, but it’s an incident of sorts, it’s an event. What’s the response plan? Who’s involved with that communication back out?
We have to understand what is the message that we want to communicate? Obviously, we don’t want to just say yeah, we’re vulnerable. Sorry, this is bad. We want to be clear about what we’ve done some transparency, what the current situation is, and Message for that has to be done in a certain way, we really have to think that through. And so you don’t want just the security team writing a message and sending it out on the blog, that might not go so well. Usually, you’re going to see legal, you’re going to see marketing, you’re going to see a lot of different groups get involved with that communication to determine how we’re going to send this information out, to make sure that we’re not saying something that could get us in hot water later on.
But at the same time, right, you’re protecting yourself. But you’re also trying to be transparent. And you know, being upfront about this is what’s going on. And communication sometimes is the hardest part. Because people want to know are you communicating too fast, or you’re not communicating fast enough? Or you’re not communicating the right information?
There’s a lot of things that go into that communication part that while outside of our typical app SEC developer realm, right, as an engineer, building applications, that’s those other departments that are really dealing with that, right. As an engineer, we’re probably more focused on Hey, can you help us to find what the risk level is? Where does this impact what are the mitigations that are in place to give us that feel good of Yeah, we feel pretty good about we’re protected against this. So that really falls more on that security side, the higher level side to do the communication. But we don’t want to overlook that.
I know that was a lot of information kind of thrown out all at once. I know everybody is super busy with dealing with the log4j event that’s going on right now. It’s a week in and it still feels like it’s day one. So you know, applaud to everybody that’s out there working your tail off getting through this, hopefully, as you go through this, if this is something that did affect you, then there’ll be some sort of retrospective afterwards and be able to identify, Okay, what are the things that we did? To get through this, let’s document those down. Let’s work on building that up. So that way, if this happens, again, we feel a bit more prepared coming into it.
If this is something that didn’t affect you, whether you looked at it and said, Nope, I don’t have that I could do a search, I know what I have in my systems doesn’t affect us. Or you didn’t look, it just didn’t affect you still take this opportunity, this should be a chance to look back at the third party component piece and analyze, determine, is this something that we’re handling? Do we have gaps around this? And ultimately ends? What is the risk appetite that we have around this? Are we doing enough? At the same time, what we’re doing, do we feel that’s enough for the risk comparison to if we don’t do any more? When this stuff happens? What’s worse, dealing with it like this, and we don’t have anything in place, or the time spent to put this all together to be able to quickly handle this when it does happen.
I know at the beginning I mentioned it’s kind of like that 100 year storm, even though it happens much more frequently than that. But we never know when this is going to occur. We could see log4j this week. And we could see a whole other component that’s used everywhere next week. We don’t know how often this is going to be so it’s better off to have a plan put in place to be able to handle this as you build your software packages.
I appreciate everybody listening. I know it’s been a while since we had a podcast come out. I’m getting back into it. We’re starting to do some more regular podcasts. So keep an eye out for those. And you know as always, you know reach out you can go to developsec.com and see the blogs, the podcasts, all that stuff. If you have questions just reach out James@ developsec.com and we’ll catch you on the next episode
You have been listening to the DevelopSec podcast with James Jardine. To find out more about how we can help you with application security. Follow us on Twitter @developsec or check out our website at www.developesec.com