Time to Exploit is Cratering

Marc Laliberte  0:00  
Hey everyone, welcome back to the 443 Security Simplified. I'm your host, Mark Le Liberty, and joining me today

Adam Winston  0:08  
is Adam Winston, Vice President of Endpoint Security and Manage Detection Response.

Marc Laliberte  0:12  
Notably, not Corey, Corey is off traveling the world ahead of us, ahead of the APAC Watch Guard Impact Conference. So this week I decided to bring on Adam to talk about, or talk with me about, a couple of really interesting security stories that popped up that I think make some of our predictions obviously true for the year. We'll start with an overview of the GitHub breach that just got disclosed last week, and then we will end with a really interesting website. I found it kind of proves that the time to patch for vulnerabilities is now effectively zero. With that, though, let's go ahead and I don't know, hack our way in. First thing I want to hit, Adam. Just last Tuesday, I was browsing around, I think I was on.. I hate to admit it, actually, I am still on the site formerly known as Twitter, but was on Twitter and noticed a pop-up from GitHub where they disclosed that they had been the victim of a breach, basically they made an ex post saying they were investigating unauthorized access to their internal repositories, and their statement they said they had no evidence that any customer data was accessed, but and they were also monitoring for follow on activities, but they had someone had made off with over 3000 internal GitHub repositories. Soon after that, Team PCP and Lapsus posted a for sale link for what they said were 4000 private repositories on an underground marketplace. They were asking for $95,000 claiming it was stolen from GitHub, and then just on Wednesday, GitHub published this blog post where they went into a bit more details about what happened. They said on Monday they detected and contained a compromise of an employee's device that had been infected with a poison VS Code extension. This one was a rather popular one called NX Console, and they confirmed that the threat actors had made off with around 3800 repositories. So, pausing here for a second, for first hot take opportunity from you, Adam, this is a that seems limited to internal GitHub repos, like no customer private repos, but this is theoretically the stuff that runs github.com GitHub Enterprise Server, like their actual infrastructure. This seems like it could be kind of bad.

Speaker 1  2:52  
Let's start with, I mean, the disclosure first off, nothing to see here, no problem, unauthorized access, no big deal. And then the hacker releases a bunch of data, and they go, well, actually, now that you can see clearly that that's not the scope of it, you've eroded completely all trusts in us, you explaining that in the public anything about about this breach whatsoever. So even though we think it's the 3800 and it's not this and it's not that well, how can I take your word for it? When five seconds ago you told me there was absolutely no problem, it was unauthorized access, no data was was leaked, and then pow, there's there's a whole bunch of repo, so I now have to trust that the attacker is trying to sell everything all at once to determine the scope of the breach, that's where we're at in terms of disclosure, so which

Marc Laliberte  3:47  
is not always the case, to like we've seen time and time again, they'll withhold stuff and then post even more for sale later,

Speaker 1  3:52  
of course, why would you sell all the art you stole in one fencing deal when you have, you know, the special scepter you could save for a real private buyer, you know, down the street, so this, this is just what they want to sell to the public. Let's be completely clear here, right? They want to sell this to any buyer on the dark web that'll post the money for this marketplace. I forget what the dollar figure was, I think

Marc Laliberte  4:13  
95,090

Speaker 1  4:14  
$5,000 So this, this is what they think is worth $95,000 What could be worth millions would be repos that they had access to that they're not going to try to sell for $95,000 on the dark web, so my first position on this is, you know, at some point these disclosure rules, and like what companies are allowed to just get away with saying, I think when it comes to stuff like this, especially when we've now, you know, a blog post, like you know what I mean, like this is

Marc Laliberte  4:43  
a Twitter post, not even a blog post, and then followed by a small blog post,

Speaker 1  4:47  
Twitter, then a blog, it just seems so amateurish, right, like I think at this point if we talk about disclosure in these, and we're what we're going to talk about, probably for the rest of the podcast was how much more. Frequent, and how much more impactful these types of things are going to become. I mean, for researchers and for people in our industry that are kind of responsible for explaining these things in detail, this level of disclosure is certainly kind of amateur, right? Like, I, you know, the fact that we had to be on Twitter, the fact that this blog post, the fact that everybody's being so cryptic, this, these things are getting swept under the rug when they have material implications for the software developers that put their repos in GitHub is a bit of a laugh, right? So, like, you know, we, I use GitHub privately, is you know, as my GitHub private repo, one of the ones that were affected, did I get an email, did they explain to me what was going on? Was there a press release? So, you know, the idea that the public's awareness of this problem is something that's just so callously and flippantly tossed to the side. I think is my first take on this story, and frankly, like to say this is a special case. This is very common for big software companies is to pretend nothing happened, and I think back to the Open AI, you know, breach that distillation, and that concept of, you know, what they got through the back doors in Azure. I think back to, you know, people that we kind of, the altruists said, you know, anthropic, that tried to say, you know, hey, they were, they were using our model to do bad things with these government agencies, right. We're almost relying on the benevolence of the company and it's in its organization to decide whether they're going to be responsible with this disclosure. So that the idea that we have to trust the company based on the culture and that there's no regulation that kind of holds people to a standard here, I think, frankly, is one of the major things that we need to learn from this, and stories like it, is that it's Microsoft we're talking about, right. GitHub is a, is not just some small startup that's, you know, protecting their IPO, or whatever, right? Like, you can come up with all the reasons that there's not responsible disclosure for other companies, but for Microsoft, right. This is one of the things we talked about, even when we were doing the debate with Corey, was, you know, is Microsoft really committed to finding bugs and explaining them, and you know, building open source library, and it's like this is a great example of how they are not right. So that's my, you mentioned

Marc Laliberte  7:11  
you mentioned the regulation piece on this, and I think there is like some good news if you are like on that bandwagon, like with the Cyber Resilience Act coming up, starting in what is it, June, July, August? In four months, any company that puts software into the European Union, so Microsoft with GitHub Enterprise Server, for example, there are strict requirements to report any security incident that could have an impact on the security of the product within 24 hours to EU regulating authorities, and then within 72 hours, send a like detailed report, including mitigation guidance, and then within, I think it's like 14 days of resolution, like a full debrief on exactly what happened. So I, I'm in the opinion that initial 24 hour report will be a lot like the one that we just got on Twitter, of, hey, something happened, we don't know yet, like what. Stay

Speaker 1  8:00  
tuned, and

Marc Laliberte  8:00  
then that, yeah, that 72 hour report will need to be a bit meatier than the blog post that we got, because it's supposed to include, like, here's what the impact to you could be, and how to mitigate it, like, because presumably I'm going on a limb and saying one of those 3800 repos is probably some component of GitHub Enterprise Server that people like Watchguard manage on prem, like, I'd want to know, like, what is their, what is the potential exposure to this beyond just, you know, oh, they could find vulnerabilities, like, is there any, I don't know, like cryptographic or authentication component that is reliant on secrecy and stuff like that, are there any secrets that were exposed that we need to take care of rotating, like, there are, we need, like you said, I think more info in four months. They're going to be required by law to provide that info or get sued that high heck if they don't by the EU regulators.

Speaker 1  8:52  
Yeah, and if that incentive is going to drive the behavior to say even something as simple as, do I believe that you only knew as much as you published in the first 24 hours, I do not right, and so I think that as these cases go, you know, forward, when it comes to the regulators having an ability to find and maybe potentially, you know, do forensic examinations on these, I'd love to know what they actually knew, and at what time, because there's something about the whole like what you have to do in 24 hours, and it's what you could have done in 24 hours, and frankly, what you could do right now, right? And so the fact that we've been kind of careless with either, you know, the investigation and the timing and releasing information that they're like, oh, we're not absolutely, you know, necessarily certain that this was, and then it's, you know, in the days that follow, you immediately realize just the full scope of almost exactly how detailed they could get into which source code repository, which library was affected by which developer, by which per, so that you know there's a really good trace here, and it's like, well, if you had all that trace, you could probably very quickly have come up to the same conclusion and disclose. Was this really, really quickly, and even now that we're sitting here with the number 3800 you know, is almost like a class action. If you were affected by, you know, it seems to be that you could have been one of the claimants here in this class action. You know, is that Cyber Resilience Act going to start including enumeration of the actual affected parties and helping them, or is it just going to be a fine for failure to comply with the procedure? Right. So, you know, I want to see more tie-ins back to the people that are actually at risk here. You and I both know that you know how it's much, much easier to find a vulnerability in source code when you have the source code, and so these packages, this, that's 3800 source code repositories. The analysis you can do now with, with a model on finding vulnerabilities at scale, which I'm sure, again, we're going to talk about more in a couple minutes. That's a huge deal, right? Like that, the potential ramifications of losing the source code now they're huge, right. And I go back to, you know, Orange Psy, and the Exchange Server, and the on-prem SharePoint to the world. When you let go the source code, I mean, it was like months between the next zero day and exploit, months between the next zero day and exploit, and they just, okay, move to the cloud, so we can patch at scale, or probably hide the problem, right, which is, you know, the other thing that I think you know probably happened there is that everybody moves from Exchange on prem to in the cloud, and now are we even hearing about the source code vulnerabilities and the breaches that are happening in the back end. So this idea that we just keep shelving these problems and trying to sweep them under the rug without being kind of fully disclosing what's going on with the with the public and those affected. Yeah, I'm hoping in a couple months it sees a different way of doing things, because we're certainly going to keep seeing this, and it's not enough for people to just say, well, I guess my credentials were taken from GitHub. It's not your credentials, it's your source code, right? And the implications of that are much more wide-reaching than some password stealer that was working on your repo, right? So, even when we talk about the fix, we're not even sure we can kind of diagnose what the fix is, because we don't even know what kind of problems we just created for these people.

Marc Laliberte  12:11  
Let's, um, like getting a bit in the weeds of like how this actually happened too, because I think that's even more interesting when we look at that. So, they mentioned it was a compromised browser extension or not browser, a VS Code extension, so one of their developers in their, their IDE had somehow received a malicious version of NX console, and that is what ultimately led to them getting their, their credentials stolen, and the threat actors, those to then scrape whatever internal repos that developer had access to, so NX console, they posted on Monday the 18th that they had suffered a their own supply chain compromise that lasted about 11 minutes from 12, or I guess 18 minutes in total. So, from at 1230 a threat actor, someone published a malicious version of one of their extensions on the Visual Studio Marketplace and the Open VSX registry, so two different locations to get extensions for your VS Code IDE. They basically noticed that published email that got sent, and within 11 minutes they responded and submitted an unpublish request, and it was taken down. I think, like, 18 minutes later, in total 36 minutes for Open VSX, because presumably they forgot to look there for a second. This malicious extension basically dropped a Python back door onto the local machine called cat.py located in the kitty directory,

Marc Laliberte  13:41  
totally

Marc Laliberte  13:45  
unassuming,

Marc Laliberte  13:46  
just a kitty cat.

Marc Laliberte  13:49  
Yeah, yep. This was a basic, like, backdoor and info stealer that, and would go scrape secrets in a bunch of different locations, so the local file system, one password using the like one password CLI AWS, if it had access to Secrets Manager, because this is an extension and a like development IDE, and those tools have access to a lot of stuff. Just like by default, when you are logged in, you want to be able to get to your One Password file or Secrets Manager to get secrets out of there. You want to be able to get to AWS Secrets Manager, because hopefully you're not checking it directly into code, but you probably have some secrets or tokens stored in like local files too, and this thing would just scrape all of them, and it had the permissions to go do it, and the interesting thing is they said the root cause for this one was a one of their developers, so one of NX, one of NX's developers suffered their own supply chain compromise too. Basically, there was a third party, now Tan Stack, which had their own supply chain compromise a couple weeks ago, where they had, I think, it was like 84 malicious versions of 42 of their packages. Uploaded the developer installed one of those packages during like the 30 minute window where it was online that stole the developer's credentials, then with that, like the threat actor was able to upload the new version of NX console, which then the Microsoft or GitHub developer downloaded. This is like a, like a three stage supply chain compromise leading to GitHub's infrastructure code getting stolen. It's pretty nuts.

Speaker 1  15:29  
Yeah, it's kind of like that rhyme. It ate the rat that killed the cat that I kind of.. I was expecting something around the kitty cat to be like, well, actually, this is a this is a compromise that was part of a chain on node, node that gets to tan, tan that gets to me, me that gets to you, GitHub that gets to the source code, and then around we go again, but yeah, I mean, if we look at the cascade, if we look at like the steps here, a lot of it is around packages, uploading packages, and the open source community, and we had a, I think you know, this concept of the supply chain attack, and these huge, you know, vast, wide-ranging source code library and review processes that are done by people, and so, you know, at the core of this is people can't QA packages at scale in the open source community, it's a volunteer exercise. They might have some funding. If anybody's worked at a nonprofit, you know how hard it is to get funding from some for some of these corporate, for some of these systems, right? You know, even if you're backed by some of the biggest philanthropic donors, you're still under-resourced in terms of trying to build this out. It's not commercial software, right? So, the whether you're the Linux Foundation or whether you're Node or whether you're any of these guys, the idea is that you have hundreds of 1000s of packages that people can develop and good, that's what we're doing, we're crowdsourcing software development to mostly hobbyists, and then they get published to a small group of reviewers that come in there, it would be interesting to know from these teams, you know, you're talking about 11 minute windows of like recognizing a published statement, and then pulling it back. You say you know there's 40 some odd compromised packages from Tan Stack that went in there in the first place, and so, like, out of 80 or so, it's like, okay, well, almost half your packages are compromised, but then you are part of even just one developer, so you could, you could see how the problem is too vast to kind of clean up at the reviewer level when it comes into these repositories, and then as the repositories filter in, even though it's 11 minutes, seems like, okay, well, why did you let it publish in the first place? Like, you could fault these guys to say, like, why wasn't the review done pre-publish instead of, you know, after and pulling it back, right? Like, but I look at it to say, well, if you had to wait for every package published, what would happen to things like bugs? What would happen to things you know, like you can't slow the entire system down in order to do, you know, a thorough review of everything, because we almost

Marc Laliberte  18:00  
also like this is a really difficult spot, too, where, like, I feel like we've collectively done a decent job of, like, hardening the workstation itself, like everyone - not everyone, most like mature organizations will have, like, good EDR running on the workstation, which would help in this case, too, but there's still a few areas that are, like, the wild wild west, like browser extensions, for example, like it feels like, like the mobile app store, like a decade ago, of just you don't know what you're going to get, or if it's going to get compromised, then a malicious update pushed out later, and then now, like extensions for IDEs, which are really powerful and really useful, like I've got probably 20 installed, but that's another big, maybe not blind spot, but overlooked area. I feel like for a lot of companies,

Speaker 1  18:44  
well, I mean, let's see it getting worse. So, like, if you look at the evolution of the IDE as well, right, and we move to a very popular program like Cursor, right? And you say, okay, is this thing dynamically loading its own extensions? Is it dynamic? It's almost like an acceptable use policy, which says, would you like me to load this library? Would you like me to load? I'm having a problem creating this JavaScript thing. Would you like me to download Node and put

Marc Laliberte  19:05  
cursor being a an AI development platform for those,

Speaker 1  19:09  
so the AI version of the same, and so I, you know, is the problem going to get better or worse, right? Because I think there was this old mentality around developers and engineers that, you know, they thought they knew better when it came to hygiene than others, or that they were much more, you know, technically savvy. So, when they were loading something like a package manager, or, sorry, a library, or a plugin, or, you know, some other extension into their, into their environment, that, you know, oh, this is trusted, I know exactly where this is coming from. I didn't get this from some, you know, shoddy repository. I didn't sideload anything, so like I've kind of followed my practice, but I'm almost always in experimentation mode, and so if I look at the way that the IDs and developers tend to function, if they see a plugin as being able to just quickly fix or do something, add something to a prototype, to a beta, something else that they're kind of playing around with. But the thought process isn't to do a lot of QA at that moment, right, and this is something that's totally been captured by the Vibe Code AI IDE, is that cursor will do very little reasoning before it immediately starts downloading other packages and immediately starting to use them, because it just figures while you're trying to get me to my objective, and the reasoning is that if I've scoured the internet and looked for packages that could do this, instead of rewriting the code, the more efficient thing to do, download the extension and just start using it, or download the plugin and just start using it. And so I think that, like, at scale, if you think about how development is evolving, and we say, okay, 25% of code is now written by AI at these larger institutions, you're starting to see the wheels come off in real time, right, because if this attack, you know, scenario was to be replayed without human reviewers of any kind, and the AI didn't reason that they needed to remove the package for security reasons at all, then this just keeps happening, you know, everywhere. So I think that if you watch the velocity, not just from the package and the human reviewers and the number of people behind these programs, and how difficult, difficult it is to find this at scale, and how useful these things are, but just how even the IDE is evolving to accelerate this problem. It is going to get worse.

Marc Laliberte  21:14  
Now, let me put on my optimist hat on this one, where there's clearly like it is going to get worse in many cases, but I feel like there's a big opportunity here for AI within software development to like close a gap that's probably wide open for many organizations, like the most mature of software development companies. They'll use like pinned versions that are hosted on their own internal like package index, they never come straight from like PyPI or NPM, they do validation before it ever becomes a part of their code, but that is not the case for like the other long tail 90% of other software engineering firms, and maybe this is somewhere where AI can help, where like before including a package, it gets a like source code review from an AI agent to then determine, is this malicious or not? Before it's allowed to run, we're not there right yet. That obviously isn't free, that costs tokens, but maybe that's another great application to kind of like use it for good to help out.

Speaker 1  22:11  
Yeah, I think about what we're really saying is that we think we found a way into the enterprise, right? I think I think this model for saying even within a company as large as Microsoft, even in software development houses as sophisticated as I'm sure a lot of these places are, that the source code that is based on a huge undercurrent, like a huge underneath the surface, a whole bunch of open source that even if I was to review each package or each library or have something that would pin, I still think I just need the one guy, right? And if I come back to this environment, who's allowed to break the rules that allows a package like this in, and who's who's the most likely to break the rules because they're experimenting is still developers, so I think attacks like this still come down to this critical one man in a room and I was I was listening to Krebs and the CISA version of Krebs, not the journalist at Black Hat, and he was saying, well, there's so many things in cybersecurity, and still in it, that still come down to one guy. There's still one guy who could technically change the root DNS config that's running for all of us to poison it, right? There's still potentially one person running a BGP node somewhere that could totally cascade an issue for the internet, right. And so, you know there's this concept of like there's still one guy that actually works to review the packages at Node that technically could fill it with poison and let us all down here, so we're sort of relying on still this kind of system that has in it built this idea that like those people aren't going to be nefarious and that we are all going to be careful, and that it just takes one compromise, one individual to turn, and there are just the checks and balances in the system fail. So, I, I take your point 100% If we had AI, or some sort of like non-corruptible entity that could take control of this package management system, and do a very thorough review on what the packages were doing before they got to the IDE, and frankly, hopefully with the AI IDEs like cursor, could you take this model even further and say, like, you're not allowed, there's going to be something policing you, and I wonder how far away we are from that future, right? Like, how many more examples of this has to happen, and who's paying for it that forced yet another cost on here, and then it's like you said, that costs tokens, and we have to be very clear here. If this was a model that was going to persist, it's a pay to play model, right? There are not going to be the opuses and the mythos is of the world that can find vulnerabilities and exploits at scale and deploy them against source code. Repository packages that create the zero day army that then are thrown at everybody and everybody somehow is going to have the money to have a, you know, token analysis engine for every package that they download and use it, right. The the open source is basically the democratization of software development, and if we have money that's going to be the delineation between secure and not secure, then there's going to be a lot of companies that are going to keep suffering this fate because they just didn't have the resources to stop it.

Marc Laliberte  25:29  
Now, I mean, some silver lining to this, like if the developer machine was running something like, let's say, I don't know, Watch Guard Endpoint Security 360 or just any solid EDR that wasn't put into audit mode, because that developer demanded it, because it was, I don't know, blocking their work, like this is the exact type of thing that it could have caught, a malicious extension dropping a Python script that goes and scrapes all your secrets, like regardless of how it got delivered, that's like red flag, like a red flag Costco, and any tool could have gotten that bad.

Speaker 1  26:02  
There is, I mean, there is a, an outsized reliance, I think, on the endpoint team to find and stop all manner of cybersecurity threats. This is easily one that we could have found and stopped on where it happened to, at the developer level. I really question how, you know, a developer sitting behind the GitHub, you know, private, you know, reposit private code is sitting there without an EDR that could have found an info stealer to grab the passwords. That's a really important, you know, a moment to be like, well, hang on a second, anybody should have been able to detect at least the info stealer part of this. Why did that get past your ED? Was nobody looking? Was there some MDR missing? Was there some component missing? Was it not up to date? Was there no, was there no EDR on that on that thing? And you know, at one, at one point you can say, well, is that not just obvious that you should do that, but I go back to how many architectures failed to deploy EDR in the first place. So, like, yes, we could capture info stealing, and most people see EDR as a desktop problem. I talked to a lot of customers who forget to put it on Amazon Linux, who do not put it on Linux at all, right, or who leave it off these kind of headless environments where, yes, it very much does run there and does the same job. It's still an operating system, if even if it's running headless, it still requires this type of thing, but they mostly think of the anti-malware as this relationship to the user, the pop-up. There's this kind of like learned behavior that anti-malware is to be put in front of your end user and not the headless containerized things that we have running in the cloud. So, I give, I give that, you know, a little bit of space to say, hey, you know, that definitely could have stopped it. Is it in the place where people who have these CI/CD pipelines that are running through headless containers or otherwise, do they see that as a place to put anti-malware? And in many cases, right, you think of, like, the Kubernetes, or these other like micro headless containers, like serverless containers, are they able to run anti-malware? And many of them are not right, hypervisors in many cases are not, and so I look at it to say, and you and I, I mean, when we leave on, you know, Saturday, Sunday, did to hit Asia and show people this full court press idea, is to say, well, there's going to be a lot of examples of places where EDR can't function, so you need a whole platform around it just to be able to to find and stop it, I mean, I think part of the reason to even here combine MDR and EDR is a very real understanding of where EDR can play and should play, and what happens if it's not there, right? What happens if it can't go there? What happens if it's not on the Android TV behind me, right? Like, and so there's so many different versions of this exploit that if I say all right, well, at this particular moment, it could have been stopped, but there are many different things that should have stopped the info stealer from going to some command and control. What allowed it to upload the credentials, right? When it went to use the credentials, we're talking about pass keys and all these other methods for changing credential management from a, from a two, from a key value pair into something a little bit more difficult to impersonate or to copy. Why is that not implemented for these resources? So there's so many things at WatchGuard that we have done to modernize, and it's still impressive which organizations have so much of this tech debt in deploying that across the system that it does, it does kind of give you a little bit of, like, well, this was entirely preventable, right? There was a lot of very basic things we could have done that would have, would have avoided this, even if the problem does seem so wide scale, right? Like, there shouldn't have been a zero, there shouldn't have been a gap in zero trust that allowed that many credentials to be sitting local in a repo, that the form of the credential taking as these kind of just secrets, key value pairs that the info stealer on the endpoint, the control channel for the attacker, somehow they were able to get out to the control channel. I'd love to know what happened with the control channel there. And then, as it's pumping its way into the repos, you know that's kind of the last mile that we're hoping the CI/CD pipeline is protected by AI. Or that AI isn't just going to circumvent that to try to get the job done faster, and you know, we're sort of saying, okay, well, that last mile we still need to talk about in this particular use case a lot, but all the other stuff still applies, and so you know, I look at it like there's a lot of lessons learned here, and I still feel like we're not learning them, even even with the predictions, even how many times you can beat it into slide wear or into conversations, you know. I really want to understand the hygiene posture here. Why were none of these zero trust things implemented? And every panel I get on, when I say the word zero trust, all the other people that are our peers in the industry, the other people running socks or telcos or whatever, they kind of have this collective growing, like, well, we're never going to get to zero trust, we're just never going to have

Marc Laliberte  30:45  
attitude,

Speaker 1  30:47  
yeah, yeah, not with that, and that's exactly what I say, I'm like, not with that attitude, but it is exactly what's required here, especially as the velocity picks up, because let's be clear, there's a lot of signals that were produced in this attack, there's a lot of artifacts, right? What if there wasn't? Then what? What if we didn't have a log for the info stealer? What if we didn't have, you know, the extension with the calls coming from outside the house with this poisoned extension? What if it was a takeover of a natural extension that actually was trusted and published and wasn't phoning home to a c2 Right, let's say it was just an actual convincing of the AI code that it should do this as part of its job, and it's just embedded secretly inside of it, and it doesn't produce any records to tell us about it. That's the world we're about to be living in, right. And so, you know, the traditional model for us, I think, will take us past this level of negligence, but I think what's coming is even more terrifying.

Marc Laliberte  31:44  
So, speaking of like what's coming, and even more terrifying, and running out of time to respond to anything, I want to move on to the next story, because I feel like you are the perfect person to talk to about this. So, the other day, like, I stumbled across this website called Zero Day clock.com.com and I thought it was super interesting for a few different reasons. First off, basically they've got this methodology where they're tracking the what they call the time to exploit, basically in the seven stages from when a vulnerability is introduced into code to when, like the attacker has, or to when the defender has it all patched. There's two stages in here, from when a vulnerability is published to when it is becoming actively exploited out there. So, the purpose of this project is to, for all vulnerabilities that are under active exploitation, which is a very small fraction, actually only 3500 out of the 235,000 CVEs that have been published since 2018 for all of those, what is the mean time to exploit from when it is disclosed to when it's under active exploit? They also, so they've taken sources from the CISA known exploitative vulnerabilities catalog, and then Vuln check, as well as another one, they've got some like statistical filtering in here, and they also track like zero day, meaning when it's under exploit before something is published, and the results were were damning. Where, if we look back over just the last like decade, almost a decade here, that mean time to exploit has been just absolutely cratering. Where back in 2018 it was 2.3 years, 2021 it was down to 10 months, 2024 it was down to 53 days, now it's down to 1.6 days, and they're predicting that if that model trends, they'll be down below an hour, potentially even a minute by 2027 so before even going into like other stats that are really interesting in here, like this alone, like a attracts this is what we've been talking about, with especially AI-fueled vulnerability discovery based off just like whatever is included in the CVE. Go hunt and find the actual issue and weaponize it, but then be like, man, less than a day now, or almost less than a day, this is, there's not enough time to patch stuff, like I, you cannot patch your entire fleet that quickly without, like, the best patch management capabilities that exist. So I'm curious, just on this first bit, Adam, is this a case of, like, I told you so from you, or what else are you thinking?

Speaker 1  34:17  
Well, so I've been arguing this position for the better part of 10 years, and I tried to see when you know the larger description of the problem I could refine to. I don't have to describe the problem, so if I kind of like go on these little tours or talks, or I get an hour with a customer or a partner or a software developer or something, I say, okay, well, of my one hour, if I have to explain the problem for the first 30 minutes, it's not really a problem, right, because I'm talking to people who don't recognize us, right, and so if you look at the format of even our discussions, right, at least that part has shrunk to just like the top two, you know. One slide now, or we're into like two minutes of describing this specific problem, and you go back to everything from Moore's Law, right, the idea that compute will grow geometrically, right, the idea that human productivity, right, will grow and innovation will grow geometrically after a certain point, right, the idea is that this curve represents what looks like a geometric, you know, arc, right? A very doubling on doubling on doubling year over year. And I think that when you're sitting in the moment, when you're looking at what period you're occupying, it doesn't feel like it, like the distance between the first iPhone and now doesn't feel like, as you know that different, but it is right. If I look at the difference between 1910 and there's nobody in the air, and there's, you know, 8 million active people in the air at any moment in time now with flight, right? It doesn't seem like at any moment that we've lived that it's been that geometric, but it has been right. And so this is another, you know, moment of just us being able to, and there's all kinds of really weird descript, you know, the frog in boiling water, whatever. Did we realize it was getting this bad? And so, if, if you were to do the top level analysis, my top level analysis, and many people that I had to convince, you know, in the venture capital community to fund Act Zero, this was the concept, was to say at some point in time we were going to be beyond the idea that we can patch something, and we're going to need artificial intelligence that can code preventions at scale or responses at scale, and because there's no other outcome, right? And so, if you just game it out, you do game theory, say, okay, well, let's play this out a little bit more. If I said, okay, there's whatever in Windows 10 or 11, like 50 million lines of source code, and for every line of source code, or every 1000 lines of source code, there's a critical vulnerability that we assume is just sitting there, but it takes a researcher, maybe you know, a few weeks, or maybe a month to find it, and then it maybe takes a QA person at the vendor maybe 20 or so hours to fix it right, as long as that cat and mouse game on the time between the vulnerability discovery and the exploit in the community is somewhat aligned from the same order of magnitude, we're okay, but let's just fast forward and say even if we could fix it at scale, so even if I could take 1.6 days from active vulnerability discovery to active exploit in the community and I could shorten it to say, well, okay, well, let's cut that in half, maybe 8.8 days, point eight days is all I need to be able to create a patch and distribute it. Let's talk about next year, or let's talk about the next six months, right? This is why we said autonomous sock by 2027 because this curve, just any data person would tell you this curve, if you just follow it along, follow the trend, we know where we're headed, right. We're having, and we're having, and we're having again, right. And so, if I say, okay, we're halved to point eight, let's say we go even a little bit further, because I feel like we leapfrogged a little, so it's a little jiggly on the curve, it's going to jiggle downwards, right, because we all know the models, and we use them every day. We know what this is going to do at scale. We can find a ton of vulnerabilities with these new models. Okay, so let's assume that I'm going to drop it to, let's be conservative point three, right? So I have a third of a day, right? I have eight hours between the discovery of a new threat that is being exploited in the wild, and I've developed a patch, and now I have to distribute it, right? So let's talk about distribution. Every time you upload or change an app on the on the devices that you have, it takes one to five minutes to produce a software update. That's one to five minutes you cannot use the app. One to five minutes for every eight hours, if you were to even just say across the apps that you're using, let's say it's an average of 100 apps, they're not all going to update at the same time, so how usable is the patch concept if all your software is just updating and downloading and creating dependency issues literally all the time, several times a day, right? It's going to be a lot of network, and even if the network increases in compute, and even if your devices increase in compute, and even if software, which there's no sign that even with all this doom and gloom about the SaaS pocalypse, that you're going to be using less software. It feels like more, right? You're going to have

Marc Laliberte  39:08  
more,

Marc Laliberte  39:08  
and like, and worse develop software too, because it's going to be vibe coded into

Speaker 2  39:12  
existence

Speaker 1  39:14  
in single, the most single use ever, right? Because if you don't have to interface with the app, it almost doesn't matter what it does, because the AI is just using this single use nonsense to get us done here, right? So, if I said, okay, well, that just increases the app scroll, increases the source code flood, increases the vulnerability, the attack surface, the update schema. Just imagine this, this rolling updates all the time, you'd say, okay, well, then obviously the way to solve this problem next year is not patching, it has to be something else, right? It has to be something that the software can remain relatively the same, or it can buy it some time to stay in that exploit window, while having detection engineering roll out something at scale that finds and blocks it, and so if you look at the R&D that's that's running behind us with the endpoint. 18, and I just came back from Spain to talk exactly about this problem, and you know, kudos to them. They're very advanced in terms of what they're able to do with agentic AI to find net new exploits at scale, turn those into detections, and now the problem becomes, how do you squeeze all those detections into the form factor that we've allowed for cybersecurity? Right, and this is the same. It was the same problem we had with the next generation firewall, or UTM. We had to split this thing out in five boxes, and it wasn't because it was hard to consolidate them. It's because the compute wouldn't allow you to do IPS decryption and firewall stateful inspection and URL filtering all in one appliance, right? It was a compute problem, right. And so you know, I think for cybersecurity, what that means is that we're going to have to rely on, hopefully, compute to level up another notch, because if I shoved every single one of these detections that we know we could use to find these zero days at scale into the endpoint, it would just crush the operating system, right? It would be, it would be too much load, right. And so, so now we have to solve for that problem, and there's a lot of tools at our disposal. Thank goodness. Right, there's the idea of distributing a lot of these things to the cloud, but that leaves us a round trip time in the several seconds to go up to the cloud, do the analysis with hyperscale, come back down, and do something on the endpoint. Right, we still want that microsecond inline block time, and so I do this thing now, where I talk about, like, if it was the internet we set up in 1996 to develop the world's first like firewall and stop detections at scale, because we needed to filter that wire, so it would still be usable. We have the same problem now as a platform, and all these applications is getting between the application and the adversary, or its exploit, and making sure we can do detection engineering at scale. What that's going to require, though, for right now, it's not like we don't know how to solve this problem. It's not like we don't have the tools to solve this problem. We are dealing with not a detection problem, not a response problem, not a platform problem. We're dealing with a scale problem, we're dealing with a compute problem, and I would argue also a commercial problem, and so the reason I put on a suit jacket, get out of the lab, and talk to people all the time about why they have to buy all the WatchGuard products isn't because it's just great for WatchGuard if you buy all of our products, it's because that platform is what the customer actually needs just to have the pieces in place to stop this, and the GitHub attack is a great example, we talked about the control channel, the identities, all the things that went wrong on the endpoint, right. Frankly, you know, the logging and disclosure that most of these companies, frankly, probably don't even have. If you're Microsoft, you might be able to do this at scale. If you're going to go to the Resilience Act and say it's every software vendor, what about the little guys who were just hacked in that 3800 or are they going to have the ability to do this? No. So we have to democratize it by flattening that, bringing down the unit cost, selling the problem is one problem, and then we have this issue where, okay, now we have to really scale it up, and we have to find ways to do that affordably, and this is when it becomes this is the challenge we need to meet in the next six months. It's not that we don't know how to solve or stop this problem. It's that we're going to need a form factor to do it in, because if you gave me all the resources in the world, I would load all the iOS out of the endpoint. Nothing would ever happen on the endpoint, but your CPU would be plagued at 100% So, so that I

Marc Laliberte  43:13  
mean, that seems like a legitimate strategy. Malware can't run if there's literally no CPU cycles for

Speaker 1  43:19  
it. I'm the malware now. Yeah, if I'm going to cause Sev ones by shutting down networks and endpoints, I don't get invited to the party, right? And so this has always been the problem in endpoint is that we are covering for the lack of a patch or the lack of a capability to avoid the problem through through preventive prevention and hygiene, and so we're going to have to continue to live that life, it's the same game, but it is going to be a compute game in my opinion,

Marc Laliberte  43:45  
and there were a couple other stats that were really interesting on here that I wanted to go over real quick too, where one of them was they're tracking the zero day rate as well, so the percentage of CVEs that were exploded before they were disclosed, and that has been skyrocketing too, like it started at 18% or so back in 2018 and it's now up to 73% Basically, almost three quarters are discovered after they are already under attack. And then also, interestingly, they're tracking the exploit rate, so just what's the overall percentage of vulnerabilities that are being exploited, and that's plummeting, and I think mostly because the volume of CVEs that we're getting, the volume of vulnerabilities discovered, yeah, it's blowing up, and so like we're in this weird situation where like defenders have to be able to spot like the true risk from the noise and then respond to it almost instantaneously, or in some, like you just said, like this is where other controls have to come in and cover up the slack, because vulnerability management and patch management is just like it is a going to be a struggling industry in the foreseeable future.

Speaker 1  44:54  
Well, it's a new tool we're not talking about, but even in misinformation campaigns, people talk about flooding the zone, you never know which. Story to pay attention to, especially, or to fact-check each one, and so it's the same thing for researchers, right? So, if you look at that, that second graph that basically says, look, we're going to, we're going to find a whole bunch of vulnerabilities at scale, and we're not going to use all of them, and if the idea is that you have to pick one up off the shelf to determine which one you're going to code into the IOA, you can't play the game of saying, well, I'm going to have to guess which one they're going to exploit next or pick which one, right? I have to load all of them, so technically, again, this comes back to the compute problem. They could fill up my stack with potentially erroneous, never used exploits that could be used legitimately. And then it's this game of saying, well, you never know which weapon I'm going to use, so you never know which defense to put into the stack up top. Zero day, this I think will change to, and we had a beautiful word for this, was with zero day, but the idea that that's the new signal, right? Like, if zero day is the new signal, is that the way that we used to be, you know, informed of these artifacts was through some sort of, you know, sigma rule or IOC, sticks, taxi, something, some sort of way of saying we're going to codify what we know to be an active exploit or what we know to be a vulnerability, and so you can look for it and fingerprint it this way, and then we're saying, well, things are moving too fast, that the actual fingerprint is just people being actively exploited, and then what does that mean? It means that some reporting of this behavior is happening based on the outcome, and is it based on the actor telling us that they've got our data that tells us that outcome, and if I was to truly be benefit of the doubt with this story around GitHub, I'd say the actual only indicator of compromise was the attacker uploading stuff to the dark web. If I was going to say, look, they were totally honest with us, there were no indicators, they did you know all the thorough research. There was no trace, and then all of a sudden they found out the data was gone, and the attacker told them how it went down, right? That's, I think, what that first graph represents. And so that 70% is actually much more alarming than the other one, because the other one is a solvable problem. The first one is, it's again dependent on the adversary's honesty, right, or forthrightness to tell us that we're being exploited, so the lack of a signal, and this is also true if people go back to white papers from Anthropic around mythos, and this idea of exploit chaining that worked without any, without any trace, right? This sort of sarin gas, there are no send senses that are going to tell you that there's danger afoot, because the model knows enough to say, well, I'm going to do something that actually doesn't create a log or an entry or any kind of change to the operating system as I cascade through it, and then I can be totally invisible, right? And so the idea that the zero day really is kind of like the distance between somebody noticing there's a cyber attack and not the researcher noticing there's a signal or finding there's an exploit or something, right, that could be a new reality, and we, I don't think we have the language for that outcome yet, the traceless version of a zero day, or the outcome of the attack being the only signal, but I think we're very quickly moving towards that reality, and this is what terrifies people most, I mean, when we go through the conversations, right, it's a simple understanding to say, okay, well, if we see this, we have to do that, and we're going to see more of this, or we're going to do more of that, but if we say we can't see any of this, what is, how do you protect yourself in a world where there's absolutely no signal, and I mean, so far our response has been batten down the hatches as if it's constantly happening, like assume you're breached, right, this was always the original tagline for MDR. Just assume they're in there and act accordingly, right? Like, what would you do if every single network connection was a compromised one? What would you do if every single program was, it was a compromised one? What would you do if every single file was potentially accessible? Right, you would, you would do a lot to reduce its existence, right? You would do a lot to say I'm not going to allow that many different things to manifest at the same time, and I will just natively shorten my probability that this is going to be an outcome that's a disaster. Right, so you know, getting rid of data, getting it offline, right? Then to move you with Mission Impossible, they move back to paper, right? Everything's analog, nothing's digital, right? And the idea is that they're sort of removing their trace from the digital world, so that it doesn't become affected if they can, and so zero trust is kind of like an early stage version of that, is saying like we're not going to let anything operate the way it had been, because there's it's just too risky to do that and cut back a lot.

Marc Laliberte  49:17  
Yeah, man, it is certainly a time to be alive right now. I don't know if it's a good, exciting, bad, stressful

Marc Laliberte  49:26  
world.

Speaker 1  49:27  
I feel like we're all shoved in the.. in this is why it's the greatest time in history to be a cybersecurity person. Yeah, and I will tell you that the problem has never been more real. It's never mattered more to people, right, because of how much we've built on top of this technology, and we have, I think, something truly difficult to solve, which, which is, I mean, and it's certainly not. It's a regrettable time. So, it was Oppenheimer that we were going to use things for that purpose. Same thing here, right? It's a regrettable time that we have to do this, because we're under attack. But if we go back to great moments in history, where we've had. Really powerful innovation. It's moments like these where we're just, we're going to be squeezed into the pressure cooker to come up with an answer of how this is going to work, or none of it does. And then we lose out on a tremendous future for everybody.

Marc Laliberte  50:14  
Yeah. Well said, Adam. Well, this has been fantastic. I'm even more stressed out than I was before we started talking an hour ago, but I do agree that this is like the best time to be in cybersecurity, because there is some bright future in silver lining for a lot of the defender capabilities. But thank you for taking some time out of your day to hop on here and give some of your insight and famous hot takes on exactly what we need to and need not be concerned about when it comes to some of the recent trends. Thanks for having me. Yeah, so hey everyone, thanks again for listening, as always. If you enjoyed today's episode, don't forget to rate, review, and subscribe. If you have any questions on today's topics, or if you have suggestions for future episode topics, you can reach out to us. I'm on Blue Sky at it's mark.me for all on Instagram at Watchguard underscore technologies, which I'm sure Adam follows every single day. But thanks again for listening. Exactly, thanks again for listening, and you will hear from us next week.