Patrick Gray: Hi, everyone, and welcome to this special Soap Box edition of the Risky Business podcast. My name is Patrick Gray. These Soap Box podcasts are wholly sponsored. And that means everyone you hear in one of these editions paid to be here. And today's guest is Andrew Morris, founder and CEO of GreyNoise. Now, GreyNoise is one of those companies that has a brief that sounds simple, but they do something that's actually quite hard to execute, right? They detect malicious mass scanning on the internet, so their customers can plug that data into their SOC to see if the IP they just got an alert on is something targeting them or targeting the whole internet.
Now, you don't even need to be a customer to get some use out of this service. If you want to know about an IP that you've seen an alert for, just head over to greynoise.io and drop it into the search box. Magic awaits. GreyNoise makes its money by selling API access to that data, basically, and yeah, its customers mostly use it for SIEM enrichment. But as we'll hear, Andrew says GreyNoise is looking at moving toward actually blocking this type of mass scanning, blocking it from hitting their customer environments. And they're even looking at working with telcos to scrub the most egregious stuff from the internet entirely.
His rationale behind wanting to do this is pretty simple. He wants to narrow that aperture through which the mass scanning can fit through, just to make it harder. But yeah, this interview isn't just about what GreyNoise is doing. It's also about the current state of mass scanning. Gone are the days when you could just increment IPv4 addresses and see everything. These days, it feels like every website and web application on the planet belongs to about 10 CDN IP addresses. So yeah, mass scanning for web these days often involves attackers starting with domain lists, and looks a little bit kind of like open web asset discovery. And then there are the dumber scans that are just looking for vulnerable devices on their own IPs, VPN gateways, open SSH, stuff like that. Basically, mass scanning sophistication is a pretty broad spectrum. So here's Andrew Morris, talking about that.
Andrew Morris: Yeah, I mean, pretty much spot on. So rewind to when the internet was many protocols, right? It was HTTP, but it was also SMTP. You've got mail servers, file transfer protocol servers, like you've got, you got fingerd, you got Gopher, right? You got all kinds of stuff, right? And so everybody knows you need a SYN-ACK, that's it. And then you do whatever the super basic protocol specific handshake is to get whatever it is you need. You batch that times 4.2 billion, and you're done. Right. And that was kind of the paradigm. Anywhere from when the internet started all the way up to when centralized hosting providers started becoming super, super big and powerful and important and ubiquitous, right? And so then, to your point exactly, basically web became its own sort of monster, because then you basically have…
Patrick Gray: You just basically rolled all of these services and turn them into web services and stuck them behind, you know, 80 and 443 webports, right?. And all of that attack surface is still there. It's just we've gone from having a bunch of little services to provide complexity, to having one really big service, which is much more complex, which didn't really solve that problem.
Andrew Morris: Not to mention, people started using HTTP and web protocols for everything under the sun for every API ever, right? Whether it did or didn't make sense.
Patrick Gray: Yeah, that's, that's kind of my point, right? We just took the entire internet and said, “Let's do this via HTTP.”
Andrew Morris: You are this protocol now. And the original protocol authors are like, “Wait, no, that's never what we meant.” So take, for example, Patrick, that you and me are doing a video and audio call over HTTP calls in our browser right now. That, in and of itself, is a really great example of this.
Patrick Gray: Yeah. Because if you asked if you asked earlier sort of engineers from back in the day, if they were asked to design a video conferencing spec, it was going to be its own client server, right? It was not going to go over the web.
Andrew Morris: Yeah. And so now, I mean, you've basically got like the two different dimensions of complexity to it. One is the shared host, where you've got one IP and many, many, many different websites. So you've got the one IP and you're looking at, basically, a gazillion domains that are behind that. Added on to the complexity of that, you basically have, pretty much, 9 things out of 10 on the internet using HTTP as a protocol for things that have nothing to do with web functionality – chat, mail, email, all these other different things like that, that are still speaking…
Patrick Gray: DNS over HTTPS for God's sakes. Like that's why, you know, some of the young-uns might wonder why we cringe at that, but it's just wrong. It's probably because, of course, that's a better way to do it. But it just feels weird, doesn't it?
Andrew Morris: It's inelegant, and it just basically creates a bunch of challenges that nobody necessarily expected. So basically, one of the reasons why mass scanning is very different now from how it was 10-ish years ago is that not only do you have to account for the horizontal, the throughput, the actual socket, packet management, packet crafting, etc. You've got all that. Fortunately, you have a few different added benefits. The internet is simply faster, cloud hosting providers make it super easy to recycle IPs, there's tooling that people have written such as Masscan, Unicornscan, Zmap, all kinds of stuff that do make it actually quite a bit easier. So you don't have to be like a really fantastic computer scientist to get some code that does the thing that you want it to do. So those things make it easier. But indeed, what makes it harder is that now you have to actually take into consideration things like, “What's the hostname? Which…”
Patrick Gray: Yeah. So the whole game now has turned into understanding the nuance of the service that you've discovered, and figuring out how to actually enumerate something useful from it. Which is a completely different game to doing an internet-wide Nmap scan and saying, “OK, well, there's a vulnerable service. Let's go pull some…off a forum and pop shell.” Right?
Andrew Morris: It's just like an iceberg, right? If you if you just scan the Internet for a thing, using just like a port scan method, like blasting SYN packets, getting them back, taking that and then doing whatever the protocol handshake is. If you do that way, looking for a thing…let's just say we're looking for a version of a WordPress plugin on the internet or something like that, if you just do it that way. And then you actually do it intelligently, where you actually harvest out all the domains that you're going to be using. And you actually start moving through validating some of those things.
Patrick Gray: This is exactly my point, which is you need a starting list of domains, not IP addresses these days, right? You can't just go scanning for a WordPress plugin, because you can't scan an IP for a WordPress plugin as half the time that IP belongs to a CDN, and there's like 600,000 WordPress sites on it that may or may not have that plugin, right?
Andrew Morris: Exactly. And chances are whatever that hosting provider that you're looking at has TLS configured in a way that if it doesn't get the valid cert and TLS initiation…it's basically just gonna be, like, BOOP – dropped. Nope, you can't come through here. So if you get it wrong, then it's not going to allow you to come through until you get it right, the way that a user would if they were browsing in a browser and typing everything in correctly. So it's very different. The people you really want to talk to about this kind of thing are the bug bounty hunters. They are good at this kind of thing.
Patrick Gray: Yeah. And that's why we've got this whole new field of asset discovery, right? There's a bunch of companies out there doing it, they got all these weird tricks to try to discover assets. You're in a position where you're observing mass scanning activity, I'm guessing a lot of it is the same old stuff, where you can just scan every IP on the internet looking for stuff like SSH, that might be you know, unpatched or old versions or whatever, right? So there's always going to be that stuff. But are you starting to see the people doing mass scanning, doing some more exotic stuff and more exotic enumeration? What's the state of the art of malicious scanning on the internet?
Andrew Morris: It's so broad. Basically, one of the really interesting things about the cat and mouse of it is that unless you are a three-letter agency that has God Mode on looking at the internet, no one really knows what is actually happening from people who are mass scanning the entirety of the internet. Because you only have a number of different perspectives, right? No one obviously, except for the major telcos, the Tier 1s, and some of the spooky spy agencies, right? No one actually knows everything that's moving over across the internet. So even just getting the right variables in place to answer that question is extremely difficult. You don't know.
And so the state of the art, I would say, it's going to vary from your step one, which is where you're going to find, “Look, this is somebody effectively zmapping the internet and getting a list of ports back. Then they're going to ask each of those ports for whatever the standard port that it is, and they're gonna interrogate that protocol. And then if they get the thing back that they're looking for, they're going to do the next thing, right?
There's some more advanced mass scanners that are intelligent enough to know, “Hey, I don't necessarily know that it's going to be the protocol or the service that normally runs out of that protocol, I should probably check for a number of different things.” Right? And so then even adding even more complexity, you're also going to have things like shared hosts and stuff like that. The state of the art is, if we're talking actively because if you want to get real wacky with it, then what you'll do is you'll actually just listen to the way that something responds passively to figure out exactly what's listening on that port just based on the responses that it sends. Which there are actually services that kind of…
Patrick Gray: Well there are, but fewer and fewer. And that was kind of what I was getting at. Now you might connect to some service and you're like, “I don't know what this is.” And you know what I mean?
Andrew Morris: You don't know where these people get this data.
Patrick Gray: Yeah, you don't even know what command to send it or what syntax, or you have no idea what it is. I mean, chances are these days, it's going to be something involving gets, but that's about as far as you're gonna get, you know? Especially if it's an API endpoint, right?
Andrew Morris: Yeah. And so now, basically, the state of the art, the people who scan the internet really well. They're basically scanning the internet from many different places, they're doing it very quickly, they're doing it very often. But they're not being so loud that they draw a ton of attention to themselves; they're doing it quietly enough that they're able to get away with it. They're also doing it almost intelligently enough that you're not really able to link two and two together, that those two IP addresses that just scanned me on two different ports, they're the same actor, it's the same system, it's the same person, it's the same group that's doing this, right? And so piecing that backwards is really tricky. It's a really fun, kind of like, you're reversing a state machine to look at who's scanning for what, where's the data going, etc. You can actually do a lot…
Patrick Gray: It's real, you know, “Pepe in the mailroom”…
Andrew Morris: I'm already feeling my eyes, like roll into the back of my head as I'm talking about it. But so the really smart ones are doing it from a gazillion places, they have support for all the different protocols. They do it fast enough to have the data be up to date, but they do it slow enough from the different hosts or the different IPs so as not to raise any unnecessary alarms. And you're going to assume that you get blocked as soon as you hit somebody the first time. So that IP is burned for the next however long, so you're going to come in the next time from somewhere else.
Or, the other people that do it really smart, they do it really dumb, right? The other way that you would do it is just is just basically make it as stupid as possible. Try to look like a script kiddie. Right? And and then yeah, you're gonna get blocked. But at least this way, no one's going to know that this is part of a really advanced system. The B team wants to look like the A team, But the A team wants to look like the D team on this…
Patrick Gray: Yeah, that makes sense. Let's just talk through like a couple of different types of activity that you've observed. Let's start on the advanced end, because that's where it's always exciting. What's the most sort of sophisticated mass scan type that you've seen? And what was it trying to discover?
Andrew Morris: Most mass scanning systems are like raw data that are a means to an end, providing raw data that are a means to an end of something. And whatever that something is, is going to highly influence my opinion of how effective I think it is. So in the more sophisticated (and this is where it gets really interesting), do you remember me describing how there's a progression, almost like a pyramid of sophistication, right? Very bottom, you have scanning for ports. Now, one step up above that you've got actually crawling for web pages. Then one step above that you've got conditional stuff. So you're only going to look for something if some set of criteria is satisfied. That's the way that you know, “Hey, look, this isn't a honeypot.” That's how you're basically going to say like, “Look, only a real system is going to respond in this way. Now I can do the advanced thing.”
When it starts to get really, really interesting to me, is basically when you effectively have a really, really good way of knowing how to fingerprint a device without the person on the other end of it knowing what device you're attempting to fingerprint. So basically, let's say, alright, there's a spectrum, you've got a switch on the internet, and you know that if you issue, for example, a head request on Port 80, it's going to come back, and it's basically going to say, “Hey, I'm a Cisco switch.” Right? But everybody knows, on both ends of that equation, exactly who you are and kind of what you're looking for. You're a scanner that is looking for a certain kind of thing. And if you get the response that you're looking for, then that's how you figure out basically what the device is. What's really interesting is undocumented behaviors, bugs, right? And like esoteric behaviors and things like this, that's going to be your, “I know that if I send a malformed packet that looks like this kind of thing, Cisco devices do this one really interesting thing on some blah, blah, blah TCP flag…” But everyone who's looking at every side of that has no idea that that's what's happening.
Patrick Gray: So it's people understand the nuance, right?
Andrew Morris: Exactly. You got to really get in the nitty gritty. Another side of it is when you've basically got kind of application-specific and version-specific vulnerabilities that are buried deep inside the application. One of the really interesting things we saw with Log4j is that first really dumb pass happened, where everyone's just like blasting out that one exploit string in user agents in different parameters. But what we started to see immediately after this, it's a perfect example, was basically tailored application-specific checks that know that only a Splunk v4.5.6 on this path is going to respond in this particular way. Anybody who sees that request is just gonna be like, “I have no idea what this is.” But if the device is the thing that the bad guy or whomever is looking for, then it's going to come back with the affirmative, and no one is going to know that that's what you're looking for. In my opinion, the way that mask scanning is interesting has less to do with the technicals and the sophistication. It has more to do with your ability to accomplish what you're trying to accomplish, without the folks on the other side of it knowing what you're looking for and where you're coming from.
Patrick Gray: It's interesting, though, because I was asking you about the sophisticated stuff. And I was expecting you to talk about this more in terms of asset discovery, and in terms of being able to find stuff that's not so easy to find anymore. But it really does sound like where we are with mass scanning is, “OK, there's all the website stuff. And that's kind of a separate discipline.” Now, it's still very much about finding the devices you want to find, but doing it in a more sophisticated and delicate way.
Andrew Morris: You can slice it into a number of different ways, but what I find is that there are categories to how you would classify or identify (Rumble is really good at this). And then there's a number of other kind of like internet scanning companies that are actually good at this, you can do things like looking at, you know, like you just mentioned favicon. There's also basically things like protocol fingerprints, and TLS fingerprints, and things like that which are one-to-one mappings. And what you find is that there's always going to be that trade-off between accuracy and breadth of utility, like this thing only works, this one method only works for this family of product, but it works every single time. Versus this is a generic way to identify things using something like a TLS fingerprint, but the problem is that it's going to have a higher false-positive ratio. And so this is where you're really making those actually calculated decisions of how you want to do it and refine it. This is when it starts to become a much more advanced thing, you're starting to actually kind of put together a proper system and tune it for efficacy, as opposed to having any one trick that's just gonna get you there, right?
It's a combination of all of these things. Some of them are more bandwidth intensive, some of them are more intrusive, some of them are actually more computationally expensive. Some of them are going to be more esoteric, some of them are less so. Some of them are going to be things that are going to raise IDs alerts, and some are not. And so this is where you start to see the really interesting compromises that happen.
Patrick Gray: Yeah, I think one thing to drive this home to listeners, when we think about some of these ransomware attacks, where people came in on vulnerable VPN border devices – how do you think they've identified those targets, right? It’s not like they picked this company, they mass scanned, they found them, they looked at the companies with the biggest revenues, and then they hit them, right? I'm guessing that's how they would have gone in via some of those Citrix bugs in PulseSecure and whatever, right?
Andrew Morris: I think you're exactly right. So this is very difficult to prove comprehensively. There are ways that you can really test this. And I've seen some of this stuff happen firsthand before. But the long and the short is that there are entire classes of bad guys out there who basically pop shells rirst, figure out what they care about later. The accesses are commoditized., right? It makes the threat model super, super wacky. I think we're still stuck as an industry and a lot of people that are making decisions in security organizations are still stuck in this universe where every attack and every compromise that happens follows like, “OK, well, first, the bad guy is going to target us and then they're going to scan our network, and then they're going to find our vulnerabilities.” And while that is certainly still true, I think we all grossly underestimate the amount of compromises that start off as being purely opportunistic, right?
Patrick Gray: Ransomware is what flipped this on its head, right? Because ransomware monetized opportunistic attacks, and that's why it became what it is today. And it totally makes sense that if you've got zero-day in some border device; I mean, step one, if I'm a ransomware actor, I'm gonna go see who's who's got that device and who's vulnerable, I'm going to collect shells, and I'm going to deploy ransomware into the most profitable companies that I've managed to rinse.
Andrew Morris: That's exactly right, you're basically going to compromise as many devices around the internet as you can. You're going to do kind of two different sides of it, but you're going to compromise as many devices as you can. And then you're going to after the fact learn about where those organizations are, what the organizations are that your accesses are in. And then what you're realistically probably going to do is sell them to your buddy, sell them to somebody else. You're going to say, “I got an access inside the State Department. It's low level privilege, so you're gonna have to do some work to actually make it useful, but I'll give it to you for 100 bucks.” And then the people who are doing a lot of the real nasty side of this are actually figuring out, “Oh, this is a municipal government, you know that these people probably have one IT person on staff, right? There's no way that they have, good security hygiene.” Some people (I can't confirm this firsthand, and really need to go back and confirm this) I've read about basically actors that are only going after organizations that have cyber insurance policies.
Patrick Gray: They actually managed to break into one of the insurers. I’m not sure how closely it tied up, but the thinking was they managed to steal a list of people about cyber insurance from the cyber insurer, and then hit them because they know that they're going to pay out. They probably knew the the extent of their coverage as well, right? So it's just unbelievable. But here's the thing, right? We've been talking about all of these sophisticated ways to do subtle things, and go deep and whatever. Now, ultimately, the thing that we're talking about having the highest impact is a really dumb scan looking for a particular version of a VPN or something…
Andrew Morris: It's the speed. It doesn't matter how sophisticated or unsophisticated the bug is, what matters is that when the bug is announced, when we become aware of the existence of the vulnerability. I'm talking, it is so fast. And I assure you, I'm not trying to scare people, right? This doesn't happen every time. It just happens enough times, and it has happened enough times that I've watched this whole thing play out a few times, and the speed is unbelievable. Tthe amount of time that it takes for a sufficiently kind of big deal vulnerability to go from being announced or disclosed to an actual the proof of concept, the proof of concept being the big thing that actually makes the difference. Because bad guys don't want to like actually do the work of writing a POC that's stable, they're just going to be looking around a GitHub in a pastebin and stuff like that trying to figure out where there's actually exploit code. And the amount of time in between that happening, the vulnerability being announced or disclosed, and the exploitation of that thing on your perimeter, is mind-bending.
Patrick Gray: It's funny, we had Mark Rogers from Okta on the show a couple of weeks ago saying exactly the same thing.
Andrew Morris: I'm telling you, it's insane. This is the kind of thing that it's really hard to talk reasonably about as a security person who really doesn't like FUD marketing, because I don't know how to say it without scaring the crap out of people. Like it's not, “The end is nigh…” It’s not, “There’s nothing you can do…”
Patrick Gray: No, it's the end is here!
Andrew Morris: It's that you just have to change the way that you think about it, you kind of really do. And a lot of people have been saying this for a long time, that you really do have to assume that, at very short notice, software that you have trusted for a long time with good vendors that really do care about security can go from being by every single measure super safe, right on the perimeter, super safe, getting the crap beat out of it all the time and being resilient, to all of a sudden some new information is available to everyone on the internet at exactly the same time, and people are going to start exploiting that thing everywhere – and they don't care where they're going to land those compromises, they don't care where they're going to get.
Patrick Gray: “Bitcoin is the thing now. Wooo!”
Andrew Morris: Because now there's 1,000 reasons why it doesn't matter. They just got to own a bunch of people, ahd some interesting targets are going to shake out of that. And that's part of what keeps this model moving. And it's just really different. We were joking about this earlier, but it feels a lot more like everyone is on the same LAN now. Yeah, it feels a lot more like that…
Patrick Gray: It feels LAN-ny these days, doesn’t it? You remember when you would hit a website and the images would load, dup-dup-dup-dup...
Andrew Morris: You you really do feel much closer to all of the hosts on the internet now. And this last year has just been more of these occurrences than I've seen before, and it's crazy. I remember when Shellshock and Heartbleed came out (I think the same year, right? Yeah, within a few months.) And I remember thinking, “Wow, it's never gonna get worse than this.” This is so bad. And we had like Shellshock- and Heartbleed-type bugs like 20 times last year, it was crazy. And so it's just bizarre. And the reason that you're seeing it, in my opinion, there's really no way to prove this because it's tricky. But the reason that you're seeing it is because it's working.
Patrick Gray: Well, it's working and it's gotten easier relative to other stuff, right? I was around in the days when you would see pretty regular IIS and Apache, like, ODOS, or in various Apache modules. And that was how you would get on. So you know, there was a big rush to fix all that stuff, and you don't see those sort of bugs too much anymore. And then there was the big rush to the client side, Internet Explorer, popping shells left and right, because there was a time when people didn't do client-side attacks, people thought client-side attacks were sort of lazy and stupid. And they quite often required someone to browse a malicious website or something and you were like, “That's kind of that's kind of stupid.” Why would you bother with with those silly things on the on the client side, right? And then, of course, client-side became all the rage. And the whole time, people like Adam Barlow were saying that it's a matter of time before they start targeting the enterprise software, right? You know, when, when penetration testing shops would actually just write zero-day for this stuff on the fly, on a gig, because it was so badly constructed. You knew it was just a matter of time before the criminal element, discovered that for themselves. And that's really where I think it is, is we've built up this mountain of technical debt in business software, and the chickens are kind of coming home to roost a little bit.
Andrew Morris: So that's true. There's a lot of different ways that you can kind of slice this particular subject, because it's so vastly complex, and there's so many different pieces that are moving. But if you stand back far enough, and you squint your eyes, and you think about it over time, there is a pattern that is sort of recurring. But if you pay attention next time, you're going to notice some of the parts of this. The folks on the GreyNoise research team, me and them chat about this all the time. A bug comes out in a piece of software, OK, like a bad bug in a super common piece of software. It comes out. And then, for a brief period of time, the entire security community, and specifically, the entire vulnerability research community, is putting an insane amount of scrutiny on that particular technology. And a bunch of people are then figuring out like, wow, the way that Windows loads fonts inside of a browser is extremely dangerous. And a bunch of people realize this at the same time.
Patrick Gray: Let's run the bad boy on the kernel, baby, what could go wrong?
Andrew Morris: It's crazy. And so you see these patterns kind of over and over again. I remember after SolarWinds...
Patrick Gray: That's totally compatible with what I was saying. There’s a pile on. People start getting results, then everyone piles on, and then people started hitting IA, and everyone piled on to that, right?
Andrew Morris: I remember what was so interesting, just to demonstrate this, was when SolarWinds happened, there wasn't even a software vulnerability in SolarWinds that was directly associated at the time with, basically the debacle. Then everyone had a look, and they found them, right? I've seen this so many times. It's like, how many different Log4j vulnerabilities came out after that one that came out? I mean, it was a lot of different ones, right? And it's not necessarily because a piece of software was written by people who don't know what they're doing, or something like that. Part of it is that this is what happens when you have many, many, many 1,000s of people actively poring over the code for something and finding bugs. And that's why the classes of bugs and other different things like this matter so much.
And here we are, we're all sitting on the same LAN, all kind of figuring this out all the time.
Patrick Gray: So one thing I want to put to you is this pivot back to back to server bugs. Because that's what these border device things are; they are services, right? So this pivot away from client-side exploitation and towards stuff like this has made a company that does stuff around mass scanning more relevant than it would have been five years ago. The timing for you is spectacular because now you are smack bang in the middle of a pretty bog standard attacker workflow, which is: discover bug, weaponized bug, scan for bug, deploy shells, fire cannons – pew pew!
Andrew Morris: And you know, and from our perspective, it's really interesting because what we do at GreyNoise is like simultaneously really cool and smart, and simultaneously really obvious and kind of dumb that no one thought about it before – which is, we’re basically just listening to stuff from a gazillion places. And when people start doing that mad dash, A) we can prove that the mad dash is happening, like “Hey, everybody, all these IDs alerts that are going off, 80% of these are people that are doing it with everyone on the internet.” Right? and B) the other ones are probably people that are coming after you. But to your point, we are in an interesting position to be able to see this kind of stuff because it is, like you said, a lot of bugs in a lot of products that I don't believe most people would have thought were going to happen, at the volume that it was going to happen, and at the frequency that it was going to happen. And in some of the products that people thought were going to happen, let alone be blasted around to the entire internet. And so it does put us in a good spot to see this kind of stuff. It's just dumb luck that we had the system built and running when a lot of these things started happening.
Patrick Gray: Well, it's interesting because you're phase one of GreyNoise’s business has been OK, we can provide you telemetry associated with with mass scans, right? Like this IP is scanning for this, this IP is scanning for that, and it's mass scan activity. And the idea was, people would pull this into their into their SIEM, use it to sort of give themselves a better understanding about what's targeting them, which mass scans are targeting, what bugs, etc. Now, before we got recording, you were saying you want to take it into the next phase, which is you just want to start blocking this stuff because you feel you've got to a point where the fidelity of your detection of mass scanning activity is good enough that you can just block.
Andrew Morris: Yes, so what we've basically figured out is the first phase of GreyNoise was the intelligence product, right? We're going to tell you after the things already happened, we're going to tell you inside your SIEM, which of the alerts that have already been raised, whether it was five minutes ago, or two seconds ago, or whatever, which of the alerts that were raised were things that were hitting everybody around the entire internet. And in gaming that out, what we figured out is yes, one of the really cool things about this is that we can help people prioritize the things that are hitting them specifically, and forget about the things that are hitting everybody, which is great. You know, it's an awesome efficiency boost that is really good for situational awareness.
Patrick Gray: Sure, but there is a limited set of very large customers who need, who are in a position to actually do something with that data. So I'm guessing most of your customers have SOCs, which is the minority of companies out there, right?
Andrew Morris: Right. And so what we've found is that a lot of our customers that find what we do to be really, really valuable, the logical next step for them is gaining enough confidence and trust in our data and being able to understand it as it relates to their networking, get the right tooling in place, and just block it all together. So that basically they're saying, “I trust GreyNoise enough, it's been correct enough times that I can now feel pretty good that I can block stuff that's inside this list, no revenue generating users, or whatever revenue generating parties that have to be able to talk to my network are going to be disrupted.” And I'm basically going to knock out of the air maybe nine out of 10 bad things that are coming towards my network at the perimeter. Is that a comprehensive defense? Absolutely not. Is that going to buy me some time when s*** is hitting the fan and there's a vulnerability that's out there – maybe there’s a patch, maybe there's no patch? Maybe...
Patrick Gray: I think it's a good example of outrunning the person next to you when the bear is chasing you personally. So I think it will actually, in some circumstances, because they will actually miss you, because you're doing this. But you know, I brought up something when you first suggested this to me where I'm like, it's never the boxes that are behind your nice firewalls, and you know, network inspection gateways and whatever, that get rinsed in these sort of things. It's always the unmanaged stuff on far flung bits of your network. To which you replied, “Well, that's why we want to partner with the telcos and do this as exact as a telco delivered thing,” which makes a lot of sense. But the question becomes, why aren't the telcos doing this themselves? Because as you already mentioned earlier, they have a lot of visibility into nefarious stuff that happens on the internet, why are they not doing this?
Andrew Morris: Internet background noise is, you want to think about it differently from just like broadly bad traffic as broadly bad traffic has like a longer half life. And command and control (C2) communications are a lot more definitive of badness and other kinds of, call it observables or indicators or whatever…
Patrick Gray: …they're focusing on that stuff.
Andrew Morris: Exactly. And that stuff is not nearly as chaotic, and as in fast of a flux, and have as short of a half life of validity. And that kind of stuff is rarely coming from a quickly cycling IP and is rarely only useful for five minutes or something like that, right? There's a few different reasons why the telcos in the in the major internet service providers don't do stuff like this. I also don't want to throw any of them under the bus, because we are building partnerships with some of them…
Patrick Gray: “Here's what our valued future partners are doing wrong…”
Andrew Morris: If you think I'm talking about you, valued Internet Service Provider partner, it's not you, it's your direct competitor. So basically, the long and the short is, there's a few reasons why they don't do it. One, the Internet Service Providers are huge, and doing anything as difficult as it is with any large organization, especially something that's new and complex, right? And the second thing is that downtime or dropping packets or dropping things, the ISPs are like allergic to this. Have you ever been to a NANOG before?
Patrick Gray: I was banned, because I snuck in there as a journalist, something like 20 years ago, and yes, I was banned.
Andrew Morris: So I don't want to get in any trouble with the NANOG crowd. But the long and the short is that, if you meet the crowd of people who really run the internet, you'll talk to them for five minutes and you'll basically be like, “Oh, running the internet is like your religion, actually moving packets across the internet at insane speed, having fantastic uptime and not dropping packets.” This is what the network crowd does, right? So as soon as anybody introduces introduces the notion of blocking something that's going to have any kind of impact on you moving that traffic from from A to B…
Patrick Gray: And I gotta tell you a story here, because and this is a bit “Old Man of the Internet”, right? I was wearing an onion on my belt, because that was the style at the time; it's that sort of story. But I think it was, one of the SQL worms back in the day, it was Slammer or Blaster, I don't remember which one. But the way this worm operated is, it had shellcode in a UDP packet of a fixed length. So I think it was quite small, like 419 bytes or something. And that's what was so bizarre, because this is back in the day when you could find enough SQL servers on the internet to actually get a worm to blow up. And, yeah, so it was it was crippling the internet. And at the time, this idea among the network operators that no, we don't touch the packets, we just deliver them, you know, we don't filter. And this is why I snuck into NANOG as a journalist, because it was kind of at this point where it's like, “Llook, that's a great philosophy. But the internet doesn't really work today, because you guys aren't filtering a fixed length UDP packet on a funny port.” Right?
And in the end, they did filter it. But as far as I know, that was the first time they take it a step like that, because the Internet was not working properly, routes were flapping around, like the whole thing just went sideways. And in the end, they had to just say, “any UDP packet of this length with these parameters…drop it.” And then everything bounced back, the internet worked again, right? Because they just took it off the backbone. But then when they figured out I was a journalist, they were like, “You. Out!” And that's how the ZDNet Sydney IP range wound up being blocked from NANOG. So that's my story that reinforces what you're saying where there is this sort of religious thing. And they saw me bringing this to light as a journalist as some kind of assault because they're a very tight knit group who don't like...
Andrew Morris: Yeah, cuz I'm like, I really don't want to get in trouble with this crowd...
Patrick Gray: ...they got a Stonecutters vibe about them (there's another Simpsons reference).
Andrew Morris: Exactly. It's like somehow, even though the internet's not that old, it's somehow like an ancient society.
Patrick Gray: Yeah. I also picture them in robes, right?
Andrew Morris: Yeah, yeah, exactly. You're like, this doesn't make any sense. But it just feels right. So basically, it is because of these people that we got here, we have this incredible group of people that are moving all this traffic. And billions of people don't even know that it's happening. Because it's just all the work that they do just works, moving all this stuff across the planet, right? It just works. And that's what we need. But to your point, sometimes our understanding of a sufficiently complex problem becomes a little bit more clear. We understand like, “Look, I don't want to mess with people's traffic. I don't even want to know what it says.” Right?
Patrick Gray: You're not suggesting doing this at the backbone. You're suggesting suggesting doing this at the edge, right? Which is a different proposition.
Andrew Morris: So I'm seeing, from my perspective, the further upstream the better, yeah. And what that means is...
Patrick Gray: So you are suggesting doing it at the core, right?
Andrew Morris: I mean, that's the way that you would really do it, that's the way that you would have the best results. But that's also where, you know, there's...
Patrick Gray: …you're gonna run into the paper who wear the robes.
Andrew Morris: Yeah, I mean, that's where it's the lowest acceptable margin of error, that's where you've got all kinds of stuff that you don't want to get wrong. And that's where it's gonna make some people feel really uncomfortable, that people are messing around with that kind of stuff, right? But my opinion is that's the place where the most good can be done. That's the place where the largest impact can happen with the least amount of effort required for all of them (the customers).
Patrick Gray: Well, here's the question: how receptive are the telcos to this?
Andrew Morris: Ah...so basically, in my experience so far, everybody wants to have the cut body, but nobody wants to lift any heavy-ass weights. So everybody wants the impact, everybody wants the effect that it creates, right? Everybody wants to have a security offering where you don't have to install any hardware software, you check a box, it's just on, it just works. And all your users are using your stuff like normal, and you don't really have to think about it, right? That's what people love. They love products that you don't really have to think about. It just works. Everybody wants that.
But as soon as you start actually talking about some of the mechanics, it scares the crap out of people. And it takes time just to build that trust with them. And for them to really kind of get it and move things forward and…mind you, also, these are old businesses that are sometimes made up…they're kind of like banks, they're made up of like many many sprawling, consolidated regionals.
Patrick Gray: That's why I would have thought offering this as a checkbox option, sort of going with the Arbor Networks business model, where they installed some Arbor equipment in the telco, and then you could route certain customers through that. That's where I would have thought it would make more sense.
Andrew Morris: So that's the thing for me. I think for the first pass of this, it makes the most sense to not route anything differently at all. Because I don't want to have to explain to people that their traffic is coming to me, but I promise I'm not looking at it, right?
Patrick Gray: Even routing through just a bit of equipment that you maintain on the telco premise, right? That's more what I had in mind.
Andrew Morris: That makes sense to me. The how of it, the long and short is that the ISPs do have means of blocking large amounts of IPs from being able to move from one network to anotherm or from the internet right to one of their customers. The do have ways of doing it.
Patrick Gray: I guess that's the point, isn't it? They don't need extra boxes to block IPs. Their a telco, they've got that stuff already.
Andrew Morris: They've got that stuff already, but there's a number of different challenges in here. One of them is the change, sort of the refresh windows, right? How often can you load a new list of definitions of IPs that you're not going to be routing traffic on, right? And then and then the other component of it is just basically, so aside from how often that actually comes, it's how large or small that list can be. Because when you're talking about core backbone switches that are moving gazillion billion zillion packets every second, you start having to get really close to the metal, and you start having to get really well optimized hardware that's probably been running in a rack somewhere for, you know, 25 years or something like that, right?
Or now what's really cool is that you can do a lot of this stuff with, like eBPF, you can do stuff with basically like kind of like bitcode that's actually pushed out to a NIC. So it's never actually having to come into the computer. There's a lot of different advances that make this like a lot more viable than it would have been 10–15 years ago. But it just takes time. And from my perspective, all I want to be able to tell people who are on the internet, I want to tell customers who have network perimeters, “Hey guys, look. You don't have to live like this. You don't have to have all of this hardware in all these different places.” The means are all there. All you have to do is this, this, this...
Patrick Gray: It sounds like you've got a bunch of different ways you can do this, right? You could maintain some hardware for some of the telcos that they could route through you to the customer, but you could give them a sort of sublist of the worst stuff…
Andrew Morris: The good thing about it is that yes, the internet is crazy noisy. Any one routable IP address on the internet is going to see unsolicited scan attack SYN communications from anywhere from 1,000 to 3,000 distinct IP addresses every day. And there’s a very particular overlap between all of those. But there is going to be like a list of ~100,000 hosts that are really noisy. And then the other ones are lower and slower, or the round robin, or whatever. So to your point, you're exactly right. And so you can still get a lot of value out of just blocking the most egregious offenders. And that's something that does actually provide quite a bit of value to the customer.
Andrew Morris: And then everyone has to go lower and slower.
Andrew Morris: Yeah, exactly. From my perspective, what it actually affords you is that it means that the only attacks that make it through are the ones that are unique and special to each customer. They're targeted, right? It involves that the bad guy has to do more work and be a little bit healthier. And that's what we want, right? We want the bad guy to have to go through a lot more and be inside of a much more narrow aperture, where human analysts can actually look at that and track that stuff down. Because that's the kind of stuff that only a human can do.
Patrick Gray: Right? Well, Andrew Morris, sounds like you've got some work to do there with the with the telcos, and I wish you the best of luck with it because anything that can cut down on the sheer volume of silly scans out there is always going to be good. It's always a pleasure to chat with you, my friend. Thank you very much.
Andrew Morris: Patrick, thank you so much for having me, I had a great time.
Patrick Gray: That was Andrew Morris of GreyNoise there with a chat about mass scanning and what he's doing about it. You can find GreyNoise at greynoise.io. And if you liked what you heard, I can obviously always recommend a chat with Andrew. It's always fun. And that is it for this edition of the Soap Box. I do hope you enjoyed it. I'll be back next week with more security news and analysis. Thanks for listening.
