How does HashiCorp Boundary pair with Cmd's eBPF monitoring solution for Linux?
By: Sean Blanton
The following transcript is pulled from Risky Business #602 — US DoJ hooks Sandworm. In this interview, host Patrick Gray speaks to CEO Jake King and CTO Mike Sample about the latest open source release from HashiCorp (HashiCorp Boundary) and how Cmd and Boundary can combine to address the challenges this year has brought to secure remote access management. They also follow up on the sophisticated Linux malware Drovorub that was discovered earlier this year. This transcript has been lightly edited for readability.
[excerpt beginning at 37:14]
Okay! It is time for this week’s sponsor interview now with Jake King and Mike Sample from Cmd Security. They make control and visibility software for Linux, but today they actually wanted to start off by talking about someone else’s product. HashiCorp has released Boundary, a cloud-neutral, remote access proxy thingy.
Jake and Mike say technologies like these are great, but with more and more remote access being provisioned in 2020, because of, you know, the thing, it’s time to get serious about setting up meaningful controls on your production systems. Mike Sample starts off here by explaining what Boundary actually is.
Boundary is basically a cloud-neutral version of SSM and Google IAP. It allows your clients to tunnel TCP connections into your intranet or cloud VPC and access the servers by SSH or whatever they need to get access to.
Now, we’ve already got a whole bunch of remote access tools and techniques for these sorts of environments. Why is someone releasing something new? What’s the problem that this one seeks to solve?
Well, I think with lots of HashiCorp technology, you see that they are cloud-neutral and very well designed. And this one, unlike Google IAP or SSM, they’re not tied to a cloud provider’s IAM system or the services that only the cloud provider has.
So you can learn this technology, apply to your own data centers or your cloud VPCs as well. And it works just as well.
So where is the IAM piece in of all of this? Do you plumb through to your SSO provider, like Okta or something? Or like, how does that work?
That’s their plan. It’s on the roadmap. Right now they just support password authentication, but they are very open about bring-your-own-identity-provider to authenticate to this thing. But the part it provides on the backside of that is basically a directory of your users, groups, the roles and permissions and the server groups they can access.
Now I’m guessing we’re talking about this because you’re anticipating that this is going to become a popular technology?
Absolutely. We’re seeing this with people [that] are setting up SSH bastions as kind of the technical persons’ VPN access to services. They’ll set up remote port forwarding and such like this to gain access to these things.
Boundary makes it a lot more friendly to your average remote worker, because it can automatically launch your SSH client or putty or curl or whatever you want without you having to point these tools at some port on local host and all that complex stuff that many people would probably curse at IT about.
So, Jake King also joins us on this call. So why is Cmd, which is essentially a Linux endpoint visibility play… like, why are you talking about this HashiCorp thing?
Yeah. Entry points Pat. I think one of the things that we’ve really noticed over the last couple of months, and I think even more so as we’ve settled into this new remote normal, is that we’re seeing dozens of different technology vendors come out with solutions that get you into Production, get you into a network, and do a pretty good job of handling not only the access and authentication, but the authorization as well. And I think where this gets really interesting for us at Cmd is we started to notice there were a lot of inconsistencies around observing actions that were being performed over these new transport techs.
And I think where this gets really exciting is for a lot of the CISOs, the CIOs out there, we’ve really added all of these new holes into the firewall for Production and we might not be actually monitoring them as well as we once thought.
Cmd is really uniquely positioned to help out in a couple of different ways here and Boundary is just a continuation of where a lot of this technology is going. We’re seeing a lot of vendors bring in entry points that are nonstandard, they’re managed (they can be managed by our operations team) and before you know it, this is the primary way that people are getting into production.
Thankfully we’ve got some capabilities to detect new entry points; the kind of things that people are using to access Production. And so being a transport layer agnostic solution focusing on capturing this data at the endpoint has been pretty awesome for us to just see how some of these technologies are being used, and how it’s changed behaviors of engineering teams that are leveraging alternative ways to get into prod. And given the new normal, it’s really kind of interesting to see.
This is interesting, right? Because there is another company that I can think of that has sponsored a couple of Risky Business episodes in the past. I won’t name them cause this is your time; this is your time, Jake! But you know, there’s a couple of competing approaches here, which is like… so what you’re saying is: “Okay, there’s all these new alternative methods for accessing production environments. So you want to be on the endpoint, we do endpoint, you know, we’re going to spit out logs, no matter how you got here.” That makes a lot of sense.
But there are competing plays that say, “We are an identity and application and protocol aware proxy and get all of your production maintainers to connect to your production environment through us, and we’ll do the logging here.”
So there seems to be two different approaches here, which is to do the logging of the proxy versus having endpoint logging and control, which is what you offer. Mike, why don’t you sell me on the endpoint approach versus the heavily logged proxy approach. Go ahead!
Sure. End-to-end encryption! I want to use the tools I like, like SSH from OpenSSH. I have a ton of scripts that use that automatically [and] I don’t want to be forced to retool those to use some other way to get in there. End-to-end encryption is a key one, and also observing terminal output is really a terrible way to see what’s going on on a Linux box. And it can be defeated so easily.
What about you Jake? Give us your best pitch for the endpoint control of it and all, and then I’ll give you my pitch. Right? Cause I’ve got one. Yeah, give us your best pitch for why endpoint versus proxy is the way to go.
You know, I think it came from where we started Cmd from, which was being able to capture obscure or obfuscated methods of trying to exfiltrate or trying to obfuscate an action over the wire.
There’s been a lot of ways we can get around TTY recording or just connection recording. And from an OS perspective, on a deployed agent perspective, we get the full footprint of what’s going on within that workload. We can see the runtime, we can see applications that are forked, files that are changed, actions that have maybe not echoed something back to the terminal that have performed something a little bit malicious or a little bit unusual.
And we can do it in such a way where you can query and obviously search for a lot of this information and compare it to other data sets that you might have.
So in contrast to using a proxy-based solution to capture a lot of the data, as it goes over the wire to the client, we’ve got the ability to look at the exact binary that was executed, the forks that it made, any files that were written by that binary; even different connections that are made from that binary. And I think where this gets really interesting for a number of the folks out there is now you’re no longer trying to reconstruct from a single side of the conversation, but you’re actually able to say exactly what’s going on on the endpoint, what’s actually occurred and what’s actually changed.
And I guess you’ve got that added control, right? Cause you know, Cmd is an endpoint control as well as a visibility tool, which gives you a further benefit. I’m going to guess that it’s probably harder to roll out Cmd across every single Linux device then it is to just whack a proxy in front of them, but it gets your further.
I mean, if we’re going to be honest, that’s the way it is, right?
Yeah, we do hear this one a little bit, and I think it’s an interesting point to bring up. If you’ve got a well-orchestrated system environment, which… look, we all want to have that beautiful orchestration Terraform, but we’re just not there yet.
Yeah, it’s a bit of a tricky one, you know? I’m of the opinion that if your engineers are accessing a workload and that workload isn’t managed, or at least isn’t managed well enough to deploy a pretty lightweight binary, you probably got bigger problems on your plate in many of these cases.
I think what I will say though, is it is a little bit more difficult, but it’s a longer term solution that gets you a lot further in the long run.
Mike, this HashiCorp stuff… surely it’s not the only new way that companies are provisioning access to their production environments in this, the “Age of Covid” and remote working. What else have you seen over the year?
It’s been the year of the identity aware proxies, right?
I would agree a hundred percent with that. We’re all working at home right now.
I think, it is definitely going in that direction and getting a bit of a handle of what’s going on in there is probably important too.
Now Jake, while I’ve got you here. I just wanted to ask you about this Linux malware that Cyber Command reported on back in August.
No one has been able to turn up a sample. You were even asking your contacts, like… does anyone know? And I reached out to a few people on your behalf to see if I could track one down for you. And I came up empty handed. So, I’m told this thing was in the wild, but in very limited circulation and no one seems to have samples.
But you must’ve been paying very close attention to Linux malware, being the number one sort of “agenda item” in a week in August for InfoSec, that’s music to your ears I would imagine. What are your feelings on that whole episode?
Absolutely. It’s been really interesting first and foremost. It’s interesting to see that we’re seeing targeted attacks against infrastructure and not necessarily just Linux endpoints that people may be using for personal use or obfuscation or just to run that live state data, run tails, and kind of “get away with what they want to get away with”.
It’s been really interesting to see a lot of the target shift to infrastructure. What I will say is I was very surprised by the sheer amount of press that the topic got. And I think what I was most interested by is just the lack of samples that were out there.
It does prove that these kinds of malware targets are, you know, generally used with scalpel level precision, and it does bring up the question often in the back of my mind: how often are we seeing these kinds of things and maybe just overlooking them?
That’s the interesting thing that came out of this for me, right? Which is like, no one had seen it, but who’s looking right? Like observability, no one is doing appropriate monitoring of their Linux environments, right? So I figured you would be all over this.
And your client might not have even seen this thing if it was once it was on the box… but it would be a box running Cmd [and as such] would be less likely to get that malware on it in the first place. But from the writeups it looks like it was pretty good stuff.
Yeah, I think the level of sophistication from the campaign was pretty clear and present in a lot of the ways that you could detect the malware.
It was connectivity to a set amount of endpoints. So connectivity, inbound to your network on a certain amount of endpoints. We actually have a great blog writeup by one of us security engineers. Derek [Betker] and the Cmd team who analyzed a lot of what was going on in the media and at least distilled it down a little bit more in some different ways.
But I think that the one challenge that we’ve identified is the kernel might be lying to you. Lots of different endpoints might be lying to you on Linux systems, and seeing the level of sophistication in a lot of the Windows variants of malware varieties that we’re seeing in the industry today.
You know, I can only imagine the complexity that we’re going to be seeing in Linux and discovering in Linux is going to be really interesting. Where I would suggest a lot of the listeners take a look at is: would you have the ability to discover payloads, even hypothetically to the level [of] sophistication as some of the malware variants, even for Windows out there? Just because, a lot of the time we’re making assumptions that we’re collecting enough information to reconstruct a particular attack or look at the way a binary may have made a change to a system.
And it’s just often you’re left holding the bag with not too many logs and maybe a little bit of an audit trail that’s locally stored.
I’d hazard a guess that in the next months and years, we’re going to see some very sophisticated adversaries popping up within Linux and Unix land. And you’re exactly right. Looking at ways to appropriately detect these kinds of attacks is going to be insanely fun for us to chase.
With you guys, you might catch them during the actual installation of the malware and then maybe during lateral movement, but even then, I mean, this was a pretty tricky one, right?
Yeah. With some of our eBPF technology, I think we have a pretty amazing vantage point on the host to capture a lot of background process information that’s running on the system.
So if there’s an active process writing to a file, executing and forking and doing different things, we’ve got a pretty incredible data model that’s gonna represent really any background task operating on the system, as well as the instrumentation from an entry point or a lateral movement.
But it’s really when the adversary acts on objectives that you’re probably going to be likely to see them crop their head off.
That was exactly what I was thinking. Eventually they’re going to stick their head up right? And that’s when you’re seeing them…
Yeah. I think it’s going to be interesting to see how traditional Unix and Linux system controls [and] native controls are going to be leveraged.
But what I think what you’re doing is subtly agreeing with me that once that particular bit of malware is on a particular host… that host is done.
It’s, it’s kind of done.
I mean, reconstruction’s always going to be fun. I think forensic analysis is going to be fun, but can you trust the system to redeploy? I dunno…
You were impressed with that work by our friends in Russia.
Relatively impressed. Relatively impressed. Can’t wait to get a sample.
Did you hear that folks? Hook Jake up he wants a sample! Mike, you have some feelings on… and that’s the name of the malware right, it’s Drovorub… you’ve got some feelings there, yea?
I do. I think people should be looking at LSMs more. And that you’ve got BPF LSM coming along and you’ve got two LSM points that would have shut down this module loading init_module and finit_module, I believe, which would have locked it in its tracks.
I’m all right. Jake King, Mike Sample – thank you so much for joining us on the show to talk about a whole bunch of topics there, it’s been a real pleasure to chat to you both.
To read more about HashiCorp Boundary, and how Boundary and Cmd combine to deliver secure service and server access management combined with scalable Production Linux observability, click here.
To learn more about the Drovorub Linux malware discovered earlier this year, you can read our advisory post here.
Ramp up your Linux defense strategies
and see what you've been missing.