Sep 9 2020 · 7 min read
The Dual-Role of Cmd to Support Ops and Security in Production Linux
By: Jennifer Ellard
The following transcript is pulled from Risky Business #591 — EncroChat user experience includes getting owned, going to prison. In this interview host Patrick Gray interviews Adam Cardillo and Curtis Simpson from HPE’s ITOC group as they discuss how Cmd supports both their Ops and Security initiatives.
[excerpt beginning at 45:09]
Patrick Gray, Host:
And this week’s show is brought to you by Cmd Security. Regular listeners would know that I really like this tech mostly because it’s really simple. It gives you visibility into what users are executing across your environment. And it also gives you the ability to restrict user actions or do things like make people MFA before they reboot a production box. Right? So this is all for Linux.
You can even do a Duo push to a Sysadmin if a user is trying to like, sudo something that your policy says is risky, right? So it’s about visibility and control and it’s quite simple. Cmd is a startup, but it is backed by Google Ventures. And, you know, as best I can tell things are going very, very well for them.
Instead of having their co-founder Jake King on the show like we’d normally do for this one, Jake suggested that we chat with a customer instead. HP Enterprise, or HPE, here in Australia, is a Cmd customer and they were happy to come on to talk about the product and what they’re doing with it. Adam Cardillo and Curtis Simpson both work for HPE IT Operations Center, or ITOC. Adam is in Engineering and Curtis is in Security and yeah, they both really like CMD. They actually heard about it on the show too so there’s a bit of symmetry here in having them back here to talk about it.
Anyway, ITOC runs customer environments, essentially like an Outsourcer or an MSP. They run things like private cloud and they found Cmd is an excellent way to “bubble wrap” the environments they manage for customers and also to help them with things like auditing, visibility and even compliance. I’ll drop you in here as Adam Cardillo explains why they bought Cmd…
Adam Cardillo, Engineering Team Lead at Hewlett Packard Enterprise (HPE):
The environment itself, that we already had, had all the logs going everywhere. We had a fair bit of visibility, but it’s not the same visibility that Cmd gives you. So one of the great things about Cmd is that control and controlling, you know, root users. So yeah, that was probably the main driver and yeah, the auditing was really the cherry on top.
So it’s like centralization of that… like centralizing all of that control, right?
So can you walk us through an example of a box you could tick because you actually got this software? Like what was one of the requirements that you were actually able to meet?
So we have a strong sort of uptime requirement in the environment. And yeah, I guess from my experience, most downtime is not caused by any malicious activity; it’s really by people. You know, changes going bad or people making mistakes and Cmd can really control that, you know, preventing people from accidentally rebooting a box, as an example.
So you were, you were protecting your uptime from your coworkers, were you? That’s a pretty [laughter]…
Yea, we have lots of different teams logging into this environment. So it could be customer logging in, it could be an application team. And they don’t typically have those sorts of permissions to cause a reboot, but…
Yeah, for the things they do manage, they can change configuration files and make changes. So I can remind people, “Hey, you’re changing this configuration file, you know that you need to change requests for this.” Or I can say, “Hey, you need approval from the application owner to make this change.” So now I can do MFA on top of that.
So there’s lots of things you can do to prevent someone or warn someone that what they’re doing is dangerous or requires change or whatever it’s going to be.
Now, you just mentioned MFA… how far down the sort of geo-integration road did you go with CMD?
We already had Duo in the environment anyway, so it was a really easy transition. So it’s really just adding that extra integration between Cmd and Duo.
So you can set up groups, right? So you can have it on specific users and the integration allows you to have different groups within Duo as well. So I guess, for example, if I wanted to reboot a server, at the moment, we’ve got it set to MFA just because, you know, the people that have those permissions to do the reboot are the senior administrators of that environment.
So they’re the technical leads of that environment, but if they accidentally type in that command it’s going to prompt them from MFA and also a terminal message… it says, “Hey, are you on the right box? Make sure you’re in the right box. Did you mean to do this?” And if so you can MFA and that will reboot the box.
You just made me think about how, you know, when you used to do a DOS delete and the Yes-Enter… when it asked you if you were sure just became muscle memory. Like, the amount of stuff that I’ve nuked by just going Y-Enter is crazy, but it’s hard to muscle memory a 2FA prompt, isn’t it?
Yeah. Well, it’s like a Duo push, right? So you have to pick up your phone and do it.
I’ve rebooted Production boxes before when I shouldn’t have, and I started thinking that if I had this on there, I would have checked to say, “Yes, I am on the right box.” You know? If you still go through that whole process and you still repeat the box then, okay, that’s really on you.
It’s interesting, right? Cause what’s emerging here is that you’re kind of using Cmd as “bubble wrap” for your environment, right? To stop things getting accidentally broken. I mean, is that a fair assessment?
Yep. So we will have all the other triggers for the monitor tracking and different sort of custom things that we’ve made… but yea it’s a big thing.
So protecting the environment from the people that manage it is a big thing because most of the time that’s what’s causing the downtime, and downtime can cost money. And we’re an MSP, right?
Well, yeah. And downtime gets yelled at as well.
So also joining us is Curtis Simpson who’s the CISO for this ITOC group. You know, we’ve just heard Adam Cardillo discussing some of the Ops benefits and, like, stopping people from making mistakes. Are you also… well, obviously you’re on this call, right? So you’re using it. It seems like maybe Ops and Security are both using CMD in your environment, is that right?
Curtis Simpson, CISO at Hewlett Packard Enteprise (HPE):
Yeah, absolutely. And it was Adam that brought it to me, I guess, from an operational perspective to say, “Hey, there’s this really cool security tool that I’m looking at. I’m trying to lab up at the moment and get hands on with. What are your thoughts? How would you use this? What do you think?”
And we kind of worked together to investigate it, to see what it was capable of and where it could fit into a few of our private cloud environments.
So, I mean, we’ve just heard about the non-security benefits, more the Ops benefits. What are some of the things you’ve done with it?
Yeah, so I’m focused heavily, as Adam is from an operational perspective, I’m focused on it from an insider threat perspective, but then also a lot of compliance. So, on the particular private cloud environment that we’re utilizing this in production, I’ve probably got three audits every year —
It’s funny, right? Cause we just heard from the Ops guy who was saying, “Oh yeah, you know, audits are just like a nice cherry on top.” And here’s the security guy coming along saying, “Audits are absolutely critical.” Right?
[laughter] Yes! So a key area for us is definitely that privileged access management because it is a shared environment.
So there’s our team, there’s an offshore team, and then there’s also a customer that has various levels of privileges throughout the environment. So they need access to different accounts, they need access to, potentially, to get to a shared account or to root for a particular project, whatever it’s going to be…
And being able to have that complete audit, to have the session management and to know what changes have taken place… everything it’s, yeah, fantastic just to be able to bolt this on.
Adam’s talked about before the, the logging, we’ve got the centralized logging, large Oscar deployment and things like that in there. But it could only go so far. So even from a logging perspective, you turn everything on… it’s very, very hard to know, the particulars around, even when you’re doing an audit or an investigation, what syntax was used in a particular command, what was changed inside of a particular file at that particular time.
So yeah, Cmd slots in perfectly and assists with a range of different things for us in that environment.
Now look, I can’t just sit here… Jake’s actually on the call, he’s lurking, right? Jake King, founder of Cmd. I can see him cause we’re doing this by video, I can see him lurking, but I got to ask… room for improvement, right? Like where can Cmd be better?
I know one of the examples with some switching shells… When, when we’re looking at this, this particular user, when they switched their shell we seem to lose the audit trail. Like, what happened between here and here?
And then the advantage that we’ve got with Cmd is that I can say to Adam, “Hey, what happened here?” He goes, talks to the Cmd guys directly and it’s fixed within a week. That’s not something that we see very often —
From vendors? Yeah, no, it’s not right. They usually just say “Thanks for the info!” and that is the last you ever hear of it. Right?
Yeah. So I guess, like, what would you say to anyone out there who’s thinking about taking a look at it?
Yeah. So from the security perspective I would say, if you’ve got any sort of large Linux environment, it’s absolutely perfect.
We use it, I guess, through a different use case to what most would in terms of those large cloud environments and lots of developers. We’re heavily focused on the smaller internal private clouds. But it’s a perfect use case for us. That is more a big win for us, for security and operations, because we are so focused on us being the primary risk [hehe] and making sure that we’re doing the right thing.
Yeah, from an ITOC perspective, from a security compliance perspective, it helps us tick a lot of boxes.
It’s like a Linux herder, isn’t it? It’s like a sheep dog. You wrangle it!
Yeah, alright, Curtis Simpson and Adam Cardillo. Thank you so much for joining me on the show, to sing the praises of Cmd, our sponsor. It’s been a real pleasure to chat to you.
Cool. Thanks, bye.
That was Adam Cardillo and Curtis Simpson of HP’s ITOC. Big thanks to them for that. And yeah, they were appearing to extol the virtues of Cmd security, and you can find them cmd.com and that is it for this week’s show…