TL;DR: Ensuring security in Production Linux environments is a major challenge. In this two-part series, we’re discussing practical steps to crawl, walk, and then run toward more secure and well-regulated production environments. Check out Part 1 for more about crawling, while this post discusses walking and running.
Securing Production Linux environments is no small task for developers. Each environment is highly customized, so there isn’t an easy, one-size-fits-all security solution or approach. Security teams need to develop customized, responsive strategies that reflect their organizations’ unique needs, processes, and products.
In this two-part series, we’re helping to answer the question on many security experts’ minds:
How do security teams gain better visibility and control in Production Linux environments without slowing down development processes?
Well, we’ve said it before and we’ll say it again: you have to crawl before you walk, and walk before you run.
In Part 1, we discussed the challenges of securing Production Linux environments and what it takes to start crawling. The watchword for organizations just getting started is visibility.
In this post, we’ll discuss the next two phases: walk and run.
By now, you’ve started collecting data and have a better sense of what’s going on; you may even have some hunches as to why it’s going on. Trends will inevitably appear, both expected and unexpected. But it’s still just data, and assumptions about what that data means are the enemy of good policies and security programs, so the next step is to actually interview users.
People are behind every Production Linux environment, and they’re probably behaving in certain ways for a reason. Every Dev, DevOps and IT team will have its own set of best practices and standard operating procedures, and security teams may have limited insight into this. Set up conversations to understand how they perceive their use of the systems, why they do things the way they do, and what’s motivating them. The goal here is not punitive—it’s to understand developers’ needs so you can help them do their jobs more securely without slowing them down.
Here’s an example:
Company A notices that a developer has set up an automated process on a production system. Every morning, the process goes in, checks a file to ensure it still exists, and sends the developer a report. The developer set this up for convenience and quality assurance—they want to make sure things are running correctly—which is a good instinct and one that shouldn’t be suppressed. But one day, the process fails. So the developer logs in, changes some things, and is back on their way… but there’s no accountability or visibility associated with that activity on a production machine, which means if they (unintentionally) introduced a major error that causes a breach or security incident, there is no way for the security team to track down the source.
At the walk stage, the goal would be to talk to this developer, understand their needs and motivations, and collaborate; build something with them that can fulfill the developer’s needs but that also fixes itself or alerts the right set of individuals should it fail, so there are no direct, unnecessary command-line changes happening without proper visibility and authorization.
Remember: this stage should be non-disruptive and collaborative. It’s not about controlling behavior (yet). It’s about working together as partners to understand and find solutions. This stage is about working together with DevOps and reassuring them that the security team wants to help without slowing down their work. After all, at the end of the day, everyone is on the same team—we all want to ship products, grow our organizations, and make end-users happy.
The good news is that humans can change their behavior. By now, your organization is working the right way. Developers aren’t running random, unmonitored scripts, and everyone is getting what they need in a safe, secure way that doesn’t slow down production.
Now, it’s time to implement controls that automatically block unsafe activities, rather than trying to deal with them manually after the fact. And since you’ve already changed problematic behaviors working as a partner, it’s easier now to addguardrails to ensure DevOps operates securely, without the security team constantly swooping in to stop activities.
Implementing controls should include the following elements:
Instituting clear and well-documented security policies
The goal in this stage is to actively block insecure behaviors and institute guardrails that make it more difficult for teams to do things that veer outside of established security policies or unintentionally introduce risk or downtime.
Production Linux Security: A Marathon, Not a Sprint
We’ll end this post with an example that illustrates the progression of crawl, walk, run:
A developer wants to use a utility tool that changes colors in production. In the “crawl” stage, your organization becomes aware of this behavior. By the “walk” stage, the organization may allow the utility tool but find ways to filter it out when the code is propagated to production machines. By the “run” stage, the tool is totally blocked in production should any preventative measures fail. Developers can be successful and thrive without putting the business at risk or taking security teams away from their day-to-day and strategic initiatives. Winning!
As Production Linux environments become more popular and common, security teams are under increasing pressure to protect code and data in these environments. This can seem like a huge challenge at first, but with a measured approach, companies can learn to crawl, walk, and then run headlong toward more secure, well-regulated Production Linux environments.