Jun 16 · 6 min read
Six Steps to Harden Linux Production Environments
By: Jake King
TL;DR: It can be a challenge to understand where a cloud vendor’s security ends and where an internal team’s responsibilities begin. Cmd will be doing a deep dive into hardening Linux production environments in the coming months. In this initial post, we walk through six steps any organization can take to get started.
Whether organizations are Fortune 500 companies or small startups just getting started, it can be challenging to identify and prioritize the right layers needed to ensure security. No cloud app is built the same, driving a highly customized approach to securing production Linux environments. With the rise of software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS), responsibilities for security patching and maintenance are not always clear. While some vendors spell out their policy, it can sometimes be a challenge to understand where a cloud vendor’s security responsibility ends and where an internal team’s responsibilities begin.
But as unique as every organization’s cloud situation is, there are also common, shared issues when it comes to Linux infrastructure and security. Some common challenges we see across company sizes and cloud environments include:
- Developing and perfecting continuous integration/continuous deployment (CI/CD) processes
- Giving the right amount of access to users in production environments
- Moving to immutable infrastructure for more timely, on-demand updates
- Ensuring images are fresh and safe
We hear these concerns so often that we recently hosted an entire Cmd Academy session on this topic. In this companion post, we’re going to walk through six steps to harden Linux production environments.
Let’s get started.
Step 1: Determine Trust Boundaries
The first thing any organization needs to do is understand the boundaries of what they can trust. Questions to ask include:
- Can you trust your CSP?
- How about your hardware?
- What about your virtual environments?
- Who has physical access, and who can tap into what machines?
Even if you can trust these different vectors for security, there is also the question of how much to trust them. Understanding the limits and capabilities of your CSP, hosting service, and data center provider is a basic and crucial part of managing risk.
When assessing boundaries, consider working with a firm to assess risk for different vendors. A trust audit can reveal where information is stored and, crucially, how it’s being stored. Once those two factors are figured out, ensure you understand where compliance needs are being met, for example SOC 2.
Step 2: Determine Where Your Responsibility Lies
Establishing boundaries as described above is a bit like drawing a map—you get a sense of where all the kingdoms are, what’s within them, and who is ruling each area. Once the map is drawn, organizations can begin to understand who is responsible for what.
If your organization runs everything on-prem, that’s an easy calculation to make—all the responsibility is yours. But with the cloud, responsibilities are shared. For example, an IaaS provider may handle more security steps, while an SaaS tool may handle security for data access, but leave everything else up to the company. In short, not every host is equal in the cloud. That’s why getting a map together is key—once you understand the boundaries of responsibility for each area, you can develop specific security plans and policies in conjunction with the buyer, and then hold them accountable for their portion of the partnership.
Step 3: Get Things up to Date—and Fast
With boundaries and responsibilities established, now it’s time to get things updated quickly. There’s an old joke in tech: “patch Tuesday, exploit Wednesday.” While this joke started with Windows, it’s true for all operating systems today, including Linux—especially in today’s fast moving, constantly evolving threat landscape. Whether it’s patching, spinning machines up or down, or implementing immutable infrastructure, the job of the security team is to make hackers’ lives harder. We’ve seen data leaks and malicious behavior stemming from all kinds of sources, like Shellshock, misconfigured S3 buckets, sudo bugs, and more.
Some key actions include:
- Determine the package footprint (don’t forget containers!)
- Identify outdated packages
- Capture the state of the system
- Understand developer behaviors (e.g. where/why developers are pinning versions)
These four checklist items are not a one-time thing, either. Organizations should continuously or regularly check for outdated packages that are exposing vulnerabilities—and don’t forget to look retroactively, lest something was missed and exposed the company to risk in the past. Organizations should also perform spot-checks on the state of their systems to ensure nothing slips by. Users who break the rules or exhibit poor judgement should be black-listed—or at the very least, educated about the risks of their behavior.
Practically speaking, organizations are experimenting with new ways to update machines. For example, some companies we’ve spoken with at Cmd have instituted policies that limit images to a 6-month lifespan.
Step 4: Map and Tune Network Interfaces
Network interfaces leave a sizable attack surface for bad guys. When organizations grow fast, a small oversight in network interfaces can quickly become a big oversight—and a big problem. That’s why organizations should take an assessment of their network interface footprint and ask:
- How many assets are in play?
- What are they?
- Are there any exposed ports?
- What is the actual network interface?
Understanding production states is key, too; whether in pre-production or staging, a quick discovery scan can reveal gaps in security.
Oftentimes, organizations will discover that developers are using ad-hoc systems. Getting a handle on these and locking them down will go a long way in hardening security. If these—or sanctioned systems that haven’t been adequately configured—are discovered in an assessment, organizations should disable unused default capabilities. Some frequent default capabilities that can be turned off include: ICMP response, forwarding, and redirects.
For best practices and benchmarks, the Center for Internet Security is a trusted resource with a helpful set of standards. Without benchmarking—both externally and internally—it’s hard to give yourself a grade on security. Another resource is the organization’s CSP(s), which often has helpful capabilities upstream from the company. Don’t forget to tap into those capabilities, too.
Step 5: Configure Sensible Records and Audit Capabilities
Good record-keeping can be the difference between stopping and solving a security issue, or letting disaster continue on—often without an organization’s knowledge. Whether an organization is being attacked by an outside source or an authorized user is misbehaving, good records can enable thorough and effective audits.
Steps organizations can take to maintain auditable records include:
- Determine the connection state in production
- Configure native logs to prevent local tampering
- Monitor logs and run against a database to assess (e.g. Athena, Snowflake, GoodData, Tableau)
- Ensure that timestamps are accurate and consistent
- Benchmark system performance wherever and whenever possible
This isn’t all just inside baseball. Beyond hardening security, good record-keeping and audit capabilities can help organizations with compliance needs for regulations like SOC2, PCI, and more.
Step 6: Step Up Authentication and Authorization
One of the best ways to harden Linux production environments is to understand who is in those environments, control their access, and monitor their behavior. The best approach is one of least privilege access—make sure that only people who really need access actually have it.
Organizations can start by categorizing users, which services they use, and their default accounts. A key way to reduce security risk is to minimize the number of services that have shell access—and we’re not just talking SSH. Consider things like SSM, Cloud Shell, and configuration management tools that touch the shell. Wherever possible, reduce and remove user privilege—especially native root access. At most organizations, that’s a level of power not a lot of users need to have.
We’re also big fans of implementing two-factor authentication throughout your production environment at Cmd – not just at the gateway. It’s a great way to ensure the correct people are getting access to crucial systems, and many 2FA solutions also allow organizations to fine-tune permissions based on workers’ specific roles or even to provide time-limited access. Sudoers or third-party tools, like Cmd, can help ensure appropriate levels of access for each organization’s specific needs.
Improving Security by Hardening Linux Production Environments
Linux production environments can be complicated, and as more companies shift to cloud-based services, boundaries and responsibilities are also becoming more interconnected and complicated. However, by implementing some security core principles, asking the right questions, and developing a clear plan, organizations of all sizes can harden their Linux production environments for safer and more compliant development processes.
Have you hardened your Linux production environments? Is your organization changing policies or procedures? Leave us a comment on LinkedIn or Twitter to share your experiences @cmd_security.
Secure your Linux Production Environment & Request a Demo of Cmd.