HashiCorp Boundary and Cmd
Cloud-Provider-Agnostic Service and Server Access Management combined with Scalable Production Linux Observability
Cloud-Provider-Agnostic Service and Server Access Management combined with Scalable Production Linux Observability
Last week HashiCorp announced the first release of Boundary, an open source proxy system that provides secure service and server access management for your staff. HashiCorp is known for high quality open source software with a strong focus on elegant design, security and scalability – we’re big fans. In this article we hope to introduce Boundary, compare it to some of the competition, and explain why it’s a great pairing with Cmd’s eBPF monitoring solution for Linux.
We’ll start by describing Boundary’s purpose, how it works and its architecture. Based on this we’ll introduce some of the competitors such as Google IAP, AWS SSM, Gravitational Teleport and how they compare. Finally, we’ll discuss why Cmd Audit is an essential addition to any of these solutions if you truly want to audit what your users and services do.
If you’ve worked in Dev, DevOps, SecOps or Ops you’re likely familiar with this daily rhythm:
Hopefully this involves MFA, and solid encryption in transit. You’ve likely seen that granting new teammates access to all these services can take longer than desired, or worse that they still had access after they moved on to a new job.
This is where Boundary comes in. It provides a TCP proxy layer in front of all your services and a consistent way to authenticate to that proxy layer before it decides to grant access to the requested service. It also solves the discovery problem that arises as your infrastructure grows. Where is the Staging4 environment we spun up last week? No problem, point your browser at Boundary’s Controller and you can find all the services and hosts.
TCP support covers the majority of services people use since most services are accessed by some protocol on top of TCP: websites (HTTPS), SSH, database clients, etc. Even services that traditionally used UDP such as DNS are moving towards TCP (DoH, DoT). Interestingly, HTTP may move from TCP to UDP when HTTP3 lands, but that’s a ways out.
Boundary supports Windows, MacOS and Linux as client platforms, with both a command line and a browser user interface. Using boundary to connect to an SSH server flows something like this:
# Authenticate to the Boundary Controller, where the credential is stashed in your platform’s keystore for future use:
boundary authenticate password -login-name=admin \
-password password -auth-method-id=ampw_1234567890
# Run boundary in the CLI so it will connect to a host randomly selected by the Boundary Controller from the Target’s host sets:
boundary connect ssh -target-id ttcp_1234567890
Note that the last step actually runs OpenSSH‘s ssh client in a somewhat novel way such that it uses the proxy connection. Running Boundary will first use your default credential and create a session with Boundary Controller, then it will start a TCP listener on an ephemeral port whose subsequent connections will be forwarded over HTTPS (wss) to a Boundary Worker and then as plain TCP on to the backend server. Then it forks and execs ssh with command line arguments to direct it to connect to its parent process’ localhost listener port, instead of directly to the host. If your staff really want to manage every aspect of their client software, Boundary can simply start the proxy localhost listener and the user can point their client at the TCP port it prints out.
Boundary provides built-in convenience like this for ssh (OpenSSH or Putty), RDP, curl and the Postgres DB client. Open source means you can ensure your favorite client is easy to use by contributing to the project.
Providing secure, highly available and manageable infrastructure for access control and proxying requires quite a few moving parts. All the HashiCorp provided components are written in Go which has proven to be the language of choice for cloud infrastructure (see Kubernetes) thanks to its readability, portability, efficiency, great system libraries, tooling and elegant model for massive concurrency. Simple is secure. The primary components of Boundary include:
One of the delightful things about Boundary is that it makes a point of clearly presenting and documenting its directory data model. It also offers a simple, ReST-ful API for interacting with it, which in turn is used in the permissions model (CRUD+list and a few actions for specific domain objects). Given that correct permissions are critical to security, having a simple data model as their basis is important.
Another nice feature of the domain model is that it contains everything you need – i.e. not just organizations, users and groups but authentication methods, projects/scopes, hosts, host sets, targets/services, sessions and the roles with permissions that can be bound to users or groups. And no, it does not offer groups nested in other groups, in case you’re suffering PTSD from some LDAP or AD mapping exercise.
Some of the distinguishing features of Boundary include:
Boundary does not offer or require an agent on hosts that are accessed through Boundary. As such, things like SSH key provisioning, SSH certificate signing and Linux PAM are left to different tools such as HashiCorp Vault. This also means that any host TCP port you wish to access through Boundary must be open to the VPC network so the Worker can access it. AWS SSM offers a websocket backchannel to the SSM Service (~Worker) so host ports listening solely on localhost (not directly network accessible) can be accessed from SSM clients. Network accessible ports also require extra work to configure source IP access rules such that only the Workers may access them so lateral movement from other hosts in the VPC is not possible.
Given that Boundary’s first release was last week, it’s still pretty green in some areas:
An important aspect of open source projects is how welcoming the community is, and from reviewing issues and forums so far, I’m happy to report a friendly, open-minded, egoless vibe.
The competition includes AWS SSM, GCP IAP, Gravitational Teleport, Strong DM and others.
AWS Systems Manager, originally called Simple Systems Manager, is usually just called SSM. While AWS SSM offers much more than proxying TCP connections, the “aws” command line tool can set up tunnels similar to the way boundary can, and because it includes an open source SSM agent (Go) on your servers it has some additional capabilities:
While SSM has some strong features, it does have some drawbacks:
aws ssm start-session \
--target instance-id \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["80"], "localPortNumber":["56789"]}'
Starting a Proxy with SSM
host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
ssh client configuration to automatically use an SSM proxy tunnel
Google’s Identity Aware Proxy’s (IAP) biggest strength is providing secured, authenticated access to HTTPS/websites; few may realize that it can forward TCP connections as well. The gcloud compute ssh INSTANCE_NAME will automatically tunnel ssh through IAP if the instance has a non-public IP address. Similar to SSM, IAP is a cloud-provider managed service only, its access policies are part of GCP IAM and there is a way to expose on premise servers with IAP albeit in a complex way requiring GCP Cloud Interconnect.
gcloud compute start-iap-tunnel INSTANCE_NAME INSTANCE_PORT \
--local-host-port=localhost:LOCAL_PORT \
--zone=ZONE
Starting a proxy with IAP
IAP does not require or offer an agent for your servers which requires less management but does complicate some aspects:
Gravitational Teleport comes in an open source community edition and a commercial enterprise edition. It is basically an outgrowth of a custom SSH client and server into a system with required, transparent bastion hosts (aka proxy hosts) that can act as SAML service providers and OpenID Connect relying parties in order to allow login from a corporate IDPs (enterprise edition).
SSH has long had local and remote TCP port forwarding that allows people to stitch together their own tunnels. Teleport does too and has some RBAC on who may use it (enterprise edition). It has some novel features like SSM-like backchannels from hosts to the proxy hosts. It also has some terminal output recording and rudimentary eBPF recording of sessions (enterprise edition).
To get the full value from Teleport you must replace putty and OpenSSH clients with tsh, and all your sshds with teleport (aka ‘nodes’). Then the teleport proxy, and authentication services must be installed as well in a highly available way that can meet your load requirements. Nodes will require a significant amount of configuration such as installing the BCC BPF tools, configuring PAM to create Linux users on first login and wiring them up to speak to the Teleport proxy and auth servers.
Teleport also has a ‘directory’ data model and web UI similar to Boundary except it’s more complex. In addition to resource CRUDL it includes concepts related to SSH, and allows regular expressions instead of basic wildcards in some places. There is a session review UI as well.
Strong DM, according to their product description, is very similar to Boundary, but is a closed-source commercial product. I have no experience with it but it’s worth mentioning as a production ready solution that is available now.
Perspective is critical and at Cmd we believe only an agent on the Linux server can have the perspective to capture what actually occurred, for both users and applications like web servers autonomously running on those servers. The language used to record what happened is equally critical for allowing reconstruction of what occurred, perhaps from years worth of data. Additionally, archival data must be stored in format that will stand the test of time and be stored on a medium resistant to disk failure and bitrot (e.g. S3, GCS, Azure Blobs). Cost of storage is also important; your bill should not be tempting you to discard valuable data.
Some of the tools introduced above (not Boundary) offer rudimentary user terminal output logging under specific configurations but not others. Even if it were recorded under all configurations, terminal output recording suffers some major drawbacks:
At Cmd we believe the lingua franca of what happened on a Linux server is the Linux process model itself, including all the glue between processes that allows data to flow: file access, network access, ttys, pipes, fifos, Unix domain sockets, etc. Some of the data that flows is also of interest such as tty output. We also believe that significant context information in the events we capture is required for fast, low-cost response to these events as they occur, and simpler analysis at a later date. For example remotely processing a file write event to /etc/shadow with a timestamp and a link to the writing process id, will take significant lookup and processing to discover what session it was in, which server it was on, which user started that session, whether it was in an SSH session, and what the geo-ip of the SSH client was.
It is far simpler, cheaper and faster if these events are decorated with this information on the server before they are persisted. Cmd is not a system call tracker, as we consider that path to be a perpetual race to keep up with new system calls (there are many variants that do the same thing like open a file) and extracting meaningful representation of what they collectively do is not something we want to ask our users to do.
We’d suggest you take a look at Boundary if you’re excited to replace a tired or home-grown solution you may have running in your own environment, or as a replacement for existing server and service access methods you’re using within your own environment. If you’re keen to see why we’re super excited about the Boundary launch, check out Cmd Free to add best-in-class observability to what we believe will be an industry standard in the near future.
To get your hands on Boundary, you can visit the project website at boundaryproject.io or head to github.com/hashicorp/boundary to get the source code. At HashiCorp Learn you’ll find resources to get started with Boundary.
Click here to sign up for your account of Cmd Free, and head to our Help Center for guides and tutorials.
Cmd provides the fastest, easiest way to know what’s happening in your Linux environment, who’s logging in, and what they are doing.
Activate Your Free Account arrow_forward