HashiCorp Boundary and Cmd

# Boundary’s Purpose: Secured Server and Service Access

If you’ve worked in Dev, DevOps, SecOps or Ops you’re likely familiar with this daily rhythm:

• start your VPN
• SSH or remote desktop to a few servers
• check some internal monitoring websites

Hopefully this involves MFA, and solid encryption in transit. You’ve likely seen that granting new teammates access to all these services can take longer than desired, or worse that they still had access after they moved on to a new job.

This is where Boundary comes in. It provides a TCP proxy layer in front of all your services and a consistent way to authenticate to that proxy layer before it decides to grant access to the requested service. It also solves the discovery problem that arises as your infrastructure grows. Where is the Staging4 environment we spun up last week? No problem, point your browser at Boundary’s Controller and you can find all the services and hosts.

TCP support covers the majority of services people use since most services are accessed by some protocol on top of TCP: websites (HTTPS), SSH, database clients, etc. Even services that traditionally used UDP such as DNS are moving towards TCP (DoH, DoT). Interestingly, HTTP may move from TCP to UDP when HTTP3 lands, but that’s a ways out.

Boundary supports Windows, MacOS and Linux as client platforms, with both a command line and a browser user interface. Using boundary to connect to an SSH server flows something like this:

One of the delightful things about Boundary is that it makes a point of clearly presenting and documenting its directory data model. It also offers a simple, ReST-ful API for interacting with it, which in turn is used in the permissions model (CRUD+list and a few actions for specific domain objects).  Given that correct permissions are critical to security, having a simple data model as their basis is important.

Another nice feature of the domain model is that it contains everything you need – i.e. not just organizations, users and groups but authentication methods, projects/scopes, hosts, host sets, targets/services, sessions and the roles with permissions that can be bound to users or groups.  And no, it does not offer groups nested in other groups, in case you’re suffering PTSD from some LDAP or AD mapping exercise.

Some of the distinguishing features of Boundary include:

• Its concept of a Session is one or more connections to a Target.  One can set an upper bound on session duration and the number connections per session to a particular Target.  The noise reduction value of this is immense to SecOps who can review a session with 1000’s of connections to a website Target over 8 hours as a single session. All while allowing traditional SSH sessions to have a connection limit of one.
• Load balancing (random) connections to a Target over the hosts in that Target. This will become even more valuable when dynamic host catalogs are supported from things like Consul.

Boundary does not offer or require an agent on hosts that are accessed through Boundary.  As such, things like SSH key provisioning, SSH certificate signing and Linux PAM are left to different tools such as HashiCorp Vault. This also means that any host TCP port you wish to access through Boundary must be open to the VPC network so the Worker can access it.  AWS SSM offers a websocket backchannel to the SSM Service (~Worker) so host ports listening solely on localhost (not directly network accessible) can be accessed from SSM clients.  Network accessible ports also require extra work to configure source IP access rules such that only the Workers may access them so lateral movement from other hosts in the VPC is not possible.

Given that Boundary’s first release was last week, it’s still pretty green in some areas:

• It doesn’t yet communicate TCP FINs through the websocket so you can end up with a hanging connection between the Worker and the Host or, if the Host goes away, the client, boundary proxy and Worker may keep the connection alive until the user terminates the client.
• Postgres is awesome and we love it, however critical infrastructure like Boundary needs to be highly available (and ideally entirely in the same region/data center as the servers it protects).  Postgres’ hot standby mode is read-only, which means new sessions and connections within those sessions cannot be inserted into the database if the main server fails – ie no one can access the infrastructure (CockroachDB might be more suitable).
• The Boundary client doesn’t support reading from stdin and writing to stdout to access the proxied connection. This would make it convenient to use as an SSH ProxyCommand.
• Only password authentication is supported at this time however bring-your-own identity provider is on the roadmap which can be found here.

An important aspect of open source projects is how welcoming the community is, and from reviewing issues and forums so far, I’m happy to report a friendly, open-minded, egoless vibe.

# The Competition

The competition includes AWS SSM, GCP IAP, Gravitational Teleport, Strong DM and others.

AWS Systems Manager (AWS SSM)

AWS Systems Manager, originally called Simple Systems Manager, is usually just called SSM. While AWS SSM offers much more than proxying TCP connections, the “aws” command line tool can set up tunnels similar to the way boundary can, and because it includes an open source SSM agent (Go) on your servers it has some additional capabilities:

• No SSH key provisioning is required on the servers if you just use the SSM console
• No additional TCP port needs to be opened on the server and it’s protected from direct in-VPC access because the agents create a websocket to the AWS SSM service and all proxied TCP ‘connections’ arrive over this websocket.
• The aws command line client can listen on port to forward, or conveniently proxy stdin and stdout as input and output of the TCP connection.  This makes setting up ssh to automatically create an SSM proxy via ssh’s ProxyCommand feature very simple.
• SSM supports CloudWatch logging and metrics.
• The SSM agent can be installed on servers in your own data center and, with the right credentials, register their websocket with the SSM service in the nearest AWS region for incoming connections, thus avoiding network routing and ingress configurations. If the network between these servers and the AWS region is poor, concurrent SSM sessions multiplexed over the same TCP connection may be sluggish.
• It can capture the terminal output on SSM sessions (but not for tunneled ssh due to end-to-end encryption). These may be helpful for compliance reasons, however tty output recordings have a number problems: they are easy to bypass by disabling terminal echo, obfuscating the commands with base64 etc, and by launching shell scripts and program interpreters where what the script or program does is not recorded.  Cmd has a better approach for this discussed in the next section.

While SSM has some strong features, it does have some drawbacks:

• There is no concept of a service offered by a pool of hosts nor load balancing over them.
• Sessions are equivalent to connections so even if you did go to the effort to proxy a website through SSM, it would be hard to tie those connections into a single session.
• Only the agent is open source; the SSM service is hosted in AWS regions and managed by AWS.
• It has a much broader attack surface than just TCP proxying that includes script evaluation.
• Access control policies are via AWS IAM.

IAP does not require or offer an agent for your servers which requires less management but does complicate some aspects:

• Since your servers must expose a listening TCP port to accept the proxied connections, you may want to only allow connections from IAP to prevent internal lateral movement from other servers.  This is fairly easy since Google shares the simple IP address range used by IAP, but it’s administrative work that’s easy to forget. Installing an agent might be easier.
• Creating a login shell on the server requires traditional SSH as there is no SSM-like console or backchannel. However, Google’s OS Login feature is available to make this a breeze.  It is an extremely robust way to manage Linux access properly as it stores short lived ssh keys in the metadata service, offers PAM modules for access and sudo privilege checking and offers nsswitch user lookup into the metadata service as well.

Gravitational Teleport

Gravitational Teleport comes in an open source community edition and a commercial enterprise edition. It is basically an outgrowth of a custom SSH client and server into a system with required, transparent bastion hosts (aka proxy hosts) that can act as SAML service providers and OpenID Connect relying parties in order to allow login from a corporate IDPs (enterprise edition).

SSH has long had local and remote TCP port forwarding that allows people to stitch together their own tunnels. Teleport does too and has some RBAC on who may use it (enterprise edition).  It has some novel features like SSM-like backchannels from hosts to the proxy hosts.  It also has some terminal output recording and rudimentary eBPF recording of sessions (enterprise edition).

To get the full value from Teleport you must replace putty and OpenSSH clients with tsh, and all your sshds with teleport (aka ‘nodes’). Then the teleport proxy, and authentication services must be installed as well in a highly available way that can meet your load requirements.  Nodes will require a significant amount of configuration such as installing the BCC BPF tools, configuring PAM to create Linux users on first login and wiring them up to speak to the Teleport proxy and auth servers.

Teleport also has a ‘directory’ data model and web UI similar to Boundary except it’s more complex. In addition to resource CRUDL it includes concepts related to SSH, and allows regular expressions instead of basic wildcards in some places. There is a session review UI as well.

Strong DM

Strong DM, according to their product description, is very similar to Boundary, but is a closed-source commercial product.  I have no experience with it but it’s worth mentioning as a production ready solution that is available now.

# Session Auditing with Cmd – See What Your Users and Services Really Did

Perspective is critical and at Cmd we believe only an agent on the Linux server can have the perspective to capture what actually occurred, for both users and applications like web servers autonomously running on those servers.  The language used to record what happened is equally critical for allowing reconstruction of what occurred, perhaps from years worth of data.  Additionally, archival data must be stored in format that will stand the test of time and be stored on a medium resistant to disk failure and bitrot (e.g. S3, GCS, Azure Blobs).  Cost of storage is also important; your bill should not be tempting you to discard valuable data.

Some of the tools introduced above (not Boundary) offer rudimentary user terminal output logging under specific configurations but not others.  Even if it were recorded under all configurations, terminal output recording suffers some major drawbacks:

• Terminal recordings are what the user saw in their terminal, not what actually happened
• If they are captured in-transit vs on the host, end-to-end encryption is not being used (there is a trusted MitM)
• It’s easy to bypass by disabling terminal echo or obfuscating commands with base64 or other standard obfuscation methods.
• Launching shell scripts and other interpreters like Python on a script file in the terminal does not show what the script does. Almost all servers have shells like bash available. See GTFObins.
• Some programs are proprietary. If you review upatr4 -p bkdoor there is no man page to tell you what it does. Or worse, someone’s PATH includes ~/bin/ as the first entry and they run date an awful lot with weird parameters.
• Transport-centric capture mechanisms don’t show the N other ways people are accessing the server. Perhaps both SSM and SSH are enabled. Perhaps your IPMI/ILO BMC is creating console or terminal sessions from its management network.
• Most often the primary workloads of the servers are applications such as web servers and databases that are started by Systemd, not a terminal session.  Wouldn’t you want to see if your application server was hacked, perhaps over public facing HTTPS, to fork and exec bash?

At Cmd we believe the lingua franca of what happened on a Linux server is the Linux process model itself, including all the glue between processes that allows data to flow: file access, network access, ttys, pipes, fifos, Unix domain sockets, etc.  Some of the data that flows is also of interest such as tty output. We also believe that significant context information in the events we capture is required for fast, low-cost response to these events as they occur, and simpler analysis at a later date. For example remotely processing a file write event to /etc/shadow with a timestamp and a link to the writing process id, will take significant lookup and processing to discover what session it was in, which server it was on, which user started that session, whether it was in an SSH session, and what the geo-ip of the SSH client was.

It is far simpler, cheaper and faster if these events are decorated with this information on the server before they are persisted.  Cmd is not a system call tracker, as we consider that path to be a perpetual race to keep up with new system calls (there are many variants that do the same thing like open a file) and extracting meaningful representation of what they collectively do is not something we want to ask our users to do.

We’d suggest you take a look at Boundary if you’re excited to replace a tired or home-grown solution you may have running in your own environment, or as a replacement for existing server and service access methods you’re using within your own environment. If you’re keen to see why we’re super excited about the Boundary launch, check out Cmd Free to add best-in-class observability to what we believe will be an industry standard in the near future.

To get your hands on Boundary, you can visit the project website at boundaryproject.io or head to  github.com/hashicorp/boundary to get the source code. At HashiCorp Learn you’ll find resources to get started with Boundary.

Click here to sign up for your account of Cmd Free, and head to our Help Center for guides and tutorials.