If you’re reading this you probably already understand that Cmd collects data about what happens on your servers, but you may not know how that data is structured, or what it contains. This post describes the data Cmd collects, and how we present it, including:
the shared data model used by Cmd Free, Cmd Audit and Cmd Control;
the types of events Cmd monitors; and,
how the Cmd web app’s terminal displays sessions and events in context.
Cmd Free, Cmd Audit, and Cmd Control
Our different product tiers use different agents. Cmd Audit and Cmd Control each have their own agent, and Cmd Free uses a restricted version of the Cmd Audit agent. They all let you set up custom alerts, and have similar monitoring capabilities. In addition to monitoring, Cmd Control can enforce custom security rules by blocking commands pending 2FA or authorization via a third-party, etc.
Both agents monitor server sessions. For each session, Cmd records the Linux events generated by users and processes, and saves those events in context. The context includes information about five highly-relevant Linux processes. Of course, this facilitates long-term auditing and analysis — it also helps make data explorations in the Cmd web app more intuitive to people familiar with Linux.
Cmd Audit uses an eBPF-based agent that (among other things) can monitor containerized workloads and Kubernetes Nodes via a DaemonSet, but this article focuses on what both agents have in common: the structure of the data they record, which is grounded in the Linux process model.
For each event, Cmd keeps track of the:
# first connected ("inception") session process;
# session leader process;
# last known user-entered process;
# parent process; and,
# the current ("self") process, in which the event occurred.
The heuristics that Cmd uses to identify these processes are not in this post. Suffice it to say Cmd users can read more.
Included in each event’s context are the PID, PPID, and PGID for each of those 5 processes, as well as a wealth of other information about each process (see the screenshots below).
Types of events Cmd monitors
Cmd separates events into the following categories:
Session leader — the event responsible for the beginning of a session (e.g. an SSH connection, an SSM session, etc.).
Process execution — a command (not a Bash builtin) executed by a non-interactive process, rather than by a remote user.
User-initiated process execution — a command executed directly by a remote, or on-terminal, human user.
Bash builtin — a builtin executed by a non-interactive process.
User-initiated bash builtin — a builtin executed directly by a remote human user.
Command completion — represents the work the shell does finding matching executables or arguments when you press tab.
While there are minor differences in the data available for each event type, there is always enough information to contextualize and investigate each event. For example, for a given event you can: view all of the user’s sessions on a server, see how they connected, or review the rest of the commands in their session.
Cmd’s Terminal 2.0
The Cmd web app terminal view is used to present and explore session data throughout the Cmd web app. It appears on the sessions, reports, and alerts pages. By default it shows the following summary data for each session:
Date connected — when the session started.
User — name of the Linux user who executed the process (or the 2FA user, to help disambiguate shared Linux accounts).
Process — name of the process responsible for the session.
Commands — number of command (process and builtin) events recorded.
Alerts — number of alert events recorded.
User IP — connection IP address of SSH connections.
Server Name — your server’s Cmd name.
Hostname — your server’s hostname.
Session ID — a Cmd-assigned identifier.
The default summary of a Bash session.
When you select a session, the terminal opens to show all the events in that session — such as commands. You can customize the terminal to show the most relevant fields for each command. For example: what command was executed, when, from what IP address, and from what working directory.
A session in the Cmd web app, expanded to show the terminal.
You can open the tabs of the details pane to view information about the active session, or about any event that you click on:
The Cmd terminal with the details pane enabled.
The details pane shows information including an event’s process context such as the SID, PID, PPID, PGID, information about the server, and related alerts.
The Command details pane displaying information related to a selected event.
The Session details pane displays data related to the selected session.