GUACAMOLE-2283: Runtime observability: add guacd process & thread naming.

[bbennett-ks]

Overview

Add runtime process and thread naming support to guacd. Connection processes now expose descriptive names (e.g., vnc user@host:5900) and threads expose meaningful names for easier debugging, profiling, and monitoring.

This change adds process title and thread naming support to guacd through three new APIs:
-guac_process_title_init(): captures the process's original argv/environ region at startup and prepares it for later process title updates.-
-guac_process_title_set(): updates the process title shown by tools such as ps, top, and htop, and also updates the main thread's comm name so process-oriented views reflect the active connection.-
-guac_thread_name_set(): assigns a descriptive name to the calling thread using the platform's thread naming facilities.

guac_process_title_set() rewrites the argv/environ memory region captured at startup, replacing the original command line with a connection-specific title. Because Linux exposes this region through /proc/<pid>/cmdline, tools such as ps, top, and htop automatically display the updated title. The main thread's comm name is also updated to keep process-oriented views consistent.

Child processes can use guac_process_title_set() to expose connection-specific names, while threads created can use guac_thread_name_set() to publish names such as vnc-main, user-input, or rdp-audio. This improves observability when debugging, profiling, or monitoring running guacd processes.

Thread Names

conn-write:     Forwards data from the connected user to the connection's child process.
conn-read:      Forwards data from the connection's child process back to the connected user.
conn-route:     Performs the protocol handshake for a new client connection and routes it to a connection process.

user-conn:      Managesa single user's connection lifecycle from handshake through disconnect.
user-input:     Readsand parses instructions from a single user's socket.
user-pending:   Periodically promotes pending users into the active connection.

client-free:    Frees a guac_client in the background, bounded by a timeout in case the free handler hangs.

display-render: Drives the display render loop, flushing completed frames to the client.
display-wrk:    Encodes and sends graphical updates for dirty layer regions.

keep-alive:     Periodically sends keep-alive NOPs on an otherwise idle socket.

k8s-in :        Reads user input and forwards it to the Kubernetes pod.
k8s-worker:     Main Kubernetes client thread; manages the websocket connection to the pod.

rdp-audio:      Flushes buffered audio input to the RDP server at the negotiated rate.
rdp-print:      Streams output from the RDP print filter process to the client.
rdp-worker:     Main RDP client thread; runs the FreeRDP connection and event loop.

ssh-in :        Reads terminal input and forwards it to the SSH server.
ssh-worker:     Main SSH client thread; runs the SSH session and drives the terminal.

telnet-in:      Reads terminal input and forwards it to the telnet server.
telnet-worker:  Main telnet client thread; runs the telnet session and event loop.

vnc-worker:     Main VNC client thread; runs the libvncclient connection and message loop.

terminal:       Renders the terminal emulator display and processes output from the remote.

Username Obfuscation

Showing the full username (e.g. vnc bbennett@host:5900) in the process title is discloses information that could be accessed by a non-admin user on the guacd server and could be used as a security exploit. Even though guacd is typically deployed on locked down systems with only admin access, the username is obfuscated to mitigate this potential security issue.

For example, vnc bbennett@host:5900, displays as vnc bb****tt@host:5900; short usernames, such as root, are completely obfuscated: **telnet **@127.0.0.1:23.

Example Output

[bbennett@rocky-9]$ ps -eo pid,ppid,comm,args --forest | sed -n '/ 272004 /,$p'
...
 272004  272002  |          gua  |                       \_ guacd -f -b 127.0.0.1 -l 4822 -L debug
 272106  272004  |          rdp  |                           \_ rdp bb****tt@192.168.238.152:3389
 272122  272004  |          ssh  |                           \_ ssh bb****tt@192.168.238.152:22
 272146  272004  |          rdp  |                           \_ rdp Br****tt@192.168.238.154:3389
 272177  272004  |          ssh  |                           \_ ssh Br****tt@192.168.238.154:22
 272188  272004  |          vnc  |                           \_ vnc bb****tt@192.168.68.50:5900
 272198  272004  |          tel  |                           \_ telnet ****@127.0.0.1:23

[bbennett@rocky-9]$  pstree -apt 272004
guacd,272004 -f -b 127.0.0.1 -l 4822 -L debug
  ├─rdp Br****,272146
  │   ├─{display-render},272392
  │   ├─{display-wrk},272157
  │   ├─{display-wrk},272158
  │   ├─{display-wrk},272159
  │   ├─{display-wrk},272160
  │   ├─{display-wrk},272162
  │   ├─{display-wrk},272163
  │   ├─{display-wrk},272165
  │   ├─{display-wrk},272166
  │   ├─{display-wrk},272167
  │   ├─{display-wrk},272168
  │   ├─{display-wrk},272169
  │   ├─{display-wrk},272170
  │   ├─{display-wrk},272171
  │   ├─{display-wrk},272172
  │   ├─{display-wrk},272173
  │   ├─{display-wrk},272174
  │   ├─{keep-alive},272151
  │   ├─{rdp-worker},272154
  │   ├─{rdp-worker},272284
  │   ├─{rdp-worker},272285
  │   ├─{rdp-worker},272286

[necouchman]

Overall looks pretty good to me, and I think this is a very useful addition to the code. I do have a few comments - mainly related to doing a few more #define constants and documenting things, but also one related to compatibility of the code across non-Linux UNIX-like platforms.

[eugen-keeper]

The formatted process titles expose connection metadata (username, host, port) in ps and /proc, where other local users may potentially see it. Do we consider that acceptable for typical guacd deployments?

[bbennett-ks]

The formatted process titles expose connection metadata (username, host, port) in ps and /proc, where other local users may potentially see it. Do we consider that acceptable for typical guacd deployments?

I had concerns as well, especially for the username, as the other information is readily available to non-root users.. My thought is that because of the Guacamole deployment model, guacd is only accessible by admins, this is acceptable. But I could see an argument for removing username.

I added a comment to proctitle.h to emphasize the security aspect.

[eugen-keeper]

LGTM

[necouchman]

The formatted process titles expose connection metadata (username, host, port) in ps and /proc, where other local users may potentially see it. Do we consider that acceptable for typical guacd deployments?

I had concerns as well, especially for the username, as the other information is readily available to non-root users.. My thought is that because of the Guacamole deployment model, guacd is only accessible by admins, this is acceptable. But I could see an argument for removing username.

I added a comment to proctitle.h to emphasize the security aspect.

This is probably okay - I would tend to agree that typical deployments are not going to grant just anyone access to the system running guacd (or a system running a set of containers that includes guacd).

Curious what @mike-jumper @corentin-soriano @sschiffli think about this, particularly if there are any objections?

[bbennett-ks]

The formatted process titles expose connection metadata (username, host, port) in ps and /proc, where other local users may potentially see it. Do we consider that acceptable for typical guacd deployments?

I had concerns as well, especially for the username, as the other information is readily available to non-root users.. My thought is that because of the Guacamole deployment model, guacd is only accessible by admins, this is acceptable. But I could see an argument for removing username.
I added a comment to proctitle.h to emphasize the security aspect.

This is probably okay - I would tend to agree that typical deployments are not going to grant just anyone access to the system running guacd (or a system running a set of containers that includes guacd).

Curious what @mike-jumper @corentin-soriano @sschiffli think about this, particularly if there are any objections?

If there are objections, I could show the UID instead of the username. Another possibility is showing the connection name.

[necouchman]

The formatted process titles expose connection metadata (username, host, port) in ps and /proc, where other local users may potentially see it. Do we consider that acceptable for typical guacd deployments?

I had concerns as well, especially for the username, as the other information is readily available to non-root users.. My thought is that because of the Guacamole deployment model, guacd is only accessible by admins, this is acceptable. But I could see an argument for removing username.
I added a comment to proctitle.h to emphasize the security aspect.

This is probably okay - I would tend to agree that typical deployments are not going to grant just anyone access to the system running guacd (or a system running a set of containers that includes guacd).
Curious what @mike-jumper @corentin-soriano @sschiffli think about this, particularly if there are any objections?

If there are objections, I could show the UID instead of the username. Another possibility is showing the connection name.

Yeah - or, here's another idea that occurred to me, if the goal is to aid in "runtime observability", what a socket opened by guacd that could be read from that contains information on the guacd process(es)? This could be one of the following variants:

• A single socket opened by the main guacd process that records information about each of the forked child processes that contains useful/required information - some of the stuff, here, and maybe some additional items (counters, maybe?). This has the advantage of limiting the number of sockets that get opened, particularly on high-volume systems. This could be some sort of uniformly formatted output (CSV, JSON, YAML, etc.) that would contain the PIDs of the forked processes and the information about each one (protocol, user, hostname, connection UUID, total bytes transferred, etc., etc.).
• A socket per forked guacd process that contains this information for that specific process, in a uniformly-formatted output. This has the advantage of being able to more easily parse information for a specific process, but would result in a lot of sockets being opened.

I'm kind of channeling the way HAProxy provides a statistics socket that can be read from and will output the information in various formats. This seems like it might achieve some of the observability requirements, but has the advantage of being able to use filesystem permissions to limit who can read the data and interact with the socket.

I'm throwing that out as an idea - I'm not saying it necessarily should completely replace what you're doing, here, I still think this PR, as-is, is useful, and provided there aren't any objections to this information being accessible to anyone logged into the guacd system, I'm still okay with it. I'm just throwing it out as an idea, perhaps for a future change or to address some of the potential concerns, here.

[bbennett-ks]

The formatted process titles expose connection metadata (username, host, port) in ps and /proc, where other local users may potentially see it. Do we consider that acceptable for typical guacd deployments?

I had concerns as well, especially for the username, as the other information is readily available to non-root users.. My thought is that because of the Guacamole deployment model, guacd is only accessible by admins, this is acceptable. But I could see an argument for removing username.
I added a comment to proctitle.h to emphasize the security aspect.

This is probably okay - I would tend to agree that typical deployments are not going to grant just anyone access to the system running guacd (or a system running a set of containers that includes guacd).

Curious what @mike-jumper @corentin-soriano @sschiffli think about this, particularly if there are any objections?

To mitigate this, I've obfuscated the username, Details in the overview.

[corentin-soriano]

[bbennett@rocky-9]$ ps -eo pid,ppid,comm,args --forest | sed -n '/ 272004 /,$p'
...
 272004  272002  |          gua  |                       \_ guacd -f -b 127.0.0.1 -l 4822 -L debug
 272106  272004  |          rdp  |                           \_ rdp bbennett@192.168.238.152:3389
 272122  272004  |          ssh  |                           \_ ssh bbennett@192.168.238.152:22
 272146  272004  |          rdp  |                           \_ rdp BradleyBennett@192.168.238.154:3389
 272177  272004  |          ssh  |                           \_ ssh BradleyBennett@192.168.238.154:22
 272188  272004  |          vnc  |                           \_ vnc bbennett@192.168.68.50:5900
 272198  272004  |          tel  |                           \_ telnet bbennett@127.0.0.1:23

The formatted process titles expose connection metadata (username, host, port) in ps and /proc, where other local users may potentially see it. Do we consider that acceptable for typical guacd deployments?

I had concerns as well, especially for the username, as the other information is readily available to non-root users.. My thought is that because of the Guacamole deployment model, guacd is only accessible by admins, this is acceptable. But I could see an argument for removing username.
I added a comment to proctitle.h to emphasize the security aspect.

This is probably okay - I would tend to agree that typical deployments are not going to grant just anyone access to the system running guacd (or a system running a set of containers that includes guacd).

Curious what @mike-jumper @corentin-soriano @sschiffli think about this, particularly if there are any objections?

I think it's not good practice, but that might not justify blocking this pull request.

The target's IP address and port are already visible via netstat with a non-privileged account so I think the biggest risk is in the leakage of the username and the username naming format.

Server access should be restricted to administrators, but an unauthorized user who has gained access to a non-privileged shell by exploiting a vulnerability in another process will be able to access this data.

[bbennett-ks]

Yeah - or, here's another idea that occurred to me, if the goal is to aid in "runtime observability", what a socket opened by guacd that could be read from that contains information on the guacd process(es)? This could be one of the following variants:

• A single socket opened by the main guacd process that records information about each of the forked child processes that contains useful/required information - some of the stuff, here, and maybe some additional items (counters, maybe?). This has the advantage of limiting the number of sockets that get opened, particularly on high-volume systems. This could be some sort of uniformly formatted output (CSV, JSON, YAML, etc.) that would contain the PIDs of the forked processes and the information about each one (protocol, user, hostname, connection UUID, total bytes transferred, etc., etc.).
• A socket per forked guacd process that contains this information for that specific process, in a uniformly-formatted output. This has the advantage of being able to more easily parse information for a specific process, but would result in a lot of sockets being opened.

I'm kind of channeling the way HAProxy provides a statistics socket that can be read from and will output the information in various formats. This seems like it might achieve some of the observability requirements, but has the advantage of being able to use filesystem permissions to limit who can read the data and interact with the socket.

I'm throwing that out as an idea - I'm not saying it necessarily should completely replace what you're doing, here, I still think this PR, as-is, is useful, and provided there aren't any objections to this information being accessible to anyone logged into the guacd system, I'm still okay with it. I'm just throwing it out as an idea, perhaps for a future change or to address some of the potential concerns, here.

I think this would be a great addition, though I believe this would be a separate issue: having a subset of the information a HAProxy-like socket interface is useful in debugging and analyze system performance and scalability issues in the context of the entire system.

I like the single socket approach, though it provides limited information (connection ID, protocol, PID (of connection process), and start/up time. There is a bi-directional socker between the guacd and the connection process that could be extended to provide the rest of the information (username, host, port...).

I'll file a Jira for this.

[bbennett-ks]

> > The formatted process titles expose connection metadata (username, host, port) in `ps` and `/proc`, where other local users may potentially see it. Do we consider that acceptable for typical **guacd** deployments?
> 
> 
> I had concerns as well, especially for the username, as the other information is readily available to non-root users.. My thought is that because of the Guacamole deployment model, `guacd` is only accessible by admins, this is acceptable. But I could see an argument for removing username.
> I added a comment to `proctitle.h` to emphasize the security aspect.


This is _probably_ okay - I would tend to agree that typical deployments are not going to grant just anyone access to the system running guacd (or a system running a set of containers that includes guacd).
Curious what @mike-jumper @corentin-soriano @sschiffli think about this, particularly if there are any objections?

I think it's not good practice, but that might not justify blocking this pull request.

The target's IP address and port are already visible via netstat with a non-privileged account so I think the biggest risk is in the leakage of the username and the username naming format.

Server access should be restricted to administrators, but an unauthorized user who has gained access to a non-privileged shell by exploiting a vulnerability in another process will be able to access this data.

Does the username obfuscation mitigation mitigate the security concernse? I just pushed this a few minutes ago & updated the overview.

[bbennett-ks]

I think this would be a great addition, though I believe this would be a separate issue: having a subset of the information a HAProxy-like socket interface is useful in debugging and analyze system performance and scalability issues in the context of the entire system.

I like the single socket approach, though it provides limited information (connection ID, protocol, PID (of connection process), and start/up time. There is a bi-directional socker between the guacd and the connection process that could be extended to provide the rest of the information (username, host, port...).

I'll file a Jira for this.

https://issues.apache.org/jira/browse/GUACAMOLE-2288