Splunk Operator: Enhancement request - use a command for the liveness probe instead of a URL #1431

Please select the type of request

Bug

Tell us more

Describe the request
Currently the liveness probe uses port 8089 to check the health status of the splunk process. In a busy indexer running smartstore downloads it may fail to respond within 30 seconds, however, this is normal behaviour so I've made the failure thresholds very large.

However, I believe there is a safer way to check the Splunk process is alive and healthy.

As an example this is a failed indexer:
Liveness probe failed: Mgmt. port is not reachable
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8089: Connection refused

(killed by the OOM killer)

This indexer is busy but will recover:
Liveness probe failed: command "/mnt/probes/livenessProbe.sh" timed out

Expected behavior
I'd suggest using a command:
/opt/splunk/bin/splunk status

I checked inside the pod and if successful you get a 0, if down you get a 3:

$ /opt/splunk/bin/splunk status
splunkd is not running.
[splunk@001 /opt/splunk]
$ echo $?
3

A running splunk instance shows a return code of 0 , the K8s docs https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ mention that non-zero is failure.

Reproduction/Testing steps
Stop the splunk instance inside the pod

K8s environment
K8s 1.28

Proposed changes(optional)
As per expected behaviour, switch the liveness probe to:
/opt/splunk/bin/splunk status