Splunk Operator: SHC bundle push false failure in FIPS mode due to strict status-string matching #1878

Please select the type of request

Bug

Tell us more

Describe the request

• In a FIPS-enabled Splunk deployer, SHC bundle push status parsing fails because the status file contains a FIPS informational line (FIPS provider enabled...) instead of (or before) the exact success string.
• The operator treats this as a bundle push error and retries, even when the output may be informational/non-fatal.

Expected behavior

• SHC bundle push should not be marked failed when output contains informational FIPS lines.
• Operator should reliably detect success/failure (and in-progress) without requiring an exact single-line stdout match.

Splunk setup on K8S

• Splunk Operator managing a SearchHeadCluster (sh) in namespace splunk.
• App Framework cluster-scope playbook in use (SHC bundle push path).
• Deployer pod is FIPS-enabled and reports:
  • FIPS provider enabled. name: OpenSSL FIPS Provider, version: 3.1.2, buildinfo: 3.1.2, status: Success
• Status file checked by operator:
  • /operator-staging/appframework/.shcluster_bundle_status.txt

Reproduction/Testing steps

1. Deploy Splunk Operator with SHC + App Framework enabled.
2. Use a FIPS-enabled Splunk deployer image/config.
3. Trigger cluster-scope app install / SHC bundle push.
4. Observe operator logs:
   • isBundlePushComplete ... SHC Bundle push status file reported an error while applying bundle
   • wrapped error: there was an error in applying SHC Bundle, err="FIPS provider enabled..."
5. Confirm current code path:
   • pkg/splunk/enterprise/afwscheduler.go (isBundlePushComplete)
   • It only treats output as success if it contains:
     • Bundle has been pushed successfully to all the cluster members.

K8s environment

• Cluster type: EKS
• Kubernetes version: v1.34.9-eks-8f14419
• Splunk Operator version/tag: 3.1.0
• Splunk Enterprise image/version: 10.4.1

Proposed changes(optional)

• Make SHC bundle status parsing resilient to informational preamble lines (including FIPS).
• Suggested options:

1. Parse for explicit success/failure markers, ignoring known informational lines.
2. Prefer command exit/status signaling over exact stdout equality.
3. If non-empty output lacks known failure markers, keep state as in-progress and re-check before hard-failing.

• Add regression tests for FIPS banner output in SHC bundle status parsing.

K8s collector data(optional)

• Please provide data collected from the K8s collectors, if any.

Additional context(optional)

• Related log observed previously in same flow:
  • ConfDeploymentException: Can't start deployment job as one is already running!
• Current behavior may produce false negatives and unnecessary retries/noise in FIPS-enabled environments.