ROX-34637: Hot reload TLS certificates in client connections

[vladbologa]

Description

clientconn.TLSConfig now uses tls.Config.GetClientCertificate to re-read the leaf certificate from disk on each TLS handshake, instead of loading it once into tls.Config.Certificates. This propagates automatically to all code paths that build on it:

• clientconn.AuthenticatedGRPCConnection
• clientconn.AuthenticatedHTTPTransport
• clientconn.GRPCConnection
• clientconn.HTTPTransport

Services that benefit (all non-Sensor Go client connections):

• Central outbound
• Scanner V4 indexer/matcher
• Admission controller
• Compliance

Additionally, the compliance node indexer had its own sync.Once-cached client cert that bypassed clientconn entirely.

User-facing documentation

• CHANGELOG.md is updated OR update is not needed
• documentation PR is created and is linked above OR is not needed

Testing and quality

• the change is production ready: the change is GA, or otherwise the functionality is gated by a feature flag
• CI results are inspected

Automated testing

• added unit tests
• added e2e tests
• added regression tests
• added compatibility tests
• modified existing tests

How I validated my change

Tested with a client admission-control connection to sensor:

• Deployed a fake gRPC sensor that logs the client certificate serial from each TLS handshake
• Scaled down the real sensor, pointed admission-control at the fake sensor via the sensor Service
• Patched tls-cert-admission-control secret with a new cert, then killed the fake sensor to force reconnections
• All 3 admission-control pods reconnected presenting the new client cert serial

[vladbologa]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR moves client certificate loading from initialization time to TLS handshake time. Instead of eagerly loading and assigning certificates to conf.Certificates, a GetClientCertificate callback now loads certificates on-demand. Error handling is adjusted: required certificates (MustUseClientCert) cause handshake failures, while optional certificates log warnings and proceed.

Changes

Client Certificate Lazy Loading

Layer / File(s)	Summary
Lazy certificate loading callback pkg/clientconn/client.go	TLSConfig replaces eager certificate loading with a GetClientCertificate callback that loads from mtls.LeafCertificateFromFile during handshakes. Load failures are handled based on certificate requirement: mandatory certificates fail the handshake, optional certificates log warnings and return empty.

🎯 3 (Moderate) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: implementing hot reload functionality for TLS certificates in client connections, which aligns with the code change from eager to lazy certificate loading.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description is well-structured and comprehensive, covering the technical changes, affected services, testing approach, and validation results.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch vb/hot-reload-client-certs

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🚀 Build Images Ready

Images are ready for commit 5872383. To use with deploy scripts:

export MAIN_IMAGE_TAG=4.11.x-1066-g587238351c