@@ -0,0 +1,47 @@
+## [1.2.12](https://github.com/Cloudzero/cloudzero-agent/compare/v1.2.11...v1.2.12) (2026-06-18)
+
+Release 1.2.12 is a maintenance release focused on deployment flexibility and reliability. Highlights include improved monitoring support, Secrets Store CSI Driver support for API key delivery, chart-wide environment variable injection (enabling egress proxy support), aggregator self-healing for persistent error states, and the ability to disable the webhook server when the experimental KubeState plugin is enabled.
+
+### Key Features
+
+- **Selectable Monitoring Discovery (ServiceMonitors or Annotations)**: When monitoring is enabled, the chart now advertises the agent's metrics via _either_ Prometheus Operator `ServiceMonitor` resources _or_ `prometheus.io/*` annotations, rather than emitting both at once. This refines the monitoring integration introduced in validation in 1.2.11. `components.monitoring.discovery.method` (`auto` | `serviceMonitors` | `annotations`, default `auto`) selects the mechanism: `serviceMonitors` renders the `ServiceMonitor` + `PrometheusRule` bundle (requires the Prometheus Operator CRDs), while `annotations` emits `prometheus.io/*` annotations only. Monitoring itself is gated behind `components.monitoring.enabled` (default `false`). See `helm/docs/monitoring-infrastructure.md` for the full reference. **Note:** the default install now emits nothing monitoring-related — see Upgrade Steps below.
+
+- **Secrets Store CSI Driver for API Key**: New `components.apiKey.secretProviderClass` value mounts the CloudZero API key via the Secrets Store CSI Driver instead of a Kubernetes Secret, allowing the key to be sourced directly from an external vault (validated against AWS Secrets Manager and Azure Key Vault). The existing `apiKey` and `existingSecretName` values continue to work and take priority. This replaces the broader `extraVolumes`/`extraVolumeMounts` mechanism from community PR #772 with a focused, single-purpose option.
+
+- **Chart-Wide Environment Injection**: New `defaults.env` value injects a list of environment variables into every container the chart manages. The most common use case is `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY` for clusters behind a corporate egress proxy — every bundled binary honors `http.ProxyFromEnvironment` — but the field covers any chart-wide binary configuration. Chart-emitted entries (e.g. `SERVER_PORT`, `NODE_NAME`) always win on name collision, and component-specific env (e.g. `server.env`) overrides `defaults.env`.
+
+- **Optional Webhook Server**: The admission webhook server can now be disabled via `components.webhookServer.enabled` (`true` | `false` | `"auto"`, default `"auto"`, which maps to enabled today). When disabled, the entire webhook surface (Deployment, Service, ConfigMap, HPA, PDB, `ValidatingWebhookConfiguration`, certificate/issuer/secret, init-cert RBAC, and the backfill job) is removed cleanly with no dangling objects. This is primarily groundwork for future deployment topologies — disabling the webhook server is only appropriate for clusters already relying on the experimental KubeState-based collection path, and is **not recommended for general use**. Default behavior is unchanged. The legacy `insightsController.enabled` value still takes precedence when explicitly set and is now marked deprecated.
+
+### Reliability Improvements
+
+- **Aggregator Self-Healing**: The aggregator now detects and recovers from a pod stuck in a persistent 5xx state. `/healthz` returns 503 when the `/collector` 5xx rate exceeds 20% (with at least 3 failures in the last 60s), so readiness pulls the pod from the Service within ~30s. A new `/livez` endpoint applies a 5-minute sticky latch to trigger a pod restart if the broken state persists. Every 5xx response now sets `Connection: close` so clients reconnect and re-balance immediately. The chart adds `aggregator.readinessProbe`/`aggregator.livenessProbe` defaults with per-container override slots.
+
+- **Backfill Memory Footprint**: The backfill job no longer reuses the webhook server's in-memory SQLite store. It now uses a streaming store that batches records (500 at a time) directly to the collector via remote_write, holding memory flat at ~53 MiB regardless of cluster size (previously ~597 MiB at 1M resources). The unnecessary pusher, housekeeper, and secret monitor no longer run during backfill.
+
+- **GC-Induced OOM Prevention**: All agent binaries now set `GOMEMLIMIT` to 90% of the detected cgroup memory limit via `automemlimit`, so the Go garbage collector reclaims aggressively before RSS reaches the pod limit. This fixes customer-reported OOM kills of the backfill job under allocation churn.
+
+### Security Improvements
+
+- **Webhook Server Credential Surface**: As part of ongoing security hardening, the webhook server no longer mounts the CloudZero API key. It now pushes metrics exclusively to the in-cluster aggregator, so the key mount, the `Authorization` header, and the in-process secret-rotation monitor have been dropped from its code path. This is an incremental reduction of the credential mount surface — the API key secret itself is unchanged, and other components that need it (the aggregator's shipper, the agent init containers, the config-loader job) are unaffected.
+
+### Build and Infrastructure
+
+- Fixed KUTTL end-to-end suites that were silently no-op'ing (command/`spec:` wrapper bugs) and wired the freshly built image into the matrix kind tests so the suites genuinely execute
+
+### Upgrade Steps
+
+To upgrade to version 1.2.12, run the following command:
+
+```sh
+helm upgrade --install <RELEASE_NAME> cloudzero/cloudzero-agent -n <NAMESPACE> --create-namespace -f configuration.example.yaml --version 1.2.12
+```
+
+The default install no longer emits any monitoring resources or `prometheus.io/*` annotations. If you relied on annotation-based scrape discovery in a prior version, restore it explicitly:
+
+```yaml
+components:
+  monitoring:
+    enabled: true
+    discovery:
+      method: annotations
+```