Skip to content

fix(proxy): build trace-collector identity from trust domain, not cluster domain#15404

Merged
adleong merged 1 commit into
linkerd:mainfrom
joedrf:fix/trace-collector-trust-domain
Jun 24, 2026
Merged

fix(proxy): build trace-collector identity from trust domain, not cluster domain#15404
adleong merged 1 commit into
linkerd:mainfrom
joedrf:fix/trace-collector-trust-domain

Conversation

@joedrf

@joedrf joedrf commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

What / why

When proxy-native tracing is enabled, the proxy exports spans to the trace collector over mesh mTLS and pins the identity it expects the collector to present, via LINKERD2_PROXY_TRACE_COLLECTOR_SVC_NAME. That value is built from .Values.clusterDomain, but a meshed collector - like every meshed workload - presents an identity suffixed with the trust domain.

On a default install these are the same string (cluster.local), so the bug is invisible. On any install where identityTrustDomain != clusterDomain, the proxy demands a SAN the collector never presents, the trace-export handshake fails (received corrupt message of type InvalidContentType on the sender; unspecific protocol error on the collector's inbound proxy), and no proxy-native spans are delivered. App-level SDK tracing to the collector is unaffected (it uses normal identity), which masks the cause.

Root cause

charts/partials/templates/_proxy.tpl defines the helper used by every identity:

{{- $trustDomain := (.Values.identityTrustDomain | default .Values.clusterDomain) -}}

The component identities (IDENTITY/DESTINATION/POLICY) all use $trustDomain, but the trace-collector identity is the lone exception - it uses .Values.clusterDomain. This is the identity the connecting proxy pins, so it must use the trust domain like the others. (The separate ..._SVC_ADDR line is the DNS address and correctly keeps using clusterDomain.)

The fix

One token: the trace-collector identity now uses $trustDomain (already in scope) instead of .Values.clusterDomain.

Scope / safety

  • No change on default installs. When identityTrustDomain is unset, $trustDomain falls back to .Values.clusterDomain, so the render is byte-identical to today. Only installs with a custom, different trust domain change - for them, from broken to working.
  • Identity only, not addressing. Verified by helm template that the collector's network address (...svc.<clusterDomain>) is untouched; only the pinned identity moves to the trust domain.

Test plan

Render (helm template, clusterDomain=mesh-dns, identityTrustDomain=my.trust):

trace-collector identity collector DNS address
before (broken) ...serviceaccount.identity.linkerd.mesh-dns ...svc.mesh-dns:4317
after (fixed) ...serviceaccount.identity.linkerd.my.trust ...svc.mesh-dns:4317 (unchanged)

Live mTLS repro (k3d, clusterDomain != identityTrustDomain, meshed otel-collector):

  • Before: sender proxy logs InvalidContentType; collector receives 0 linkerd-proxy spans while an app-SDK OTLP control keeps delivering spans.
  • After: InvalidContentType stops; linkerd-proxy spans arrive. Toggled off and on (and back) plus negative/positive controls isolate the identity name as the sole cause.

Note for maintainers - data plane needs a rebuilt image

This partial renders both control-plane component proxies (fixed at helm upgrade) and data-plane sidecars - but the latter are stamped from the template baked into the proxy-injector image. So the chart change fixes control-plane proxies; application sidecars pick up the fix only once a proxy-injector image is built from this commit and rolled out. Worth a release-note callout. Until then, operators can work around per-workload via proxy.additionalEnv overriding LINKERD2_PROXY_TRACE_COLLECTOR_SVC_NAME (duplicate env, last-wins).

…ster domain

LINKERD2_PROXY_TRACE_COLLECTOR_SVC_NAME was built from .Values.clusterDomain,
while every other proxy identity (identity/destination/policy) uses the trust
domain helper ($trustDomain = identityTrustDomain | default clusterDomain).

On installs where identityTrustDomain differs from clusterDomain, the proxy
pins a collector identity the meshed collector never presents, so the trace
export mTLS handshake fails (received corrupt message of type InvalidContentType)
and no proxy-native spans are delivered. App-SDK OTLP is unaffected, which masks
the cause. Use $trustDomain to match the sibling identities.

No behavior change when identityTrustDomain is unset (falls back to
clusterDomain, byte-identical render). Only the pinned identity changes; the
collector's DNS address still uses clusterDomain.

Signed-off-by: Joe <joe@buoyant.io>
@joedrf joedrf marked this pull request as ready for review June 22, 2026 16:16
@joedrf joedrf requested a review from a team as a code owner June 22, 2026 16:16

@adleong adleong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch! thanks for the fix

@adleong adleong merged commit 8f6a4ec into linkerd:main Jun 24, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants