Skip to content

Skip element semantic-equality walk when no element type implements semantic equality (fixes O(n²) on large sets)#1315

Open
troymjones wants to merge 2 commits into
hashicorp:mainfrom
troymjones:fix-set-semantic-equality-o-n2
Open

Skip element semantic-equality walk when no element type implements semantic equality (fixes O(n²) on large sets)#1315
troymjones wants to merge 2 commits into
hashicorp:mainfrom
troymjones:fix-set-semantic-equality-o-n2

Conversation

@troymjones

Copy link
Copy Markdown

Problem

ValueSemanticEqualitySetElements (and the List/Map equivalents) walk collection elements to run recursive semantic-equality reconciliation on every ReadResource. For sets, the walk introduced in #1064 (v1.14.0, fixing order-sensitivity from #1061) compares every proposed element against every remaining prior element: O(n²). It runs even when no element type implements any *ValuableWithSemanticEquals interface, in which case every recursive call is a guaranteed no-op and the entire pass is dead work. (List is index-matched and Map is key-matched, so those are O(n), but they still make N no-op calls in the same case.)

Real impact: a cloudflare_list with ~3,251 items (no field implements semantic equality) takes 10m+ per plan/create with the provider pegged above 100% CPU. This is the underlying cause behind the broad slowdown symptom in #775 (there mis-attributed to allocations/logging).

Fix

Add typeMightHaveSemanticEquals(ctx, attr.Type): it takes a representative t.ValueType(ctx), type-asserts against every basetypes.*ValuableWithSemanticEquals interface, and recurses into element types (list/set/map/tuple) and attribute types (object). It returns false only when the entire type tree is concrete and provably free of semantic-equals implementers, and is conservative on dynamic types (returns true). Each of the Set/List/Map element functions returns early when the element type cannot contain a semantic-equals implementer, leaving the proposed value unchanged.

Behavior preservation

  • The collection-level path (a type implementing SetValuableWithSemanticEquals etc.) runs before *Elements and is untouched.
  • When no element implements semantic equality, the old walk always ended with updatedElements == false and returned the proposed value unchanged, byte-identical to the early return.
  • Dynamic types and any custom type implementing a semantic-equals interface still take the full walk.

Benchmarks

New internal/fwschemadata/value_semantic_equality_benchmark_test.go. SetNested, plain string/int64 fields, no semantic equality:

N before after speedup
100 82.6 ms 17.6 µs ~4,700x
1,000 8.96 s 149 µs ~60,000x
3,251 98.9 s 0.53 ms ~185,000x

Allocations at N=3,251: 990,666,514 → 18 (~46 GB → ~116 KB). A guard benchmark/test with an element type that does implement semantic equality confirms the reconciliation path still runs and still swaps proposed→prior.

Tests

Existing internal/fwschemadata tests pass. go build ./..., go vet, and gofmt -l are clean. Added TestBenchmarkSetNestedSemanticEqualsStillApplies (element-level semantic-equals path still applies) and TestBenchmarkSetNestedNoSemanticEqualsUnchanged (no-equality output unchanged).

Refs #775, #1064, #1061.

Fixes #1314

…emantic equality

ValueSemanticEqualitySetElements compares every proposed element against every
prior element (O(n^2)) on each ReadResource, and the Set/List/Map element walks
run even when no element type implements semantic equality, where every
recursive comparison is a guaranteed no-op. Add a static
typeMightHaveSemanticEquals pre-check and return early from the element walks
when the element type tree contains no semantic-equals implementer.

Behavior-preserving: the collection-level semantic-equality path is untouched,
and dynamic types stay conservative (full walk). Benchmarks: a 3,251-element
SetNested with no semantic equality drops from ~99s to ~0.5ms per pass.

Refs hashicorp#775, hashicorp#1064, hashicorp#1061
@troymjones
troymjones requested a review from a team as a code owner July 21, 2026 19:14
@hashicorp-cla-app

hashicorp-cla-app Bot commented Jul 21, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@hashicorp-cla-app

Copy link
Copy Markdown

CLA assistant check

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

Have you signed the CLA already but the status is still pending? Recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ValueSemanticEqualitySetElements walks all N² element pairs even when no element type implements semantic equality

1 participant