Skip element semantic-equality walk when no element type implements semantic equality (fixes O(n²) on large sets)#1315
Open
troymjones wants to merge 2 commits into
Conversation
…emantic equality ValueSemanticEqualitySetElements compares every proposed element against every prior element (O(n^2)) on each ReadResource, and the Set/List/Map element walks run even when no element type implements semantic equality, where every recursive comparison is a guaranteed no-op. Add a static typeMightHaveSemanticEquals pre-check and return early from the element walks when the element type tree contains no semantic-equals implementer. Behavior-preserving: the collection-level semantic-equality path is untouched, and dynamic types stay conservative (full walk). Benchmarks: a 3,251-element SetNested with no semantic equality drops from ~99s to ~0.5ms per pass. Refs hashicorp#775, hashicorp#1064, hashicorp#1061
|
Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement Learn more about why HashiCorp requires a CLA and what the CLA includes Have you signed the CLA already but the status is still pending? Recheck it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ValueSemanticEqualitySetElements(and the List/Map equivalents) walk collection elements to run recursive semantic-equality reconciliation on everyReadResource. For sets, the walk introduced in #1064 (v1.14.0, fixing order-sensitivity from #1061) compares every proposed element against every remaining prior element: O(n²). It runs even when no element type implements any*ValuableWithSemanticEqualsinterface, in which case every recursive call is a guaranteed no-op and the entire pass is dead work. (List is index-matched and Map is key-matched, so those are O(n), but they still make N no-op calls in the same case.)Real impact: a
cloudflare_listwith ~3,251 items (no field implements semantic equality) takes 10m+ per plan/create with the provider pegged above 100% CPU. This is the underlying cause behind the broad slowdown symptom in #775 (there mis-attributed to allocations/logging).Fix
Add
typeMightHaveSemanticEquals(ctx, attr.Type): it takes a representativet.ValueType(ctx), type-asserts against everybasetypes.*ValuableWithSemanticEqualsinterface, and recurses into element types (list/set/map/tuple) and attribute types (object). It returnsfalseonly when the entire type tree is concrete and provably free of semantic-equals implementers, and is conservative on dynamic types (returnstrue). Each of the Set/List/Map element functions returns early when the element type cannot contain a semantic-equals implementer, leaving the proposed value unchanged.Behavior preservation
SetValuableWithSemanticEqualsetc.) runs before*Elementsand is untouched.updatedElements == falseand returned the proposed value unchanged, byte-identical to the early return.Benchmarks
New
internal/fwschemadata/value_semantic_equality_benchmark_test.go. SetNested, plain string/int64 fields, no semantic equality:Allocations at N=3,251: 990,666,514 → 18 (~46 GB → ~116 KB). A guard benchmark/test with an element type that does implement semantic equality confirms the reconciliation path still runs and still swaps proposed→prior.
Tests
Existing
internal/fwschemadatatests pass.go build ./...,go vet, andgofmt -lare clean. AddedTestBenchmarkSetNestedSemanticEqualsStillApplies(element-level semantic-equals path still applies) andTestBenchmarkSetNestedNoSemanticEqualsUnchanged(no-equality output unchanged).Refs #775, #1064, #1061.
Fixes #1314