Reduce transformer peak memory for staged site imports - #816
Merged
Conversation
Preserve canonical source identities while hashing normalized files incrementally instead of concatenating every text and base64 payload into a second bundle-sized string. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced a live 256 MiB compiler exhaustion, implemented the streaming hash contract, and verified legacy hash compatibility and normalizer idempotence. Chris Huber reviewed and remains responsible for the change.
Preserve RuntimeDeclarations canonical SHA-256 digests while streaming arrays and escaped strings into the hash context, avoiding payload-sized canonical JSON allocations during staged artifact preparation. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced a live staged-plan memory exhaustion, implemented the canonical streaming encoder, reviewed edge-case compatibility, and added bounded 64 MiB coverage. Chris Huber reviewed and remains responsible for the change.
Build the WordPress compatibility stylesheet before reference reports and asset manifests increase the compiler heap. The generated asset and final ordering remain unchanged. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced a live 256 MiB compiler exhaustion, reordered the independent compilation phase, and ran the WordPress site plan contract. Chris Huber reviewed and remains responsible for the change.
Build the compatibility-inclusive stylesheet from the cached authored CSS instead of reconstructing the same long-string deduplication map. Preserve stylesheet order and compatibility output while reducing compiler peak memory. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the next live 256 MiB compiler exhaustion, implemented cache reuse, and ran site-plan and stylesheet contracts. Chris Huber reviewed and remains responsible for the change.
Guard escaped JSON, inline style, and block-comment canonicalization passes by their literal transport markers. Preserve rewrite behavior while avoiding repeated full-string allocations for large markup without those transports. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced a live 256 MiB asset canonicalization exhaustion, implemented marker guards, and ran site-plan and stylesheet contracts. Chris Huber reviewed and remains responsible for the change.
Avoid a second full-markup regex allocation when serialized url and action keys are absent. Preserve route rewriting for HTML attributes and JSON route transports. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the live 2.1 MiB no-match allocation, implemented the transport guard, and ran site-plan and shared-shell contracts. Chris Huber reviewed and remains responsible for the change.
Use a whitespace span check before font source analysis instead of trimming the entire stylesheet. Preserve accepted CSS and source hashes while avoiding a full-size temporary string. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the live 1.2 MiB stylesheet allocation, implemented the non-copying check, and ran site-plan and stylesheet tests. Chris Huber reviewed and remains responsible for the change.
Scan serialized url and action references for an actually routable local target before running the full-string replacement. This preserves route canonicalization while avoiding a multi-megabyte copy for markup containing only external or otherwise inert JSON URLs. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the canonical 256 MiB failure, implemented the bounded preflight scan, and ran WordPress site-plan and shared-shell contracts. Chris Huber reviewed and remains responsible for the change.
Preflight full-content asset canonicalization callbacks and preserve the original string when no declared reference changes. This bounds peak memory for large escaped RichText payloads while retaining real HTML, srcset, style, and block JSON substitutions. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the 256 MiB canonicalizer failure, implemented change-aware replacement, added large escaped-markup coverage, and ran site-plan and shared-shell contracts. Chris Huber reviewed and remains responsible for the change.
After document-relative resolution misses, resolve link and script metadata against the artifact entry root. This lets packaged external documents bind shared collected assets without weakening fail-closed validation for unresolved declarations. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the redirected external-page metadata mismatch, implemented entry-root fallback resolution, and added contract coverage. Chris Huber reviewed and remains responsible for the change.
Keep change-aware scanning only for the unicode-escaped attribute passes that exhausted memory. Restore direct replacements for ordinary tags, escaped quotes, styles, and block comments to avoid multiplying canonicalization work on large pages. AI assistance: OpenAI gpt-5.6-sol via OpenCode analyzed the 60-minute Flaming Pear checkpoint, isolated the broad preflight performance regression, narrowed the optimization, and reran site-plan and shared-shell contracts. Chris Huber reviewed and remains responsible for the change.
Scan CSS rules one at a time and stop once heading and body font roles are known. This avoids copying the stylesheet with trim and materializing every rule during font planning. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the 256 MiB font-planning failure, implemented bounded rule scanning, added a large-stylesheet regression, and ran transformer and site-plan contracts. Chris Huber reviewed and remains responsible for the change.
Preserve staged asset ownership through compiled site plans and enqueue page-owned CSS only on its canonical WordPress route. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the ownership loss, implemented the scoped enqueue contract, and added deterministic coverage; Chris Huber reviewed and remains responsible for the change.
Retain runtime ownership only for CSS assets, avoiding per-asset nested scope metadata for images and fonts that are never enqueued. AI assistance: OpenAI gpt-5.6-sol via OpenCode analyzed the 256 MiB canonical failure, isolated retained non-CSS scope metadata as the headroom regression, and verified the canonical suite; Chris Huber reviewed and remains responsible for the change.
Project inferred inline CSS ownership and derive front-page loading from the canonical root route instead of transient batch entrypoints. AI assistance: OpenAI gpt-5.6-sol via OpenCode traced the canonical network inventory, identified leaked support-page styles, implemented route-owned loading, and verified the full canonical suite; Chris Huber reviewed and remains responsible for the change.
Preflight WordPress block-comment canonicalization so unchanged large documents retain their existing string instead of allocating a full replacement buffer. AI assistance: OpenAI gpt-5.6-sol via OpenCode and parallel research agents isolated the no-op PCRE allocation, implemented the bounded path, and added constrained-memory coverage; Chris Huber reviewed and remains responsible for the change.
Keep generated non-entry CSS scoped to its source route and derive streamed JSON array shape after canonical key ordering. AI assistance: OpenAI gpt-5.6-sol via OpenCode and a parallel review agent identified the uncovered ownership and digest cases, implemented the fixes, and ran canonical, parity, and packaging suites; Chris Huber reviewed and remains responsible for the change.
Exercise sparse integer-key runtime declarations explicitly so streamed hashing remains byte-compatible with canonical JSON object encoding. AI assistance: OpenAI gpt-5.6-sol via OpenCode and a final review agent identified the exact sparse-key coverage gap and verified the focused test; Chris Huber reviewed and remains responsible for the change.
AI assistance: OpenAI GPT-5.6-sol with OpenCode was used to diagnose the Composer provisioning failure, update the advisory policy, and verify the exact CI Composer version. Chris Huber reviewed and remains responsible for every line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
A representative multi-page Weebly import intermittently exhausted a 256 MiB PHP limit during staged compilation and asset canonicalization. The failures moved through several allocation sites as earlier peaks were removed. This change fixes the owning transformer paths rather than raising memory or exposing tuning controls.
Verification
composer test:canonicalcomposer test:parity(270 fixtures)composer test:packagingThe dependent Static Site Importer PR contains the resumable batch lifecycle and public ability integration used for the full runs.
Compatibility
Canonical hash tests preserve existing digests, including reordered and sparse integer keys. Parity fixtures and package installation remain unchanged. Generated CSS now retains its source route instead of becoming global.
AI assistance
OpenAI gpt-5.6-sol via OpenCode was used to trace allocation failures, implement bounded transformations, run integration evidence, and perform parallel pre-PR reviews. Chris Huber reviewed the changes and remains responsible for every line.