[TRTLLM-14628][perf] Incremental, streamed artifact copy-back into the checkout - #17538
Draft
brnguyen2 wants to merge 6 commits into
Draft
[TRTLLM-14628][perf] Incremental, streamed artifact copy-back into the checkout#17538brnguyen2 wants to merge 6 commits into
brnguyen2 wants to merge 6 commits into
Conversation
Developers increasingly build from checkouts on network filesystems (Lustre, NFS, GPFS), which are slow for metadata-heavy workloads. The build currently writes high-churn state into the checkout: the CMake build dir (default cpp/build*), the build venv (~70k files), the setuptools wheel staging tree and *.egg-info, extension-module object files, and (by default) the ccache directory. Add --build_root DIR (env: TRTLLM_BUILD_ROOT) to build_wheel.py. When set, all of the above default under DIR so it can be pointed at fast node-local storage while the checkout stays on shared storage. Each piece remains individually overridable (--build_dir, CCACHE_DIR, TRTLLM_WHEEL_STAGING_DIR). Only final artifacts (tensorrt_llm/libs, include, bindings, stubs, wheels) are still written into the checkout. setup.py learns TRTLLM_WHEEL_STAGING_DIR, redirecting setuptools build_base and egg_base out of the source tree. Behavior without --build_root is unchanged. Documented in docs/source/installation/build-from-source.md along with CCACHE_DIR / CONAN_HOME / --use-3rdparty-cache guidance for shared-storage workflows. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
…o the checkout Builds on --build_root: with --hermetic, the remaining source-tree writes are redirected so the checkout can be mounted read-only: - Generated FMHA kernel sources move to <build_root>/fmha-gen, consumed via a new TRTLLM_FMHA_GEN_DIR CMake cache variable (default: the historical in-source location). The generator runs from a scratch copy since it writes ./generated, ./temp and ./obj relative to itself, and generated kernels now include fused_multihead_attention_common.h without a "../" prefix so they compile from either location. - The configured executor/version.h can be written to the build tree via TRTLLM_VERSION_H_INCLUDE_DIR instead of cpp/include. - The wheel is assembled from a staging copy of the Python package under <build_root>/package (sources staged, compiled artifacts and stubs installed there), with the wheel landing in <build_root>/dist by default. - Missing submodules become an error instead of an in-place 'git submodule update', and --version-override is rejected (it would edit tensorrt_llm/version.py in the checkout). Editable-install workflows (--skip_building_wheel, --linking_install_binary, --install) are incompatible with --hermetic by design: they import compiled artifacts from the checkout. Default behavior without --hermetic is unchanged. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
fused_multihead_attention_v2.h (#include "cubin/fmha_cubin.h") is included transitively by sources in other targets (e.g. common_src/attentionOp.cpp), so the generated-FMHA include dir must be global when TRTLLM_FMHA_GEN_DIR redirects generation out of the source tree. Found by the read-only-checkout validation build: in-tree builds resolve the header includer-relative and never see the problem. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
…e checkout Every build cleared tensorrt_llm/include (~9k files) and the deep_gemm/ deep_ep/flash_mla Python trees and re-copied them file by file, even when nothing changed. On network filesystems (Lustre/NFS/GPFS) this per-file copy storm dominates incremental rebuild time; it also defeated the FMHA include-tree generation stamps, which were deleted by the clear on every build. Replace the rmtree+copytree pattern with sync_tree: - a missing destination is populated via one streamed tar pipeline (posix format to preserve sub-second mtimes) instead of per-file copies; - an existing destination is mirrored by size/mtime comparison — only changed files are rewritten and entries missing from the source are deleted, so warm rebuilds cause almost no destination I/O; - symlinks are dereferenced as before (copytree(symlinks=False) semantics). tensorrt_llm/include is no longer cleared up front: its subtrees are synced with deletion or guarded by generation stamps (which now actually take effect). deep_ep is removed explicitly when a build does not produce it; the linking-install mode handles a leftover copy-mode directory. The hermetic staging tree reuses sync_tree, making repeated staging incremental as well. Final artifacts that are few and large (libs/*.so, bindings, wheels) remain plain copies. Wheel contents are unchanged. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
Collaborator
Author
|
Warm-rebuild benchmark for this change (same node and checkout per config, identical build flags, single GPU architecture, 64 compile jobs; each variant timed twice, values in seconds of total
The savings are pure copy-back elimination: ~3 minutes per warm rebuild in-tree and ~80 seconds in the |
setup.py's root-level find_packages() ships examples.configs.database; without examples/ in the staging tree the hermetic wheel was missing those four files (found by file-list comparison against a conventional build's wheel). Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
…8-archive-transfer Signed-off-by: Brian Nguyen <brnguyen@nvidia.com> # Conflicts: # scripts/build_wheel.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stacked on #17524 and #17525 — only the top commit is new here; will be rebased as those merge.
Every build cleared
tensorrt_llm/include(~9k files) and thedeep_gemm/deep_ep/flash_mlaPython trees out of the checkout and re-copied them file by file, even when nothing changed. When the checkout lives on a network filesystem (Lustre, NFS, GPFS), this per-file copy storm dominates warm rebuild time; the clear also deleted the FMHA include-tree generation stamps every build, so that skip-if-unchanged mechanism never actually fired.This PR replaces the rmtree+copytree pattern with
sync_tree:copytree(symlinks=False)semantics), and wheel contents are byte-identical in file list.tensorrt_llm/includeis no longer cleared up front (its subtrees are synced-with-deletion or stamp-guarded, and the stamps now take effect). The hermetic staging tree from #17525 reusessync_tree, making repeated staging incremental too. Final artifacts that are few and large (libs/*.so, bindings, wheels) remain plain copies.Test Coverage
sync_treeunit tests: cold populate + symlink dereference, warm no-op (no mtime churn), change/deletion propagation, file↔dir type swaps, same-directory guard, symlink destinations, exclude patterns.PR Checklist