[WIP] Add declared-io task cache backed by buildxl

Directory.Build.props

@@ -2,6 +2,10 @@
 <Project>
   <Import Project="Sdk.props" Sdk="Microsoft.DotNet.Arcade.Sdk" />
 
+  <PropertyGroup Condition="'$(DotNetBuildSourceOnly)' != 'true'">
+    <DefineConstants>$(DefineConstants);FEATURE_BUILDXL_TASK_CACHE</DefineConstants>
+  </PropertyGroup>
+
   <PropertyGroup Condition="'$(CopyrightNetFoundation)' != ''">
     <Copyright>$(CopyrightNetFoundation)</Copyright>
     <PackageLicenseExpression>MIT</PackageLicenseExpression>

MSBuild.slnx

@@ -57,6 +57,9 @@
   <Project Path="src/Build/Microsoft.Build.csproj">
     <Platform Solution="*|x64" Project="x64" />
   </Project>
+  <Project Path="src/Build.TaskCache/Microsoft.Build.TaskCache.csproj">
+    <Platform Solution="*|x64" Project="x64" />
+  </Project>
   <Project Path="src/BuildCheck.UnitTests/Microsoft.Build.BuildCheck.UnitTests.csproj">
     <Platform Solution="*|ARM64" Project="arm64" />
     <Platform Solution="*|x64" Project="x64" />

NuGet.config

@@ -19,12 +19,23 @@
     <add key="dotnet11" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11/nuget/v3/index.json" />
     <add key="dotnet11-transport" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet11-transport/nuget/v3/index.json" />
     <add key="vs-impl" value="https://pkgs.dev.azure.com/azure-public/vside/_packaging/vs-impl/nuget/v3/index.json" />
+    <!-- Public upstream used by Microsoft.MSBuildCache; restricted to its cache dependencies. -->
+    <add key="msbuildcache" value="https://pkgs.dev.azure.com/msbuildcache/public/_packaging/msbuildcache/nuget/v3/index.json" />
   </packageSources>
   <packageSourceMapping>
+    <packageSource key="msbuildcache">
+      <package pattern="Microsoft.BuildXL.*" />
+      <package pattern="RocksDb*" />
+      <package pattern="RuntimeContracts" />
+      <package pattern="protobuf-net.Grpc*" />
+      <package pattern="CopyOnWrite" />
+    </packageSource>
     <packageSource key="arcade">
       <package pattern="*" />
     </packageSource>
     <packageSource key="dotnet-public">
+      <package pattern="Microsoft.BuildXL.Processes" />
+      <package pattern="Microsoft.BuildXL.*" />
       <package pattern="*" />
     </packageSource>
     <packageSource key="dotnet-tools">

documentation/specs/task-cache-buildxl.md

@@ -0,0 +1,67 @@
+# BuildXL cache storage
+
+Use BuildXL LocalCache for CAS and memoization while MSBuild owns execution,
+fingerprints, and result replay. There is no alternative backend or cache daemon.
+
+## Dependencies
+
+Package-based API hosts reference `Microsoft.Build.TaskCache` alongside
+`Microsoft.Build`. It carries the optional implementation and managed/native
+runtime closure; ordinary engine consumers do not need cache-specific feeds.
+Transitive package targets copy native files for both build and publish.
+
+`Microsoft.BuildXL.Cache.MemoizationStore.Library` supplies LocalCache and
+RocksDB memoization. `RocksDbNative` is referenced directly because its native
+copy targets are not transitive. Dependency versions and feed mappings are
+maintained in the repository's package configuration.
+
+The packaged native assets cover Windows/Linux/macOS x64. Linux requires glibc
+and libstdc++; runtime validation is currently Linux-only. Source-build excludes
+the dependencies and consuming implementation through `FEATURE_BUILDXL_TASK_CACHE`.
+Ordinary source-built MSBuild works; opting into task caching reports MSB1077.
+
+## Layout and API boundary
+
+Under the configured root:
+
+```text
+buildxl/
+  owner.lock
+  cache/
+    memoization/
+    ... BuildXL-managed content and metadata ...
+```
+
+The invocation key becomes a `StrongFingerprint` with a fixed empty-content
+selector. `GetContentHashListAsync` looks up results;
+`AddOrGetContentHashListAsync` publishes the manifest hash and artifact hashes.
+`PutFileAsync` hashes/copies artifacts; `PutStreamAsync` stores manifests.
+Each publication has its own artifact list. Copy insertion and private output
+staging prevent writable inode sharing with CAS content.
+
+One coordinator-owned cache/session holds exclusive directory ownership for the
+top-level build. Acquisition is fail-fast. Workers use the existing coordinator
+connection rather than opening the database themselves. Concurrent operations
+do not serialize task execution. Shutdown drains calls, closes the session/cache,
+then releases ownership. A leftover lock file is not a held OS lock.
+
+## Retention and maintenance
+
+The build-long `ImplicitPin.PutAndGet` session protects content until shutdown.
+Lookup explicitly pins all referenced hashes; already-evicted content is a miss.
+BuildXL enforces a 10 GiB content quota. Pinned content can exhaust that quota;
+insertion failures are build errors, not silent cache bypass.
+
+At cache startup, run metadata GC if the last successful collection is at least
+an hour old or its recorded time is in the future. Keep the timestamp in
+`owner.lock`; do not run a periodic GC timer during the build.
+The 64 MB metadata target is size-driven with older last-access records as
+eviction candidates, not a TTL. Metadata removal and CAS eviction are separate.
+Database bookkeeping means these limits do not cap total directory size.
+
+## Bootstrap
+
+`AddBootstrapTaskCacheDependencies` merges the cache dependency closure into
+the bootstrap SDK's runtime manifests and copies required assets, preserving
+existing SDK package versions. Baseline manifests allow replacement of cache
+additions on subsequent builds. This integration is excluded from source-build.

documentation/specs/task-cache-decisions.md

@@ -0,0 +1,89 @@
+# Task-cache design decisions
+
+MSBuild owns task execution and result semantics. BuildXL supplies content
+storage and memoization, not execution or dependency discovery.
+
+## Scope and configuration
+
+- Cache individual task invocations. Preserve target timestamp skipping and
+  output inference; skipped targets do not consult the task cache.
+- Require `-taskCache` or `BuildParameters.TaskCache`. Configuring a directory
+  alone enables nothing.
+- Configure the shared storage root through `-buildCacheDirectory:<path>` or
+  `BuildParameters.BuildCacheDirectory`. Resolve it once for the top-level build.
+  Project properties and environment variables cannot override it.
+- Use platform-default user cache directories when configuration is unspecified.
+  Do not expose the directory as a project property.
+- Use only BuildXL storage. Do not add a runner, persistent service, sandbox,
+  access monitoring, learned dependencies, or a separate timing facility.
+
+## Task contract
+
+- Describe file I/O with unconditional class-level annotations naming task
+  parameters. Inputs and outputs must be disjoint within an invocation.
+- Trust the author to provide complete conservative dependencies and deterministic
+  behavior. Structural validation cannot prove the absence of hidden effects.
+- Resolve the requested task normally and inspect its actual type. Never
+  substitute tasks to make an invocation cacheable.
+- Use a separate restricted task when an existing task's parameter surface
+  cannot meet the contract. Keep existing invocations unchanged.
+- Allow conservative output-path supersets. Record whether each candidate exists
+  after execution, including outputs removed or not produced.
+
+## Keys and results
+
+- Hash task implementation, bound inputs, execution context/environment, declared
+  input contents and absence, output paths, and referenced output-parameter names.
+  Workers compute input hashes; revalidate inputs before publication.
+- Include engine `ModuleVersionId` in every key. This isolates serialization
+  changes between engine binaries, so the cache format has no separate version
+  fields or migration promise.
+- Reuse `TaskParameter` and `BinaryTranslator` for typed values and item metadata.
+  Keep one manifest per invocation, containing file records, output values,
+  and supported warnings. See the [format reference](task-cache.md).
+- Capture every referenced output getter after successful execution, regardless
+  of binding conditions. Getters must be safe to read then. Read each once before
+  cleanup; publish afterward so cleanup failures cannot produce a reusable result.
+- Keep binding conditions and destination property/item names outside the cached
+  result. Apply current bindings to independent restored values.
+
+## Ownership and publication
+
+- One coordinator owns LocalCache and its session for the top-level build.
+  Acquire directory ownership without waiting; competing builds fail.
+  Ownership remains held while tools execute, without serializing task execution.
+- Workers access the owner through existing node communication. Exchange paths,
+  hashes, and result metadata, not artifact contents through in-memory transport.
+- Keep session pins for the build. Await each publication in the task path;
+  do not add a background publication queue.
+- Let CAS hash output files during insertion. Use the returned hashes rather
+  than prehashing and rereading the same outputs in MSBuild.
+- At shutdown, stop new operations, cancel active calls, and await completion
+  before closing storage and releasing ownership. Do not abandon active calls
+  through a separate shutdown timeout.
+
+## Diagnostics and failure recovery
+
+- Cache standard warnings from execution and output getters before warning-policy
+  conversion. Replay them in the current task context under current warning policy.
+- Setup/cleanup diagnostics and unsupported warning subclasses prevent caching
+  rather than being replayed incompletely or twice. Raw errors, failed tasks,
+  exceptions, and cancellation prevent publication. Do not replay ordinary messages.
+- Missing entries or evicted content are misses. Corruption and operational
+  failures fail the build, without task-execution fallback.
+- Validate and stage all artifacts before replacing destinations. Honor
+  cancellation before replacement, then finish replacement without cancellation
+  interruption. Filesystem errors remain fatal.
+- Do not implement multi-file rollback. Preserve the primary error during cleanup
+  and require clean/rebuild recovery after partial restoration; timestamp skipping
+  prevents a guarantee of automatic incremental recovery.
+
+## Storage policy
+
+Use a 10 GiB content quota and build-long pins. Check metadata collection only
+when opening the cache, with a one-hour minimum interval and a 64 MB size target.
+Metadata eviction is based on size and last access, not a fixed expiration age.
+
+Exclude BuildXL dependencies and implementation from source-build. Permit the
+packaged Windows/Linux/macOS x64 assets; runtime validation is currently Linux-only.
+See the [storage reference](task-cache-buildxl.md).

documentation/specs/task-cache.md

@@ -0,0 +1,117 @@
+# Task-cache implementation and format
+
+The [decision record](task-cache-decisions.md) explains the design.
+The [user guide](../wiki/Task-Cache.md) describes usage and task-author requirements.
+
+## Execution and ownership
+
+After conditions, batching, and parameter binding, `TaskInvocationCache` keys
+the invocation and queries storage. A hit restores files and typed outputs;
+normal MSBuild binding applies the current `<Output>` elements.
+Target timestamp skipping remains unchanged.
+
+`TaskCacheOwner` holds one BuildXL cache/session from `BeginBuild` to `EndBuild`.
+`TaskCacheClient` routes worker requests through existing node communication.
+Workers hash inputs and restore files locally; the owner stores files by path.
+No artifact bytes are transported through node messages.
+Unix output staging files are created owner-only before writing content;
+recorded final permissions are applied after validation.
+
+Requests carry an ID, operation, and key. Responses complete waiters directly
+on receipt, independently of scheduler work that might be waiting for a task.
+Cancellation waits for operation completion; connection loss releases waiters.
+Shutdown drains active calls before closing storage. Worker protocol compatibility
+is checked by the handshake, independently of the on-disk format.
+
+`BuildManager` resets configuration/results between cache-enabled build sessions
+so an old in-memory build result cannot bypass invocation processing.
+
+## Invocation key
+
+The key is SHA-256 over an encoded invocation descriptor and declared inputs:
+
+- Engine/task module identities and task type.
+- Project/toolset, runtime/platform, culture, and task execution directory.
+- Bound parameter values and effective declared file parameters.
+- Referenced output-parameter names, effective environment, and output paths.
+- Each declared input's absolute path, existence flag, and content hash.
+
+Names are sorted where ordering is not meaningful; item metadata is canonicalized.
+Item inputs retain both the transport representation and task-visible custom
+metadata values, so inherited expressions and literal metadata remain distinct.
+File bytes are hashed directly, not embedded in the descriptor. Engine module
+identity isolates incompatible encodings; there are no explicit format-version
+fields or cross-engine migration guarantees.
+
+## Result manifest
+
+`TaskCacheStore` writes one manifest per invocation. BuildXL memoization maps
+the invocation key to a content-hash list: manifest first, then present artifacts.
+The manifest itself is an opaque CAS object.
+
+Integers use `BinaryWriter`'s little-endian representation. Booleans occupy one
+byte. A *manifest string* is an Int32 UTF-8 byte length followed by its bytes.
+
+| Field, in serialization order | Encoding |
+| --- | --- |
+| Output-path count | Int32 |
+| For each output: absolute path | Manifest string |
+| Output exists | Boolean |
+| Digest, only if present | Manifest string: uppercase hexadecimal SHA-256 |
+| Original last-write time | Int64 UTC ticks; zero if absent |
+| Unix permissions | Int32, masked to `0x1ff`; zero if absent/not captured |
+| Task-state byte count | Int32 |
+| Task state | Bytes, described below |
+| Checksum | SHA-256 of all preceding manifest bytes |
+
+Output records follow sorted, deduplicated paths and must match the current
+contract exactly. An absent record deletes that destination on restore.
+Stored timestamps are validated, but restored files receive the current time.
+Limits: 64 MiB per manifest and 1 MiB per manifest string.
+
+### Typed outputs
+
+Task state begins with an Int32 output count. Entries are ordinally sorted by
+task parameter name. Each contains:
+
+1. Name via `BinaryWriter.Write(string)` (7-bit UTF-8 byte length).
+2. Int32 value byte count.
+3. Value bytes from `TaskParameter.Translate` through `BinaryTranslator`.
+
+Supported values are nulls, the primitive/array subset accepted by
+`TaskInvocationCache.IsSupportedParameterType`, and task items with metadata.
+Type tags and restored values are validated; arbitrary task-defined objects
+are not deserialized. Each binding receives a fresh value.
+
+### Warnings
+
+Typed outputs are followed by an Int32 warning count and these records:
+
+1. Subcategory, Code, File, Message, HelpKeyword, SenderName, HelpLink:
+   Int32 UTF-8 byte length and bytes, with -1 meaning null.
+2. LineNumber, ColumnNumber, EndLineNumber, EndColumnNumber: Int32 each.
+
+Null/empty strings remain distinct. UTF-8 is strict. Limits are 1,024 warnings,
+1 MiB per string, and 16 MiB for the warning section. Invalid bounds, negative
+positions, malformed encoding, truncation, and trailing state are rejected
+before restoration.
+
+Only exact `BuildWarningEventArgs` instances are supported. Snapshot formatted
+fields before asynchronous logging or policy conversion, not raw argument
+objects. Replay regenerates task context, project association, timestamp,
+and thread identity. Live replay preserves HelpLink; the existing binlog
+encoding's omission of that field is unchanged.
+
+## Capture and publication
+
+The diagnostic watch spans initialization through cleanup. Replayable warning
+capture covers only execution and referenced output getters on a cache miss.
+Setup, binding, input-getter, cleanup, and unsupported diagnostics block caching.
+Raw errors are observed before `ContinueOnError` conversion.
+
+Capture each referenced getter once before cleanup, including getters whose
+binding conditions are false. Publish after cleanup and input revalidation.
+On a hit, validate state and restore files, then replay warnings before current
+output binding. Current warning policy applies without entering the key.
+
+See [BuildXL storage](task-cache-buildxl.md) for lifecycle and maintenance.

documentation/wiki/Contributing-Tasks.md

@@ -17,6 +17,10 @@ Review the existing documentation on [Task Writing](https://learn.microsoft.com/
 
 Tasks are generally simple and should not require much effort to develop.  If you find a task becoming very complicated, consider breaking it up into smaller tasks which can be run together in a target.
 
+For file I/O annotations and task-cache author requirements, see
+[Task invocation caching](Task-Cache.md). That documentation applies to task
+authors generally, not just contributors to this repository.
+
 ## Developing unit tests
 Contributed tasks must have unit tests in place to prove they work and to prevent regressions caused by other code changes.  There are a lot of examples in the [Microsoft.Build.Tasks.UnitTests](https://github.com/dotnet/msbuild/tree/main/src/Tasks.UnitTests) project.  Please provide a reasonable amount of test coverage so ensure the quality of the product.
 
@@ -25,5 +29,3 @@ You can document the new task in the [visualstudio-docs](https://github.com/Micr
 
 ## Ship schedule
 MSBuild ships regularly with Visual Studio.  It also is updated in Preview releases.  Once your contribution is merged, expect it to be available in the next release.
-
-

documentation/wiki/Results-Cache.md

@@ -4,6 +4,10 @@ MSBuild uses caching to speed up builds. It does this by remembering the outcome
 
 ![MSBuild Cache Flow](CacheFlow.png)
 
+The experimental [task invocation cache](Task-Cache.md) is a separate persistent
+cache. Unlike the in-memory structures described below, its entries survive
+build completion and process exit.
+
 ## `ResultsCache` (The Core Cache Component)
 
 `ResultsCache` is the primary storage mechanism where MSBuild keeps the outcomes of its build targets.