Fix Windows reserve-only page allocations

[Weiteng Chen (CvvT)]

Some applications may reserve large chunk of memory (e.g., 4GB) upfront but only uses a small portion of it. When allocating memory with no permission, we should only reserve memory without committing any pages on Windows platform. We then commit reserved pages when accessible permissions are later requested.

[Weidong Cui (wdcui)]

gpt/opus found the following two issues:

1.  MAP_POPULATE | PROT_NONE  can still call  do_prefetch_on_range  after  reserve_and_maybe_commit  leaves the range uncommitted. This occurs in the  MEM_FREE  path around lines 1772–1787 and the anonymous fallback around lines 1797–1808.  PrefetchVirtualMemory  requires a range accessible to the target process, and failure currently triggers an assertion. Please skip prefetching when  initial_permissions.is_empty() .

2. The collision check around lines 1710–1726 only treats  MEM_COMMIT  regions as occupied. Now that a live  PROT_NONE  mapping can remain  MEM_RESERVE , a subsequent allocation may treat its address as available.  Vmem::insert_mapping  protects the normal  NoReplace  path through its VMA map, but  Hint  and direct provider behavior can still incorrectly reuse the range. The provider needs a way to distinguish live reserve-only mappings from reusable reserved padding/decommitted regions and include live mappings in collision handling.

[Weidong Cui (wdcui)]

Weiteng Chen (@CvvT), do you plan to address the comment above?

[Weiteng Chen (CvvT)]

Weiteng Chen (Weiteng Chen (@CvvT)), do you plan to address the comment above?

Yes, I am still working on it.

[Weiteng Chen (CvvT)]

• MAP_POPULATE | PROT_NONE  can still call  do_prefetch_on_range  after  reserve_and_maybe_commit  leaves the range uncommitted. This occurs in the  MEM_FREE  path around lines 1772–1787 and the anonymous fallback around lines 1797–1808.  PrefetchVirtualMemory  requires a range accessible to the target process, and failure currently triggers an assertion. Please skip prefetching when  initial_permissions.is_empty() .

Now we skip do_prefetch_on_range if PROT_NONE is requested.

[Weiteng Chen (CvvT)]

2. The collision check around lines 1710–1726 only treats  MEM_COMMIT  regions as occupied. Now that a live  PROT_NONE  mapping can remain  MEM_RESERVE , a subsequent allocation may treat its address as available.  Vmem::insert_mapping  protects the normal  NoReplace  path through its VMA map, but  Hint  and direct provider behavior can still incorrectly reuse the range. The provider needs a way to distinguish live reserve-only mappings from reusable reserved padding/decommitted regions and include live mappings in collision handling.

Add tracking for live reserve-only mappings for collision detection and update tracking after permission changes and deallocation.

[github-actions]

🤖 SemverChecks 🤖 No breaking API changes detected

Note: this does not mean API is unchanged, or even that there are no breaking changes; simply, none of the detections triggered.