From cba076baf8906d0296610f7b30390c899a2882ef Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 5 Nov 2026 13:40:26 +0900 Subject: [PATCH 01/40] ci: preserve the macOS bootstrap compiler during a clean rebuild --- ...heterogeneous-build-implementation-plan.md | 128 ++++++++++++++++++ .github/workflows/release.yml | 6 + 2 files changed, 134 insertions(+) diff --git a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md index 5373a95e7..ce960f4be 100644 --- a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md +++ b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md @@ -310,3 +310,131 @@ platform 0.23.7 的一致版本组已在本地准备，待两个前置公开后 `/home/speak/workspace/mcpp-885-realpath-and-final-core-diagnostic-evidence.tar.gz`， 152 个文件，SHA256 为 `4d7b783c27bb25dc75c354af72e98dc4b47037ab75043f792a28dceeeb10083a`。 + +## 13. 核心最终审查与发布启动（2026-11-05） + +核心 PR #888 的最终提交 `850bbf53b5cf8a98dc8b6e3da894da77059cfbc4` 已通过完整 +CI `41947858860` 第 4 次尝试：74 项成功，1 项不适用的 iOS 清单跳过。Windows +与 macOS 的 E2E 1015 均实际通过 Unicode 通道及清理控制。ARM 交叉构建早先 +遇到 runner 关闭，SIGTERM 143 日志保留；另一次清理挂起的失败未取得日志，原因 +仍未证明。最终相同提交的 ARM 构建及 QEMU 运行通过，没有修改测试或交叉目标。 + +完整 CI 成功后，另行逐项完成用户要求的生态审查，覆盖通用核心与插件边界、每个 +消费者的普通依赖闭包、宿主/目标分离、版本与协议兼容、五种 action 角色、取消 +及失败后的通道清理、实际依赖版本中的有界文件读取、中英文说明、平台覆盖、 +TLS 和交叉封装。核心没有阻塞发现；插件进度写入的 Windows UTF-8 问题在 #52 +处理，不把未通过完整性能验收的引擎视为已完成。审查记录为 +`/tmp/mcpp-885-core850-post-ci-ecosystem-review.json`。 + +#888 已于 2026-11-05 03:13:21 UTC 合入，提交为 +`1e6d1b3f9c68510af7d2d5b68bbf7aaef1a89e73`。合入树与被审查提交的树一致。 +对应 main CI `41960218833` 已先取消并确认终止，随后在合入提交创建 +`v2026.11.5.1`，启动 Release `41960453765`。本检查点只证明发布已启动；完整 +资产、双端镜像、索引传播、实际公开安装、bootstrap pin 和 main CI 重启仍须按 +第 6 节的顺序闭环。 + +## 14. 已公开依赖组与消费者边界（2026-11-05） + +musl 0.27.6、LLVM runtime 0.18.7 与 platform 0.23.7 已分别合入、发布并进入 +公开索引。三份源码的 GitHub/GitCode 字节一致。普通 GLOBAL/CN 消费者实际 +通过 C++/import std、ThinLTO、TLS 线程隔离、canonical path 及 ARM 交叉封装 +控制；这些消费者不使用本地路径或索引重定向。三个发布后 main CI 分别为 +`41956257730`、`41956262835`、`41956268179`，共 24 项检查成功；验证时没有 +`MCPP_SOURCE_REF`。核心默认保持已验证的 0.23.6，显式 0.23.7 集成另有实证。 + +插件 #52 的 `26ff5430eda7a957bffe39ca6869801ad5d1b672` 完整 CI +`41959654833` 六项全部成功。Windows ACP 1252 上普通 CRT `getenv` 的确丢失 +Unicode 通道字节；辅助写入端已改用 os.env 的 UTF-8 接口。Windows 原生、macOS +及 Linux 的关闭、Unicode 写入与不可写路径控制通过。插件最终新核心集成 pin +和发布仍等待核心版本公开可安装。 + +引擎 #16 的 `a13953808f4ec4704288adca3ae095ba858bce86` 已使用公开 platform +0.23.7。冻结 Linux 对照 `41959812911` 运行原始工作负载，16 个语义会话、就绪、 +热 CPU、缺页及路径对等通过；冷 CPU 分别为 15.08 / 12.32 = 1.224（x64）和 +10.65 / 9.11 = 1.169（ARM64），仍高于 1.15。不得据其他检查通过提升台账或 +发布，也不重跑未修改的性能检查寻求偶然通过。服务器 #75 的原生构建和测试 +归档共 34 项成功；严格引擎候选接纳仍拒绝未合格的上游。 + +## 15. 当前产物的逐进程定位（2026-11-05） + +以 a139 对应的原生与 universal 资产 SHA、ZIP CRC、包内身份和 ELF 成员位置 +固定输入。两路完整冷工作负载均包含 304 个单元、三个文件和 176 个编译 worker， +实际语义就绪通过。strace 按 PID 分类，独立 snapshot 后台进程另记；跟踪后的 +CPU 与耗时不作原性能门槛的替代。当前 image 的 `fstatfs` 共 1,050 次，较上一轮 +约 68,000 次明显减少，证明 realpath 修正确实进入了当前产物。 + +每个 image worker 除 owner-death 监视外，还由 Linux 后端的单位内存监督启动 +一个线程，立即遍历整个宿主 `/proc`，打开每个可见 PID 的 `stat` 再判断进程组。 +该观察属于宿主运行器的后端实现，不需要在通用 mcpp 核心添加进程或 HPC 语义。 + +实验原型先以 `getpgid` 排除确认属于其他组的进程；查询失败仍读原记录，可能的 +组成员仍从同一份 RSS 记录复核进程组。原型保留全部进程组成员、首次立即采样、 +100/200/500 ms 的原采样间隔、组内内存求和与全组终止。实际组件控制包括成员 +加入/退出、组长结束后的成员、EPERM/ENOSYS 后备和合计超限时终止所有成员而 +保留其他组；全部通过。一次采样的 `stat` 打开数从 37 降到 2，12 组交替组件 +CPU 中位比为 0.244。该结果仅属于组件。 + +两份相同编译参数、仅后端过滤逻辑不同的运行器，随后对同一 a139 image 运行 +原始 304 单元、三个文件、176 worker 的完整冷工作负载。八组交替测试的冷 CPU +中位数为基线 5.600 s、候选 5.385 s（比值 0.962）；公开运行器对照为 5.575 s。 +完整生命周期 CPU 中位数为 5.639234 s 与 5.4300145 s。就绪时间中位数为 +3.880 s 与 4.034 s，没有得到就绪时间改善的证据。所有语义会话通过，单组测量 +存在波动；这些本地结果不能证明原 CI 的冷 CPU 门槛已通过。 + +Linux 后端改动、0.24.2 版本及说明复用同一 PR #50，提交为 +`6bd65128c53d1bc1fc99f2ddcb984e12325b3490`。另一个冻结诊断 +`41966671816` 在 Linux x64/ARM64 上运行原生引擎、公开运行器、同参数基线和 +候选运行器；保留原工作负载、三次热会话与未修改的接纳脚本。诊断不提升引擎 +台账，也不替代正式生产者的完整资格；后端尚未发布。 + +另以同一批 BitstreamReader/Support ThinLTO 对象和当前 176 份 PCM，比较最终 +LLD 的默认、显式 O2 与 O3。默认产物与显式 O2 字节一致；十二组交替组件测试 +中 O3/O2 的 CPU 中位比为 1.013，没有得到该组件的优化收益证据，未据此修改 +引擎链接参数。该组件沿用实验中的 platform 0.23.5，其结果不推广到完整引擎。 + +## 16. 运行器后端的采用依据与发布次序（2026-11-05） + +冻结诊断 `41966671816` 两个 Linux job 已成功，资产 ZIP 的 SHA256、CRC 与展开 +文件已逐项核对，并用 a139 的未修改脚本重新计算六组接纳结果。x64 同一宿主的 +公开运行器冷 CPU 为 15.45 / 13.05 = 1.184，同编译参数基线为 +15.33 / 13.05 = 1.175，均失败；过滤候选为 14.77 / 13.05 = 1.132，通过。 +ARM64 候选为 10.99 / 9.89 = 1.111；该轮公开和基线运行器也通过，分别为 +1.126、1.124，不能把 ARM64 的通过全部归因于过滤。两宿主共 32 个语义会话 +通过，候选的就绪、冷热 CPU、缺页及路径对等均按原门槛通过。这是采用修正的 +对照证据，不是正式生产者或联合 UX 的资格提升。 + +后端 PR #50 的 GCC/LLVM 完整 CI `41966412791` 已通过，随后另行审查资源、 +错误、组成员、ABI、跨架构与说明，合入为 +`59a3a59b53625f144b17c96e8a3b07eb617cef1a`，并发布 0.24.2。GitHub/GitCode +源码同为 164569 字节，SHA256 为 +`bfe9f4fc34b9d2c4d321939bb00f4da7c8f444abb38bb12756da7b5d57717ee9`。 +索引 PR #713 已准备并提交，实际公开源码消费者、精确锁与本地 lint 通过；尚须 +合入、发布索引并确认普通客户端可安装。 + +运行器改动复用原 #885 全局审查产生的 openkal-image PR #25。其 okprof 环境 +传递修正已有五项完整 CI 通过；合入当前 main 0.4.4 后，保留 Windows 映射尾部 +修正，再准备四包同版 0.4.5、Linux 0.24.2 与说明。公开依赖未完成索引前不推送 +要求安装该版本的最终 CI。预编译运行器仍使用正式 hosted 平台与同一精确版本 +组，不通过更换运行库或跳过版本组检查缩短发布链。 + +核心 Release `41960453765` 的两项生态 canary 与 Linux x86_64 静态包已通过， +ARM64、macOS 与 Windows 正在并行构建。完整镜像、索引和实际安装尚待完成， +bootstrap pin 和已取消的 main CI 因此尚未前移或重启。 + +## 17. macOS 发布自举生命周期修正（2026-11-05） + +Release 的 Windows 构建成功，macOS job `127048731858` 在第二阶段失败。第一阶段 +实际输出 2026.11.5.1，随后从 `target/.../bin/mcpp` 调用 `build --no-cache`；清理 +删除了正在自举的编译器路径，Ninja 再调用该路径生成依赖记录时报告文件不存在。 +这是发布流程的编译器生命周期问题，不能按基础设施波动原样重跑。 + +发布流程先将第一阶段可执行文件复制到 RUNNER_TEMP，再执行原来的清理重建。 +实际候选编译器与 LLVM 23.1.3 的 import std 小工程复现了负向控制：编译器位于 +输出目录时被删除并失败；同一编译器复制到目录外后，完整清理重建通过，生成 +程序实际运行输出 `stage1 lifetime held`。该 Linux 控制不能替代修复后 macOS +发布 job 的原生验证。 + +已上传的 Linux x86_64 与 Windows 资产保留。修复后的 macOS 构建仍取原 tag +的源码，完成原部署版本、静态 libc++、路径、封装及 smoke 检查；发布清单和 +镜像继续要求其他三个平台的实际构建成功，不以补建单个平台替代整组发布门槛。 +原 tag、源代码和此前通过的核心 CI 身份保持可核对。 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6f4cc7522..95507d9b0 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -538,6 +538,12 @@ jobs: "$MCPP" build STAGE1=$(find target -path "*/bin/mcpp" | head -1) STAGE1=$(cd "$(dirname "$STAGE1")" && pwd)/$(basename "$STAGE1") + # --no-cache removes target/, including the stage-1 compiler. Ninja + # invokes that compiler again for dependency scanning, so keep the + # bootstrap outside the tree the second stage is about to clear. + STAGE1_COPY=$(mktemp -d "$RUNNER_TEMP/mcpp-stage1.XXXXXX")/mcpp + cp "$STAGE1" "$STAGE1_COPY" + STAGE1="$STAGE1_COPY" "$STAGE1" --version # Stage 2: this release's mcpp rebuilds itself — flags.cppm's From 11dc0d0e85d7352c0afa264c13f9c703118c1643 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 5 Nov 2026 14:39:40 +0900 Subject: [PATCH 02/40] ci: close out 2026.11.5.1 publication and local GitCode mirroring --- ...heterogeneous-build-implementation-plan.md | 29 +++++++++++++++++++ .agents/skills/mcpp-release/SKILL.md | 19 ++++++++++++ .xlings.json | 2 +- 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md index ce960f4be..3bb278d5a 100644 --- a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md +++ b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md @@ -438,3 +438,32 @@ Release 的 Windows 构建成功，macOS job `127048731858` 在第二阶段失 的源码，完成原部署版本、静态 libc++、路径、封装及 smoke 检查；发布清单和 镜像继续要求其他三个平台的实际构建成功，不以补建单个平台替代整组发布门槛。 原 tag、源代码和此前通过的核心 CI 身份保持可核对。 + +## 18. 发布资源的本地镜像与索引收尾（2026-11-05） + +修复后的原生 macOS job、原发布的其他三个平台与两项生态 canary 均已成功。 +恢复流程 `41969495679` 的源码校验、macOS 构建、完整发布清单和生态发布四项 +全部成功；tag 仍指向 `1e6d1b3f9c68510af7d2d5b68bbf7aaef1a89e73`。 +四个平台安装包已独立下载并核对清单哈希与封装。 + +用户指定后续资源在 GitHub 产出时立即使用本地 gtc 补齐 GitCode，不等待 CI +mirror。该流程已补齐本次全部 21 项发布资源，包含安装包、别名、校验文件、 +源码、脚本与清单；每项均实际 GET 下载并校验 SHA256。此前 GitCode 下载为 +404 的 Linux x64 与 ARM64 包，本地上传分别耗时约 11 s 与 13 s。资源已有且 +字节相同则保留，同名内容冲突须查明，不以删除重传处理。 + +索引 PR `openxlings/xim-pkgindex#1108` 的三个 latest 条目、四个平台资源地址 +及 SHA256 已与完整清单核对，合入为 +`785e62895aa4216361f031424033394f6ef8ba4e`。按用户此前指定的本地直接发布 +流程完成双端资产和指针，实际公开下载的索引包为 1758038 字节，SHA256 为 +`e35e26ec319db22e7fbfb2f98cb158a61e3eff5162778187940764cbf285053b`。 +随后 GLOBAL 与 CN 两种配置的隔离客户端均从普通公开索引安装精确版本和 +latest，实际执行输出 `mcpp 2026.11.5.1`；安装后的 ELF 与公开安装包逐字节 +对应，SHA256 为 `cb665b0ceb1856a1e588faba183c3de29b9f5b455d6e9fd2944d14175235bced`。 +客户端保留产品自身的镜像回退行为。bootstrap pin 在这些验证之后前移，核心 +同一收尾 PR #889 继续要求最终提交 CI 和生态审查；完成后重启 main CI。 + +运行器源码 0.4.5 已复用 PR #25 发布，Linux 0.24.2 和四个 image 包 0.4.5 的 +索引分别经 PR #713、#714 合入、发布并通过普通客户端验证。musl 0.27.7 复用 +版本组推进的 PR #85；runtime 0.18.8 与 platform 0.23.8 继续按公开前置依赖 +依次验证。源码对照诊断不能替代引擎正式生产者与联合 UX 的资格验证。 diff --git a/.agents/skills/mcpp-release/SKILL.md b/.agents/skills/mcpp-release/SKILL.md index abe28c571..8e98c8673 100644 --- a/.agents/skills/mcpp-release/SKILL.md +++ b/.agents/skills/mcpp-release/SKILL.md @@ -209,6 +209,25 @@ multipart/无断点续传，**单文件拆不开** —— 这正是必须先把 再回调 GitCode API，回调失败就返回 `code:400 ... EOF`，而**对象其实已落盘**。 判定上传成功**只能靠回探下载 URL**，脚本就是这么做的。 +### 本地 GitCode 补齐（2026-11-05 用户指定流程） + +每个发布资源在 GitHub 产出后立即下载到本地，以本地已配置的 `gtc` 补齐 +`xlings-res/mcpp` 的同版本 GitCode release，不等待 CI mirror。资源集合取自 +GitHub release 的实际资产清单，包括版本安装包、无版本别名、校验文件、源码、 +安装脚本与发布清单；后续新增资源继续补齐。资源镜像 tag 不带 `v`。 + +```bash +gtc release upload xlings-res/mcpp --tag +``` + +上传前校验本地文件的来源、大小和 SHA256；GitCode 已存在且下载校验一致的 +资源直接保留。缺失资源可并发上传，上传后实际 GET 下载并复核大小和 SHA256， +不以 `gtc` 的退出码代替下载回验。同名资源内容不一致时先查明原因，不删除或 +覆盖既有发布字节。记录每项资产的来源、哈希、上传及下载验证结果。 + +本地补齐与 CI 构建、mirror 并行进行。索引发布仍要求四个平台构建成功、完整 +发布清单通过以及所有被索引资源实际可下载；这些条件满足后可继续索引收尾。 + ### 发布后的收尾（必须做完，否则用户装不到） ```bash diff --git a/.xlings.json b/.xlings.json index 116255d3d..f3a65ae90 100644 --- a/.xlings.json +++ b/.xlings.json @@ -1,5 +1,5 @@ { "workspace": { - "mcpp": "2026.11.3.1" + "mcpp": "2026.11.5.1" } } From 7192039d7138a3044b09fb2de0370fc94321df8b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 5 Nov 2026 20:11:19 +0900 Subject: [PATCH 03/40] fix: load cached action metadata before buffered ninja output --- ...heterogeneous-build-implementation-plan.md | 21 ++++++++++ src/build/progress.cppm | 7 +++- tests/unit/test_build_progress.cpp | 38 +++++++++++++++++++ 3 files changed, 64 insertions(+), 2 deletions(-) diff --git a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md index 3bb278d5a..38de97ac6 100644 --- a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md +++ b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md @@ -467,3 +467,24 @@ latest，实际执行输出 `mcpp 2026.11.5.1`；安装后的 ELF 与公开安 索引分别经 PR #713、#714 合入、发布并通过普通客户端验证。musl 0.27.7 复用 版本组推进的 PR #85；runtime 0.18.8 与 platform 0.23.8 继续按公开前置依赖 依次验证。源码对照诊断不能替代引擎正式生产者与联合 UX 的资格验证。 + +## 19. 缓存构建的内部进度首次读取（2026-11-05） + +收尾 CI 的 `1013_optional_action_progress_preserves_outputs` 暴露了一个核心显示 +缺口。缓存构建不重新规划，其步骤归属记录原先只在 Ninja 首次输出状态时读取； +异步轮询在记录尚未加载时直接跳过 action-start 文件。Ninja 缓冲长动作输出时， +合法进度文件虽已存在，界面仍不能显示正在执行的动作和内部阶段。该问题不说明 +动作本身停止，也不能解释引擎冷启动 CPU 的性能差额。 + +修正使轮询先检查 action-start 是否新增内容，再按需读取已有归属记录。无工作 +的缓存构建仍不预读记录；快照计数仍不能完成动作，输出后置条件、命令退出码和 +取消处理保持原契约。没有通过延长工作、隐藏断言或调低门槛修复正式测试。 + +固定原构建的 448 个对象，仅重编译进度模块并链接诊断二进制。新增测试在没有 +任何 Ninja stdout 或完成记录时要求观察到动作及内部阶段：原模块在两秒期限内 +失败，修正模块通过；完整进度测试文件的 25 项全部通过。原 1013 E2E 与额外的 +短动作观测控制均通过，包含各 role 的输出、增量不重跑、失败和取消。初次八路 +全量本地编译曾被资源回收终止，不记为通过；单模块控制也不替代正式 CI。 + +源代码和回归测试继续复用 PR #889。其最新完整 CI 和随后独立生态审查仍是合入 +前置条件，已发布 tag 的源码与资产不作改写。 diff --git a/src/build/progress.cppm b/src/build/progress.cppm index 9e18040fa..21b9b06d7 100644 --- a/src/build/progress.cppm +++ b/src/build/progress.cppm @@ -733,7 +733,8 @@ struct Build::Impl { namespace { // The fast path has no plan: it reads the record the plan wrote, when ninja -// reports its first step, so a build with nothing to do reads nothing. +// reports its first step or an action writes its start, so a build with +// nothing to do reads nothing. Ninja may buffer stdout until an action ends. void ensure_record(Build::Impl& b) { if (b.record || b.recordTried) return; b.recordTried = true; @@ -1004,11 +1005,13 @@ void read_log(Report& r, Build::Impl& b, std::vector& out) { // Reads the start file of an open pass; model lock held. A check or prepare // action that starts names its package: that is work. void read_starts(Report& r, Build::Impl& b, std::vector& out) { - if (!b.inPass || !b.record) return; + if (!b.inPass) return; std::error_code ec; const auto path = b.dir / kStartsFile; const auto size = std::filesystem::file_size(path, ec); if (ec || size <= b.startsOffset) return; + ensure_record(b); + if (!b.record) return; std::ifstream f(path, std::ios::binary); f.seekg(static_cast(b.startsOffset)); std::string text(static_cast(size - b.startsOffset), '\0'); diff --git a/tests/unit/test_build_progress.cpp b/tests/unit/test_build_progress.cpp index f0fc2d0b9..965f8c905 100644 --- a/tests/unit/test_build_progress.cpp +++ b/tests/unit/test_build_progress.cpp @@ -3,6 +3,7 @@ import std; import mcpp.ui; import mcpp.build.progress; +import mcpp.build.action_progress; // The build's report (.agents/docs/2026-09-29-build-progress-display-design.md, // revised by .agents/docs/2026-09-30-build-output-refinement-design.md): its @@ -331,6 +332,43 @@ TEST(ProgressText, ThePhaseIsAlignedWithTheVerbs) { // ─── When a package is named (revision 3, §5.3) ────────────────────────── +TEST(ProgressModel, AnActionStartLoadsTheFastPathRecordBeforeNinjaWritesStdout) { + namespace telemetry = mcpp::build::action_progress; + Tmp tmp; + Record rec; + rec.packages = {{"progress-no-stdout", true, "progress-no-stdout", 0, 1, "v0.0.0 (.)", "project"}}; + rec.owner = {{"gen/check.stamp", 0}}; + rec.step = {{"gen/check.stamp", 0}}; + rec.actions = {{"gen/check.stamp", "check-action"}}; + rec.steps = 1; + write_record(tmp.path, rec); + Build build(tmp.path); // The cached path has a record on disk, not set_record(). + build.pass_begin(); + const auto env = build.environment(); + const auto root = std::ranges::find(env, telemetry::kRootEnv, &decltype(env)::value_type::first); + ASSERT_NE(root, env.end()); + std::ofstream(telemetry::channel(root->second, "gen/check.stamp")) + << R"({"v":1,"seq":1,"phase":"internal-check","done":1,"total":2})"; + const auto unixMs = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + append(tmp.path / kStartsFile, std::format("gen/check.stamp\t{}\n", unixMs)); + testing::internal::CaptureStderr(); + open(false); + const auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(2); + std::string row; + do { + row = status_row(); + if (row.contains("internal-check 1/2")) break; + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } while (std::chrono::steady_clock::now() < deadline); + close(); + (void)testing::internal::GetCapturedStderr(); + EXPECT_TRUE(build.has_record()); + EXPECT_NE(row.find("check-action"), std::string::npos) << row; + EXPECT_NE(row.find("internal-check 1/2"), std::string::npos) << row; + // No Build::status() or completed ninja-log entry was supplied. +} + TEST(ProgressModel, APackageIsNamedOnceWhenItsFirstStepFinishes) { mcpp::ui::disable_color(); Tmp tmp; From 05d94de5d92bb66effb7620952b43709a82ab751 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 5 Nov 2026 21:09:15 +0900 Subject: [PATCH 04/40] chore: bump version to 2026.11.5.2 --- mcpp.toml | 2 +- modules/versioning/src/version.cppm | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/mcpp.toml b/mcpp.toml index b2b34b1f0..8d04a856b 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.11.5.1" +version = "2026.11.5.2" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index 104e219f6..582a25ec6 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.11.5.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.11.5.2"; } // namespace mcpp From 08fb971ffc95b0cdfc8f217af680e5289cb708e2 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 5 Nov 2026 21:11:36 +0900 Subject: [PATCH 05/40] docs: align patch release provenance and emergency procedure --- ...heterogeneous-build-implementation-plan.md | 19 +++++++ .agents/skills/mcpp-release/SKILL.md | 52 ++++++++----------- CHANGELOG.md | 16 ++++++ 3 files changed, 57 insertions(+), 30 deletions(-) diff --git a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md index 38de97ac6..9e71deca5 100644 --- a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md +++ b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md @@ -488,3 +488,22 @@ latest，实际执行输出 `mcpp 2026.11.5.1`；安装后的 ELF 与公开安 源代码和回归测试继续复用 PR #889。其最新完整 CI 和随后独立生态审查仍是合入 前置条件，已发布 tag 的源码与资产不作改写。 + +## 20. 收尾 CI 后的生态审查与补丁版本（2026-11-05） + +提交 `7192039d` 的 CI `42016930743` 已完成：74 项执行成功，另有一项按原条件 +跳过。随后单独复核了 action-start 与缓存归属记录的时序、无工作路径、遥测与 +退出状态的独立性、四宿主构建、原 1013 E2E、发布自举的执行路径和公开 bootstrap。 +该显示修正不改变 action 图、指纹输入、消费者工具链或 accelerator 语义，也不 +承担插件 CMake 参数序列化的职责。 + +审查要求两项收尾：进度显示的代码修正应使用新补丁版本，不能改写公开的 +2026.11.5.1；发布技能中残留的直推 main、移动旧 tag 和删除旧 release 步骤应 +与 PR 规范统一。版本号在同一提交内改为 2026.11.5.2，真源位于 `mcpp.toml` +和 `modules/versioning/src/version.cppm`。bootstrap 保持已完成公开安装验证的 +2026.11.5.1，运行时推导的 MCPP_PIN 不作手工修改。 + +新 CHANGELOG 与发布技能已对齐。旧版本文件路径同时改为当前模块路径，紧急 +修复也明确要求 PR、新版本及原发布门禁。新最终提交仍须完成完整 CI 和随后 +生态审查；仅在通过后合入并创建新 tag，按用户指定顺序优先 release，及时用 +本地 gtc 补 GitCode，完成索引和实际安装后再恢复 main CI。 diff --git a/.agents/skills/mcpp-release/SKILL.md b/.agents/skills/mcpp-release/SKILL.md index 8e98c8673..1fffd2b5d 100644 --- a/.agents/skills/mcpp-release/SKILL.md +++ b/.agents/skills/mcpp-release/SKILL.md @@ -18,7 +18,7 @@ description: Use when releasing a new version of mcpp — bumps version, creates 跨方案的序是单调的：`0.0.109` < `2026.7.27.1`，第一段从 `0` 变 `2026`，不存在回退。 -> 比较逻辑见 `src/version_req.cppm`。它支持 4 段；**改动那里时务必保证 `str()` 精确回写**， +> 比较逻辑见 `modules/versioning/src/version_req.cppm`。它支持 4 段；**改动那里时务必保证 `str()` 精确回写**， > `src/pm/resolver.cppm` 用它重建依赖版本串，会流向 lock 文件与 xlings wire 地址。 > 尤其 `.0` 结尾的版本不能塌成三段。 @@ -31,7 +31,7 @@ mcpp 有 **三个持久化版本位置**，以及 `ci-fresh-install` 的一个 **第一组：正在构建的版本**（发布时改，走 bump PR） 1. `mcpp.toml` → `[package].version` — 构建系统读取的项目版本，release.yml 由它推导 tag -2. `src/version.cppm` → `MCPP_VERSION` — 编译期硬编码常量（`--version` 输出、BMI 指纹、E0006 索引底线比较） +2. `modules/versioning/src/version.cppm` → `MCPP_VERSION` — 编译期硬编码常量（`--version` 输出、BMI 指纹、E0006 索引底线比较） 这两处必须**在同一个 commit 里**一起改：`tests/e2e/01_help_and_version.sh` 交叉比对 `mcpp.toml` 与 `mcpp --version`，只改一处 CI 立刻红。 @@ -59,7 +59,7 @@ unexpected`。那是调用它的 shell 的问题，不是脚本的缺陷 —— `#!/usr/bin/env bash`，CI 也是用 `bash` 调的。别据此把这条 guard 当成坏的而跳过： 它是唯一能机器化捕捉 pin 漂移的东西。 -也不要通过修改文档或 workflow 绕开动态 `MCPP_PIN` 设计。`src/platform/xlings/xlings.cppm` 的 +也不要通过修改文档或 workflow 绕开动态 `MCPP_PIN` 设计。`src/provider/xlings/xlings.cppm` 的 `pinned::kXlingsVersion` 仍是 xlings 版本的唯一真源。 ## 发布步骤 @@ -89,7 +89,7 @@ NEW_VERSION="2026.7.27.1" git checkout -b "chore/bump-$NEW_VERSION" sed -i "s/^version.*=.*/version = \"$NEW_VERSION\"/" mcpp.toml -sed -i "s/MCPP_VERSION = \".*\"/MCPP_VERSION = \"$NEW_VERSION\"/" src/version.cppm +sed -i "s/MCPP_VERSION = \".*\"/MCPP_VERSION = \"$NEW_VERSION\"/" modules/versioning/src/version.cppm # 校验：mcpp.toml 与 MCPP_VERSION 相等，.xlings.json 不领先于正在构建的版本。 bash .github/tools/check_version_pins.sh @@ -302,14 +302,14 @@ gh workflow run bump-formula.yml -R mcpp-community/homebrew-mcpp | 症状 | 原因 | 修复 | |------|------|------| -| `mcpp X.Y.Z-1` 但 tag 是 `vX.Y.Z` | `fingerprint.cppm` 版本未更新 | 更新 `MCPP_VERSION`，重新打 tag | +| `mcpp X.Y.Z-1` 但 tag 是 `vX.Y.Z` | `modules/versioning/src/version.cppm` 版本未更新 | 在 PR 中同步版本号；已发布版本使用新 tag | | bump PR 里**所有** CI job 都红在 bootstrap，报 `package 'mcpp@X.Y.Z' not found` | 把 `.xlings.json` bootstrap pin 一起 bump 了，CI 去装一个还没发布的版本 | 把 `.xlings.json` 回退到上一个已发布版本；不要修改运行时推导的 `MCPP_PIN` | | main 上**所有** workflow 都红在 bootstrap，报 `package 'mcpp@X.Y.Z' not found`，而该版本确实已发布并已合入索引 | pin 推得比索引指针稳定早；客户端读 CDN artifact 不读 git 树 | 等 `xim-index-latest.json` 的 `index_version` 等于索引 main 短 SHA 后重跑；不要改 pin | | `xlings install mcpp`（不带 `@版本`）装到**上一个**版本，而索引文件里明明有新版本 | 同一平台表里两个 `["latest"]` 键（手写索引 PR 追加而非替换），Lua 后写的赢 | 删掉陈旧的那行，使每个平台表只剩一个 `["latest"]`。**精确 pin 仍可用，所以 mcpp 自己的 CI 不会报这个** | | 自查 `--version` 显示旧版本，但源码已改 | `target//<指纹>/` 的指纹随版本变，`ls \| head -1` 取到了上一次构建的目录 | 用 `ls -dt … \| head -1` 取最新构建 | | Smoke test 输出旧版本 | CI 缓存了旧的 sandbox/target | 删除 GitHub Actions cache 后重跑 | -| e2e `01_help_and_version.sh` 挂 | 只改了 `mcpp.toml` 没改 `fingerprint.cppm`（它把两者交叉比对） | 同步两处正在构建的版本；注意这个 e2e 只在部分分片里跑，可能表现为"只有某个平台红" | -| xlings bootstrap 失败 | xlings 版本不兼容 | 改 `src/platform/xlings/xlings.cppm::kXlingsVersion`（唯一真源），再核对引用它的 workflow 与脚本；当前 pin-check 脚本修复前不能依赖它完成扫描 | +| e2e `01_help_and_version.sh` 挂 | 只改了 `mcpp.toml` 没改 `modules/versioning/src/version.cppm`（它把两者交叉比对） | 同步两处正在构建的版本；注意这个 e2e 只在部分分片里跑，可能表现为"只有某个平台红" | +| xlings bootstrap 失败 | xlings 版本不兼容 | 改 `src/provider/xlings/xlings.cppm::kXlingsVersion`（唯一真源），再核对引用它的 workflow 与脚本；当前 pin-check 脚本修复前不能依赖它完成扫描 | | macOS/Windows 构建失败 | 需要等 Linux job 先完成 | 检查 Linux job 是否成功 | | `slim: FAIL: ... still not stripped` | strip 工具没生效／被 pack 覆盖 | 别绕过断言——它就是为了拦住 34.8MB 的 tarball 再次发出去 | | mirror leg 报 `missing/unverified` | 资产没传上去或还没传播 | 先 GET 核验（**必须 GET，`curl -I` 会骗你**），gitcode 用 `gitcode.com` 直链而非 `api.` 主机；确认缺件后本地补传再 `gh run rerun --failed`（脚本幂等，已验证的资产会跳过） | @@ -331,38 +331,30 @@ gh workflow run release.yml --ref "v$NEW_VERSION" ## 紧急修复发版 -如果发版后发现问题需要紧急修复： +已发布版本的修复仍走 PR 和新补丁版本，不直接推 main，不移动已有 tag，不删除 +已发布资源后用不同字节替换。紧急程度不改变源码与公开安装包的对应关系。 -```bash -# 1. 在 main 上修复 -git checkout main && git pull -# ... 修改代码 ... -git commit -m "fix: " -git push origin main - -# 2. 更新 tag 指向新 commit（包含修复） -git tag -d "v$NEW_VERSION" -git tag "v$NEW_VERSION" -git push origin "v$NEW_VERSION" --force - -# 3. 删除旧 release（如果已创建） -gh release delete "v$NEW_VERSION" --yes - -# 4. 重跑 release -gh workflow run release.yml --ref "v$NEW_VERSION" -``` +1. 从最新 main 建修复分支，将修正、第一组的两处版本号与新 CHANGELOG 条目提交 + 到 PR。bootstrap 继续使用已发布版本。 +2. 完成实际 required checks 和独立生态审查后合入。只为已验证的新版本创建新 tag， + 并确认它指向审查后的提交。 +3. 完成原有四平台 release 检查、双端资源校验、索引发布和普通客户端安装验证。 + GitCode 资源按本地及时补齐流程处理。 +4. 新版发布、镜像与索引均可实际下载后，再通过 PR 前移 bootstrap pin。 -**注意**：如果修复涉及版本号变化，应该 bump 到新的 patch 版本而不是覆盖旧 tag。 +若原版本只是构建或上传尚未完成，可以在原 tag 源码不变的前提下恢复失败步骤。 +同名资源仅在字节相同并完成下载校验时视为已存在；不同内容须先查明原因，不能 +以删除重传或重指 tag 绕过冲突。 ## 文件清单 | 文件 | 版本相关内容 | |------|-------------| | `mcpp.toml` | `version = "X.Y.Z"` — 项目版本，release.yml 由它推导 tag | -| `src/version.cppm` | `MCPP_VERSION = "X.Y.Z"` — 编译期版本常量 | +| `modules/versioning/src/version.cppm` | `MCPP_VERSION = "X.Y.Z"` — 编译期版本常量 | | `.xlings.json` | `workspace.mcpp` — CI bootstrap 装哪个 mcpp（发布**后**才 bump） | | `.github/workflows/ci-fresh-install.yml` | `MCPP_PIN` — 由 `wait-index` 从最新 release 推导，**从不手工 bump** | -| `src/platform/xlings/xlings.cppm` | `kXlingsVersion` — xlings pin 的**唯一真源** | +| `src/provider/xlings/xlings.cppm` | `kXlingsVersion` — xlings pin 的**唯一真源** | | `.github/tools/check_version_pins.sh` | 版本/pin 校验 guard（**用 `bash` 跑，不能用 `sh`**） | | `.github/tools/slim_linux_payload.sh` | linux 载荷 strip + 断言 | | `.github/tools/mirror_res.sh` | 双端镜像（并发上传 + leg deadline + 完整性 gate） | @@ -371,7 +363,7 @@ gh workflow run release.yml --ref "v$NEW_VERSION" | `install.sh` | 安装脚本（随 release 发布） | | `CHANGELOG.md` | Release notes 来源（按 `## [X.Y.Z]` 提取）；条目、commit 与 release notes 一律用英文（2026.9.28.3 起） | -> **注意版本 bump 的两个阶段**：`mcpp.toml` + `fingerprint.cppm` 在发版**前**改 +> **注意版本 bump 的两个阶段**：`mcpp.toml` + `modules/versioning/src/version.cppm` 在发版**前**改 > （它们定义要发什么）；`.xlings.json` 只在发版成功、镜像并进索引后才可更新 > （它指定 bootstrap 使用的已发布版本）。`MCPP_PIN` 是被测版本的运行时推导值， > 不属于任何手工 bump 阶段。这些关系由 `bash .github/tools/check_version_pins.sh` 校验。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 29813fe1f..551967ffd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,22 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. +## [2026.11.5.2] - 2026-11-05 + +### Fixed + +- Cached builds load action ownership when an action starts, so internal progress + appears while Ninja is still buffering stdout. Idle builds retain lazy loading; + command results and output checks remain authoritative. +- The macOS clean bootstrap keeps its stage-one compiler outside the output tree + removed by the second stage. + +### Changed + +- CI bootstraps from the published, mirrored and indexed 2026.11.5.1 release. +- Release guidance preserves published tags and uses a new reviewed patch version + for fixes. GitCode assets are supplemented locally as they become available. + ## [2026.11.5.1] - 2026-11-05 ### Added From 627786aec7edf9c87cdcef2e768e073c1cf78b50 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 5 Nov 2026 22:00:45 +0900 Subject: [PATCH 06/40] fix: redraw action snapshots between Ninja status lines --- ...heterogeneous-build-implementation-plan.md | 22 ++++++++++++ CHANGELOG.md | 2 ++ src/build/progress.cppm | 16 ++++++--- tests/unit/test_build_progress.cpp | 34 +++++++++++++++++++ 4 files changed, 70 insertions(+), 4 deletions(-) diff --git a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md index 9e71deca5..6f3747eb6 100644 --- a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md +++ b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md @@ -507,3 +507,25 @@ latest，实际执行输出 `mcpp 2026.11.5.1`；安装后的 ELF 与公开安 修复也明确要求 PR、新版本及原发布门禁。新最终提交仍须完成完整 CI 和随后 生态审查；仅在通过后合入并创建新 tag，按用户指定顺序优先 release，及时用 本地 gtc 补 GitCode，完成索引和实际安装后再恢复 main CI。 + +## 21. 内部快照与界面刷新之间的通知（2026-11-05） + +最终版本提交 `08fb971f` 的 CI `42025082619` 在原 1013 E2E 首次终端输出 +断言失败。缓存归属记录的首次读取修正仍有效；这次进一步核对发现，异步轮询 +接受内部快照后并未请求界面刷新。无 Ninja 输出和其他叙述时，界面依靠一秒 +时钟刷新，而原 fixture 的阶段只持续 0.75 秒，因而可能在实际绘制前结束。 +此前同一测试通过不能排除该时序缺口，也不能用原样重试代替修正。 + +快照读取现在报告是否接受了新的序号，轮询在释放模型锁后请求现有 UI frame。 +各构建的结果采用非短路聚合，仍读取全部活跃动作；无更新、无效或旧序号不 +请求额外 frame。原十帧每秒上限、命令完成、快照非权威性以及锁顺序均保持。 + +新增回归检查实际渲染输出：先绘制初始 frame，再发布快照，在一秒时钟刷新 +之前结束观察；既不提供 Ninja stdout，也不补充其他叙述。仅替换进度模块的 +固定对象对照中，上一模块在此检查失败，修正后完整 26 项进度检查通过。 +正常 1013 E2E 保持原 0.25 秒更新间隔及全部断言，并通过修正后的诊断二进制。 +另有 0.001 秒的历史短动作探针，其失败不代表原 1013 测试失败；局部探针的 +时序结果不能替代实际输出回归与最终完整 CI。正式 E2E 未加入等待观察的握手。 + +该补充继续使用 PR #889 与 2026.11.5.2。新的最终提交仍须完整 CI 以及其后 +独立生态审查，通过后才合入和发布；CLion 两仓库的兼容性结果不受此门禁替代。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 551967ffd..afd519cfa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,8 @@ - Cached builds load action ownership when an action starts, so internal progress appears while Ninja is still buffering stdout. Idle builds retain lazy loading; command results and output checks remain authoritative. +- New action snapshots request a normal UI frame, so short phases can appear + between Ninja status lines without waiting for the clock redraw. - The macOS clean bootstrap keeps its stage-one compiler outside the output tree removed by the second stage. diff --git a/src/build/progress.cppm b/src/build/progress.cppm index 21b9b06d7..a01b43006 100644 --- a/src/build/progress.cppm +++ b/src/build/progress.cppm @@ -1035,14 +1035,18 @@ void read_starts(Report& r, Build::Impl& b, std::vector& out) { // Read only active graph actions. Snapshot counts never complete an action; // the ninja log removes it from running only after the command has finished. -void read_action_progress(Build::Impl& build) { - if (build.progressDir.empty()) return; +bool read_action_progress(Build::Impl& build) { + if (build.progressDir.empty()) return false; + bool changed = false; for (auto& action : build.running) { auto snapshot = action_progress::read( action_progress::channel(build.progressDir, action.stamp)); - if (snapshot && (!action.snapshot || snapshot->seq > action.snapshot->seq)) + if (snapshot && (!action.snapshot || snapshot->seq > action.snapshot->seq)) { action.snapshot = std::move(snapshot); + changed = true; + } } + return changed; } // The builds whose Build object is alive: the frame, the poll and the status @@ -1178,6 +1182,7 @@ mcpp::ui::Frame frame() { void poll() { auto& r = report(); std::vector out; + bool progressChanged = false; { std::lock_guard lock(r.m); if (r.game && r.keys) @@ -1191,10 +1196,13 @@ void poll() { for (auto const& b : live_builds(r)) { read_starts(r, *b, out); read_log(r, *b, out); - read_action_progress(*b); + progressChanged |= read_action_progress(*b); } } write_lines(out); + // A snapshot can arrive between Ninja status lines. Wake the normal + // frame writer instead of waiting for its one-second clock redraw. + if (progressChanged) mcpp::ui::touch_region(); } // The status row's screen, as MCPP_PROGRESS asks: `random` (the default) or diff --git a/tests/unit/test_build_progress.cpp b/tests/unit/test_build_progress.cpp index 965f8c905..80cf11676 100644 --- a/tests/unit/test_build_progress.cpp +++ b/tests/unit/test_build_progress.cpp @@ -369,6 +369,40 @@ TEST(ProgressModel, AnActionStartLoadsTheFastPathRecordBeforeNinjaWritesStdout) // No Build::status() or completed ninja-log entry was supplied. } +TEST(ProgressModel, ANewSnapshotRedrawsBeforeTheClockHeartbeat) { + namespace telemetry = mcpp::build::action_progress; + Tmp tmp; + Record rec; + rec.packages = {{"progress-redraw", true, "r", 0, 1, "", "project"}}; + rec.owner = {{"gen/redraw.stamp", 0}}; + rec.actions = {{"gen/redraw.stamp", "a"}}; + Build build(tmp.path); + build.set_record(rec); + build.pass_begin(); + const auto env = build.environment(); + const auto root = std::ranges::find(env, telemetry::kRootEnv, &decltype(env)::value_type::first); + ASSERT_NE(root, env.end()); + const auto unixMs = std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()).count(); + append(tmp.path / kStartsFile, std::format("gen/redraw.stamp\t{}\n", unixMs)); + const bool wasLive = mcpp::ui::live_progress(); + mcpp::ui::set_live_progress(true); + testing::internal::CaptureStderr(); + open(false); + // Let the initial half-second display guard pass. This line then draws + // the frame immediately before publication, resetting the clock redraw. + std::this_thread::sleep_for(std::chrono::milliseconds(550)); + mcpp::ui::info("Probe", "before snapshot publication"); + std::ofstream(telemetry::channel(root->second, "gen/redraw.stamp")) + << R"({"v":1,"seq":1,"phase":"snapshot-redraw","done":1,"total":2})"; + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + close(); + const auto out = testing::internal::GetCapturedStderr(); + mcpp::ui::set_live_progress(wasLive); + EXPECT_NE(out.find("snapshot-redraw 1/2"), std::string::npos) << out; + // No Ninja stdout, completion or other narration requested a redraw. +} + TEST(ProgressModel, APackageIsNamedOnceWhenItsFirstStepFinishes) { mcpp::ui::disable_color(); Tmp tmp; From e530c80b9c7ef4a0c097797578c97a0417b9359f Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 5 Nov 2026 22:51:54 +0900 Subject: [PATCH 07/40] fix: exclude consumer link inputs from foreign-build contracts --- ...heterogeneous-build-implementation-plan.md | 36 +++++++++++++++++++ CHANGELOG.md | 3 ++ docs/30-build-mcpp.md | 2 +- docs/zh/30-build-mcpp.md | 2 +- src/build/compile_commands.cppm | 8 +++-- src/build/flags.cppm | 18 ++++++++-- ...982_an_action_reads_the_builds_contract.sh | 5 +++ 7 files changed, 67 insertions(+), 7 deletions(-) diff --git a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md index 6f3747eb6..16dcf4174 100644 --- a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md +++ b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md @@ -529,3 +529,39 @@ latest，实际执行输出 `mcpp 2026.11.5.1`；安装后的 ELF 与公开安 该补充继续使用 PR #889 与 2026.11.5.2。新的最终提交仍须完整 CI 以及其后 独立生态审查，通过后才合入和发布；CLion 两仓库的兼容性结果不受此门禁替代。 + +## 22. 外部构建契约的消费输入与参数边界（2026-11-05） + +插件 PR #52 的 CI 后审查复现了两项不同责任的问题。其 CMake 适配将 JSON argv +拼成原生 flags 字符串，只为带空格的词增加引号，因而丢失字面引号、反斜杠、 +空参数和 tab 的边界。插件改为生成目标 toolchain 与按内容命名的编译器 response +files，通过 CMake 的原生 options 交给各生成器转义；不向核心增加 CMake 原语。 + +另一项最小反例在实际 musl 跨构建中成立：同一 CMake 子工程既安装静态库，又 +构建一个独立可执行程序。核心契约将消费程序尚待安装的库包含在 link.args 中， +使该独立程序在子工程安装前就依赖这份库，链接因文件不存在而失败。修正后的 +契约复用同一目标 flags 派生，只排除消费包的链接标志和库；目标、sysroot、平台 +与 profile 的设置仍由原派生逻辑产生，消费程序自己的实际链接不作删减。 +此处的通用边界属于核心，argv 到 CMake 的表示属于插件。 + +局部验证只重编 flags 和 compile_commands 两个模块，分别约 4.1 s 与 5.7 s， +复用已有对象以及此前验证的进度模块。旧核心配合新插件仍在同一自依赖处失败； +修正核心后，静态库、独立子工程程序和最终 musl 消费程序均实际运行通过。 +扩展原 982 E2E，要求消费包的标志不进入外部契约、同时仍存在于消费程序的 +build.ninja；旧模块失败，新模块通过。该诊断二进制不替代正式版本构建。 + +插件的真实编译器检查覆盖 GCC、Clang 的 Ninja／Make、C／C++ 作用域、 +try_compile、字面引号、反斜杠、美元符号、分号、tab、空字符串、重复选项次序、 +不变参数不重编，以及参数变化后的增量失效。clang-cl 的 response 文件实际 +生成 COFF，并核对字面数据；这不能等同原生 MSVC 全生成器资格。 + +还确认两项边界。纯 CMake 4.4.2／Ninja 工程在源码路径含单引号时也拆错依赖 +路径；这一负向控制不含 mcpp、toolchain 或 response files，不归为新插件缺陷。 +测试仍在 response-file 路径保留单引号，源码路径覆盖空格、Unicode 和美元符号。 +旧 CMake cache 则不会因后来指定 toolchain 而开始加载它，新参数表示因此使用 +独立构建目录，避免沿用旧 CMAKE_*_FLAGS。两项结果分别记录，不能靠重试掩盖。 + +核心继续复用 #889 和 2026.11.5.2，插件继续复用 #52。核心最终完整 CI 及其后 +生态审查、发布和公开安装仍须先完成；随后对齐插件 CI 的公开核心 pin，再验证 +完整插件候选。现有 Windows E2E shard 在下载 artifact 时 ECONNRESET，尚未运行 +测试；不据此判为源代码回归，也不在仍有新源码待验证时原样重复整个 CI。 diff --git a/CHANGELOG.md b/CHANGELOG.md index afd519cfa..8a7754dca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,9 @@ command results and output checks remain authoritative. - New action snapshots request a normal UI frame, so short phases can appear between Ninja status lines without waiting for the clock redraw. +- Foreign-build contracts exclude the consuming package's link flags and + libraries. A subproject can link its own executable without depending on + an installation that the same subproject has yet to produce. - The macOS clean bootstrap keeps its stage-one compiler outside the output tree removed by the second stage. diff --git a/docs/30-build-mcpp.md b/docs/30-build-mcpp.md index 99f2cfbf5..bf197dccd 100644 --- a/docs/30-build-mcpp.md +++ b/docs/30-build-mcpp.md @@ -1097,7 +1097,7 @@ none to rely on), and the only interpolations are a closed set: | `${mcpp.out_dir}` | the build output directory | | `${mcpp.bin_dir}` | where produced binaries land: `bin/`, or, for an action a workspace member's build program declares, that member's product directory (2026.9.29.4+) | | `${mcpp.compile_db}` | path to `compile_commands.json` (what clang-tidy's `-p` wants) | -| `${mcpp.contract}` *(2026.10.30.1+)* | path to the configuration's contract, `mcpp-contract.json`, beside its compile database: the compiler of each language with the words every unit is compiled with (target, sysroot, target-side headers, platform and profile flags), and the link driver, its words and the archiver. A foreign build system reads it to compile for the target as the engine does; a package's own include directories are not in it, nor the engine's C++ dialect and module words (`-std=`, `-fmodules`), which are its choice for its own units | +| `${mcpp.contract}` *(2026.10.30.1+)* | path to the configuration's contract, `mcpp-contract.json`, beside its compile database: the compiler of each language with the words every unit is compiled with (target, sysroot, target-side headers, platform and profile flags), and the link driver, its words and the archiver. A foreign build system reads it to compile for the target as the engine does; a package's own include directories are not in it, nor the engine's C++ dialect and module words (`-std=`, `-fmodules`), which are its choice for its own units. Since 2026.11.5.2, link arguments describe the target configuration; the consuming package's link flags and libraries are excluded, since they can name outputs the foreign build has yet to produce | | `${mcpp.target_file:}` | the built file of target `` | | `${mcpp.stage_dir}` *(2026.9.11.1+)* | the tree `mcpp pack` staged, absolute. `artifact` role only, and only under `mcpp pack --pkg-format ` | | `${mcpp.self}` *(2026.9.13.1+)* | the engine's own executable, absolute. An action's command is an argv with no shell, so a build program has no portable way to copy a file; the engine is present wherever a build runs, and `${mcpp.self} stage --verify content --output ` copies one file, creates the destination's parent, and writes only when the bytes differ. That argument shape is a contract from 2026.9.13.1 on; a build program that names it declares that release as its floor | diff --git a/docs/zh/30-build-mcpp.md b/docs/zh/30-build-mcpp.md index 52707fa39..c128ec11f 100644 --- a/docs/zh/30-build-mcpp.md +++ b/docs/zh/30-build-mcpp.md @@ -933,7 +933,7 @@ mcpp 会写出 `<暂存树>.stage-manifest` —— 一个兄弟文件，永不 | `${mcpp.out_dir}` | 构建输出目录 | | `${mcpp.bin_dir}` | 产出的二进制所在目录:`bin/`;工作区成员的构建程序声明的 action 中为该成员的产物目录(2026.9.29.4+) | | `${mcpp.compile_db}` | `compile_commands.json` 的路径（clang-tidy 的 `-p` 要的就是它） | -| `${mcpp.contract}` *(2026.10.30.1+)* | 配置的契约 `mcpp-contract.json` 的路径，与该配置的编译数据库同目录：每种语言的编译器与每个单元都带的词（目标、sysroot、目标侧的头文件、平台与 profile 的标志），以及链接驱动、它的词与归档工具。外部构建系统读它，以引擎的方式为目标编译；包自己的包含目录不在其中，引擎为自己的单元选择的 C++ 方言与模块词（`-std=`、`-fmodules`）也不在其中 | +| `${mcpp.contract}` *(2026.10.30.1+)* | 配置的契约 `mcpp-contract.json` 的路径，与该配置的编译数据库同目录：每种语言的编译器与每个单元都带的词（目标、sysroot、目标侧的头文件、平台与 profile 的标志），以及链接驱动、它的词与归档工具。外部构建系统读它，以引擎的方式为目标编译；包自己的包含目录不在其中，引擎为自己的单元选择的 C++ 方言与模块词（`-std=`、`-fmodules`）也不在其中。自 2026.11.5.2 起，链接参数描述目标配置，不含消费包自己的链接标志和库；这些库可能正是外部构建尚待生成的产物 | | `${mcpp.target_file:}` | target `` 构建出的文件 | | `${mcpp.stage_dir}` *(2026.9.11.1+)* | `mcpp pack` 暂存出的那棵树，绝对路径。仅 `artifact` role 可用，且仅在 `mcpp pack --pkg-format ` 下可用 | | `${mcpp.self}` *(2026.9.13.1+)* | 引擎自己的可执行文件，绝对路径。action 的命令是没有 shell 的 argv，构建程序因此没有可移植的拷贝手段；引擎在构建运行的每台机器上都在，`${mcpp.self} stage --verify content --output ` 拷贝一个文件、创建目标的父目录、只在字节不同时写入。这个参数形状自 2026.9.13.1 起是契约；写下它的构建程序即以该版本为下限 | diff --git a/src/build/compile_commands.cppm b/src/build/compile_commands.cppm index d256bc0cb..bdcd37555 100644 --- a/src/build/compile_commands.cppm +++ b/src/build/compile_commands.cppm @@ -204,7 +204,7 @@ write_compile_commands(const BuildPlan& plan, const CompileFlags& flags); // calls mcpp back. A tool for the host is built in the host graph, whose // configuration has its own contract. // -// {"schema": 1, "target": "", "dialect": "", +// {"schema": 1, "target": "", "dialect": "", // "compile": {"c": {"program", "args"}, "cxx": {...}, "asm": {...}}, // "link": {"program", "args", "archiver"}} std::filesystem::path contract_path(const BuildPlan& plan); @@ -901,8 +901,10 @@ std::expected write_build_contract(const BuildPlan& plan, con doc["compile"]["asm"] = tool(flags.ccBinary, flags.as, true); target_side(doc["compile"]["asm"], plan.contractTargetSide.cflags, true); for (auto const& w : plan.contractTargetSide.asmflags) doc["compile"]["asm"]["args"].push_back(w); - auto link = tool(flags.ldDriver.empty() ? flags.cxxBinary : flags.ldDriver, flags.ld, false); - link["archiver"] = flags.arBinary.generic_string(); + const auto contractFlags = compute_build_contract_flags(plan); + auto link = tool(contractFlags.ldDriver.empty() ? contractFlags.cxxBinary : contractFlags.ldDriver, + contractFlags.ld, false); + link["archiver"] = contractFlags.arBinary.generic_string(); doc["link"] = std::move(link); const auto path = contract_path(plan); std::error_code ec; diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 9f3fa39c5..dd4729a0c 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -188,6 +188,9 @@ std::string render_link_intent_flags( LinkIntentFlavor flavor); CompileFlags compute_flags(const BuildPlan& plan); +// A foreign build inherits target configuration, not the consuming +// package's link inputs (which may be outputs of that foreign build). +CompileFlags compute_build_contract_flags(const BuildPlan& plan); // The object format the TARGET produces (#647 E3). One derivation, read by // `compute_flags` for the runtime contract table and the link-line shape, and @@ -707,7 +710,7 @@ std::string program_msvc_crt_linkage(const mcpp::manifest::Manifest& m, ? "static" : "dynamic"; } -CompileFlags compute_flags(const BuildPlan& plan) { +CompileFlags compute_flags_impl(const BuildPlan& plan, bool packageLinkFlags) { CompileFlags f; // Central query points for per-toolchain decisions — prefer these over @@ -1289,7 +1292,10 @@ CompileFlags compute_flags(const BuildPlan& plan) { // `__cxa_throw` and the personality first, and libc++abi answered the rest // (mcpp#838, measured: `catch (const std::exception&)` missed). std::string user_ldflags; - for (auto const& word : mcpp::manifest::flag_words(plan.manifest.buildConfig.ldflags)) { + const auto linkWords = packageLinkFlags + ? mcpp::manifest::flag_words(plan.manifest.buildConfig.ldflags) + : std::vector{}; + for (auto const& word : linkWords) { const auto rendered = ninja_command_word(normalize_ldflag(plan.projectRoot, word)); if (plan.cxxIsland && std::ranges::find(plan.cxxIsland->namedWords, word) != plan.cxxIsland->namedWords.end()) { @@ -2637,4 +2643,12 @@ CompileFlags compute_flags(const BuildPlan& plan) { return f; } +CompileFlags compute_flags(const BuildPlan& plan) { + return compute_flags_impl(plan, true); +} + +CompileFlags compute_build_contract_flags(const BuildPlan& plan) { + return compute_flags_impl(plan, false); +} + } // namespace mcpp::build diff --git a/tests/e2e/982_an_action_reads_the_builds_contract.sh b/tests/e2e/982_an_action_reads_the_builds_contract.sh index 145157b2a..001b4a4a5 100755 --- a/tests/e2e/982_an_action_reads_the_builds_contract.sh +++ b/tests/e2e/982_an_action_reads_the_builds_contract.sh @@ -21,6 +21,7 @@ cat > build.mcpp <<'EOF' #include import mcpp; int main() { + mcpp::link_flag("-Wl,--defsym,mcpp_contract_consumer=1"); const std::string out = std::string(mcpp::out_dir()) + "/seen-contract.json"; mcpp::action a; a.id = "copy-contract"; @@ -47,8 +48,12 @@ assert any(a.startswith("-O") for a in cxx), cxx assert not any(a.startswith("-std=") or a == "-fmodules" or a.startswith("-fmodule-") for a in cxx), cxx assert os.path.isabs(d["link"]["program"]) and isinstance(d["link"]["args"], list), d["link"] assert "archiver" in d["link"], d["link"] +assert not any("mcpp_contract_consumer" in a for a in d["link"]["args"]), d["link"] print("contract:", d["target"], d["dialect"], d["compile"]["cxx"]["program"]) PY cfg=$(dirname "$(find target -name compile_commands.json -not -path '*/.build-mcpp/*' | head -1)") [[ -f "$cfg/mcpp-contract.json" ]] || { echo "FAIL: no contract beside the compile database"; exit 1; } +grep -q 'mcpp_contract_consumer=1' "$cfg/build.ninja" || { + echo "FAIL: excluding package link inputs from the contract removed them from the consumer"; exit 1; +} echo "PASS: 982" From 202aa9fe8767effdbf8f56bad0479032016a607c Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 02:03:16 +0900 Subject: [PATCH 08/40] fix: bound split compilation and track active download progress --- ...i-publication-and-compiler-cap-closeout.md | 132 +++++++++++++ .agents/docs/README.md | 7 +- CHANGELOG.md | 9 + docs/04-mcpp-toml.md | 4 + docs/25-home-indices-and-store.md | 8 + docs/zh/04-mcpp-toml.md | 3 + docs/zh/25-home-indices-and-store.md | 5 + src/build/ninja_backend.cppm | 54 +++--- src/build/schedule/detach_codegen.cppm | 79 ++++++-- src/cli.cppm | 25 +++ src/provider/xlings/xlings.cpp | 24 ++- src/provider/xlings/xlings.cppm | 31 ++- tests/e2e/234_bmi_schedule_on.sh | 45 ++++- .../e2e/helpers/check_shared_compiler_cap.py | 177 ++++++++++++++++++ tests/unit/test_detach_status.cpp | 15 ++ tests/unit/test_network.cpp | 41 ++++ tests/unit/test_ninja_backend.cpp | 25 +++ 17 files changed, 633 insertions(+), 51 deletions(-) create mode 100644 .agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md create mode 100644 tests/e2e/helpers/check_shared_compiler_cap.py diff --git a/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md new file mode 100644 index 000000000..98266e4e7 --- /dev/null +++ b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md @@ -0,0 +1,132 @@ +--- +subject: build-progress-and-scheduling +status: active +--- + +# BMI publication, compiler capacity and active download budgets + +Audience: maintainers closing the 2026.11.5.2 candidate and reviewing PR #889. +This record separates three failures exposed during candidate validation. +It does not qualify a diagnostic object substitution as a released binary. + +## Failure boundaries + +| Failure | Confirmed cause | Owner | +|---|---|---| +| An importer cannot find a BMI after its interface is touched | GCC unlinks the old final path before renaming the new BMI; an absent identity was accepted as publication | Core detached compilation | +| A build exceeds its requested compiler count | Extra Ninja slots serve publication and join edges, while ordinary compilers bypass the detached semaphore | Core Ninja scheduling | +| A newly activated download is immediately considered stalled | Its queued state was observed earlier; starting without a source attempt or byte change retained the queue-entry timestamp | Core provider progress tracking | + +No compiler toolchain payload, plugin semantic or timeout constant is changed +by these corrections. + +## BMI publication + +GCC 16.1 tracing shows an `unlink(final)` immediately before +`rename(temporary, final)`. The observed interval was approximately 26 +microseconds. A changed identity alone is therefore insufficient: publication +requires a present file with a changed identity. The adapter still preserves +the previous BMI for equivalence checking and failure recovery. + +The causal control amplifies only that existing unlink interval to 200 +milliseconds. The previous core reproduces the candidate failure: the middle +module cannot read the core module. Replacing only the detached-compilation +implementation makes the same real GCC workload succeed. A separate +deterministic publisher confirms the boundary: the old phase returns zero +while the BMI is absent; the corrected phase returns only after publication. + +An exit status of zero without the required BMI is diagnosed as a failure. +The previous BMI is restored. E2E 234 retains its cold execution, graph-shape, +incremental build and stale-token assertions, and adds both publication cases. + +## Shared compiler permits + +The candidate generated `sched_cap = 8`, while its scheduling policy allows +48 Ninja slots. The BMI supervisor used that semaphore; ordinary C/C++ +compilation and compiler-driven scanning did not. A full local candidate +build exceeded eight compiler processes and encountered memory-pressure +kills. Its interrupted output is not build qualification. + +The selected correction retains one actual compiler budget. Under the +detached strategy, ordinary built-in compilation and scanning use an internal +argv executor holding the same permit until the real child exits. BMI +supervisors continue to hold their permit across early publication and code +generation. Join edges acquire no permit. A holder never waits for another +permit, so waiting joins cannot form a permit dependency cycle. + +The internal executor introduces no user mode and no argument serialization. +It receives the compiler's actual argv, inherits the existing streams and +returns its status. Existing response files retain their original owner and +encoding. Ordinary completion and interrupt cleanup release a permit once; +release cannot remove a permit already acquired by another compiler. +Failure to create or acquire the semaphore is reported rather than followed +by an unbounded compile. A failed supervisor start releases its permit. + +The mixed control uses a real generated Ninja graph and GCC 16.1. A driver +records active invocations and remains alive briefly after BMI publication, +amplifying overlap with ordinary C and C++ units. With a requested cap of +one, the previous graph reaches six active drivers. Corrected controls use +caps of one and two, retain the extra Ninja slots, and observe peaks of +exactly one and two. They also check no-op builds, ordinary compiler failure, +a module failure after BMI publication, subsequent recovery, ordinary +compiler cancellation, child termination and permit cleanup. E2E 234 retains +the full-set stale-token reclamation check. + +These local controls use selected recompiled objects and an extracted copy of +the production dispatch branch. They establish causality and small-load +behavior. The complete current sources still require the ordinary CI build +and all required checks before release. + +## Download activation and deadline reasons + +The pinned xlings 2026.10.26.1 provider reports queued files in each batch +with `started = false`. Once a worker obtains a slot, `started` becomes true +before a source attempt or byte count necessarily changes. That transition +starts the active budget. Waiting in the queue consumes no active budget; +repeated samples of an already active file do not renew it. A byte increase +or source change retains its existing renewal semantics. + +An isolated provider invocation retained raw NDJSON from one download stream. +Replaying the same 347 snapshots through the actual C++ provider modules gives: + +| Module | Expired snapshots | First expiration | +|---|---:|---:| +| Previous state machine | 11 | 95.751 seconds | +| Corrected activation transition | 0 | none | + +The first false expiration belongs to a queued entry activated at 95.751 +seconds with source attempt zero. Its inherited age was 95.651 seconds. +With queue time excluded, the maximum active idle interval at the observed +snapshots is 7.735 seconds. The isolated network invocation reached its +150-second observer budget; it was stopped by the experiment, not by a +provider progress deadline, and does not establish a completed graphics run. + +The failed candidate CI log contains rendered status rather than raw NDJSON. +Its timing is consistent with activation of the final queued entries. It +does not identify which raw file state first expired, and is not presented as +proof of a particular file's network behavior. Progress on an independent +download remains insufficient to renew a genuinely stalled file. + +The provider source was followed through its HTTP implementation: the +connect and idle environment budgets are consumed below the downloader +wrapper. They remain eight and thirty seconds, respectively; the parent's +per-file budget remains forty-five seconds. No budget is increased to hide +the state error. Progress timeouts now identify the file, interval and source +attempt; process deadlines remain distinct. Time-injected unit cases cover +queue waits longer than forty-five seconds, genuine per-file stalls, source +fallback and completion. + +## Qualification sequence + +The local controls precede the complete PR CI. The graphics job is accepted +only after its shader, rendered-image and pixel assertions actually run. +After every required check succeeds, an independent ecosystem review +precedes merge and release. Published tags remain unchanged. + +Plugin PR #52 has independently passed all six jobs against the published +2026.11.5.1 core. Its final public core pin and real subproject-sidecar fixture +follow ordinary installation of the new release, in the same plugin PR. + +Public records contain synthetic assertions and aggregate evidence. Raw +provider streams, user logs, usernames and private paths remain outside the +repository and pull-request description. diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 6f97f7255..977f265a6 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -349 records. +350 records. ## By subject @@ -34,6 +34,10 @@ Records that declare one. Everything else is listed by date below. - [构建依赖的一条管线：配置与产物两轴、包的实例、库接口、兼容系统与工作空间收尾（#797）](2026-10-11-build-dependencies-one-pipeline-design.md) — active - [Four upstream asks from a UI framework: what each one is under mcpp's design, and the combined plan](2026-09-13-four-upstream-asks-from-a-ui-framework.md) — landed +### build-progress-and-scheduling + +- [BMI publication, compiler capacity and active download budgets](2026-11-06-bmi-publication-and-compiler-cap-closeout.md) — active + ### design - [升级的连续性：打包修订不在使用路径上替换载荷、替换失败不拆掉可用状态、home 中的悬空记录自愈、真实的升级 lane](2026-10-17-upgrade-continuity-design.md) — active @@ -167,6 +171,7 @@ Records that declare one. Everything else is listed by date below. ### 2026-11 +- [BMI publication, compiler capacity and active download budgets](2026-11-06-bmi-publication-and-compiler-cap-closeout.md) — active - [异构构建语义与插件边界补充方案（mcpp#885、mcpp#887）](2026-11-05-heterogeneous-build-semantics-and-plugin-boundaries-supplement.md) — active - [异构构建补充方案的第二轮调研与实验](2026-11-05-heterogeneous-build-research-and-experiments.md) — active - [生态 885 与异构构建补充方案的实现依赖](2026-11-05-heterogeneous-build-implementation-plan.md) — active diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a7754dca..14a1728f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,15 @@ an installation that the same subproject has yet to produce. - The macOS clean bootstrap keeps its stage-one compiler outside the output tree removed by the second stage. +- GCC BMI publication requires a present replacement file; its unlink-before-rename + interval no longer releases importers. A successful compiler that produces no + BMI fails with a diagnostic and restores the previous BMI. +- Ordinary compiler invocations share the detached GCC compilers' concurrency + permits, so the additional Ninja scheduling slots do not exceed `--jobs`. +- Queued downloads begin their progress budget when a worker starts them, + rather than inheriting time spent waiting for a slot. +- Provider progress timeouts identify the stalled download and source attempt, + independently of progress on other files and provider process deadlines. ### Changed diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index 18be4d992..b87545a07 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1048,6 +1048,10 @@ with two behaviours has to be. `bmi_schedule` decides when importers are unblocked. +With GCC and `bmi_schedule = "on"`, ordinary compilation and detached code +generation share the same total `jobs` limit (2026.11.5.2+). Additional +scheduling and join tasks do not increase the number of compiler processes. + | value | | |---|---| | `"auto"` | **the default, and it currently means OFF** | diff --git a/docs/25-home-indices-and-store.md b/docs/25-home-indices-and-store.md index efcb51133..998d25276 100644 --- a/docs/25-home-indices-and-store.md +++ b/docs/25-home-indices-and-store.md @@ -87,6 +87,14 @@ rules are in [SPEC-010](specs/home-initialization.md). without installing tools. `self config --mirror auto|CN|GLOBAL` updates only local configuration, so it remains usable while the network is broken. +An install tracks byte progress and source changes separately for each active +download. Time spent waiting for a download slot does not consume that budget. +Progress on another file does not renew a stalled file's budget. +From 2026.11.5.2, a stalled-download diagnostic names the file, elapsed interval +and source attempt. Provider preparation and process timeouts are reported +separately. Mirror selection +uses `mcpp self config --mirror auto`, `CN` or `GLOBAL`. + ### Retention of build outputs (2026.10.31.3+) The build cache under `build-cache/v1/` is shared by every project of the home; diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 6ca9b8f09..d4020d43c 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -972,6 +972,9 @@ bmi_schedule = "off" # auto (default, = off) | on | off `bmi_schedule` 决定 importer 何时被解除阻塞。 +使用 GCC 且 `bmi_schedule = "on"` 时，普通编译与分离的代码生成共用同一个 `jobs` 总上限 +（2026.11.5.2+）。额外的调度和等待任务不增加编译器进程数量。 + | 取值 | | |---|---| | `"auto"` | **默认值，目前意味着关闭** | diff --git a/docs/zh/25-home-indices-and-store.md b/docs/zh/25-home-indices-and-store.md index 6e1a8556d..cb8e25aeb 100644 --- a/docs/zh/25-home-indices-and-store.md +++ b/docs/zh/25-home-indices-and-store.md @@ -71,6 +71,11 @@ Windows 首次显式 `self init` 时，若 D 是可写的本地固定磁盘， `self env`、`self config`、`self doctor` 和 `toolchain list` 读取本地状态，不安装工具。 `self config --mirror auto|CN|GLOBAL` 仅更新本地配置，因此网络故障时仍可使用。 +安装过程分别跟踪每个活动下载的字节进度与来源切换，等待下载槽位的时间不消耗该预算。 +其他文件取得进度不会延长停滞文件的预算。 +从 2026.11.5.2 起，下载停滞诊断会标明文件、停滞时长与来源尝试次数；provider 准备阶段与进程超时单独报告。 +镜像选择使用 `mcpp self config --mirror auto`、`CN` 或 `GLOBAL`。 + ### 构建产物的保留期（2026.10.31.3+） `build-cache/v1/` 下的构建缓存由 home 中所有工程共享；工程的 `target/` 属于工程自身。两者各有保留期， diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 5b975d105..75fca0ba6 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -1466,6 +1466,12 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // dyndep is a precondition: without it nothing declares BMIs as outputs, so // there is no BMI edge for importers to depend on. const bool splitBmi = plan.scheduleTag == "detach-codegen" && dyndep; + // Ninja has extra slots for BMI publication and join edges. Ordinary + // compilers must share the supervisors' semaphore rather than use that + // larger slot count as their own concurrency bound. + const std::string compilerPrefix = splitBmi + ? std::format("$mcpp __compiler-cap --sem .mcpp-sched --cap {} -- ", plan.scheduleCompilerCap) + : std::string{}; // The other split shape. Clang publishes no BMI early — it writes the BMI // at the end of the compile — so the GCC trick of releasing the file // mid-compile has nothing to release. What clang has instead is a driver @@ -2057,9 +2063,9 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, if constexpr (mcpplibs::os::is_windows) { // Windows: skip BMI restat optimization (requires POSIX shell). const std::string payload = " $local_includes"; - append(std::format(" command = {}$cxx{} $cxxflags $unit_cxxflags" + append(std::format(" command = {}{}$cxx{} $cxxflags $unit_cxxflags" " $module_output $module_lang {}{}\n", - win_depfile_filter, rsp_ref(payload), mmd_flag, + win_depfile_filter, compilerPrefix, rsp_ref(payload), mmd_flag, compile_tail)); append_rspfile(payload); append_cxx_deps(); @@ -2071,7 +2077,7 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // `-x c++` / `-x c++-module` is POSITIONAL on GNU drivers: it // must precede `-c $in` (which compile_tail carries) or it // applies to nothing. - "$cxx $local_includes $cxxflags $unit_cxxflags" + "{}$cxx $local_includes $cxxflags $unit_cxxflags" " $module_output $module_lang {}{}{} && " // `$mcpp bmi-equal`, not `cmp -s`: GCC stamps a wall clock into // the BMI content, so a byte compare NEVER reports "unchanged" @@ -2083,7 +2089,7 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, "mv \"$bmi_out.bak\" \"$bmi_out\"; " "else " "rm -f \"$bmi_out.bak\"; " - "fi\n", mmd_flag, compile_tail, mmd_filter)); + "fi\n", compilerPrefix, mmd_flag, compile_tail, mmd_filter)); append_cxx_deps(); } append(" description = MOD $out\n"); @@ -2202,14 +2208,14 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, append("rule cxx_object\n"); if constexpr (mcpplibs::os::is_windows) { const std::string payload = " $local_includes"; - append(std::format(" command = {}$cxx{} $cxxflags $unit_cxxflags {}{}\n", - win_depfile_filter, rsp_ref(payload), mmd_flag, + append(std::format(" command = {}{}$cxx{} $cxxflags $unit_cxxflags {}{}\n", + win_depfile_filter, compilerPrefix, rsp_ref(payload), mmd_flag, compile_tail)); append_rspfile(payload); } else { append(std::format( - " command = $cxx $local_includes $cxxflags $unit_cxxflags {}{}{}\n", - mmd_flag, compile_tail, mmd_filter)); + " command = {}$cxx $local_includes $cxxflags $unit_cxxflags {}{}{}\n", + compilerPrefix, mmd_flag, compile_tail, mmd_filter)); } append(" description = OBJ $out\n"); append_cxx_deps(); @@ -2220,8 +2226,8 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, if (need_c_rule) { append("rule c_object\n"); const std::string payload = " $local_includes"; - append(std::format(" command = $cc{} $cflags $unit_cflags {}{}\n", - rsp_ref(payload), c_mmd_flag, compile_tail)); + append(std::format(" command = {}$cc{} $cflags $unit_cflags {}{}\n", + compilerPrefix, rsp_ref(payload), c_mmd_flag, compile_tail)); append_rspfile(payload); append(" description = CC $out\n"); append_cxx_deps(); @@ -2241,16 +2247,16 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // absent depfile as an error. So `.s` keeps the pre-#257 shape. const std::string payload = " $local_includes"; append("rule asm_object\n"); // .S — preprocessed, tracks #include - append(std::format(" command = $cc{} $asmflags $unit_asmflags {}{}\n", - rsp_ref(payload), c_mmd_flag, compile_tail)); + append(std::format(" command = {}$cc{} $asmflags $unit_asmflags {}{}\n", + compilerPrefix, rsp_ref(payload), c_mmd_flag, compile_tail)); append_rspfile(payload); append(" description = AS $out\n"); append_cxx_deps(); append("\n"); append("rule asm_object_raw\n"); // .s — not preprocessed, no depfile - append(std::format(" command = $cc{} $asmflags $unit_asmflags {}\n", - rsp_ref(payload), compile_tail)); + append(std::format(" command = {}$cc{} $asmflags $unit_asmflags {}\n", + compilerPrefix, rsp_ref(payload), compile_tail)); append_rspfile(payload); append(" description = AS $out\n\n"); } @@ -2259,8 +2265,8 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // NASM (.asm): its own fixed flag spelling — deliberately outside // CommandDialect. -MD/-MQ feed ninja's header tracking for %include. append("rule nasm_object\n"); - append(" command = $nasm -f $nasmfmt $local_includes $nasmflags " - "$unit_asmflags -MD $out.d -MQ $out -o $out $in\n"); + append(std::format(" command = {}$nasm -f $nasmfmt $local_includes $nasmflags " + "$unit_asmflags -MD $out.d -MQ $out -o $out $in\n", compilerPrefix)); append(" deps = gcc\n"); append(" depfile = $out.d\n"); append(" description = NASM $out\n\n"); @@ -2278,9 +2284,9 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // [resources].extra-inputs, and lands on this edge as implicit inputs. append("rule rc_object\n"); if (plan.rcStyle == "msvc") - append(" command = $rc /nologo $rcflags /fo $out $in\n"); + append(std::format(" command = {}$rc /nologo $rcflags /fo $out $in\n", compilerPrefix)); else - append(" command = $rc -O coff $rcflags -o $out $in\n"); + append(std::format(" command = {}$rc -O coff $rcflags -o $out $in\n", compilerPrefix)); append(" description = RC $out\n\n"); } @@ -2562,20 +2568,20 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, if (msvcDeps) { // MSVC: compiler-integrated P1689 via /scanDependencies (scan // only — no codegen); /TP because our module units are .cppm. - append(std::format(" command = $cxx_driver{} $cxxflags $unit_cxxflags " + append(std::format(" command = {}$cxx_driver{} $cxxflags $unit_cxxflags " "/scanDependencies $out /TP /c $in /Fo:$compile_target\n", - rsp_ref(scanPayload))); + compilerPrefix, rsp_ref(scanPayload))); // No $unit_lang here: /TP above already applies to every input, // and cl has no per-suffix recognition problem to fix — a module // interface is identified by /interface at COMPILE time, which // the scan does not do. } else if (plan.scanDepsPath.empty()) { // GCC path: compiler-integrated P1689 scanning. - append(std::format(" command = $cxx_driver{} $cxxflags $unit_cxxflags -fmodules " + append(std::format(" command = {}$cxx_driver{} $cxxflags $unit_cxxflags -fmodules " "-fdeps-format=p1689r5 " "-fdeps-file=$out -fdeps-target=$deps_target " "-M -MM -MF $out.dep $unit_lang -E $in -o $compile_target\n", - rsp_ref(scanPayload))); + compilerPrefix, rsp_ref(scanPayload))); } else { // Clang path: clang-scan-deps writes the P1689 JSON itself via -o // (LLVM 17+), like the GCC and MSVC branches above write theirs @@ -2585,10 +2591,10 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // CreateProcess — which any package with a large include-dir list // overruns (#261: 48 -I entries at a deep consumer path). append(std::format( - " command = $scan_deps -format=p1689 -o $out -- " + " command = {}$scan_deps -format=p1689 -o $out -- " "$cxx_driver{} $cxxflags $unit_cxxflags $unit_lang -c $in " "-o $compile_target\n", - rsp_ref(scanPayload))); + compilerPrefix, rsp_ref(scanPayload))); } append_rspfile(scanPayload); append(" description = SCAN $out\n\n"); diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 6ec0e2fed..880f9d787 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -26,9 +26,12 @@ // BMI to `.gcm~` and rename()s it into place — verified with strace: // // openat("gcm.cache/x.gcm~", O_RDWR|O_CREAT|O_TRUNC) = 5 +// unlink("gcm.cache/x.gcm") = 0 // rename("gcm.cache/x.gcm~", "gcm.cache/x.gcm") = 0 // -// so the final path is atomically complete-or-absent. Confirmed three further +// GCC unlinks the previous final path immediately before that rename, so the +// path can briefly be absent. Only a PRESENT changed identity is publication. +// Confirmed three further // ways: the early snapshot is byte-identical to the finished file, and a real // downstream importer compiles against it and exits 0. Clang, by contrast, // writes the BMI straight to the final path with O_TRUNC — which is exactly why @@ -136,6 +139,11 @@ int supervise(const std::filesystem::path& slot, // must not pass on. int await_unit(const std::filesystem::path& slot, const std::filesystem::path& object); +// Ordinary compiler invocations share the detached compilers' actual bound. +// Their original argv and streams are retained; no shell is introduced. +int run_capped(const std::vector& argv, + const std::filesystem::path& semaphore, int maxCompilers); + } // namespace mcpp::build::schedule::detach // --------------------------------------------------------------------------- @@ -195,21 +203,25 @@ std::filesystem::path acquire_token(const std::filesystem::path& dir, int cap) { if (dir.empty() || cap <= 0) return {}; std::error_code ec; std::filesystem::create_directories(dir, ec); + if (ec) { + std::println(std::cerr, "mcpp: cannot create compiler semaphore {}: {}", dir.string(), ec.message()); + return {}; + } const auto started = std::chrono::steady_clock::now(); constexpr auto kLimit = std::chrono::hours(2); for (;;) { for (int i = 0; i < cap; ++i) { const auto tok = dir / std::to_string(i); if (std::filesystem::create_directory(tok, ec) && !ec) return tok; + if (ec) { + std::println(std::cerr, "mcpp: cannot acquire compiler token {}: {}", tok.string(), ec.message()); + return {}; + } } if (std::chrono::steady_clock::now() - started > kLimit) { - // Proceeds WITHOUT the cap rather than failing: the semaphore bounds - // memory pressure, it is not a correctness property, so an unbounded - // compile is a worse build and a failed one is no build at all. std::println(std::cerr, "mcpp: no compiler slot became free in 2h — all {} tokens in {} " - "are held by processes that are gone. Continuing without the " - "concurrency cap; remove that directory to restore it.", + "are still occupied; refusing to exceed the concurrency cap.", cap, dir.string()); return {}; } @@ -268,8 +280,9 @@ void copy_first_rule(const std::filesystem::path& from, const std::filesystem::p // // reproducible at `-j1`, on four of six scenarios of the generated fixture. // -// The previous BMI is now COPIED aside instead, so the file is continuously -// readable and GCC's own atomic rename is what replaces it. Publication is +// The previous BMI is now COPIED aside instead, so this adapter does not +// introduce a missing-file interval. GCC itself unlinks the old path before +// its rename; that interval must not release importers. Publication is // detected by the identity below changing, which is exactly the question the // existence check was a poor proxy for. struct BmiIdentity { @@ -403,6 +416,7 @@ int compile_release_at_bmi(const CompileRequest& req) { if (!req.bmi.empty()) std::ofstream(suffixed(req.slot, ".bmi"), std::ios::trunc) << req.bmi.string(); const auto token = acquire_token(req.semaphore, req.maxCompilers); + if (!req.semaphore.empty() && req.maxCompilers > 0 && token.empty()) return 1; // The supervisor reads the SAME argv file rather than receiving the command // on its own command line: one representation, no re-quoting, and no limit @@ -411,7 +425,11 @@ int compile_release_at_bmi(const CompileRequest& req) { "--slot", req.slot.string(), "--command-file", req.commandFile.string()}; if (!token.empty()) { sup.push_back("--token"); sup.push_back(token.string()); } - if (!spawn_detached(sup)) return 2; + if (!spawn_detached(sup)) { + std::filesystem::remove(token, ec); + restore_backup(req.bmi); + return 2; + } // BOUNDED, for the same reason `await_unit` is — and this loop waits on the // SAME supervisor. The spawn succeeding only says a process was created; if @@ -429,9 +447,11 @@ int compile_release_at_bmi(const CompileRequest& req) { constexpr auto kHardLimit = std::chrono::hours(2); for (;;) { - // Published = the file's identity is no longer the one we snapshotted. - // For a unit with no previous BMI that reduces to "it now exists". - if (!req.bmi.empty() && bmi_identity(req.bmi) != before) { + // GCC unlinks an existing BMI immediately before renaming its new + // temporary file. Absence is a transition, never publication: treating + // it as a changed identity releases importers inside that unlink gap. + const auto published = bmi_identity(req.bmi); + if (published.present && published != before) { settle_bmi(req.bmi); copy_first_rule(req.depFrom, req.depTo, req.bmiTarget.empty() ? req.bmi.string() : req.bmiTarget); @@ -445,6 +465,12 @@ int compile_release_at_bmi(const CompileRequest& req) { // it back rather than leaving the unit with no BMI at all. restore_backup(req.bmi); } else { + if (!req.bmi.empty() && !file_exists(req.bmi)) { + std::println(std::cerr, "mcpp: compiler reported success but {} is missing", + req.bmi.string()); + restore_backup(req.bmi); + return 1; + } // THE COMPILER CAN FINISH BETWEEN THE TWO CHECKS ABOVE. // // The loop tests `file_exists(bmi)` first and `rc` second, so a @@ -578,4 +604,33 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o return 0; } +int run_capped(const std::vector& argv, + const std::filesystem::path& semaphore, int maxCompilers) { + if (argv.empty()) return 2; + struct Lease { + std::filesystem::path token; + // Completion and interrupt cleanup may race. A second release must + // never remove a permit that another compiler has already acquired. + std::atomic released{false}; + void release() { + if (released.exchange(true) || token.empty()) return; + std::error_code ec; + std::filesystem::remove(token, ec); + } + ~Lease() { release(); } + }; + auto lease = std::make_shared(); + lease->token = acquire_token(semaphore, maxCompilers); + if (!semaphore.empty() && maxCompilers > 0 && lease->token.empty()) return 1; + const int cleanup = mcpp::run::add_cleanup([lease] { lease->release(); }); + struct Cleanup { + int handle; + std::shared_ptr lease; + ~Cleanup() { mcpp::run::remove_cleanup(handle); lease->release(); } + } guard{cleanup, lease}; + return mcpp::run::execute(argv, { + .input = mcpp::run::Input::inherit, + .streams = mcpp::run::Streams::inherit}).exit_code; +} + } // namespace mcpp::build::schedule::detach diff --git a/src/cli.cppm b/src/cli.cppm index 157a50c59..75917d4e5 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -18,6 +18,7 @@ export module mcpp.cli; import std; import mcpp.build.depfile; +import mcpp.build.schedule.detach_codegen; import mcpplibs.cmdline; import mcpp.cli.cmd_build; import mcpp.cli.cmd_cache; @@ -1492,6 +1493,30 @@ int run(int argc, char** argv) { // check and no `prepare`. On the MSVC ABI the first build after upgrading // to 2026.9.28.2 re-runs each action once, because its command line gained // the directory. + // Internal split-schedule executor. Everything after `--` is the + // compiler's argv, including flags with the same names as these options. + if (std::string_view(argv[1]) == "__compiler-cap") { + std::filesystem::path semaphore; + int cap = 0, i = 2; + for (; i < argc && std::string_view(argv[i]) != "--"; ++i) { + const std::string_view option = argv[i]; + if ((option != "--sem" && option != "--cap") || i + 1 >= argc) { + std::println(stderr, "error: __compiler-cap: expected --sem --cap -- ..."); + return 2; + } + const std::string_view value = argv[++i]; + if (option == "--sem") semaphore = value; + else { + const auto [end, ec] = std::from_chars(value.data(), value.data() + value.size(), cap); + if (ec != std::errc{} || end != value.data() + value.size()) return 2; + } + } + if (semaphore.empty() || cap <= 0 || i + 1 >= argc) return 2; + std::vector command; + for (++i; i < argc; ++i) command.emplace_back(argv[i]); + return mcpp::build::schedule::detach::run_capped(command, semaphore, cap); + } + // `mcpp depfile-filter --raw --out -- ...` // (internal: written into build.ninja for GCC on a Windows host, WS2 of // the 2026-09-28 design). Runs the compile with inherited stdio and no diff --git a/src/provider/xlings/xlings.cpp b/src/provider/xlings/xlings.cpp index b12386882..2c6b8706f 100644 --- a/src/provider/xlings/xlings.cpp +++ b/src/provider/xlings/xlings.cpp @@ -1150,6 +1150,7 @@ call(const Env& env, std::string_view capability, const bool installing = capability == "install_packages"; const bool syncing = capability == "sync_index" || capability == "update_packages"; InstallDeadline deadline; + std::string progressDeadlineReason; std::map sourceAttempts; inv.options.streams = mcpp::run::Streams::output_error_kept; inv.options.on_line = [&](std::string_view line) { @@ -1196,14 +1197,19 @@ call(const Env& env, std::string_view capability, : syncing ? std::chrono::duration_cast(index_refresh_timeout()) : std::chrono::milliseconds{0}; inv.options.idle = std::chrono::duration_cast(kInterfaceIdleTimeout); - inv.options.should_stop = [&] { return installing && deadline.expired(); }; + inv.options.should_stop = [&] { + if (!installing) return false; + progressDeadlineReason = deadline.failure_reason(); + return !progressDeadlineReason.empty(); + }; const auto ran = mcpp::run::execute(inv.argv, inv.options); const int rc = ran.exit_code; const bool timedOut = ran.timed_out; if (timedOut) { result.exitCode = result.exitCode != 0 ? result.exitCode : 124; - const auto message = std::format( - "provider {} exceeded its progress/process deadline and was stopped", capability); + const auto message = progressDeadlineReason.empty() + ? std::format("provider {} exceeded its process deadline and was stopped", capability) + : std::format("provider {} stopped: {}", capability, progressDeadlineReason); if (!result.error) result.error = ErrorEvent{ installing || syncing ? "E_NETWORK" : "E_PROVIDER_TIMEOUT", message, installing || syncing ? "check proxy settings or use mcpp self config --mirror auto|CN|GLOBAL" @@ -1484,6 +1490,7 @@ int install_with_progress(const Env& env, std::string_view target, std::map sourceAttempts; mcpp::ui::DownloadProgress download; InstallDeadline deadline; + std::string progressDeadlineReason; auto handle_line = [&](std::string_view line) { deadline.observe(line); @@ -1522,7 +1529,10 @@ int install_with_progress(const Env& env, std::string_view target, inv.options.on_line = handle_line; inv.options.total = std::chrono::duration_cast(kDirectInstallTimeout); inv.options.idle = std::chrono::duration_cast(kInterfaceIdleTimeout); - inv.options.should_stop = [&] { return deadline.expired(); }; + inv.options.should_stop = [&] { + progressDeadlineReason = deadline.failure_reason(); + return !progressDeadlineReason.empty(); + }; const auto ran = mcpp::run::execute(inv.argv, inv.options); const bool idleTimedOut = ran.timed_out; const int closeRc = ran.exit_code; @@ -1531,8 +1541,10 @@ int install_with_progress(const Env& env, std::string_view target, else download.finish_failed(); } if (idleTimedOut) { - mcpp::log::warn("xlings", "package install exceeded its progress/process deadline and was stopped"); - if (!quiet) print_status("Network", "Download stopped: no byte progress or process deadline exceeded; check proxy/mirror settings"); + const auto reason = progressDeadlineReason.empty() ? "provider process deadline exceeded" + : progressDeadlineReason; + mcpp::log::warn("xlings", std::format("package install stopped: {}", reason)); + if (!quiet) print_status("Network", std::format("Download stopped: {}; check proxy/mirror settings", reason)); return resultExitCode > 0 ? resultExitCode : 124; } return (resultExitCode != -1) ? resultExitCode : closeRc; diff --git a/src/provider/xlings/xlings.cppm b/src/provider/xlings/xlings.cppm index db36797be..ba8c3c81e 100644 --- a/src/provider/xlings/xlings.cppm +++ b/src/provider/xlings/xlings.cppm @@ -508,17 +508,29 @@ public: for (const auto& f : progress.files) { auto [it, added] = files_.try_emplace(f.name); auto& state = it->second; - if (added || f.sourceAttempt != state.attempt || f.downloadedBytes > state.bytes) + const bool inactive = f.finished || !f.started; + // Queued files appear in the batch before obtaining a download + // slot. Their active budget begins when they start, even before + // the first source attempt or byte has been reported. + if (added || (state.finished && !inactive) + || f.sourceAttempt != state.attempt || f.downloadedBytes > state.bytes) state.progress = now; state.bytes = f.downloadedBytes; state.attempt = f.sourceAttempt; - state.finished = f.finished || !f.started; + state.finished = inactive; } } + std::string failure_reason(std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now()) const { + for (const auto& [name, state] : files_) { + if (!state.finished && now - state.progress >= idle_) + return std::format("download '{}' made no byte progress or source change for {}s (attempt {})", + name, std::chrono::duration_cast(now - state.progress).count(), + state.attempt); + } + return {}; + } bool expired(std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now()) const { - return std::ranges::any_of(files_, [&](const auto& f) { - return !f.second.finished && now - f.second.progress >= idle_; - }); + return !failure_reason(now).empty(); } }; @@ -534,9 +546,14 @@ public: : downloads_(idle), started_(start), preparation_(preparation) {} void observe(std::string_view line, std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now()); + std::string failure_reason(std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now()) const { + if (auto reason = downloads_.failure_reason(now); !reason.empty()) return reason; + if (!downloadSeen_ && !localWork_ && now - started_ >= preparation_) + return "provider preparation made no download or local-work progress"; + return {}; + } bool expired(std::chrono::steady_clock::time_point now = std::chrono::steady_clock::now()) const { - return downloads_.expired(now) - || (!downloadSeen_ && !localWork_ && now - started_ >= preparation_); + return !failure_reason(now).empty(); } }; diff --git a/tests/e2e/234_bmi_schedule_on.sh b/tests/e2e/234_bmi_schedule_on.sh index 5bf3dfc2b..669383ca4 100755 --- a/tests/e2e/234_bmi_schedule_on.sh +++ b/tests/e2e/234_bmi_schedule_on.sh @@ -189,4 +189,47 @@ BIN=$(ls -t target/*/*/bin/schedon 2>/dev/null | head -1) out=$("$BIN") [ "$out" = "8" ] || { echo "FAIL: binary printed '$out' after the stale-token build"; exit 1; } -echo "234 bmi_schedule OK (detach-codegen, tokens reclaimed)" +echo "== 5. a BMI removal is not publication ==" +# GCC unlinks the previous .gcm immediately before renaming .gcm~ into place. +# Amplify only that interval: a polling loop must not release an importer +# while the module is absent. This deterministic publisher exercises the same +# supervisor and two-phase join without depending on a microsecond race. +mkdir publication +cd publication +printf 'old BMI' > published.gcm +cat > publish.sh <<'EOF' +rm published.gcm +sleep 0.2 +printf 'new BMI' > next.gcm +mv next.gcm published.gcm +sleep 0.05 +printf object > unit.o +EOF +printf '"%s" publish.sh\n' "$BASH" > command.txt +"$MCPP" bmi-compile --bmi published.gcm --slot unit --self "$MCPP" \ + --command-file command.txt +published=0 +[ ! -f published.gcm ] || published=1 +# Join even when the assertion will fail, so no publisher survives cleanup. +"$MCPP" bmi-await --slot unit --object unit.o +[ "$published" = 1 ] || { + echo "FAIL: phase 1 released an importer while the BMI was absent"; exit 1; } +[ "$(cat published.gcm)" = "new BMI" ] || { + echo "FAIL: phase 1 did not wait for the new BMI"; exit 1; } +echo " absence withheld; new BMI published and object joined" + +printf '"%s" -c "rm published.gcm"\n' "$BASH" > command.txt +if "$MCPP" bmi-compile --bmi published.gcm --slot unit --self "$MCPP" \ + --command-file command.txt > missing.log 2>&1; then + echo "FAIL: a successful compiler without a BMI was accepted"; exit 1 +fi +grep -q 'compiler reported success but.*is missing' missing.log || { + echo "FAIL: the missing BMI was not diagnosed"; cat missing.log; exit 1; } +[ "$(cat published.gcm)" = "new BMI" ] || { + echo "FAIL: the previous BMI was not restored after missing publication"; exit 1; } +echo " missing publication diagnosed and previous BMI restored" + +echo "== 6. ordinary and detached compilers share the actual cap ==" +python3 "$_root/tests/e2e/helpers/check_shared_compiler_cap.py" "$MCPP" "$TMP/cap-load" + +echo "234 bmi_schedule OK (detach-codegen, tokens reclaimed, publication checked)" diff --git a/tests/e2e/helpers/check_shared_compiler_cap.py b/tests/e2e/helpers/check_shared_compiler_cap.py new file mode 100644 index 000000000..7ff96e741 --- /dev/null +++ b/tests/e2e/helpers/check_shared_compiler_cap.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +"""Check actual GCC/Ninja mixed compilation and the shared permit lifetime.""" +import argparse +import json +import os +from pathlib import Path +import re +import shlex +import shutil +import signal +import subprocess +import sys +import time + + +def checked(command, cwd, log, env=None): + result = subprocess.run(command, cwd=cwd, env=env, stdout=log, stderr=subprocess.STDOUT, timeout=60) + if result.returncode: + log.flush() + raise AssertionError(f"command failed ({result.returncode}): " + + Path(log.name).read_text(errors="replace")[-5000:]) + + +def state(path): + return json.loads(path.read_text()) if path.exists() else {"active": [], "peak": 0, "calls": 0} + + +def settled(path, semaphore): + deadline = time.monotonic() + 5 + while time.monotonic() < deadline: + if not state(path)["active"] and not list(semaphore.iterdir()): + return + time.sleep(0.02) + raise AssertionError("compilers or permits survived the completed workload") + + +def check_mixed(mcpp, root, cap, ninja): + project = root / f"mixed-{cap}" + (project / "src").mkdir(parents=True) + (project / "mcpp.toml").write_text('''[package] +name = "cap_probe" +version = "0.1.0" +[toolchain] +default = "gcc@16.1.0" +[build] +bmi_schedule = "on" +''') + (project / "src/core.cppm").write_text("export module cap.core;\nexport int core_value() { return 7; }\n") + (project / "src/mid.cppm").write_text("export module cap.mid;\nimport cap.core;\nexport int value() { return core_value(); }\n") + (project / "src/main.cpp").write_text("import cap.mid;\nint main() { return value() == 7 ? 0 : 1; }\n") + for index in range(6): + (project / f"src/ordinary_{index}.cpp").write_text(f"int ordinary_{index}() {{ return {index}; }}\n") + (project / f"src/c_{index}.c").write_text(f"int c_{index}(void) {{ return {index}; }}\n") + with (project / "build.log").open("w") as log: + checked([mcpp, "build", "--release", "--configure-only", "-j", str(cap)], project, log) + graphs = list(project.glob("target/*/*/build.ninja")) + assert len(graphs) == 1 + graph = graphs[0] + build = graph.parent + counter = project / "counter.json" + wrapper = project / "compiler-driver" + wrapper.write_text("#!" + sys.executable + r''' +import fcntl,json,os,subprocess,sys,time +from pathlib import Path +counter=Path(os.environ["MCPP_CAP_COUNTER"]) +def update(start): + with counter.with_suffix(".lock").open("a") as lock: + fcntl.flock(lock,fcntl.LOCK_EX) + record=json.loads(counter.read_text()) if counter.exists() else {"active":[],"peak":0,"calls":0} + if start: + record["active"].append(os.getpid());record["calls"]+=1 + record["peak"]=max(record["peak"],len(record["active"])) + else:record["active"].remove(os.getpid()) + temporary=counter.with_suffix(".tmp");temporary.write_text(json.dumps(record));temporary.replace(counter) +update(True) +try: + result=subprocess.run(sys.argv[1:]).returncode + time.sleep(0.15) # keep the driver alive after BMI publication + failed=os.environ.get("MCPP_CAP_TEST_FAIL") + if failed and "-c" in sys.argv and any(arg.endswith("/"+failed) for arg in sys.argv): + print("controlled compiler failure",file=sys.stderr);result=17 +finally:update(False) +sys.exit(result) +''') + wrapper.chmod(0o755) + text = graph.read_text() + word = shlex.quote(str(wrapper)).replace("$", "$$") + text = re.sub(r"^(cxx|cc|cxx_driver)\s*= (.+)$", lambda match: + match[1] + " = " + word + " " + match[2], text, flags=re.MULTILINE) + graph.write_text(text) + env = os.environ.copy() + env["MCPP_CAP_COUNTER"] = str(counter) + command = [ninja, "-j", str(cap * 6)] + checked(command, build, log, env) + semaphore = build / ".mcpp-sched" + settled(counter, semaphore) + measured = state(counter) + assert measured["peak"] == cap, f"actual compiler driver peak {measured['peak']} differs from cap {cap}" + checked([str(next(build.glob("bin/cap_probe")))], build, log, env) + checked(command, build, log, env) + assert state(counter)["calls"] == measured["calls"], "no-op build launched compilers" + for failed in ("ordinary_0.cpp", "core.cppm"): + checked([ninja, "-t", "clean"], build, log, env) + env["MCPP_CAP_TEST_FAIL"] = failed + result = subprocess.run(command, cwd=build, env=env, stdout=log, stderr=subprocess.STDOUT, timeout=60) + assert result.returncode != 0, "compiler failure was swallowed" + log.flush() + assert "controlled compiler failure" in Path(log.name).read_text(), "compiler diagnostic was lost" + settled(counter, semaphore) + del env["MCPP_CAP_TEST_FAIL"] + checked(command, build, log, env) + settled(counter, semaphore) + assert state(counter)["peak"] <= cap, "failure recovery exceeded the compiler cap" + print(f"PASS: mixed GCC modules/C++/C cap={cap}, joined without deadlock, no-op and both failure recoveries") + + +def alive(pid): + try: + os.kill(pid, 0) + status = Path(f"/proc/{pid}/stat") + return not status.exists() or status.read_text().split(")", 1)[1].split()[0] != "Z" + except ProcessLookupError: + return False + + +def check_cancellation(mcpp, root): + semaphore = root / "cancel-semaphore" + semaphore.mkdir() + pid_file = root / "child.pid" + command = [mcpp, "__compiler-cap", "--sem", str(semaphore), "--cap", "1", "--"] + child = [sys.executable, "-c", "import os,time;from pathlib import Path;" + f"Path({str(pid_file)!r}).write_text(str(os.getpid()));time.sleep(60)"] + process = subprocess.Popen([*command, *child], cwd=root) + pid = None + try: + deadline = time.monotonic() + 5 + while not pid_file.exists() and time.monotonic() < deadline: + time.sleep(0.01) + assert pid_file.exists(), "controlled compiler did not start" + pid = int(pid_file.read_text()) + process.send_signal(signal.SIGINT) + assert process.wait(timeout=5) != 0 + deadline = time.monotonic() + 5 + while alive(pid) and time.monotonic() < deadline: + time.sleep(0.01) + assert not alive(pid), "cancellation left the compiler alive" + assert not list(semaphore.iterdir()), "cancellation retained its compiler permit" + result = subprocess.run([*command, sys.executable, "-c", "raise SystemExit(7)"], cwd=root) + assert result.returncode == 7 and not list(semaphore.iterdir()), "failure lost its status or leaked a permit" + words = ["space value", 'literal "quote"', "back\\slash", "$literal", "", "Ω"] + result = subprocess.run([*command, sys.executable, "-c", "import json,sys;print(json.dumps(sys.argv[1:]))", *words], + cwd=root, capture_output=True, text=True) + assert result.returncode == 0 and json.loads(result.stdout) == words, "compiler argument words changed" + finally: + if process.poll() is None: + process.kill() + process.wait() + if pid is not None and alive(pid): + os.kill(pid, signal.SIGKILL) + print("PASS: compiler cancellation ends its child and releases the permit; argv and failure status retained") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("mcpp", type=Path) + parser.add_argument("fixture", type=Path) + args = parser.parse_args() + args.fixture.mkdir(parents=True) + ninja = shutil.which("ninja") + assert ninja, "Ninja is required" + for cap in (1, 2): + check_mixed(str(args.mcpp.resolve()), args.fixture.resolve(), cap, ninja) + check_cancellation(str(args.mcpp.resolve()), args.fixture.resolve()) + + +if __name__ == "__main__": + main() diff --git a/tests/unit/test_detach_status.cpp b/tests/unit/test_detach_status.cpp index 43df4a2b4..79fc0583f 100644 --- a/tests/unit/test_detach_status.cpp +++ b/tests/unit/test_detach_status.cpp @@ -40,3 +40,18 @@ TEST(DetachStatus, AWrittenStatusIsTheUnits) { std::ofstream(t.slot.string() + ".rc") << "0\n"; EXPECT_EQ(detach::await_unit(t.slot, {}), 0); } + +TEST(DetachStatus, OrdinaryCompilerStartFailureReleasesItsPermit) { + Slot s; + const auto semaphore = s.dir / "permits"; + EXPECT_EQ(detach::run_capped({(s.dir / "missing-compiler").string()}, semaphore, 1), 127); + EXPECT_TRUE(fs::is_empty(semaphore)); +} + +TEST(DetachStatus, AnUnavailableSemaphoreRefusesUnboundedCompilation) { + Slot s; + const auto semaphore = s.dir / "not-a-directory"; + std::ofstream(semaphore) << "occupied"; + EXPECT_NE(detach::run_capped({"unused-compiler"}, semaphore, 1), 0); + EXPECT_TRUE(fs::is_regular_file(semaphore)); +} diff --git a/tests/unit/test_network.cpp b/tests/unit/test_network.cpp index e5f398fd6..e30940548 100644 --- a/tests/unit/test_network.cpp +++ b/tests/unit/test_network.cpp @@ -120,6 +120,47 @@ TEST(NetworkCheck, ASilentSourceIsLeftBeforeTheInstallStops) { EXPECT_TRUE(watchdog.expired(left + x::kDownloadStallTimeout)); } +TEST(NetworkCheck, AnotherDownloadCannotRenewAStalledFilesBudget) { + using namespace std::chrono; + namespace x = mcpp::provider::xlings; + const auto start = steady_clock::now(); + x::InstallDeadline deadline(seconds(120), seconds(45), start); + deadline.observe(R"({"kind":"data","dataKind":"download_progress","payload":{"stream":"install:1","files":[{"name":"stalled","started":true,"finished":false,"sourceAttempt":1,"downloadedBytes":0},{"name":"healthy","started":true,"finished":false,"sourceAttempt":1,"downloadedBytes":1}]}})", start); + deadline.observe(R"({"kind":"data","dataKind":"download_progress","payload":{"stream":"install:1","files":[{"name":"stalled","started":true,"finished":false,"sourceAttempt":1,"downloadedBytes":0},{"name":"healthy","started":true,"finished":false,"sourceAttempt":1,"downloadedBytes":100}]}})", start + seconds(44)); + EXPECT_FALSE(deadline.expired(start + seconds(44))); + EXPECT_EQ(deadline.failure_reason(start + seconds(45)), + "download 'stalled' made no byte progress or source change for 45s (attempt 1)"); + deadline.observe(R"({"kind":"data","dataKind":"download_progress","payload":{"stream":"install:1","files":[{"name":"stalled","started":true,"finished":false,"sourceAttempt":2,"downloadedBytes":0},{"name":"healthy","started":true,"finished":true,"sourceAttempt":1,"downloadedBytes":100}]}})", start + seconds(46)); + EXPECT_TRUE(deadline.failure_reason(start + seconds(90)).empty()); + EXPECT_EQ(deadline.failure_reason(start + seconds(91)), + "download 'stalled' made no byte progress or source change for 45s (attempt 2)"); + deadline.observe(R"({"kind":"data","dataKind":"download_progress","payload":{"stream":"install:1","files":[{"name":"stalled","started":true,"finished":true,"sourceAttempt":2,"downloadedBytes":1}]}})", start + seconds(92)); + EXPECT_TRUE(deadline.failure_reason(start + hours(1)).empty()); +} + +TEST(NetworkCheck, AQueuedFilesBudgetStartsWhenItBecomesActive) { + using namespace std::chrono; + namespace x = mcpp::provider::xlings; + const auto start = steady_clock::now(); + x::DownloadWatchdog watchdog(seconds(45)); + mcpp::provider::BootstrapProgress progress; + progress.files.push_back({.name = "queued", .started = false}); + watchdog.observe(progress, start); + EXPECT_FALSE(watchdog.expired(start + seconds(96))); + // Neither bytes nor sourceAttempt changes when the worker obtains a slot. + progress.files.front().started = true; + watchdog.observe(progress, start + seconds(96)); + EXPECT_FALSE(watchdog.expired(start + seconds(140))); + EXPECT_EQ(watchdog.failure_reason(start + seconds(141)), + "download 'queued' made no byte progress or source change for 45s (attempt 0)"); + progress.files.front().sourceAttempt = 1; + watchdog.observe(progress, start + seconds(142)); + EXPECT_FALSE(watchdog.expired(start + seconds(186))); + progress.files.front().finished = true; + watchdog.observe(progress, start + seconds(187)); + EXPECT_FALSE(watchdog.expired(start + hours(1))); +} + TEST(NetworkProcess, ProgressCancellationIsPolledWhileOutputContinues) { using namespace std::chrono; const auto start = steady_clock::now(); diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index a8b59735f..9c875de04 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -123,6 +123,31 @@ TEST(NinjaBackend, BitcodeExportToolsComeFromTheSelectedLlvmAndAreQuotedAsWords) EXPECT_EQ(emit_ninja_string(plan).find("coff_args ="), std::string::npos); } +TEST(NinjaBackend, OrdinaryCompilersShareTheDetachedScheduleBound) { + auto plan = minimal_plan(); + plan.toolchain.version = "16.1.0"; + plan.scheduleTag = "detach-codegen"; + plan.scheduleCompilerCap = 2; + plan.compileUnits.push_back({.source = "src/value.c", .kind = mcpp::SourceKind::C, + .object = "obj/value.o"}); + plan.compileUnits.push_back({.source = "src/value.S", .kind = mcpp::SourceKind::GasAsm, + .object = "obj/value-asm.o"}); + plan.compileUnits.push_back({.source = "src/value.asm", .kind = mcpp::SourceKind::NasmAsm, + .object = "obj/value-nasm.o"}); + const auto text = emit_ninja_string(plan); + auto rule = [&](std::string_view name) { + const auto start = text.find("rule " + std::string(name) + "\n"); + EXPECT_NE(start, std::string::npos) << name; + return start == std::string::npos ? std::string{} : text.substr(start, text.find("\n\n", start) - start); + }; + for (const auto name : {"cxx_module", "cxx_object", "c_object", "asm_object", "asm_object_raw", "nasm_object", "cxx_scan"}) + EXPECT_NE(rule(name).find("__compiler-cap --sem .mcpp-sched --cap 2 -- "), std::string::npos) << name; + EXPECT_EQ(rule("cxx_module_bmi").find("__compiler-cap"), std::string::npos) + << "the BMI supervisor already holds a permit"; + EXPECT_EQ(rule("cxx_module_obj").find("__compiler-cap"), std::string::npos) + << "a join must not wait for a second permit"; +} + TEST(NinjaBackend, ObjectiveCSourceUsesCObjectRuleAndCFlags) { auto plan = minimal_plan(); plan.compileUnits.push_back({ From ffcf49d51ef69235389a7241705afbd90943afe6 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 02:07:48 +0900 Subject: [PATCH 09/40] fix: dispatch compiler argv before global option scans --- ...i-publication-and-compiler-cap-closeout.md | 3 +- src/cli.cppm | 50 ++++++++++--------- .../e2e/helpers/check_shared_compiler_cap.py | 3 +- 3 files changed, 30 insertions(+), 26 deletions(-) diff --git a/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md index 98266e4e7..de187f7e6 100644 --- a/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md +++ b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md @@ -56,7 +56,8 @@ permit, so waiting joins cannot form a permit dependency cycle. The internal executor introduces no user mode and no argument serialization. It receives the compiler's actual argv, inherits the existing streams and -returns its status. Existing response files retain their original owner and +returns its status. It dispatches before global option scans so child flags +cannot configure mcpp. Existing response files retain their original owner and encoding. Ordinary completion and interrupt cleanup release a permit once; release cannot remove a permit already acquired by another compiler. Failure to create or acquire the semaphore is reported rather than followed diff --git a/src/cli.cppm b/src/cli.cppm index 75917d4e5..3c5f43f14 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -274,6 +274,32 @@ int cmd_toolchain_any(const mcpplibs::cmdline::ParsedArgs& p) { int run(int argc, char** argv) { namespace cl = mcpplibs::cmdline; + + // Internal split-schedule executor. Everything after `--` is the + // compiler's argv, including flags with the same names as these options. + // Dispatch before global option scans: child words never configure mcpp. + if (argc > 1 && std::string_view(argv[1]) == "__compiler-cap") { + std::filesystem::path semaphore; + int cap = 0, i = 2; + for (; i < argc && std::string_view(argv[i]) != "--"; ++i) { + const std::string_view option = argv[i]; + if ((option != "--sem" && option != "--cap") || i + 1 >= argc) { + std::println(stderr, "error: __compiler-cap: expected --sem --cap -- ..."); + return 2; + } + const std::string_view value = argv[++i]; + if (option == "--sem") semaphore = value; + else { + const auto [end, ec] = std::from_chars(value.data(), value.data() + value.size(), cap); + if (ec != std::errc{} || end != value.data() + value.size()) return 2; + } + } + if (semaphore.empty() || cap <= 0 || i + 1 >= argc) return 2; + std::vector command; + for (++i; i < argc; ++i) command.emplace_back(argv[i]); + return mcpp::build::schedule::detach::run_capped(command, semaphore, cap); + } + ReportUnnarrowablePaths reportUnnarrowable_; // What a build that waits for this one is told it waits for (mcpp#839): @@ -1493,30 +1519,6 @@ int run(int argc, char** argv) { // check and no `prepare`. On the MSVC ABI the first build after upgrading // to 2026.9.28.2 re-runs each action once, because its command line gained // the directory. - // Internal split-schedule executor. Everything after `--` is the - // compiler's argv, including flags with the same names as these options. - if (std::string_view(argv[1]) == "__compiler-cap") { - std::filesystem::path semaphore; - int cap = 0, i = 2; - for (; i < argc && std::string_view(argv[i]) != "--"; ++i) { - const std::string_view option = argv[i]; - if ((option != "--sem" && option != "--cap") || i + 1 >= argc) { - std::println(stderr, "error: __compiler-cap: expected --sem --cap -- ..."); - return 2; - } - const std::string_view value = argv[++i]; - if (option == "--sem") semaphore = value; - else { - const auto [end, ec] = std::from_chars(value.data(), value.data() + value.size(), cap); - if (ec != std::errc{} || end != value.data() + value.size()) return 2; - } - } - if (semaphore.empty() || cap <= 0 || i + 1 >= argc) return 2; - std::vector command; - for (++i; i < argc; ++i) command.emplace_back(argv[i]); - return mcpp::build::schedule::detach::run_capped(command, semaphore, cap); - } - // `mcpp depfile-filter --raw --out -- ...` // (internal: written into build.ninja for GCC on a Windows host, WS2 of // the 2026-09-28 design). Runs the compile with inherited stdio and no diff --git a/tests/e2e/helpers/check_shared_compiler_cap.py b/tests/e2e/helpers/check_shared_compiler_cap.py index 7ff96e741..c66182e24 100644 --- a/tests/e2e/helpers/check_shared_compiler_cap.py +++ b/tests/e2e/helpers/check_shared_compiler_cap.py @@ -147,7 +147,8 @@ def check_cancellation(mcpp, root): assert not list(semaphore.iterdir()), "cancellation retained its compiler permit" result = subprocess.run([*command, sys.executable, "-c", "raise SystemExit(7)"], cwd=root) assert result.returncode == 7 and not list(semaphore.iterdir()), "failure lost its status or leaked a permit" - words = ["space value", 'literal "quote"', "back\\slash", "$literal", "", "Ω"] + words = ["space value", 'literal "quote"', "back\\slash", "$literal", "", "Ω", + "--system", "--home", "--deny", "--toolchain", "-j200", "--sem", "--cap"] result = subprocess.run([*command, sys.executable, "-c", "import json,sys;print(json.dumps(sys.argv[1:]))", *words], cwd=root, capture_output=True, text=True) assert result.returncode == 0 and json.loads(result.stdout) == words, "compiler argument words changed" From d76ffb6187a190323337cdf241cc65e546c1b992 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 03:15:37 +0900 Subject: [PATCH 10/40] fix: own detached compilers through cancellation and rebuilds --- ...i-publication-and-compiler-cap-closeout.md | 68 ++++ CHANGELOG.md | 5 + docs/04-mcpp-toml.md | 3 + docs/zh/04-mcpp-toml.md | 2 + src/build/execute.cppm | 15 + src/build/ninja_backend.cppm | 16 +- src/build/schedule/detach_codegen.cppm | 378 ++++++++++++++---- src/cli.cppm | 11 +- src/cli/cmd_build.cppm | 12 +- src/run/run.cpp | 13 + tests/e2e/234_bmi_schedule_on.sh | 28 ++ .../helpers/check_owned_schedule_lifetime.py | 280 +++++++++++++ .../e2e/helpers/check_shared_compiler_cap.py | 19 +- tests/unit/test_detach_status.cpp | 125 +++++- tests/unit/test_ninja_backend.cpp | 2 +- tests/unit/test_process_run_exec.cpp | 17 + 16 files changed, 910 insertions(+), 84 deletions(-) create mode 100644 tests/e2e/helpers/check_owned_schedule_lifetime.py diff --git a/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md index de187f7e6..4d109c249 100644 --- a/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md +++ b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md @@ -131,3 +131,71 @@ follow ordinary installation of the new release, in the same plugin PR. Public records contain synthetic assertions and aggregate evidence. Raw provider streams, user logs, usernames and private paths remain outside the repository and pull-request description. + +## Correction, 2026-11-06: ownership across cancellation and generations + +The shared-cap change exposed an existing lifetime gap. A real Ninja control +cancelled the top-level build after BMI publication. Its detached compiler +remained alive 200 milliseconds later. Deleting the previous directory tokens +at the next build could therefore admit new compilers over the requested cap. +The correction below supersedes directory-token ownership in the earlier +shared-permit description. + +Each detached-schedule backend drive owns a runtime generation. Normal graph +execution and cached replay, including quiet replay, open the same scope. Its +owner has both the operating system's process identity with birth time and an +exclusive kernel lock. A PID alone, a surviving record, or an absent birth time +cannot establish ownership. Closing the lock revokes the drive even while its +process is alive. The generation is passed only in the child environment after +cache decoding; it never enters an action, build fingerprint or saved graph. + +Every phase-one helper, supervisor, join helper and ordinary compiler executor +registers a kernel-locked actor before modifying build state and rechecks the +owner after registration. The actor covers status publication, BMI equivalence +settlement and backup restoration, as well as compiler execution. Interrupt +callbacks retain that actor until their final write. A delayed helper from a +closed generation returns cancellation before changing any build state. + +A stable gate serializes generations for one canonical output directory. The +next generation waits for all previous actor locks, including actors that have +already released a compiler permit. It retains busy state and refuses admission +if the actors do not drain within the cleanup bound. Runtime locks reside under +the work directory's `.mcpp/compiler-runs`, outside the target output tree and +independently of `MCPP_HOME`. Cleaning output files cannot unlink a live permit. + +Actual compiler permits are stable kernel-locked files. A supervisor acquires +its permit itself and holds it until the child has stopped; phase one does not +transfer a lock or consume a second permit. Ordinary compilation uses the same +slots. Files left after process death are reused by locking, never reclaimed by +unlinking. A typed busy result is retried; permission and other failures stop +the invocation. This also replaces the directory-creation contention path that +reported access denied in the Windows-host GCC cross-build. + +Child cancellation uses the existing launcher stop predicate. A further causal +control found that inherited or discarded streams skipped that predicate when +no absolute deadline was set. The launcher now polls child completion in that +case, retaining its argv, streams, exit status and existing deadline. No timeout +is increased. Failed or cancelled compiler invocations withdraw partial outputs; +subsequent ordinary builds recover missing objects and BMIs. + +The permanent real-Ninja control checks: + +- Module and ordinary compilation under top-level SIGINT and SIGKILL, followed + immediately by another generation; the old compiler has stopped before the + new drive is admitted. +- No-op builds across generations, retaining compiler count and object mtime; + normal and quiet cached replay retain their graph, objects and BMIs. +- Changed homes and a symlinked target directory retain the work-directory + runtime barrier, independently of the disposable output location. +- Owner-lock closure while the owner process remains alive. +- Missing and incorrect process birth identities, rejected before state writes. +- Delayed phase-one BMI settlement and backup restoration after compiler exit; + both still block the next generation until their final state write. +- Unregistered old phase-one and supervisor processes delayed until after a + successful new generation; neither can alter its BMI, object or status. + +The controls execute the actual helper and launcher implementations with a +small synthetic compiler. The separate GCC mixed workload retains the real +compiler cap, late failure, recovery, argv and no-op assertions. Selected-object +controls establish causality; final qualification remains complete current-source +CI and independent ecosystem review. diff --git a/CHANGELOG.md b/CHANGELOG.md index 14a1728f2..65f60a8e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,11 @@ BMI fails with a diagnostic and restores the previous BMI. - Ordinary compiler invocations share the detached GCC compilers' concurrency permits, so the additional Ninja scheduling slots do not exceed `--jobs`. +- Cancelled detached builds stop their compilers and finish state cleanup before + another generation starts. Stable kernel permits retain live ownership across + immediate rebuilds; runtime generations leave no-op build identities unchanged. +- Child stop predicates apply with inherited and discarded streams, including + executions without an absolute deadline. - Queued downloads begin their progress budget when a worker starts them, rather than inheriting time spent waiting for a slot. - Provider progress timeouts identify the stalled download and source attempt, diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index b87545a07..c8dc29009 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1051,6 +1051,9 @@ with two behaviours has to be. With GCC and `bmi_schedule = "on"`, ordinary compilation and detached code generation share the same total `jobs` limit (2026.11.5.2+). Additional scheduling and join tasks do not increase the number of compiler processes. +Cancelling the build ends its ordinary and detached compilers. An immediately +following build waits for their cleanup before starting another compiler; +partial outputs are rebuilt. An unchanged build still performs no compilation. | value | | |---|---| diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index d4020d43c..9934dab4c 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -974,6 +974,8 @@ bmi_schedule = "off" # auto (default, = off) | on | off 使用 GCC 且 `bmi_schedule = "on"` 时，普通编译与分离的代码生成共用同一个 `jobs` 总上限 （2026.11.5.2+）。额外的调度和等待任务不增加编译器进程数量。 +取消构建会结束本轮普通编译器和分离的编译器。紧接着的构建等待它们完成清理后 +才启动新的编译器；未完成的输出会重新构建。输入不变的构建仍不进行编译。 | 取值 | | |---|---| diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 95a454bbd..eb34971c4 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -61,6 +61,7 @@ import mcpp.runtime.launch_env; import mcpp.run; import mcpp.home.files; import mcpp.build.schedule.policy; // resolve_jobs — one answer to "how many at once" +import mcpp.build.schedule.detach_codegen; import mcpp.fetcher.progress; import mcpp.project; import mcpp.ui; @@ -2166,6 +2167,14 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, childEnv.emplace_back(runtimeEnvKey, runtimeEnvValue); } + std::optional compilerRun; + if (mcpp::build::read_schedule(ninjaPath) == "detach-codegen") { + auto opened = mcpp::build::schedule::detach::RunScope::begin(outputDir); + if (!opened) { mcpp::ui::error(opened.error()); return 1; } + compilerRun.emplace(std::move(*opened)); + compilerRun->environment(childEnv); + } + auto t0 = std::chrono::steady_clock::now(); // capture_exec merges stderr into the captured output (replacing `2>&1`), // so is_stale_ninja_failure / filter_ninja_output still see ninja errors. @@ -2213,6 +2222,12 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, out = std::move(r.output); status = r.exit_code; } + if (compilerRun) { + if (auto settled = compilerRun->finish(); !settled) { + mcpp::ui::error(settled.error()); + return 1; + } + } if (status != 0) { if (!reported && is_stale_ninja_failure(out)) return std::nullopt; diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 75fca0ba6..b66d7cc75 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -58,6 +58,7 @@ import mcpp.ui; import mcpp.log; import mcpp.build.action_progress; import mcpp.build.progress; +import mcpp.build.schedule.detach_codegen; export namespace mcpp::build { @@ -1470,7 +1471,7 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // compilers must share the supervisors' semaphore rather than use that // larger slot count as their own concurrency bound. const std::string compilerPrefix = splitBmi - ? std::format("$mcpp __compiler-cap --sem .mcpp-sched --cap {} -- ", plan.scheduleCompilerCap) + ? std::format("$mcpp __compiler-cap --sem .mcpp-sched --cap {} --output $out -- ", plan.scheduleCompilerCap) : std::string{}; // The other split shape. Clang publishes no BMI early — it writes the BMI // at the end of the compile — so the GCC trick of releasing the file @@ -2108,7 +2109,7 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // wrong in this file before. append("rule cxx_module_bmi\n"); append(" command = $mcpp bmi-compile --bmi $bmi_out --slot $slot" - " --self $mcpp --sem $sched_sem --cap $sched_cap" + " --self $mcpp --sem $sched_sem --cap $sched_cap --object $obj_out" " --command-file $out.cmd --dep-from $scan_dep --dep-to $out.d\n"); append(std::format( " rspfile = $out.cmd\n" @@ -4538,6 +4539,12 @@ std::expected NinjaBackend::build(const BuildPlan& plan return std::unexpected(BuildError{std::format("cannot create output dir '{}': {}", plan.outputDir.string(), ec.message()), plan.outputDir}); + std::optional compilerRun; + if (!opts.dryRun && plan.scheduleTag == "detach-codegen") { + auto opened = mcpp::build::schedule::detach::RunScope::begin(plan.outputDir); + if (!opened) return std::unexpected(BuildError{opened.error(), plan.outputDir}); + compilerRun.emplace(std::move(*opened)); + } if (!opts.dryRun && plan.scheduleTag == "detach-codegen") reclaim_stale_schedule_tokens(plan.outputDir); @@ -4881,6 +4888,7 @@ std::expected NinjaBackend::build(const BuildPlan& plan nenv.emplace_back(runtimeEnv->first, runtimeEnv->second); for (auto& ev : plan.toolchain.envOverrides) nenv.emplace_back(ev.key, ev.value); + if (compilerRun) compilerRun->environment(nenv); // THE STAGED CACHE IS PLACED BY A PASS OF ITS OWN, before the graph that // reads it. ninja 1.12.1 loads a dyndep file whose producer is up to date @@ -5008,6 +5016,10 @@ std::expected NinjaBackend::build(const BuildPlan& plan ninjaExit = cap.exit_code; } stage("ninja"); + if (compilerRun) { + if (auto settled = compilerRun->finish(); !settled) + return std::unexpected(BuildError{settled.error(), plan.outputDir}); + } bool ok = (ninjaExit == 0) && !buildTimedOut; // Every package gets its final line before what the build has to say // after ninja: validations, advice, `Finished`. diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 880f9d787..55d3f982c 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -74,19 +74,44 @@ import mcpplibs.os.process; import mcpp.build.command_line; import mcpp.run; import mcpp.build.stage; +import mcpp.home; +import mcpp.home.files; +import mcpp.invocation; +import mcpp.toolchain.fingerprint; export namespace mcpp::build::schedule::detach { +inline constexpr std::string_view kRunEnv = "MCPP_COMPILER_RUN"; + +// One runtime generation, outside the build identity. Its actor barrier covers +// every helper that may write state, not only helpers holding compiler slots. +class RunScope { +public: + static std::expected begin(const std::filesystem::path& outputDir); + RunScope(RunScope&&) noexcept; + RunScope& operator=(RunScope&&) noexcept; + ~RunScope(); + void environment(std::vector>& env) const; + std::expected finish(); + const std::filesystem::path& directory() const; +private: + struct State; + std::unique_ptr state_; + explicit RunScope(std::unique_ptr state); +}; + struct CompileRequest { // The BMI this compile publishes. Empty for a unit that produces none, in // which case phase 1 simply waits like an ordinary edge. std::filesystem::path bmi; + std::filesystem::path object; // `.log` and `.rc` live beside this path. std::filesystem::path slot; // Absolute path to the mcpp binary, re-invoked as the supervisor. A spawn // rather than a fork is what keeps this portable. std::filesystem::path self; - // Directory of concurrency tokens. Empty disables the cap (hazard 2). + // Standalone permit directory. Managed drives use their generation's + // stable slots; no compiler acquires a second permit (hazard 2). std::filesystem::path semaphore; int maxCompilers{0}; // The compiler invocation, as ONE shell command line. @@ -130,8 +155,8 @@ int compile_release_at_bmi(const CompileRequest& req); // The supervisor. Runs the compiler to completion with its output redirected, // then records the status. Never invoked directly by a build edge. int supervise(const std::filesystem::path& slot, - const std::filesystem::path& semaphoreToken, - std::string_view command); + const std::filesystem::path& semaphore, int maxCompilers, + const std::filesystem::path& object, std::string_view command); // Phase 2 — blocks until the compiler for `slot` finished, replays what it // wrote, and propagates its status. `object`, when given, must exist: a @@ -142,7 +167,8 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o // Ordinary compiler invocations share the detached compilers' actual bound. // Their original argv and streams are retained; no shell is introduced. int run_capped(const std::vector& argv, - const std::filesystem::path& semaphore, int maxCompilers); + const std::filesystem::path& semaphore, int maxCompilers, + const std::filesystem::path& output = {}); } // namespace mcpp::build::schedule::detach @@ -160,6 +186,110 @@ bool file_exists(const std::filesystem::path& p) { return std::filesystem::exists(p, ec); } +struct Owner { + bool managed = false; + std::filesystem::path generation; + mcpplibs::os::process::program program; + + static Owner inherited() { + Owner owner; + const auto value = mcpp::invocation::get(kRunEnv); + if (!value || value->empty()) return owner; + owner.managed = true; + owner.generation = mcpp::home::from_utf8(*value); + std::ifstream record(owner.generation / "owner"); + std::string version; + if (!(record >> version >> owner.program.id >> owner.program.started_ns) + || version != "mcpp-compiler-run-v1") owner.program = {}; + return owner; + } + bool active() const { + if (!managed) return true; // standalone internal helpers + if (program.id == 0 || program.started_ns == 0 + || !mcpplibs::os::process::alive(program)) return false; + // A surviving file is not a surviving owner. A normal drive may also + // have ended while its process remains alive to report the outcome. + auto released = mcpplibs::os::fs::lock::acquire(generation / "owner.lock", + mcpplibs::os::fs::lock::kind::exclusive, false); + return !released && released.error().code == mcpplibs::os::errc::again; + } + std::filesystem::path permits(const std::filesystem::path& fallback) const { + return managed ? generation.parent_path() / "slots" : fallback; + } +}; + +struct Actor { + Owner owner = Owner::inherited(); + std::filesystem::path path; + std::optional lock; + bool ready = false; + Actor() { + if (!owner.active()) return; + if (!owner.managed) { ready = true; return; } + const auto self = mcpplibs::os::process::self(); + if (!self || self->id == 0 || self->started_ns == 0) return; + static std::atomic sequence{0}; + const auto actors = owner.generation / "actors"; + std::error_code ec; + std::filesystem::create_directories(actors, ec); + if (ec) return; + const auto name = std::format("actor-{}-{}-{}.lock", self->id, self->started_ns, sequence++); + path = actors / name; + auto registered = mcpplibs::os::fs::lock::acquire(path, + mcpplibs::os::fs::lock::kind::exclusive, false); + if (registered) lock.emplace(std::move(*registered)); + // Registration precedes the final owner check. A late helper may + // create a record after closure, but cannot write build state. + ready = lock.has_value() && owner.active(); + } + ~Actor() { + lock.reset(); + if (!path.empty()) { + std::error_code ec; + std::filesystem::remove(path, ec); + } + } +}; + +struct InterruptCleanup { + int handle; + ~InterruptCleanup() { mcpp::run::remove_cleanup(handle); } +}; + +bool actors_idle(const std::filesystem::path& generation) { + const auto actors = generation / "actors"; + std::error_code ec; + if (!std::filesystem::exists(actors, ec)) return !ec; + std::filesystem::directory_iterator entries(actors, ec); + if (ec) return false; + for (const auto& entry : entries) { + const auto name = mcpp::home::spelling(entry.path().filename()); + if (!name.ends_with(".lock")) continue; + auto free = mcpplibs::os::fs::lock::acquire(entry.path(), + mcpplibs::os::fs::lock::kind::exclusive, false); + if (!free) return false; + } + return true; +} + +bool drain_actors(const std::filesystem::path& generation) { + const auto limit = std::chrono::steady_clock::now() + std::chrono::seconds(5); + do { + if (actors_idle(generation)) return true; + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } while (std::chrono::steady_clock::now() < limit); + return false; +} + +void withdraw_outputs(const std::filesystem::path& slot, + const std::filesystem::path& object = {}) { + std::error_code ec; + std::ifstream named(suffixed(slot, ".bmi")); + if (std::string bmi; named && std::getline(named, bmi) && !bmi.empty()) + std::filesystem::remove(bmi, ec); + if (!object.empty()) std::filesystem::remove(object, ec); +} + // The status a supervisor left. `.rc` appears by a rename once its number is // written, so one that holds no number is a write that failed (a full disk): // the supervisor's status is lost, and the unit is taken as failed rather than @@ -185,37 +315,34 @@ void write_rc(const std::filesystem::path& slot, int rc) { std::filesystem::rename(tmp, suffixed(slot, ".rc"), ec); } -// A counting semaphore made of directories. `mkdir` is atomic on every -// filesystem mcpp targets, it needs no daemon and no shared memory. A holder -// never waits for another token, so this cannot deadlock between holders. -// -// WHAT IT CAN DO IS OUTLIVE ITS HOLDER. A token is released by the -// supervisor that took it; a supervisor killed before its cleanup (Ctrl-C, the -// OOM killer, a reboot) leaves the directory behind, and nothing in this file -// ever reclaims one. The reclaim is therefore done ONCE PER BUILD, in prepare, -// before ninja is spawned — the only moment at which no token can have a live -// owner. Doing it here instead would race every other compiler in the build. -// -// The wait is bounded anyway. If a token is somehow still unreleasable, the -// honest outcome is a failure that names the directory, not a build that stops -// producing output and never returns. -std::filesystem::path acquire_token(const std::filesystem::path& dir, int cap) { - if (dir.empty() || cap <= 0) return {}; +// Stable kernel lock slots, shared by ordinary and detached compilers. The +// lock ends with its holder even after SIGKILL; its file is never unlinked as +// a way of reclaiming a live permit. No holder acquires a second permit. +std::expected, int> +acquire_permit(const Actor& actor, const std::filesystem::path& requested, int cap) { + const auto dir = actor.owner.permits(requested); + if (!actor.ready) return std::unexpected(130); + if (dir.empty() || cap <= 0) { + if (actor.owner.managed) return std::unexpected(2); + return std::optional{}; + } std::error_code ec; std::filesystem::create_directories(dir, ec); if (ec) { std::println(std::cerr, "mcpp: cannot create compiler semaphore {}: {}", dir.string(), ec.message()); - return {}; + return std::unexpected(1); } const auto started = std::chrono::steady_clock::now(); constexpr auto kLimit = std::chrono::hours(2); for (;;) { + if (!actor.owner.active()) return std::unexpected(130); for (int i = 0; i < cap; ++i) { - const auto tok = dir / std::to_string(i); - if (std::filesystem::create_directory(tok, ec) && !ec) return tok; - if (ec) { - std::println(std::cerr, "mcpp: cannot acquire compiler token {}: {}", tok.string(), ec.message()); - return {}; + auto permit = mcpplibs::os::fs::lock::acquire(dir / std::format("slot-{}.lock", i), + mcpplibs::os::fs::lock::kind::exclusive, false); + if (permit) return std::optional{std::move(*permit)}; + if (permit.error().code != mcpplibs::os::errc::again) { + std::println(std::cerr, "mcpp: {}", permit.error().message()); + return std::unexpected(1); } } if (std::chrono::steady_clock::now() - started > kLimit) { @@ -223,7 +350,7 @@ std::filesystem::path acquire_token(const std::filesystem::path& dir, int cap) { "mcpp: no compiler slot became free in 2h — all {} tokens in {} " "are still occupied; refusing to exceed the concurrency cap.", cap, dir.string()); - return {}; + return std::unexpected(1); } std::this_thread::sleep_for(std::chrono::milliseconds(5)); } @@ -361,32 +488,125 @@ bool spawn_detached(const std::vector& argv) { // the one pair of quotes `/s` removes, #425 #832), its output and errors into // `logPath`. int run_to_completion(std::string_view command, - const std::filesystem::path& logPath) { + const std::filesystem::path& logPath, const Owner& owner) { // Created before the compiler starts: its existence is what tells the // waiters that a supervisor started. std::ofstream log(logPath, std::ios::binary | std::ios::trunc); log.flush(); auto sh = mcpp::run::shell(command); - auto r = mcpp::run::execute(sh.argv, { .one_string = sh.one_string }); + auto r = mcpp::run::execute(sh.argv, { + .should_stop = [&] { return !owner.active(); }, .one_string = sh.one_string }); // A failure carries the command it ran. Phase 2 replays this file, so the // failure arrives with the exact invocation attached -- without it a // remote failure can only be guessed at, which is how #425 cost a CI cycle // to even localise. if (r.exit_code != 0) log << "+ " << command << '\n'; log << r.output; - return r.exit_code; + return owner.active() ? r.exit_code : 130; } } // namespace +struct RunScope::State { + std::filesystem::path root, generation; + std::optional gate; + std::optional owner; + bool closed = false; + std::string failure; + std::expected close() { + if (closed) return failure.empty() ? std::expected{} + : std::unexpected(failure); + closed = true; + if (generation.empty()) return {}; + owner.reset(); + if (!drain_actors(generation)) { + failure = "previous compiler helpers have not stopped; refusing to start another generation"; + return std::unexpected(failure); + } + std::error_code ec; + std::filesystem::remove_all(generation, ec); + return {}; + } + ~State() { (void)close(); } +}; + +RunScope::RunScope(std::unique_ptr state) : state_(std::move(state)) {} +RunScope::RunScope(RunScope&&) noexcept = default; +RunScope& RunScope::operator=(RunScope&&) noexcept = default; +RunScope::~RunScope() = default; + +std::expected RunScope::begin(const std::filesystem::path& outputDir) { + const auto self = mcpplibs::os::process::self(); + if (!self || self->id == 0 || self->started_ns == 0) + return std::unexpected("cannot identify the compiler-run owner and its birth time"); + std::error_code ec; + const auto spelled = std::filesystem::absolute(outputDir, ec).lexically_normal(); + if (ec) return std::unexpected("cannot identify the compiler-run output directory"); + const auto canonical = std::filesystem::weakly_canonical(spelled, ec); + if (ec) return std::unexpected("cannot identify the compiler-run output directory"); + auto state = std::make_unique(); + // Locate work BEFORE resolving output symlinks: target/ may point + // at a disposable cache. Locks remain beside target/, independently of + // clean and MCPP_HOME; the canonical output still identifies the slots. + const auto target = spelled.parent_path().parent_path(); + const auto work = std::filesystem::weakly_canonical( + target.filename() == "target" ? target.parent_path() : spelled.parent_path(), ec); + if (ec) return std::unexpected("cannot identify the compiler-run work directory"); + state->root = work / ".mcpp" / "compiler-runs" + / mcpp::toolchain::hash_string(mcpp::home::spelling(canonical)); + std::filesystem::create_directories(state->root, ec); + if (ec) return std::unexpected("cannot create compiler-run runtime state: " + ec.message()); + state->gate = mcpp::home::FileLock::acquire(state->root, "gate.lock", "", + std::chrono::milliseconds(300), {}); + if (!state->gate) return std::unexpected("cannot acquire the compiler-run generation gate"); + std::filesystem::directory_iterator previous(state->root, ec); + if (ec) return std::unexpected("cannot inspect previous compiler runs"); + for (const auto& entry : previous) { + if (!entry.path().filename().native().starts_with(std::filesystem::path("run-").native())) continue; + const auto limit = std::chrono::steady_clock::now() + std::chrono::seconds(5); + bool free = false; + do { + auto oldOwner = mcpplibs::os::fs::lock::acquire(entry.path() / "owner.lock", + mcpplibs::os::fs::lock::kind::exclusive, false); + if (oldOwner && actors_idle(entry.path())) { free = true; break; } + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } while (std::chrono::steady_clock::now() < limit); + if (!free) return std::unexpected("previous compiler helpers are still active; their state was retained"); + std::filesystem::remove_all(entry.path(), ec); + if (ec) return std::unexpected("cannot retire previous compiler-run state"); + } + static std::atomic sequence{0}; + state->generation = state->root / std::format("run-{}-{}-{}", self->id, self->started_ns, sequence++); + std::filesystem::create_directories(state->generation, ec); + if (ec) return std::unexpected("cannot create a compiler-run generation"); + auto owner = mcpplibs::os::fs::lock::acquire(state->generation / "owner.lock", + mcpplibs::os::fs::lock::kind::exclusive, false); + if (!owner) return std::unexpected("cannot lock the compiler-run owner: " + owner.error().message()); + state->owner.emplace(std::move(*owner)); + const auto record = std::format("mcpp-compiler-run-v1\n{} {}\n", self->id, self->started_ns); + if (!mcpplibs::os::fs::write_atomic(state->generation / "owner", record)) + return std::unexpected("cannot publish compiler-run ownership"); + return RunScope(std::move(state)); +} + +void RunScope::environment(std::vector>& env) const { + std::erase_if(env, [](const auto& entry) { return entry.first == kRunEnv; }); + env.emplace_back(kRunEnv, mcpp::home::spelling(state_->generation)); +} +std::expected RunScope::finish() { return state_->close(); } +const std::filesystem::path& RunScope::directory() const { return state_->generation; } + int compile_release_at_bmi(const CompileRequest& req) { if (req.command.empty() || req.self.empty()) return 2; + Actor actor; + if (!actor.ready) return 130; std::error_code ec; if (!req.slot.parent_path().empty()) std::filesystem::create_directories(req.slot.parent_path(), ec); std::filesystem::remove(suffixed(req.slot, ".rc"), ec); std::filesystem::remove(suffixed(req.slot, ".rc.tmp"), ec); + std::filesystem::remove(suffixed(req.slot, ".log"), ec); // Keep the previous BMI for the equivalence check, and LEAVE THE ORIGINAL // IN PLACE — see BmiIdentity for why moving it away is what broke importers. @@ -415,18 +635,17 @@ int compile_release_at_bmi(const CompileRequest& req) { // fails (see `await_unit`). if (!req.bmi.empty()) std::ofstream(suffixed(req.slot, ".bmi"), std::ios::trunc) << req.bmi.string(); - const auto token = acquire_token(req.semaphore, req.maxCompilers); - if (!req.semaphore.empty() && req.maxCompilers > 0 && token.empty()) return 1; - // The supervisor reads the SAME argv file rather than receiving the command // on its own command line: one representation, no re-quoting, and no limit // on how long a compiler command may be. std::vector sup{req.self.string(), "bmi-supervise", "--slot", req.slot.string(), "--command-file", req.commandFile.string()}; - if (!token.empty()) { sup.push_back("--token"); sup.push_back(token.string()); } + if (!req.semaphore.empty()) { sup.push_back("--sem"); sup.push_back(req.semaphore.string()); } + if (req.maxCompilers > 0) { sup.push_back("--cap"); sup.push_back(std::to_string(req.maxCompilers)); } + if (!req.object.empty()) { sup.push_back("--object"); sup.push_back(req.object.string()); } + if (!actor.owner.active()) return 130; if (!spawn_detached(sup)) { - std::filesystem::remove(token, ec); restore_backup(req.bmi); return 2; } @@ -447,6 +666,7 @@ int compile_release_at_bmi(const CompileRequest& req) { constexpr auto kHardLimit = std::chrono::hours(2); for (;;) { + if (!actor.owner.active()) return 130; // GCC unlinks an existing BMI immediately before renaming its new // temporary file. Absence is a transition, never publication: treating // it as a changed identity releases importers inside that unlink gap. @@ -455,7 +675,7 @@ int compile_release_at_bmi(const CompileRequest& req) { settle_bmi(req.bmi); copy_first_rule(req.depFrom, req.depTo, req.bmiTarget.empty() ? req.bmi.string() : req.bmiTarget); - return 0; // importers may proceed + return actor.owner.active() ? 0 : 130; // importers may proceed only in this run } if (const auto rc = read_rc(req.slot)) { if (*rc != 0) { // failed before publishing a BMI @@ -469,6 +689,7 @@ int compile_release_at_bmi(const CompileRequest& req) { std::println(std::cerr, "mcpp: compiler reported success but {} is missing", req.bmi.string()); restore_backup(req.bmi); + if (!req.object.empty()) std::filesystem::remove(req.object, ec); return 1; } // THE COMPILER CAN FINISH BETWEEN THE TWO CHECKS ABOVE. @@ -493,7 +714,7 @@ int compile_release_at_bmi(const CompileRequest& req) { copy_first_rule(req.depFrom, req.depTo, req.bmiTarget.empty() ? req.bmi.string() : req.bmiTarget); } - return *rc; + return actor.owner.active() ? *rc : 130; } const auto waited = clock::now() - started; // No log file means no supervisor ever opened one. Distinguished from @@ -520,8 +741,20 @@ int compile_release_at_bmi(const CompileRequest& req) { } int supervise(const std::filesystem::path& slot, - const std::filesystem::path& semaphoreToken, - std::string_view command) { + const std::filesystem::path& semaphore, int maxCompilers, + const std::filesystem::path& object, std::string_view command) { + auto actor = std::make_shared(); + if (!actor->ready) return 130; + auto settled = std::make_shared>(false); + auto settle = [actor, settled, slot, object](int rc) { + if (settled->exchange(true)) return; + if (rc != 0 || !actor->owner.active()) withdraw_outputs(slot, object); + write_rc(slot, rc); + }; + // The callback retains the actor lease until its last state write, even + // when interrupt cleanup and the normal return race on different threads. + InterruptCleanup cleanup{mcpp::run::add_cleanup([settle] { settle(130); })}; + { std::ofstream started(suffixed(slot, ".log"), std::ios::trunc); } // EVERY EXIT FROM HERE MUST LEAVE AN `.rc`, INCLUDING THE ONES THAT DO // NOT RUN A COMPILER. Phase 1 and phase 2 both wait on that file; a // supervisor that bails without writing one is indistinguishable from one @@ -531,23 +764,25 @@ int supervise(const std::filesystem::path& slot, if (command.empty()) { std::println(std::cerr, "mcpp: bmi-supervise got an empty command for {}", slot.string()); - if (!semaphoreToken.empty()) { - std::error_code ec; - std::filesystem::remove(semaphoreToken, ec); - } - write_rc(slot, 2); + settle(2); return 0; } - const int rc = run_to_completion(command, suffixed(slot, ".log")); - if (!semaphoreToken.empty()) { - std::error_code ec; - std::filesystem::remove(semaphoreToken, ec); + auto permit = acquire_permit(*actor, semaphore, maxCompilers); + if (!permit) { + if (actor->owner.active()) settle(permit.error()); + return permit.error(); } - write_rc(slot, rc); + // Acquire in the supervisor: no lock-handle handoff, and no interval in + // which an early-returning phase 1 could release the compiler's permit. + if (!actor->owner.active()) return 130; + const int rc = run_to_completion(command, suffixed(slot, ".log"), actor->owner); + settle(mcpp::run::ending() ? 130 : rc); return 0; // the supervisor's own status is not the compiler's } int await_unit(const std::filesystem::path& slot, const std::filesystem::path& object) { + Actor actor; + if (!actor.ready) return 130; // BOUNDED. An unbounded wait here turns "phase 1 never started a compiler" // into a build that hangs forever with no output — which is strictly worse // than a failure, because nothing says what to look at. If no supervisor @@ -559,6 +794,7 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o std::optional rc; while (!(rc = read_rc(slot))) { + if (!actor.owner.active()) return 130; const auto waited = clock::now() - started; if (!file_exists(suffixed(slot, ".log")) && waited > kNoSupervisorGrace) { std::println(std::cerr, @@ -605,32 +841,32 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o } int run_capped(const std::vector& argv, - const std::filesystem::path& semaphore, int maxCompilers) { + const std::filesystem::path& semaphore, int maxCompilers, + const std::filesystem::path& output) { if (argv.empty()) return 2; - struct Lease { - std::filesystem::path token; - // Completion and interrupt cleanup may race. A second release must - // never remove a permit that another compiler has already acquired. - std::atomic released{false}; - void release() { - if (released.exchange(true) || token.empty()) return; + auto actor = std::make_shared(); + if (!actor->ready) return 130; + InterruptCleanup cleanup{mcpp::run::add_cleanup([actor, output] { + if (!output.empty()) { std::error_code ec; - std::filesystem::remove(token, ec); + std::filesystem::remove(output, ec); } - ~Lease() { release(); } - }; - auto lease = std::make_shared(); - lease->token = acquire_token(semaphore, maxCompilers); - if (!semaphore.empty() && maxCompilers > 0 && lease->token.empty()) return 1; - const int cleanup = mcpp::run::add_cleanup([lease] { lease->release(); }); - struct Cleanup { - int handle; - std::shared_ptr lease; - ~Cleanup() { mcpp::run::remove_cleanup(handle); lease->release(); } - } guard{cleanup, lease}; - return mcpp::run::execute(argv, { + })}; + auto permit = acquire_permit(*actor, semaphore, maxCompilers); + if (!permit) return permit.error(); + if (!actor->owner.active()) return 130; + const auto result = mcpp::run::execute(argv, { .input = mcpp::run::Input::inherit, - .streams = mcpp::run::Streams::inherit}).exit_code; + .streams = mcpp::run::Streams::inherit, + .should_stop = [&] { return !actor->owner.active(); }}); + if (!actor->owner.active() || result.exit_code != 0) { + if (!output.empty()) { + std::error_code ec; + std::filesystem::remove(output, ec); + } + return actor->owner.active() ? result.exit_code : 130; + } + return result.exit_code; } } // namespace mcpp::build::schedule::detach diff --git a/src/cli.cppm b/src/cli.cppm index 3c5f43f14..ef0bc1868 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -280,15 +280,17 @@ int run(int argc, char** argv) { // Dispatch before global option scans: child words never configure mcpp. if (argc > 1 && std::string_view(argv[1]) == "__compiler-cap") { std::filesystem::path semaphore; + std::filesystem::path output; int cap = 0, i = 2; for (; i < argc && std::string_view(argv[i]) != "--"; ++i) { const std::string_view option = argv[i]; - if ((option != "--sem" && option != "--cap") || i + 1 >= argc) { + if ((option != "--sem" && option != "--cap" && option != "--output") || i + 1 >= argc) { std::println(stderr, "error: __compiler-cap: expected --sem --cap -- ..."); return 2; } const std::string_view value = argv[++i]; if (option == "--sem") semaphore = value; + else if (option == "--output") output = value; else { const auto [end, ec] = std::from_chars(value.data(), value.data() + value.size(), cap); if (ec != std::errc{} || end != value.data() + value.size()) return 2; @@ -297,7 +299,7 @@ int run(int argc, char** argv) { if (semaphore.empty() || cap <= 0 || i + 1 >= argc) return 2; std::vector command; for (++i; i < argc; ++i) command.emplace_back(argv[i]); - return mcpp::build::schedule::detach::run_capped(command, semaphore, cap); + return mcpp::build::schedule::detach::run_capped(command, semaphore, cap, output); } ReportUnnarrowablePaths reportUnnarrowable_; @@ -1438,6 +1440,7 @@ int run(int argc, char** argv) { .subcommand(cl::App("bmi-compile") .description("(internal) Compile a module interface and return when its BMI is published") .option(cl::Option("bmi").takes_value().value_name("PATH").help("BMI this unit publishes")) + .option(cl::Option("object").takes_value().value_name("PATH")) .option(cl::Option("slot").takes_value().value_name("PATH").help("where .log/.rc are kept")) .option(cl::Option("self").takes_value().value_name("PATH").help("path to mcpp, re-invoked as supervisor")) .option(cl::Option("sem").takes_value().value_name("DIR").help("concurrency token directory")) @@ -1449,7 +1452,9 @@ int run(int argc, char** argv) { .subcommand(cl::App("bmi-supervise") .description("(internal) Run a compiler to completion and record its status") .option(cl::Option("slot").takes_value().value_name("PATH")) - .option(cl::Option("token").takes_value().value_name("PATH")) + .option(cl::Option("sem").takes_value().value_name("PATH")) + .option(cl::Option("cap").takes_value().value_name("N")) + .option(cl::Option("object").takes_value().value_name("PATH")) .option(cl::Option("command-file").takes_value().value_name("PATH")) .action(wrap_rc(cmd_bmi_supervise))) .subcommand(cl::App("bmi-await") diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index f858966b5..fb7496683 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -2314,6 +2314,7 @@ export int cmd_bmi_compile(const mcpplibs::cmdline::ParsedArgs& parsed) { std::filesystem::path{opt_value(parsed, name)}); }; req.bmi = file_path("bmi"); + req.object = file_path("object"); req.bmiTarget = opt_value(parsed, "bmi"); req.slot = file_path("slot"); req.self = std::filesystem::path{opt_value(parsed, "self")}; @@ -2340,8 +2341,13 @@ export int cmd_bmi_compile(const mcpplibs::cmdline::ParsedArgs& parsed) { export int cmd_bmi_supervise(const mcpplibs::cmdline::ParsedArgs& parsed) { const auto slot = mcpplibs::os::fs::extended( std::filesystem::path{opt_value(parsed, "slot")}); - const auto token = mcpplibs::os::fs::extended( - std::filesystem::path{opt_value(parsed, "token")}); + const auto semaphore = mcpplibs::os::fs::extended( + std::filesystem::path{opt_value(parsed, "sem")}); + const auto object = mcpplibs::os::fs::extended( + std::filesystem::path{opt_value(parsed, "object")}); + int cap = 0; + if (const auto value = opt_value(parsed, "cap"); !value.empty()) + std::from_chars(value.data(), value.data() + value.size(), cap); const auto command = read_command_file(mcpplibs::os::fs::extended( std::filesystem::path{opt_value(parsed, "command-file")})); // Only `--slot` is checked here. An empty COMMAND is handed to supervise() @@ -2352,7 +2358,7 @@ export int cmd_bmi_supervise(const mcpplibs::cmdline::ParsedArgs& parsed) { std::println(stderr, "error: bmi-supervise needs --slot"); return 2; } - return mcpp::build::schedule::detach::supervise(slot, token, command); + return mcpp::build::schedule::detach::supervise(slot, semaphore, cap, object, command); } // Phase 2: join the detached compiler before anything reads its object. diff --git a/src/run/run.cpp b/src/run/run.cpp index 9f603f882..d6e11dbd0 100644 --- a/src/run/run.cpp +++ b/src/run/run.cpp @@ -299,6 +299,19 @@ Outcome launch(const Spec& s) { } drain_error(std::chrono::milliseconds(100)); } + } else if (s.shouldStop && *s.shouldStop) { + // Inherited/discarded streams have no read loop. Cancellation still + // belongs to the invocation, so poll the child without changing its + // streams or waiting for a separate absolute deadline. + for (;;) { + if ((*s.shouldStop)() + || (s.total.count() > 0 && clock::now() - start >= s.total)) { + stop = true; + break; + } + auto waited = p->wait_for(std::chrono::milliseconds(20)); + if (!waited || waited->has_value()) break; + } } else if (s.total.count() > 0) { auto w = p->wait_for(s.total); if (w && !w->has_value()) stop = true; diff --git a/tests/e2e/234_bmi_schedule_on.sh b/tests/e2e/234_bmi_schedule_on.sh index 669383ca4..a58690688 100755 --- a/tests/e2e/234_bmi_schedule_on.sh +++ b/tests/e2e/234_bmi_schedule_on.sh @@ -64,6 +64,7 @@ macos = "llvm@${LLVM_VERSION}" windows = "llvm@${LLVM_VERSION}" [build] +default-profile = "release" bmi_schedule = "on" EOF @@ -232,4 +233,31 @@ echo " missing publication diagnosed and previous BMI restored" echo "== 6. ordinary and detached compilers share the actual cap ==" python3 "$_root/tests/e2e/helpers/check_shared_compiler_cap.py" "$MCPP" "$TMP/cap-load" +echo "== 7. runtime generations preserve normal and quiet cached replay ==" +python3 - "$MCPP" "$TMP" "$TMP/$BUILD_DIR" <<'PY_REPLAY' +from pathlib import Path +import hashlib +import subprocess +import sys +mcpp, root, output = sys.argv[1], Path(sys.argv[2]), Path(sys.argv[3]) +graph = output / "build.ninja" +graph_before = (graph.stat().st_mtime_ns, graph.read_bytes()) +def identity(): + return {str(path.relative_to(output)): (path.stat().st_mtime_ns, + hashlib.sha256(path.read_bytes()).hexdigest()) + for pattern in ("obj/**/*.o", "gcm.cache/**/*.gcm") + for path in output.glob(pattern)} +before = identity() +assert before, "no compiler outputs to verify" +for flags in ([], ["--quiet"]): + result = subprocess.run([mcpp, "build", *flags], cwd=root, + capture_output=True, text=True, timeout=60) + assert result.returncode == 0, result.stdout + result.stderr + assert identity() == before, "cached replay recompiled unchanged inputs" + assert (graph.stat().st_mtime_ns, graph.read_bytes()) == graph_before, "cached replay rewrote its graph" +assert b"MCPP_COMPILER_RUN" not in graph_before[1] +assert b".mcpp/compiler-runs" not in graph_before[1] +print("PASS: normal/quiet cached replay retains graph, objects and BMIs without persisting generations") +PY_REPLAY + echo "234 bmi_schedule OK (detach-codegen, tokens reclaimed, publication checked)" diff --git a/tests/e2e/helpers/check_owned_schedule_lifetime.py b/tests/e2e/helpers/check_owned_schedule_lifetime.py new file mode 100644 index 000000000..a07383042 --- /dev/null +++ b/tests/e2e/helpers/check_owned_schedule_lifetime.py @@ -0,0 +1,280 @@ +#!/usr/bin/env python3 +"""Exercise run generations with real Ninja, signals and delayed state writers.""" +import argparse +import fcntl +import os +from pathlib import Path +import shlex +import shutil +import signal +import subprocess +import time + + +def until(predicate, description, timeout=6): + end = time.monotonic() + timeout + while time.monotonic() < end: + if predicate(): + return + time.sleep(0.01) + raise AssertionError(description) + + +def alive(pid): + try: + fields = Path(f"/proc/{pid}/stat").read_text().split(")", 1)[1].split() + return fields[0] != "Z" + except FileNotFoundError: + return False + + +def integer(path): + try: + return int(path.read_text().split()[0]) + except (FileNotFoundError, ValueError, IndexError): + return 0 + + +class Case: + def __init__(self, root, name, binary, kind="module", linked=False, **extra): + self.out = root / name / "target/host/fingerprint" + if linked: + cache = root / name / "cache-host" + cache.mkdir(parents=True) + self.out.parents[1].mkdir() + self.out.parent.symlink_to(cache, target_is_directory=True) + self.out.mkdir(parents=True) + self.env = os.environ.copy() + self.env.update(GTEST_FILTER="DetachLifetime.SubprocessDriver", + MCPP_TEST_LIFETIME_ROLE="root", MCPP_TEST_LIFETIME_KIND=kind, + MCPP_TEST_LIFETIME_OUTPUT=str(self.out)) + self.env.update(extra) + self.binary = binary + self.roots = [] + self.logs = [] + word = shlex.quote(str(binary)) + prefix = "setsid --wait " if extra.get("MCPP_TEST_LIFETIME_BLOCK") else "" + (self.out / "compile.command").write_text(f"env MCPP_TEST_LIFETIME_ROLE=compiler {word}") + if kind == "module": + graph = (f"rule phase\n command = {prefix}env MCPP_TEST_LIFETIME_ROLE=phase {word}\n restat = 1\n" + f"rule await\n command = env MCPP_TEST_LIFETIME_ROLE=await {word}\n" + "build published.gcm: phase source\nbuild module.o: await published.gcm\ndefault module.o\n") + else: + graph = (f"rule ordinary\n command = env MCPP_TEST_LIFETIME_ROLE=ordinary {word}\n" + "build ordinary.o: ordinary source\ndefault ordinary.o\n") + (self.out / "build.ninja").write_text(graph) + (self.out / "source").write_text("input") + + def start(self, delay=20000, **extra): + env = dict(self.env, MCPP_TEST_LIFETIME_DELAY=str(delay), **extra) + log = (self.out / f"run-{len(self.roots)}.log").open("w") + self.logs.append(log) + child = subprocess.Popen([str(self.binary)], cwd=self.out, env=env, + stdout=log, stderr=subprocess.STDOUT) + self.roots.append(child) + return child + + def ready(self, child): + until(lambda: integer(self.out / "root.ready") == child.pid, + f"new root did not cross generation barrier: {self.out.name}") + + def compiler(self): + until(lambda: integer(self.out / "compiler.ready") != 0, "compiler did not start") + return integer(self.out / "compiler.ready") + + def success(self, child): + assert child.wait(timeout=8) == 0, "new generation failed: " + self.diagnostics() + obj = "module.o" if self.env["MCPP_TEST_LIFETIME_KIND"] == "module" else "ordinary.o" + assert (self.out / obj).read_text() == "complete" + + def diagnostics(self): + for log in self.logs: + log.flush() + return "\n".join(path.read_text()[-3000:] for path in self.out.glob("run-*.log")) + + def close(self): + # Only PIDs recorded by this private fixture may be ended here. + for child in self.roots: + if child.poll() is None: + child.kill() + child.wait(timeout=5) + for name in ("compiler.ready", "phase.pid"): + pid = integer(self.out / name) + if pid and alive(pid): + os.kill(pid, signal.SIGKILL) + for log in self.logs: + log.close() + + +def cancellation(root, binary, sig, kind, linked=False): + case = Case(root, f"cancel-{sig.name}-{kind}" + ("-linked" if linked else ""), binary, kind, linked) + try: + old = case.start() + case.ready(old) + if linked: + assert (case.out.parents[2] / ".mcpp/compiler-runs").is_dir(), "runtime locks followed the target symlink" + compiler = case.compiler() + old.send_signal(sig) + # Begin immediately: no waiting for old compiler death before launch. + new = case.start(delay=30, MCPP_HOME=str(root / "changed-home")) + case.ready(new) + assert not alive(compiler), "new generation admitted a live old compiler" + case.success(new) + old.wait(timeout=5) + calls = (case.out / "compiler.runs").read_text() + obj = case.out / ("module.o" if kind == "module" else "ordinary.o") + stamp = obj.stat().st_mtime_ns + noop = case.start(delay=30) + case.success(noop) + assert (case.out / "compiler.runs").read_text() == calls, "generation caused no-op compilation" + assert obj.stat().st_mtime_ns == stamp + assert not list(case.out.glob("*.bak")), "old backup survived recovery" + print(f"PASS: {kind} {sig.name}" + (" with linked target" if linked else "") + + ", changed home, immediate restart, output recovery and unchanged no-op") + finally: + case.close() + + +def owner_closure(root, binary): + case = Case(root, "owner-lock-closure", binary, MCPP_TEST_LIFETIME_CLOSE="1") + try: + child = case.start() + case.ready(child) + compiler = case.compiler() + until(lambda: (case.out / "owner.closed").exists(), "scope did not close") + assert alive(child.pid), "control did not retain the owner process" + assert not alive(compiler), "owner file and live process overrode closed kernel lock" + assert child.wait(timeout=5) != 0 + print("PASS: releasing owner lock revokes helpers while owner process is still alive") + finally: + case.close() + + +def blocked_writer(root, binary, library, restore): + case = Case(root, "restore-barrier" if restore else "settle-barrier", binary, + MCPP_TEST_LIFETIME_BLOCK="1", LD_PRELOAD=str(library)) + try: + (case.out / "published.gcm").write_text("same BMI") + # Make the source newer than the previously published BMI. + time.sleep(0.02) + (case.out / "source").touch() + old = case.start(delay=20, MCPP_TEST_LIFETIME_FAIL="19" if restore else "0") + case.ready(old) + until(lambda: (case.out / "writer.entered").exists(), "phase1 did not reach controlled rename") + compiler = case.compiler() + until(lambda: not alive(compiler), "compiler still alive at state-writer control") + old.send_signal(signal.SIGINT) + new = case.start(delay=30, LD_PRELOAD="", MCPP_TEST_LIFETIME_BLOCK="") + time.sleep(0.15) + assert integer(case.out / "root.ready") != new.pid, "new generation bypassed old phase1 writer" + (case.out / "writer.release").write_text("go") + case.ready(new) + case.success(new) + assert not (case.out / "published.gcm.bak").exists() + assert (case.out / "unit.rc").read_text().strip() == "0" + print("PASS: old phase1 " + ("restore" if restore else "settle") + " writes block next generation after compiler permit release") + finally: + (case.out / "writer.release").write_text("go") + case.close() + + +def invalid_owner(root, binary): + for identity in ("0 0", f"{os.getpid()} 0", f"{os.getpid()} 1"): + case = Case(root, "invalid-" + identity.replace(" ", "-"), binary) + try: + generation = case.out / "forged-run" + generation.mkdir() + (generation / "owner").write_text("mcpp-compiler-run-v1\n" + identity + "\n") + env = dict(case.env, MCPP_TEST_LIFETIME_ROLE="phase", MCPP_COMPILER_RUN=str(generation)) + with (generation / "owner.lock").open("a+") as lock: + fcntl.lockf(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + result = subprocess.run([str(binary)], cwd=case.out, env=env, capture_output=True, timeout=5) + assert result.returncode == 130, "invalid owner identity was accepted" + assert not (case.out / "compiler.ready").exists() + assert not (case.out / "unit.bmi").exists(), "late rejected phase1 wrote state" + finally: + case.close() + print("PASS: missing birth and reused PID identities fail closed before state writes") + + +def late_registration(root, binary, supervisor): + case = Case(root, "late-supervisor" if supervisor else "late-phase1", binary) + delayed = None + try: + old = case.start() + case.ready(old) + case.compiler() + generation = next((case.out.parents[2] / ".mcpp/compiler-runs").glob("*/run-*")) + gate = case.out / "late.release" + command = [str(binary)] + if supervisor: + command += ["bmi-supervise", "--slot", "unit", "--sem", "permits", "--cap", "1", + "--object", "module.o", "--command-file", "compile.command"] + env = dict(case.env, MCPP_TEST_LIFETIME_ROLE="phase", MCPP_COMPILER_RUN=str(generation)) + waiter = ("import os,sys,time;from pathlib import Path;" + "gate=Path(sys.argv[1]);\nwhile not gate.exists():time.sleep(.01)\n" + "os.execv(sys.argv[2],sys.argv[2:])") + delayed = subprocess.Popen(["python3", "-c", waiter, str(gate), *command], + cwd=case.out, env=env, start_new_session=True, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + old.send_signal(signal.SIGINT) + new = case.start(delay=30) + case.ready(new) + case.success(new) + names = ("published.gcm", "module.o", "unit.rc", "unit.bmi", "compiler.runs") + before = {name: ((case.out / name).read_bytes(), (case.out / name).stat().st_mtime_ns) + for name in names} + gate.write_text("go") + assert delayed.wait(timeout=5) == 130, "late old helper entered the new run" + after = {name: ((case.out / name).read_bytes(), (case.out / name).stat().st_mtime_ns) + for name in names} + assert before == after, "late helper changed new-generation artifacts or status" + assert not (case.out / "published.gcm.bak").exists() + print("PASS: unregistered old " + ("supervisor" if supervisor else "phase1") + + " cannot mutate new generation") + finally: + if delayed is not None and delayed.poll() is None: + delayed.kill() + delayed.wait() + case.close() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("binary", type=Path) + parser.add_argument("fixture", type=Path) + args = parser.parse_args() + assert shutil.which("ninja") and shutil.which("setsid"), "Ninja and setsid required" + # Extend only the phase1 rename window; the actual C++ helper remains intact. + source = args.fixture / "delay-writer.c" + source.write_text(r'''#define _GNU_SOURCE +#include +#include +#include +#include +#include +int rename(const char *from,const char *to) { + static int (*next)(const char*,const char*); if(!next) next=dlsym(RTLD_NEXT,"rename"); + if(getenv("MCPP_TEST_LIFETIME_BLOCK") && strstr(from,"published.gcm.bak")) { + FILE *f=fopen("writer.entered","w"); if(f){fputs("entered",f);fclose(f);} + for(int i=0;i<3000 && access("writer.release",F_OK);++i) usleep(10000); + } + return next(from,to); +} +''') + library = args.fixture / "delay-writer.so" + subprocess.run(["/usr/bin/cc", "-shared", "-fPIC", str(source), "-o", str(library), "-ldl"], check=True, timeout=20) + invalid_owner(args.fixture, args.binary) + for kind in ("module", "ordinary"): + for sig in (signal.SIGINT, signal.SIGKILL): + cancellation(args.fixture, args.binary, sig, kind) + cancellation(args.fixture, args.binary, signal.SIGKILL, "module", linked=True) + owner_closure(args.fixture, args.binary) + for restore in (False, True): + blocked_writer(args.fixture, args.binary, library, restore) + for supervisor in (False, True): + late_registration(args.fixture, args.binary, supervisor) + + +if __name__ == "__main__": + main() diff --git a/tests/e2e/helpers/check_shared_compiler_cap.py b/tests/e2e/helpers/check_shared_compiler_cap.py index c66182e24..7a547ed82 100644 --- a/tests/e2e/helpers/check_shared_compiler_cap.py +++ b/tests/e2e/helpers/check_shared_compiler_cap.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 """Check actual GCC/Ninja mixed compilation and the shared permit lifetime.""" import argparse +import fcntl import json import os from pathlib import Path @@ -25,10 +26,22 @@ def state(path): return json.loads(path.read_text()) if path.exists() else {"active": [], "peak": 0, "calls": 0} +def permits_free(semaphore): + for entry in semaphore.iterdir(): + if not entry.is_file(): + return False + try: + with entry.open("r+b") as lock: + fcntl.lockf(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + return False + return True + + def settled(path, semaphore): deadline = time.monotonic() + 5 while time.monotonic() < deadline: - if not state(path)["active"] and not list(semaphore.iterdir()): + if not state(path)["active"] and permits_free(semaphore): return time.sleep(0.02) raise AssertionError("compilers or permits survived the completed workload") @@ -144,9 +157,9 @@ def check_cancellation(mcpp, root): while alive(pid) and time.monotonic() < deadline: time.sleep(0.01) assert not alive(pid), "cancellation left the compiler alive" - assert not list(semaphore.iterdir()), "cancellation retained its compiler permit" + assert permits_free(semaphore), "cancellation retained its compiler permit" result = subprocess.run([*command, sys.executable, "-c", "raise SystemExit(7)"], cwd=root) - assert result.returncode == 7 and not list(semaphore.iterdir()), "failure lost its status or leaked a permit" + assert result.returncode == 7 and permits_free(semaphore), "failure lost its status or leaked a permit" words = ["space value", 'literal "quote"', "back\\slash", "$literal", "", "Ω", "--system", "--home", "--deny", "--toolchain", "-j200", "--sem", "--cap"] result = subprocess.run([*command, sys.executable, "-c", "import json,sys;print(json.dumps(sys.argv[1:]))", *words], diff --git a/tests/unit/test_detach_status.cpp b/tests/unit/test_detach_status.cpp index 79fc0583f..f4a6b7bfa 100644 --- a/tests/unit/test_detach_status.cpp +++ b/tests/unit/test_detach_status.cpp @@ -2,6 +2,11 @@ import std; import mcpp.build.schedule.detach_codegen; +import mcpp.run; +import mcpp.invocation; +import mcpplibs.os.fs; +import mcpplibs.os.platform; +import mcpplibs.os.process; // A detached compile reports its status in `.rc`, which appears by a // rename once the number is written. One that holds no number is a write that @@ -45,7 +50,9 @@ TEST(DetachStatus, OrdinaryCompilerStartFailureReleasesItsPermit) { Slot s; const auto semaphore = s.dir / "permits"; EXPECT_EQ(detach::run_capped({(s.dir / "missing-compiler").string()}, semaphore, 1), 127); - EXPECT_TRUE(fs::is_empty(semaphore)); + auto free = mcpplibs::os::fs::lock::acquire(semaphore / "slot-0.lock", + mcpplibs::os::fs::lock::kind::exclusive, false); + EXPECT_TRUE(free); } TEST(DetachStatus, AnUnavailableSemaphoreRefusesUnboundedCompilation) { @@ -55,3 +62,119 @@ TEST(DetachStatus, AnUnavailableSemaphoreRefusesUnboundedCompilation) { EXPECT_NE(detach::run_capped({"unused-compiler"}, semaphore, 1), 0); EXPECT_TRUE(fs::is_regular_file(semaphore)); } + + +TEST(DetachStatus, KernelSlotsDistinguishContentionFromFailure) { + Slot s; + const auto path = s.dir / "slot.lock"; + { + auto held = mcpplibs::os::fs::lock::acquire(path, + mcpplibs::os::fs::lock::kind::exclusive, false); + ASSERT_TRUE(held); + auto busy = mcpplibs::os::fs::lock::acquire(path, + mcpplibs::os::fs::lock::kind::exclusive, false); + ASSERT_FALSE(busy); + EXPECT_EQ(busy.error().code, mcpplibs::os::errc::again); + } + EXPECT_TRUE(fs::exists(path)); + auto reused = mcpplibs::os::fs::lock::acquire(path, + mcpplibs::os::fs::lock::kind::exclusive, false); + EXPECT_TRUE(reused); +} + +namespace { +std::string lifetime_env(std::string_view key, std::string fallback = {}) { + return mcpp::invocation::get(key).value_or(std::move(fallback)); +} +int lifetime_driver() { + const auto role = lifetime_env("MCPP_TEST_LIFETIME_ROLE"); + const auto self = mcpp::invocation::self_exe_path(); + const auto argv = ::testing::internal::GetArgvs(); + if (argv.size() > 1 && argv[1] == "bmi-supervise") { + auto value = [&](std::string_view key) { + auto it = std::ranges::find(argv, key); + return it != argv.end() && ++it != argv.end() ? *it : std::string{}; + }; + const auto cmd = value("--command-file"); + std::ifstream in(cmd); + std::string command{std::istreambuf_iterator(in), {}}; + return detach::supervise(value("--slot"), value("--sem"), + std::stoi(value("--cap")), value("--object"), command); + } + if (role == "root") { + auto run = detach::RunScope::begin(lifetime_env("MCPP_TEST_LIFETIME_OUTPUT", fs::current_path().string())); + if (!run) { std::println(std::cerr, "{}", run.error()); return 2; } + std::ofstream("root.ready") << mcpplibs::os::process::self()->id; + mcpp::run::Env env; + run->environment(env); + // Test an owner whose process is alive but whose drive has finished. + std::jthread closer; + if (lifetime_env("MCPP_TEST_LIFETIME_CLOSE") == "1") { + closer = std::jthread([&] { + while (!fs::exists("compiler.ready")) + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + auto done = run->finish(); + std::ofstream("owner.closed") << (done ? "closed" : done.error()); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + }); + } + auto result = mcpp::run::execute({"ninja", "-j12"}, {.env = env}); + if (closer.joinable()) closer.join(); + auto done = run->finish(); + return done ? result.exit_code : 2; + } + if (role == "phase") { + std::ofstream("phase.pid") << mcpplibs::os::process::self()->id; + detach::CompileRequest request; + request.bmi = "published.gcm"; request.object = "module.o"; + request.slot = "unit"; request.self = self; + request.semaphore = "permits"; request.maxCompilers = 1; + request.commandFile = "compile.command"; + std::ifstream in(request.commandFile); + request.command.assign(std::istreambuf_iterator(in), {}); + return detach::compile_release_at_bmi(request); + } + if (role == "await") return detach::await_unit("unit", "module.o"); + if (role == "ordinary") + return detach::run_capped({"env", "MCPP_TEST_LIFETIME_ROLE=compiler", self.string()}, + "permits", 1, "ordinary.o"); + if (role == "compiler") { + const auto process = mcpplibs::os::process::self(); + std::ofstream("compiler.ready") << process->id << ' ' << process->started_ns; + std::ofstream("compiler.runs", std::ios::app) << process->id << '\n'; + const bool ordinary = lifetime_env("MCPP_TEST_LIFETIME_KIND") == "ordinary"; + const auto object = ordinary ? "ordinary.o" : "module.o"; + std::ofstream(object) << "partial"; + const int fail = std::stoi(lifetime_env("MCPP_TEST_LIFETIME_FAIL", "0")); + if (!ordinary && fail != 19) { + std::ofstream("published.gcm.new") << "same BMI"; + fs::rename("published.gcm.new", "published.gcm"); + } + std::this_thread::sleep_for(std::chrono::milliseconds( + std::stoi(lifetime_env("MCPP_TEST_LIFETIME_DELAY", "20000")))); + if (!fail) std::ofstream(object) << "complete"; + return fail; + } + return 2; +} +} // namespace + +// This copy is an actual detached supervisor/phase1/ordinary compiler under +// Ninja. It is selected through GTEST_FILTER inherited by detached children. +TEST(DetachLifetime, SubprocessDriver) { + if (lifetime_env("MCPP_TEST_LIFETIME_ROLE").empty()) GTEST_SKIP() << "subprocess driver"; + mcpp::run::install(); + const int rc = lifetime_driver(); + std::cout.flush(); std::cerr.flush(); + std::_Exit(rc); +} + +TEST(DetachLifetime, OwnerCancellationAndImmediateRestart) { + if constexpr (!mcpplibs::os::is_linux) GTEST_SKIP() << "Linux signal and LD_PRELOAD controls"; + Slot temporary; + const auto helper = fs::current_path() / "tests/e2e/helpers/check_owned_schedule_lifetime.py"; + ASSERT_TRUE(fs::exists(helper)); + const auto result = mcpp::run::execute({"python3", helper.string(), + mcpp::invocation::self_exe_path().string(), temporary.dir.string()}); + EXPECT_EQ(result.exit_code, 0) << result.output; +} diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 9c875de04..1c2a27f60 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -141,7 +141,7 @@ TEST(NinjaBackend, OrdinaryCompilersShareTheDetachedScheduleBound) { return start == std::string::npos ? std::string{} : text.substr(start, text.find("\n\n", start) - start); }; for (const auto name : {"cxx_module", "cxx_object", "c_object", "asm_object", "asm_object_raw", "nasm_object", "cxx_scan"}) - EXPECT_NE(rule(name).find("__compiler-cap --sem .mcpp-sched --cap 2 -- "), std::string::npos) << name; + EXPECT_NE(rule(name).find("__compiler-cap --sem .mcpp-sched --cap 2 --output $out -- "), std::string::npos) << name; EXPECT_EQ(rule("cxx_module_bmi").find("__compiler-cap"), std::string::npos) << "the BMI supervisor already holds a permit"; EXPECT_EQ(rule("cxx_module_obj").find("__compiler-cap"), std::string::npos) diff --git a/tests/unit/test_process_run_exec.cpp b/tests/unit/test_process_run_exec.cpp index 2abb0b583..2812a0540 100644 --- a/tests/unit/test_process_run_exec.cpp +++ b/tests/unit/test_process_run_exec.cpp @@ -19,6 +19,23 @@ namespace process = mcpp::run; // change mcpp's own environment — that mutation is exactly what leaked into // /bin/sh and crashed it on newer-glibc hosts. The addition is the started // program's alone; this invocation's statement is what it was. +TEST(RunExec, StopPredicateAppliesWithoutCapturedStreams) { + for (const auto streams : {process::Streams::inherit, process::Streams::discard}) { + const auto started = std::chrono::steady_clock::now(); + int checks = 0; + const auto result = process::execute({"/bin/sh", "-c", "sleep 60"}, { + .streams = streams, .total = std::chrono::seconds(3), + .should_stop = [&] { + ++checks; + return std::chrono::steady_clock::now() - started > std::chrono::milliseconds(50); + }}); + EXPECT_GT(checks, 0); + EXPECT_TRUE(result.timed_out); + EXPECT_NE(result.exit_code, 0); + EXPECT_LT(std::chrono::steady_clock::now() - started, std::chrono::seconds(1)); + } +} + TEST(RunExec, DoesNotMutateParentEnvironment) { mcpp::invocation::ScopedEnv leak("MCPP_TEST_LEAK", "sentinel"); // Use /bin/sh (present on Linux AND macOS) — /bin/true lives at From 728db275131bd96c182bf465219a25490a1d1803 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 03:21:19 +0900 Subject: [PATCH 11/40] fix: retain child bounds after captured stream EOF --- ...i-publication-and-compiler-cap-closeout.md | 22 +++++++ CHANGELOG.md | 3 +- src/run/run.cpp | 36 +++++------ tests/unit/test_process_run_exec.cpp | 63 +++++++++++++++++++ 4 files changed, 105 insertions(+), 19 deletions(-) diff --git a/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md index 4d109c249..b5907ce62 100644 --- a/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md +++ b/.agents/docs/2026-11-06-bmi-publication-and-compiler-cap-closeout.md @@ -199,3 +199,25 @@ small synthetic compiler. The separate GCC mixed workload retains the real compiler cap, late failure, recovery, argv and no-op assertions. Selected-object controls establish causality; final qualification remains complete current-source CI and independent ecosystem review. + +### Addendum, 2026-11-06: the final child wait retains its bounds + +A final launcher control closes both captured streams before the child sleeps. +The preceding correction still returned success after 2.002 seconds instead of +observing the fifty-millisecond stop request: stream EOF had led to an unbounded +child wait. The same boundary also bypassed an absolute deadline after EOF. + +One shared predicate now applies during output reads, remaining error reads and +the final child wait. It retains the original absolute deadline and stop request; +observed output retains its idle budget after EOF. Activity on a separately +captured error stream renews that observed-output budget. Inherited and discarded +streams retain no inferred idle observation. Child completion is checked during +the final bounded wait, preserving already completed child statuses. + +A finite matrix covers all six stream modes, ordinary and signalled exits, +completion with bounds configured, stop requests before and after one or both +streams close, absolute deadlines after closure, captured idle expiry and live +stderr after stdout closure. Its fifty-three cases pass, together with the +original launcher tests and the direct EOF regression. Top-level signal behavior +is additionally covered by the real-Ninja lifecycle control. No budget, compiler +argument or output stream is changed to make a cancellation succeed. diff --git a/CHANGELOG.md b/CHANGELOG.md index 65f60a8e0..d09d02374 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,7 +27,8 @@ another generation starts. Stable kernel permits retain live ownership across immediate rebuilds; runtime generations leave no-op build identities unchanged. - Child stop predicates apply with inherited and discarded streams, including - executions without an absolute deadline. + executions without an absolute deadline. Captured stream EOF retains stop, + absolute and observed-output idle bounds until the child exits. - Queued downloads begin their progress budget when a worker starts them, rather than inheriting time spent waiting for a slot. - Provider progress timeouts identify the stalled download and source attempt, diff --git a/src/run/run.cpp b/src/run/run.cpp index d6e11dbd0..a77e26a6c 100644 --- a/src/run/run.cpp +++ b/src/run/run.cpp @@ -241,6 +241,14 @@ Outcome launch(const Spec& s) { auto lastOutput = start; bool stop = false; const bool keepErr = s.out == Output::stdout_err_kept; + const bool captured = s.out == Output::merged || s.out == Output::stdout_only + || s.out == Output::stdout_err_shown || keepErr; + auto should_stop = [&] { + const auto now = clock::now(); + return (s.total.count() > 0 && now - start >= s.total) + || (captured && s.idle.count() > 0 && now - lastOutput >= s.idle) + || (s.shouldStop && *s.shouldStop && (*s.shouldStop)()); + }; bool errDone = !keepErr; // The error stream, read as it arrives so that a program writing much of // it is never stopped on a full pipe while its output is read. @@ -250,17 +258,15 @@ Outcome launch(const Spec& s) { if (!e) { errDone = true; return; } if (!e->has_value()) return; if ((*e)->empty()) { errDone = true; return; } + lastOutput = clock::now(); out.error_output += **e; }; - if (s.out == Output::merged || s.out == Output::stdout_only || s.out == Output::stdout_err_shown - || keepErr) { + if (captured) { std::string acc; std::optional endedAt; for (;;) { const auto now = clock::now(); - if ((s.total.count() > 0 && now - start >= s.total) - || (s.idle.count() > 0 && now - lastOutput >= s.idle) - || (s.shouldStop && *s.shouldStop && (*s.shouldStop)())) { + if (should_stop()) { stop = true; break; } @@ -290,7 +296,7 @@ Outcome launch(const Spec& s) { // bounds allow. while (!stop && !errDone) { const auto now = clock::now(); - if (s.total.count() > 0 && now - start >= s.total) { stop = true; break; } + if (should_stop()) { stop = true; break; } if (!endedAt) { if (auto w = p->try_wait(); w && w->has_value()) endedAt = now; } else if (now - *endedAt > std::chrono::milliseconds(500)) { @@ -299,22 +305,16 @@ Outcome launch(const Spec& s) { } drain_error(std::chrono::milliseconds(100)); } - } else if (s.shouldStop && *s.shouldStop) { - // Inherited/discarded streams have no read loop. Cancellation still - // belongs to the invocation, so poll the child without changing its - // streams or waiting for a separate absolute deadline. + } + if (!stop && (s.total.count() > 0 || (captured && s.idle.count() > 0) + || (s.shouldStop && *s.shouldStop))) { + // Stream EOF is independent of child completion. Keep the same bounds + // through the final wait, including inherited and discarded streams. for (;;) { - if ((*s.shouldStop)() - || (s.total.count() > 0 && clock::now() - start >= s.total)) { - stop = true; - break; - } auto waited = p->wait_for(std::chrono::milliseconds(20)); if (!waited || waited->has_value()) break; + if (should_stop()) { stop = true; break; } } - } else if (s.total.count() > 0) { - auto w = p->wait_for(s.total); - if (w && !w->has_value()) stop = true; } if (stop) { p->end_unit(); diff --git a/tests/unit/test_process_run_exec.cpp b/tests/unit/test_process_run_exec.cpp index 2812a0540..d8fca99a5 100644 --- a/tests/unit/test_process_run_exec.cpp +++ b/tests/unit/test_process_run_exec.cpp @@ -19,6 +19,69 @@ namespace process = mcpp::run; // change mcpp's own environment — that mutation is exactly what leaked into // /bin/sh and crashed it on newer-glibc hosts. The addition is the started // program's alone; this invocation's statement is what it was. +TEST(RunExec, StreamCompletionAndChildWaitKeepTheirBounds) { + struct Case { + const char* name; + const char* command; + int total = 0; + int idle = 0; + bool stop = false; + int exit = -1; + }; + const std::vector cases{ + {"exit", "exit 7", 0, 0, false, 7}, + {"signal", "kill -TERM $$", 0, 0, false, 143}, + {"already-exited-bounded", "exit 7", 3000, 0, true, 7}, + {"open-stop", "sleep 2", 0, 0, true}, + {"both-EOF-stop", "exec 1>&- 2>&-; sleep 2", 0, 0, true}, + {"stdout-EOF-stop", "exec 1>&-; sleep 2", 0, 0, true}, + {"both-EOF-total", "exec 1>&- 2>&-; sleep 2", 50}, + {"stdout-EOF-total", "exec 1>&-; sleep 2", 50}, + }; + auto exercise = [](process::Streams streams, const Case& test) { + SCOPED_TRACE(std::format("stream={} case={}", static_cast(streams), test.name)); + const auto started = std::chrono::steady_clock::now(); + process::Options options; + options.streams = streams; + options.total = std::chrono::milliseconds(test.total); + options.idle = std::chrono::milliseconds(test.idle); + if (test.stop) options.should_stop = [&] { + return std::chrono::steady_clock::now() - started > std::chrono::milliseconds(50); + }; + const auto result = process::execute({"/bin/sh", "-c", test.command}, options); + if (test.exit >= 0) { + EXPECT_EQ(result.exit_code, test.exit); + EXPECT_FALSE(result.timed_out); + } else { + EXPECT_NE(result.exit_code, 0); + EXPECT_TRUE(result.timed_out); + } + EXPECT_LT(std::chrono::steady_clock::now() - started, std::chrono::seconds(1)); + }; + for (const auto streams : {process::Streams::inherit, process::Streams::discard, + process::Streams::merged, process::Streams::output, + process::Streams::output_error_shown, process::Streams::output_error_kept}) { + for (const auto& test : cases) exercise(streams, test); + if (streams != process::Streams::inherit && streams != process::Streams::discard) + exercise(streams, {"both-EOF-idle", "exec 1>&- 2>&-; sleep 2", 0, 50}); + } + exercise(process::Streams::output_error_kept, + {"stderr-activity", "exec 1>&-; for i in 1 2 3 4 5 6 7 8; do printf x >&2; sleep .03; done; exit 7", + 0, 100, false, 7}); +} + +TEST(RunExec, StopPredicateAppliesAfterCapturedStreamsClose) { + const auto started = std::chrono::steady_clock::now(); + const auto result = process::execute({"/bin/sh", "-c", "exec 1>&- 2>&-; sleep 2"}, { + .streams = process::Streams::merged, + .should_stop = [&] { + return std::chrono::steady_clock::now() - started > std::chrono::milliseconds(50); + }}); + EXPECT_TRUE(result.timed_out); + EXPECT_NE(result.exit_code, 0); + EXPECT_LT(std::chrono::steady_clock::now() - started, std::chrono::seconds(1)); +} + TEST(RunExec, StopPredicateAppliesWithoutCapturedStreams) { for (const auto streams : {process::Streams::inherit, process::Streams::discard}) { const auto started = std::chrono::steady_clock::now(); From dc5e75ebcc258785d0a0ca2e11e31c784caaabf2 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 07:40:23 +0900 Subject: [PATCH 12/40] fix: recover compiler attempts only after all writers retire --- .../2026-11-06-compiler-attempt-recovery.md | 189 ++++++ CHANGELOG.md | 8 + docs/04-mcpp-toml.md | 6 +- docs/specs/concurrency.md | 42 +- docs/zh/04-mcpp-toml.md | 5 +- src/build/execute.cppm | 10 +- src/build/ninja_backend.cppm | 30 +- src/build/schedule/compiler_transaction.cpp | 97 ++++ src/build/schedule/compiler_transaction.cppm | 539 ++++++++++++++++++ src/build/schedule/detach_codegen.cppm | 301 ++++++++-- src/cli.cppm | 6 +- src/run.cppm | 4 + src/run/run.cpp | 52 +- tests/e2e/234_bmi_schedule_on.sh | 4 + .../helpers/check_cached_object_recovery.py | 59 ++ .../check_compiler_transaction_faults.py | 191 +++++++ .../helpers/check_owned_schedule_lifetime.py | 147 +++-- .../e2e/helpers/check_shared_compiler_cap.py | 69 ++- tests/e2e/helpers/record_close_failure.c | 27 + tests/unit/test_compiler_transaction.cpp | 222 ++++++++ tests/unit/test_detach_status.cpp | 52 +- tests/unit/test_ninja_backend.cpp | 2 +- tests/unit/test_process_run_exec.cpp | 87 ++- 23 files changed, 1995 insertions(+), 154 deletions(-) create mode 100644 .agents/docs/2026-11-06-compiler-attempt-recovery.md create mode 100644 src/build/schedule/compiler_transaction.cpp create mode 100644 src/build/schedule/compiler_transaction.cppm create mode 100644 tests/e2e/helpers/check_cached_object_recovery.py create mode 100644 tests/e2e/helpers/check_compiler_transaction_faults.py create mode 100644 tests/e2e/helpers/record_close_failure.c create mode 100644 tests/unit/test_compiler_transaction.cpp diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md new file mode 100644 index 000000000..fd254d24c --- /dev/null +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -0,0 +1,189 @@ +# Compiler attempt lifetime and output recovery + +Status: source implementation and focused acceptance in progress. This record +supplements the release candidate's compiler scheduling corrections. Native +source-group API checks do not qualify core or the released dependency group. + +## Problem and responsibility + +A compiler driver may start a descendant that continues writing after the +supervisor and directly bound driver have ended. Releasing a kernel permit +when its holder dies therefore does not prove that its compiler unit is quiet. +A separate failure occurs after early BMI publication: Ninja has recorded the +BMI edge while the object/status never completes, so a later build can enter +an orphan await instead of recompiling. + +Core owns the attempt journal, concurrency reservation and recovery of its +outputs. OS owns declared units, sealed admission and observation through an +original native reference. OpenKal and the native backends own the kernel +identity and completion semantics. Plugins do not manage process units. The +core implementation uses those generic capabilities and does not inspect +native process identifiers, process groups or platform job handles. + +## Runtime protocol + +One build drive owns a generation and an actor barrier. Generation identity is +runtime state; it changes neither build identity nor Ninja command hashes. +Phase one, ordinary wrappers, supervisors and await helpers acquire actor +leases before their final owner check. Recovery waits for every actor, including +phase-one BMI settling and status writers, rather than only compiler permits. + +Each attempt publishes a checksummed prepared record before modifying outputs. +It resolves parent directories in the original execution context and records +absolute BMI, object, dependency and status identities. A leaf symlink is not +resolved into an unrelated output target. A previous BMI is copied into a +protected baseline; its content identity is certified after the copy completes. +Separate immutable stage records avoid competing writers replacing one record. + +A helper acquires a kernel slot, declares its execution unit, publishes that +unit and its logical slot reservation, and registers with the live owner. The +owner retains the original native observation reference before acknowledging +registration over existing directory-local blocking IPC. Only then may the +helper publish admission and start the compiler. Record, registration or ACK +failure does not start an unregistered compiler. + +An unlocked kernel slot whose reservation remains occupied or unknown is not +available. A hard-killed supervisor therefore cannot admit a replacement while +its compiler descendant continues writing. The live owner's listener also +notices controller loss, so an orphan descendant holding Ninja's output pipe +does not leave the build waiting indefinitely for EOF. + +After the direct compiler exits, retirement seals admission and establishes +that no external writer remains. On systems that permit leaving a unit, the +helper uses the build owner's full program identity as a continuation; the +short-lived BMI publisher is insufficient. A dead, reused or inaccessible +continuation, including a different native session, yields unknown. Retirement +does not grant termination or admission authority over the continuation unit. +The helper publishes its quiet-unit certificate before atomic status and +releases its logical reservation only after this barrier. + +## Recovery and boundaries + +Recovery follows the actor barrier and requires either a verified retirement +certificate, positive native emptiness, or proof that admission never occurred. +A successful status also needs matching output identities. An incomplete early +BMI publication is removed to force a new BMI edge; its verified prior baseline +is retained with a certificate for that retry's equivalence/failure handling. +A successful cached object carries a quiet completion proof so a later missing +object can invalidate the BMI without inferring unit death from leader absence. + +Malformed records, invalid stage order, write refusal, occupied units and +unknown observation retain reservations and evidence. Recovery is repeatable +between file operations. If every original native reference holder is forcibly +lost before retirement can be certified, Windows may no longer identify the +original job. That case remains a bounded refusal and is not automatic recovery. +A C observation job has an observe/close caller precondition; the OS observer +class hides every native control path. + +## Validation layers + +The OpenKal actual-source API workflow +[42110324914](https://github.com/mcpplibs/openkal/actions/runs/42110324914) +and OS actual-source lifecycle workflow +[42110397318](https://github.com/mcpplibs/os/actions/runs/42110397318) +passed on Linux, macOS and Windows. They cover declared-unit registration, +blocking ACK failure/timeout, driver descendants, sealed retirement, retained +observation, an expired publisher, forged continuation birth and observer +control rejection. Linux additionally kills the continuation at native migration +entry and immediately after successful migration; both report unknown. + +Each OS native run recorded two batches of eight paired startup samples. The +median paired ACK increments were 0.175 ms on Linux, 0.409 ms on macOS and +2.508 ms on Windows. Listener start/wake/close medians were 0.177, 0.289 and +0.721 ms respectively. All samples, including negative deltas and tail values, +are retained in the run artifacts. These measurements describe the source +consumer, rather than a complete build or core no-op benchmark. + +Core acceptance includes checksummed records, baseline staging/corruption, +atomic status gaps, late failures, missing cached objects, repeated recovery, +write refusal and changed directory symlinks. The actual helper matrix also +covers supervisor-only hard loss with a still-writing descendant, a free kernel +slot with a retained logical reservation, owner cancellation and immediate +restart, phase-one state writers, and normal/quiet cached replay. Syscall-boundary +fault controls belong to the test driver and add no product fault hooks. + +Final acceptance still requires the completed core fault matrix, actual GCC +mixed-cap/replay controls, updated user/spec documentation, the coherent public +OpenKal/backend/runtime/platform/OS version group and full core CI. Core receives +an independent ecosystem review after those checks and before merge/release. + +## 2026-11-06 cached object admission closeout + +A completed generation already leaves a quiet completion certificate beside +each module's object slot. A later missing object originally made cached await +remove its BMI and fail, requiring a second invocation. The corrected boundary +reuses those existing certificates under the configuration's object tree after +old owners, actors and units are quiet and before any new Ninja reader starts. +Only a certificate's own declared object is checked. A missing object invokes +the existing certified-baseline invalidation; Ninja then rebuilds in its normal +graph on that same invocation. There is no additional success index, source +dirty model, dry run or alternate compiler entry. + +Certificate names are bound to their slot/object paths. Absolute BMI scope, +field values, checksums, regular certificate files and unchanged BMI parents +are checked before invalidation. Symlinked state does not authorize recovery. +The checksum detects accidental corruption; it is not authentication against +a user who can already replace the build tool and its outputs. + +The transaction suite passes 19 cases. The actual fresh engine recovers one +and two deleted completed module objects in a single build and preserves the +following no-op. Actual CLI controls reject a corrupt certificate before Ninja +starts and reject a directory symlink without changing an unrelated artifact. +The implementation's 30 warm admission samples have medians of 0.057 ms for +two real certificates, 2.200 ms for the core object tree's 1908 entries without +these new certificates, and 1.089 ms for 200 format-correct synthetic records +with real object existence checks. The latter is a scale fixture, not evidence +that those synthetic records came from real retirements. + +A separate paired protocol control preserves the actual GCC argument multiset, +cap two and Ninja twelve. Eight no-op pairs have a median paired difference of +-0.041 ms. Five small cold-build pairs have a median new/old ratio of 1.101 and +a median paired increase of 250 ms. The comparison includes the complete old +protocol/OS and the current source group, and its controlled compiler driver +sleeps 150 ms after real compilation. It does not isolate IPC, establish an +absence of build overhead, or extrapolate to large module workloads. + +## 2026-11-06 process-visible publication scope + +The recovery contract covers owner, supervisor and driver process loss while +the filesystem continues operating. It requires complete atomic visibility, +checked write/close outcomes and rejection of damaged records. It does not +promise persistence across power loss or a filesystem/storage service failure. +The OS library's existing durable `write_atomic` operation remains unchanged. + +The runtime publisher creates an exclusive same-directory temporary, writes +and flushes the user-space stream, checks its explicit close, then uses the +existing generic replacing rename. Failed write/close/rename does not publish +the final name or acknowledge admission. It removes only the temporary it +created. This composition preserves process-loss visibility without flushing +each short-lived runtime record to storage. + +A controlled libc close returns EIO after closing the actual temporary; the +publisher rejects the replacement, preserves the previous record and removes +its temporary. The transaction suite passes 20 functional cases plus one +subprocess-driver skip. The same candidate passes the actual lifecycle and +syscall-boundary fault controls, including occupied descendants, reservation +retention and restart. + +A frozen current-protocol durable binary/driver is the comparison baseline. +Five paired small cold builds preserve source, actual GCC argument multiset, +cap two and Ninja twelve; the candidate/durable median ratio is 0.900, with a +median paired reduction of 232 ms. Eight no-op pairs have a median paired +reduction of 1.639 ms. These samples use blocking child waits, a timeout guard +and alternating order. The earlier timeout-based sampler rounded short waits +and its small no-op delta was within that measurement's noise. + +A corresponding syscall control records 200 fsync calls and 554 ms of summed +call wall-time for the durable protocol, and zero fsync calls for the candidate. +The sampled process wall-times are 2.868 and 2.668 seconds. Summed syscall +wall-time includes concurrent threads and is not the build wall-time. These +small-workload results support this publication choice; they are not an +extrapolation to large module builds or a general benchmark of IPC. + +The capacity counter also distinguishes a forcibly ended driver from live +activity. It records each original pid/birth pair, preserves entries whose +finally handler did not execute, and counts only the original live processes. +A deterministic cancellation control kills a real wrapper, retains that +history, then starts a command returning seven without inflating peak one. +The prior false failure and its dead-process/empty-reservation evidence remain +part of the experiment record. diff --git a/CHANGELOG.md b/CHANGELOG.md index d09d02374..49c4e4474 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,14 @@ - Cancelled detached builds stop their compilers and finish state cleanup before another generation starts. Stable kernel permits retain live ownership across immediate rebuilds; runtime generations leave no-op build identities unchanged. +- Compiler units retain their concurrency reservation after a supervisor is + forcibly ended. Recovery waits for descendants and final state writers; + occupied or unknown units retain their outputs and fail within a bound. +- Missing completed module objects are recovered in the same build before Ninja + reads their BMIs. Corrupt completion evidence and redirected cache state refuse + recovery without authorizing changes to unrelated outputs. +- Captured idle bounds consume queued progress from both streams before + classifying silence, including progress buffered during a slow line callback. - Child stop predicates apply with inherited and discarded streams, including executions without an absolute deadline. Captured stream EOF retains stop, absolute and observed-output idle bounds until the child exits. diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index c8dc29009..b02ca68f1 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1053,7 +1053,11 @@ generation share the same total `jobs` limit (2026.11.5.2+). Additional scheduling and join tasks do not increase the number of compiler processes. Cancelling the build ends its ordinary and detached compilers. An immediately following build waits for their cleanup before starting another compiler; -partial outputs are rebuilt. An unchanged build still performs no compilation. +partial outputs are rebuilt. A missing object from a completed module is +recreated by that build. An unchanged build still performs no compilation. +If a forcibly ended build leaves a compiler unit whose completion cannot be +verified, the next build reports an occupied or unknown unit and retains its +outputs; it does not start a replacement compiler in that slot. | value | | |---|---| diff --git a/docs/specs/concurrency.md b/docs/specs/concurrency.md index 59b3c65ef..3030c33d6 100644 --- a/docs/specs/concurrency.md +++ b/docs/specs/concurrency.md @@ -4,10 +4,10 @@ |---|---| | 规范编号 | SPEC-011 | | 标题 | 同一工程上的并发运行:构建目录锁、原子发布、乐观写入与取消 | -| 状态 | 评审中 v1.0 | -| 版本 | 1.0 | -| 最后修改 | 2026-10-26 | -| 对应实现 | mcpp >= 2026.10.26.1 | +| 状态 | 评审中 v1.1 | +| 版本 | 1.1 | +| 最后修改 | 2026-11-06 | +| 对应实现 | 基础保证: mcpp >= 2026.10.26.1; §5.1: mcpp >= 2026.11.5.2 | | 相关设计文档 | `.agents/docs/2026-10-26-control-contract-runtime-and-ide-ecosystem-design.md` §3、§4、§5.4 | | 相关 issue | #839 | | 使用文档 | [50 — 机器可读输出](../zh/50-machine-output.md) §9、§11 | @@ -79,13 +79,41 @@ CI。在 2026.10.24.1 上,同一配置的两次冷构建相隔 0.1–0.5 s 启 1. SIGINT、SIGTERM、SIGHUP 结束修改段中的运行时,mcpp **必须**先结束它启动的进程组, 再写下 `cancelled` 的结果记录,然后按信号的默认动作结束(退出状态见 - [SPEC-003](exit-codes.md) §3.7)。记录的这一形态在修改段开始时预先生成,信号处理器 - 只调用异步信号安全的 `open`、`write`、`close`、`rename`。 + [SPEC-003](exit-codes.md) §3.7)。记录的这一形态在修改段开始时预先生成。 + 结束请求由 OS 库接收;运行时的等待线程结束执行单元并发布取消记录。 2. 锁由进程结束释放。构建目录保持一致:ninja 的日志只记录完成的步骤,被中断的步骤在 下一次构建中重做。 3. SIGKILL 不经过处理器;`status` 为 `running` 而 `pid` 已不存在的记录,读者**应当** 读作"被强制结束"。 +### 5.1 分离编译的执行单元与恢复 + +以下保证适用于 GCC 的 `bmi_schedule = "on"`（2026.11.5.2+）: + +1. 普通编译与分离代码生成**必须**共用实际的 `jobs` 总上限。等待任务与调度任务 + **禁止**作为编译名额的替代。内核锁已释放而原编译单元仍被占用或状态未知时, + 对应名额**禁止**重新分配。 +2. 真实编译器启动前,其控制程序**必须**发布输出身份、执行单元及名额保留记录, + 并取得活所有者已保留原生观察引用的确认。登记、写入或确认失败时**禁止**启动编译器。 +3. 直接编译器或控制程序结束**禁止**被解释为所有后代写入者已经结束。正常状态发布 + **必须**在封闭执行单元的接纳并确认外部成员已结束之后进行。 +4. 下一次构建恢复输出前**必须**等待旧代所有状态写入者及原编译单元。恢复使用 + 原执行上下文中确定的绝对输出身份,**禁止**依据恢复时的 cwd/home 或重定向后的 + 目标目录符号链接重新解释路径。运行代身份**禁止**进入构建指纹或 Ninja 命令身份。 +5. 未完成的早发布 BMI **必须**使下一次构建重新执行 BMI 边,同时保留经认证的 + 先前 BMI 作为恢复基线。成功状态**必须**与已经安静的执行单元及输出身份相符。 + 重复恢复**必须**保持安全。已认证完成的模块缺 object 时，**必须**在新 Ninja + 读取 BMI 之前失效对应 BMI，使该次构建正常重建。损坏证书、目录符号链接或 + 已重定向的 BMI 父目录**禁止**授权删除其他输出。损坏记录、非法阶段顺序或 + 写入失败**禁止**猜测成功。 +6. 原单元被占用或不能被确认时,恢复**必须**在有限等待后明确失败并保留名额及证据。 + 所有原生引用持有者在认证完成之前都被强制结束时,平台可能只能报告未知; + 此边界**不保证**自动恢复,**禁止**把名字消失或数值标识复用解释为空单元。 + +7. 本节的恢复范围是文件系统持续工作时的进程异常退出；**不保证**断电或存储服务 + 故障后的持久性。运行记录**必须**完整写入并检查关闭结果后才原子替换最终名字， + 确认消息**禁止**先于该替换。OS 库的通用持久化写入接口保持其既有语义。 + ## 6. 全局一侧 全局 `MCPP_HOME` 保持既有的细粒度保护:索引刷新使用非阻塞锁,BMI 缓存按条目加锁, @@ -99,3 +127,5 @@ CI。在 2026.10.24.1 上,同一配置的两次冷构建相隔 0.1–0.5 s 启 | 版本 | 日期 | 变更 | |---|---|---| | 1.0 | 2026-10-26 | 首版:三段、构建目录锁、原子发布、`mcpp.lock` 的乐观写入、取消(#839) | + +| 1.1 | 2026-11-06 | 分离编译的单元观察、共享名额、绝对输出事务与有限恢复边界;对齐取消请求的 OS 接收方式。 | diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 9934dab4c..30009a59b 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -975,7 +975,10 @@ bmi_schedule = "off" # auto (default, = off) | on | off 使用 GCC 且 `bmi_schedule = "on"` 时，普通编译与分离的代码生成共用同一个 `jobs` 总上限 （2026.11.5.2+）。额外的调度和等待任务不增加编译器进程数量。 取消构建会结束本轮普通编译器和分离的编译器。紧接着的构建等待它们完成清理后 -才启动新的编译器；未完成的输出会重新构建。输入不变的构建仍不进行编译。 +才启动新的编译器；未完成的输出会重新构建。已完成模块的 object 缺失时， +该次构建会重新生成它。输入和输出均不变的构建仍不进行编译。 +若被强制结束的构建留下了无法确认已经完成的编译单元，下一次构建报告该单元 +仍被占用或状态未知，并保留其输出；该名额不会启动替代编译器。 | 取值 | | |---|---| diff --git a/src/build/execute.cppm b/src/build/execute.cppm index eb34971c4..b2ae52b3a 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -2175,6 +2175,7 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, compilerRun->environment(childEnv); } + const auto compilerStopped=[&] {return compilerRun&&compilerRun->failed();}; auto t0 = std::chrono::steady_clock::now(); // capture_exec merges stderr into the captured output (replacing `2>&1`), // so is_stale_ninja_failure / filter_ninja_output still see ninja errors. @@ -2200,11 +2201,12 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, if (scanArgv) run = mcpp::build::run_ninja_reporting(*scanArgv, childEnv, std::chrono::milliseconds{0}, report, verbose, prefixes, - mcpp::build::progress::PassKind::Scan); + mcpp::build::progress::PassKind::Scan,compilerStopped); // A failed scan ends the build with its own output. if (run.exitCode == 0 && !run.timedOut) run = mcpp::build::run_ninja_reporting(argv, childEnv, std::chrono::milliseconds{0}, - report, verbose, prefixes); + report, verbose, prefixes, + mcpp::build::progress::PassKind::Work,compilerStopped); out = std::move(run.output); status = run.exitCode; reported = run.reported; @@ -2217,8 +2219,8 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, // (build progress design 2026-09-29, §6.4). childEnv.emplace_back(std::string(mcpp::build::progress::kStartsEnv), ""); mcpp::run::RunResult r; - if (scanArgv) r = mcpp::run::capture_exec(*scanArgv, childEnv); - if (r.exit_code == 0) r = mcpp::run::capture_exec(argv, childEnv); + if (scanArgv) r = mcpp::run::execute(*scanArgv,{.env=childEnv,.should_stop=compilerStopped}); + if (r.exit_code == 0) r = mcpp::run::execute(argv,{.env=childEnv,.should_stop=compilerStopped}); out = std::move(r.output); status = r.exit_code; } diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index b66d7cc75..adf995d76 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -124,7 +124,8 @@ NinjaRun run_ninja_reporting(const std::vector& argv, bool verbose, std::span commandPrefixes, mcpp::build::progress::PassKind kind = - mcpp::build::progress::PassKind::Work); + mcpp::build::progress::PassKind::Work, + const std::function& shouldStop = {}); // The goal of the scan pass (build wall-time plan, W2): the dyndep file of // every unit whose scan waits on no action. Present in build.ninja when the @@ -1385,7 +1386,8 @@ NinjaRun run_ninja_reporting(const std::vector& argv, mcpp::build::progress::Build& progress, bool verbose, std::span commandPrefixes, - mcpp::build::progress::PassKind kind) { + mcpp::build::progress::PassKind kind, + const std::function& shouldStop) { NinjaRun run; for (auto& kv : progress.environment()) env.push_back(std::move(kv)); progress.pass_begin(kind); @@ -1424,8 +1426,8 @@ NinjaRun run_ninja_reporting(const std::vector& argv, }; // A ninja that cannot be started is reported through `on_line`, into the // output the failure report reads. - run.exitCode = mcpp::run::stream_exec(argv, env, deadline, on_line, - &run.timedOut); + const auto result=mcpp::run::execute(argv,{.env=env,.total=deadline,.on_line=on_line,.should_stop=shouldStop}); + run.exitCode=result.exit_code;run.timedOut=result.timed_out; progress.pass_end(); return run; } @@ -1471,7 +1473,7 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // compilers must share the supervisors' semaphore rather than use that // larger slot count as their own concurrency bound. const std::string compilerPrefix = splitBmi - ? std::format("$mcpp __compiler-cap --sem .mcpp-sched --cap {} --output $out -- ", plan.scheduleCompilerCap) + ? std::format("$mcpp __compiler-cap --sem .mcpp-sched --cap {} --output $out --depfile $out.d -- ", plan.scheduleCompilerCap) : std::string{}; // The other split shape. Clang publishes no BMI early — it writes the BMI // at the end of the compile — so the GCC trick of releasing the file @@ -4889,6 +4891,7 @@ std::expected NinjaBackend::build(const BuildPlan& plan for (auto& ev : plan.toolchain.envOverrides) nenv.emplace_back(ev.key, ev.value); if (compilerRun) compilerRun->environment(nenv); + const auto compilerStopped=[&] {return compilerRun&&compilerRun->failed();}; // THE STAGED CACHE IS PLACED BY A PASS OF ITS OWN, before the graph that // reads it. ninja 1.12.1 loads a dyndep file whose producer is up to date @@ -4916,7 +4919,7 @@ std::expected NinjaBackend::build(const BuildPlan& plan std::string(kStagedCacheGoal)}; (void)run_ninja_reporting(pre, nenv, preDeadline, *report, opts.verbose, command_prefixes(flags, plan), - mcpp::build::progress::PassKind::Placement); + mcpp::build::progress::PassKind::Placement,compilerStopped); } else { std::vector pre{ninjaProgram, "--quiet", "-C", plan.outputDir.string(), std::string(kStagedCacheGoal)}; @@ -4926,7 +4929,7 @@ std::expected NinjaBackend::build(const BuildPlan& plan preEnv.emplace_back(std::string(mcpp::build::progress::kStartsEnv), ""); preEnv.emplace_back(std::string(mcpp::build::action_progress::kRootEnv), ""); preEnv.emplace_back(std::string(mcpp::build::action_progress::kFileEnv), ""); - (void)mcpp::run::capture_exec_deadline(pre, preEnv, preDeadline, nullptr); + (void)mcpp::run::execute(pre,{.env=preEnv,.total=preDeadline,.should_stop=compilerStopped}); } stage("ninja-staged-cache"); } @@ -4968,7 +4971,7 @@ std::expected NinjaBackend::build(const BuildPlan& plan if (report) { auto run = run_ninja_reporting(scan, nenv, scanDeadline, *report, opts.verbose, command_prefixes(flags, plan), - mcpp::build::progress::PassKind::Scan); + mcpp::build::progress::PassKind::Scan,compilerStopped); if (run.exitCode != 0 || run.timedOut) { out = std::move(run.output); ninjaExit = run.exitCode; @@ -4981,8 +4984,8 @@ std::expected NinjaBackend::build(const BuildPlan& plan scanEnv.emplace_back(std::string(mcpp::build::progress::kStartsEnv), ""); scanEnv.emplace_back(std::string(mcpp::build::action_progress::kRootEnv), ""); scanEnv.emplace_back(std::string(mcpp::build::action_progress::kFileEnv), ""); - auto cap = mcpp::run::capture_exec_deadline( - scan, scanEnv, scanDeadline, &buildTimedOut); + auto cap = mcpp::run::execute(scan,{.env=scanEnv,.total=scanDeadline,.should_stop=compilerStopped}); + buildTimedOut=cap.timed_out; if (cap.exit_code != 0 || buildTimedOut) { out = std::move(cap.output); ninjaExit = cap.exit_code; @@ -5000,7 +5003,8 @@ std::expected NinjaBackend::build(const BuildPlan& plan // `out`, `ninjaExit`, `buildTimedOut` and `reported` are the scan's. } else if (report) { auto run = run_ninja_reporting(nargv, nenv, deadline, *report, opts.verbose, - command_prefixes(flags, plan)); + command_prefixes(flags, plan), + mcpp::build::progress::PassKind::Work,compilerStopped); out = std::move(run.output); ninjaExit = run.exitCode; buildTimedOut = run.timedOut; @@ -5010,8 +5014,8 @@ std::expected NinjaBackend::build(const BuildPlan& plan quietEnv.emplace_back(std::string(mcpp::build::progress::kStartsEnv), ""); quietEnv.emplace_back(std::string(mcpp::build::action_progress::kRootEnv), ""); quietEnv.emplace_back(std::string(mcpp::build::action_progress::kFileEnv), ""); - auto cap = mcpp::run::capture_exec_deadline( - nargv, quietEnv, deadline, &buildTimedOut); + auto cap = mcpp::run::execute(nargv,{.env=quietEnv,.total=deadline,.should_stop=compilerStopped}); + buildTimedOut=cap.timed_out; out = std::move(cap.output); ninjaExit = cap.exit_code; } diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp new file mode 100644 index 000000000..6ccacafe4 --- /dev/null +++ b/src/build/schedule/compiler_transaction.cpp @@ -0,0 +1,97 @@ +module mcpp.build.schedule.compiler_transaction; +import std; + +namespace mcpp::build::schedule::transaction { + +bool replace_record(const fs::path& path,std::string_view bytes) { + // No process may observe a partial record. This protects process loss, + // not power loss; do not change the OS library's durable write_atomic. + static const auto who=os::process::self(); + static std::atomic sequence{0}; + if(!who||!who->id||!who->started_ns)return false; + const auto temporary=path.parent_path()/named(std::format(".mcpp-record-{}-{}-{}", + who->id,who->started_ns,sequence++)); + std::ofstream output(temporary,std::ios::binary|std::ios::out|std::ios::noreplace); + if(!output)return false; // Never remove a name this invocation did not create. + output.write(bytes.data(),static_cast(bytes.size())); + output.flush(); + const bool written=bool(output); + output.close(); + const bool closed=bool(output); + if(written&&closed&&os::fs::rename(temporary,path,true))return true; + std::error_code ec;fs::remove(temporary,ec);return false; +} +bool publish(const fs::path& path,std::string_view tag,const Fields& fields) { + return replace_record(path,pack(tag,fields)); +} +namespace { +std::expected admit_completed(const fs::path& certificate, + const fs::path& output_directory) { + auto record=read(certificate,"mcpp-completed-unit-v1"); + if(!record||!*record)return std::unexpected("cannot read a cached compiler completion certificate"); + const auto& v=**record; + if(v.size()!=6||!decimal(v[2])||!decimal(v[3]) + ||!decimal(v[4])||!decimal(v[5])) + return std::unexpected("invalid cached compiler completion fields"); + const auto name=spelling(certificate); + const auto slot=named(std::string_view(name).substr(0,name.size()-std::string_view(".completed").size())); + const auto object=named(std::string_view(name).substr(0,name.size()-std::string_view(".sched.completed").size())); + if(named(v[1])!=object)return std::unexpected("cached compiler certificate does not name its own object"); + const auto bmi=named(v[0]);const auto cache=output_directory/"gcm.cache"; + const auto relative=bmi.lexically_relative(cache); + if(!bmi.is_absolute()||bmi!=bmi.lexically_normal()||relative.empty()||relative.is_absolute() + ||*relative.begin()==".."||relative==".") + return std::unexpected("cached compiler certificate names a BMI outside this configuration"); + std::error_code ec; + const bool present=fs::exists(object,ec); + if(ec)return std::unexpected("cannot inspect a cached compiler object"); + if(present)return {}; + // Parent identity was fixed by the original attempt. A newly redirected + // cache directory cannot authorize touching a different BMI or baseline. + const auto current=output_identity(bmi); + if(!current||*current!=bmi)return std::unexpected("cached compiler BMI directory identity changed"); + const auto state=fs::symlink_status(bmi,ec); + if(ec&&ec!=std::errc::no_such_file_or_directory) + return std::unexpected("cannot inspect the cached compiler BMI"); + if(state.type()!=fs::file_type::not_found&&!fs::is_regular_file(state)) + return std::unexpected("cached compiler BMI is not a regular output"); + // Reuse the same certified-baseline invalidation as an exceptional join. + // Ninja has not evaluated this BMI edge yet, so it can rebuild normally. + auto invalidated=cached_result(slot,object); + if(!invalidated)return std::unexpected(invalidated.error()); + if(!*invalidated||**invalidated!=1)return std::unexpected("cached compiler completion changed during admission"); + return {}; +} +} + +std::expected admit_cached_outputs(const fs::path& output_directory) { + const auto objects=output_directory/"obj"; + std::error_code ec;const auto root=fs::symlink_status(objects,ec); + if(root.type()==fs::file_type::not_found)return {}; + if(ec||!fs::is_directory(root))return std::unexpected("cannot inspect the compiler object directory"); + fs::recursive_directory_iterator entries(objects,fs::directory_options::none,ec),end; + std::vector certificates; + if(ec)return std::unexpected("cannot enumerate cached compiler outputs"); + for(;entries!=end;entries.increment(ec)) { + if(ec)return std::unexpected("cannot enumerate cached compiler outputs"); + const auto path=entries->path(); + const bool certificate=spelling(path.filename()).ends_with(".sched.completed"); + const auto state=entries->symlink_status(ec); + if(ec)return std::unexpected("cannot inspect cached compiler state"); + if(fs::is_symlink(state)) { + if(certificate||entries->is_directory(ec)) + return std::unexpected("symlinked cached compiler state cannot authorize recovery"); + ec.clear();continue; + } + if(!certificate)continue; + if(!fs::is_regular_file(state))return std::unexpected("cached compiler certificate is not a regular file"); + certificates.push_back(path); + } + if(ec)return std::unexpected("cannot enumerate cached compiler outputs"); + // Invalidation removes its certificate. Finish enumeration first so those + // removals cannot change which other cached objects this admission checks. + for(const auto& path:certificates) + if(auto admitted=admit_completed(path,output_directory);!admitted)return admitted; + return {}; +} +} // namespace mcpp::build::schedule::transaction diff --git a/src/build/schedule/compiler_transaction.cppm b/src/build/schedule/compiler_transaction.cppm new file mode 100644 index 000000000..17431459c --- /dev/null +++ b/src/build/schedule/compiler_transaction.cppm @@ -0,0 +1,539 @@ +// Compiler attempts are runtime transactions. A free kernel lock is not proof +// that an earlier writer is gone; the runtime reservation names its unit. +export module mcpp.build.schedule.compiler_transaction; +import std; +import mcpplibs.os.fs; +import mcpplibs.os.process; +import mcpplibs.os.ipc; +import mcpplibs.os.dirs; +import mcpp.home; +import mcpp.build.stage; +import mcpp.toolchain.fingerprint; + +export namespace mcpp::build::schedule::transaction { +namespace fs = std::filesystem; +namespace os = mcpplibs::os; +using Fields = std::vector; +using Record = std::expected,std::string>; +using ObserverMap = std::map; + +template bool decimal(std::string_view text) { + T value{};const auto parsed=std::from_chars(text.data(),text.data()+text.size(),value); + return !text.empty()&&parsed.ec==std::errc{}&&parsed.ptr==text.data()+text.size(); +} +bool digest(std::string_view text) { + return text.size()==16&&std::ranges::all_of(text,[](unsigned char c) { + return (c>='0'&&c<='9')||(c>='a'&&c<='f'); + }); +} +bool output_stamp(std::string_view size,std::string_view time) { + return (size=="missing"&&time=="missing") + ||(decimal(size)&&decimal(time)); +} + +std::string pack(std::string_view tag, const Fields& fields) { + std::string body = std::string(tag) + "\n" + std::to_string(fields.size()) + "\n"; + for (const auto& field : fields) body += std::to_string(field.size()) + "\n" + field; + return body + "\n" + mcpp::toolchain::hash_string(body) + "\n"; +} +// Atomic visibility across process loss; storage durability is not promised. +bool replace_record(const fs::path& path,std::string_view bytes); +bool publish(const fs::path& path,std::string_view tag,const Fields& fields); +Record read(const fs::path& path, std::string_view tag) { + std::ifstream input(path,std::ios::binary); + if (!input) { + std::error_code ec; + if (!fs::exists(path,ec) && !ec) return std::optional{}; + return std::unexpected("cannot read compiler transaction record"); + } + std::string bytes(65537,'\0'); + input.read(bytes.data(),bytes.size());bytes.resize(static_cast(input.gcount())); + if(input.bad())return std::unexpected("cannot read compiler transaction record"); + if (bytes.size() < 18 || bytes.size() > 65536 || bytes[bytes.size()-18] != '\n' || bytes.back() != '\n') + return std::unexpected("corrupt compiler transaction record"); + const auto body = std::string_view(bytes).substr(0,bytes.size()-18); + if (mcpp::toolchain::hash_string(body) != std::string_view(bytes).substr(bytes.size()-17,16)) + return std::unexpected("compiler transaction checksum mismatch"); + std::size_t at=0; + auto word = [&]() -> std::optional { + const auto end=body.find('\n',at); if (end==body.npos) return {}; + const auto value=body.substr(at,end-at);at=end+1;return value; + }; + auto number = [&]() -> std::optional { + const auto text=word();if(!text)return {}; + std::size_t value=0;const auto result=std::from_chars(text->data(),text->data()+text->size(),value); + if(result.ec!=std::errc{}||result.ptr!=text->data()+text->size())return {};return value; + }; + const auto schema=word();const auto count=number(); + if(!schema||*schema!=tag||!count||*count>32)return std::unexpected("unknown compiler transaction schema"); + Fields fields; + for(std::size_t i=0;i<*count;++i) { + const auto n=number();if(!n||*n>body.size()-at)return std::unexpected("truncated compiler transaction record"); + fields.emplace_back(body.substr(at,*n));at+=*n; + } + if(at!=body.size())return std::unexpected("unexpected compiler transaction record data"); + return std::optional{std::move(fields)}; +} +std::expected output_identity(const fs::path& path) { + if(path.empty())return fs::path{}; + std::error_code ec;const auto absolute=fs::absolute(path,ec).lexically_normal(); + if(ec)return std::unexpected("cannot resolve a compiler output identity"); + const auto parent=fs::weakly_canonical(absolute.parent_path(),ec); + if(ec)return std::unexpected("cannot resolve a compiler output directory"); + return parent/absolute.filename(); +} +std::string spelling(const fs::path& path) { return mcpp::home::spelling(path); } +fs::path named(std::string_view path) { return mcpp::home::from_utf8(path); } +fs::path suffix(const fs::path& path,std::string_view tail) { return named(spelling(path)+std::string(tail)); } +std::optional identity(const fs::path& path) { + if(path.empty())return Fields{"", ""}; + std::error_code ec;const auto size=fs::file_size(path,ec);if(ec)return {}; + const auto time=fs::last_write_time(path,ec);if(ec)return {}; + return Fields{std::to_string(size),std::to_string(time.time_since_epoch().count())}; +} +bool copy_atomic(const fs::path& from,const fs::path& to) { + std::error_code ec;const auto temp=suffix(to,".mcpp-restore"); + fs::copy_file(from,temp,fs::copy_options::overwrite_existing,ec);if(ec)return false; + const auto time=fs::last_write_time(from,ec);if(ec)return false; + fs::last_write_time(temp,time,ec);if(ec)return false; + return bool(os::fs::rename(temp,to,true)); +} +struct Attempt { + fs::path directory, bmi, object, slot, dep; + bool ordinary=false, before_present=false; + std::string before_size,before_time,before_hash; + static std::expected load(const fs::path& directory) { + auto record=read(directory/"prepared","mcpp-attempt-v1"); + if(!record)return std::unexpected(record.error()); + if(!*record||(**record).size()!=9)return std::unexpected("missing or invalid prepared compiler transaction"); + const auto& v=**record; + if((v[0]!="ordinary"&&v[0]!="bmi")||(v[5]!="0"&&v[5]!="1"))return std::unexpected("invalid compiler transaction fields"); + Attempt a; a.directory=directory;a.ordinary=v[0]=="ordinary";a.bmi=named(v[1]);a.object=named(v[2]);a.slot=named(v[3]);a.dep=named(v[4]); + a.before_present=v[5]=="1";a.before_size=v[6];a.before_time=v[7];a.before_hash=v[8]; + if(a.object.empty()||(!a.ordinary&&(a.slot.empty()||a.bmi.empty())) + ||(a.before_present&&(!decimal(v[6])||!decimal(v[7])||!digest(v[8]))) + ||(!a.before_present&&(!v[6].empty()||!v[7].empty()||!v[8].empty()))) + return std::unexpected("invalid prepared compiler output identity"); + for(const auto& p:{a.bmi,a.object,a.slot,a.dep})if(!p.empty()&&!p.is_absolute())return std::unexpected("relative compiler output in transaction"); + return a; + } + static std::expected prepare(const fs::path& generation,bool ordinary, + const fs::path& bmi,const fs::path& object,const fs::path& slot,const fs::path& dep) { + Attempt a;a.ordinary=ordinary; + auto b=output_identity(bmi),o=output_identity(object),s=output_identity(slot),d=output_identity(dep); + if(!b||!o||!s||!d)return std::unexpected("cannot identify compiler transaction outputs"); + a.bmi=*b;a.object=*o;a.slot=*s;a.dep=*d; + if(a.object.empty()||(!ordinary&&(a.bmi.empty()||a.slot.empty()))) + return std::unexpected("missing compiler transaction output identity"); + if(!a.bmi.empty())if(auto old=identity(a.bmi)) { + a.before_present=true;a.before_size=(*old)[0];a.before_time=(*old)[1];a.before_hash=mcpp::toolchain::hash_file(a.bmi); + } + const auto who=os::process::self();if(!who||!who->id||!who->started_ns)return std::unexpected("cannot identify compiler transaction writer"); + static std::atomic sequence{0}; + a.directory=generation/"attempts"/std::format("{}-{}-{}",who->id,who->started_ns,sequence++); + std::error_code ec;fs::create_directories(a.directory,ec);if(ec)return std::unexpected("cannot create compiler transaction"); + if(!publish(a.directory/"prepared","mcpp-attempt-v1",{ordinary?"ordinary":"bmi",spelling(a.bmi),spelling(a.object),spelling(a.slot),spelling(a.dep),a.before_present?"1":"0",a.before_size,a.before_time,a.before_hash})) + return std::unexpected("cannot publish compiler transaction before output modification"); + if(!a.slot.empty()&&!publish(suffix(a.slot,".transaction"),"mcpp-attempt-pointer-v1",{spelling(a.directory)})) + return std::unexpected("cannot publish compiler transaction pointer"); + return a; + } + fs::path backup() const { return directory/"baseline.bmi"; } + std::expected prepare_backup() const { + if(bmi.empty())return {}; + fs::path source;std::string hash; + if(before_present) {source=bmi;hash=before_hash;} + else { + auto carried=read(suffix(bmi,".bak.cert"),"mcpp-carried-baseline-v1"); + if(!carried)return std::unexpected(carried.error()); + if(*carried) { + const auto& v=**carried; + if(v.size()!=2||v[0]!=spelling(bmi)||!digest(v[1])||!identity(suffix(bmi,".bak")) + ||v[1]!=mcpp::toolchain::hash_file(suffix(bmi,".bak"))) + return std::unexpected("invalid carried compiler baseline"); + source=suffix(bmi,".bak");hash=v[1]; + } + } + if(source.empty())return {}; + std::error_code ec;fs::copy_file(source,backup(),fs::copy_options::overwrite_existing,ec); + if(ec)return std::unexpected("cannot copy the previous BMI"); + const auto time=fs::last_write_time(source,ec);if(ec)return std::unexpected("cannot read the previous BMI time"); + fs::last_write_time(backup(),time,ec);if(ec)return std::unexpected("cannot retain the previous BMI time"); + if(!identity(backup())||mcpp::toolchain::hash_file(backup())!=hash)return std::unexpected("the copied compiler baseline cannot be verified"); + if(!publish(directory/"baseline","mcpp-baseline-v1",{spelling(bmi),hash}))return std::unexpected("cannot certify the previous BMI"); + return {}; + } + std::expected baseline() const { + auto record=read(directory/"baseline","mcpp-baseline-v1");if(!record)return std::unexpected(record.error()); + if(!*record)return false; + if((**record).size()!=2||(**record)[0]!=spelling(bmi)||!identity(backup()) + ||(**record)[1]!=mcpp::toolchain::hash_file(backup())) + return std::unexpected("the certified compiler baseline is incomplete or changed"); + return true; + } + std::expected settle() const { + if(bmi.empty())return {}; + auto certificate=read(directory/"baseline","mcpp-baseline-v1"); + if(!certificate)return std::unexpected(certificate.error()); + if(!*certificate)return {}; + if(auto certified=baseline();!certified)return std::unexpected(certified.error()); + if(mcpp::build::stage::bmi_equivalent(bmi,backup())&&!copy_atomic(backup(),bmi)) + return std::unexpected("cannot settle an equivalent BMI"); + return {}; + } + bool publication() const { return publish(directory/"published","mcpp-publication-v1",{}); } + bool result(int rc) const { return publish(directory/"result","mcpp-result-v1",{std::to_string(rc)}); } + bool retired(int rc) const { + auto output=identity(object);if(!output)output=Fields{"missing","missing"}; + return publish(directory/"retired","mcpp-retired-v1",{std::to_string(rc),(*output)[0],(*output)[1]}); + } + std::expected register_unit(os::process::owned_unit& unit,const fs::path& actor,const fs::path& reservation) const; +}; + +std::expected,std::string> for_slot(const fs::path& generation,const fs::path& slot) { + auto pointer=read(suffix(slot,".transaction"),"mcpp-attempt-pointer-v1"); + if(!pointer)return std::unexpected(pointer.error()); + if(!*pointer)return std::optional{}; + if((**pointer).size()!=1)return std::unexpected("invalid compiler transaction pointer"); + const auto directory=named((**pointer)[0]); + if(directory.parent_path()!=generation/"attempts")return std::unexpected("compiler transaction belongs to another generation"); + auto attempt=Attempt::load(directory);if(!attempt)return std::unexpected(attempt.error()); + return std::optional{std::move(*attempt)}; +} +std::expected validate(const Attempt& attempt) { + struct Shape {const char* file;const char* tag;std::size_t fields;}; + std::set present; + for(const auto& shape:std::array{ + Shape{"baseline","mcpp-baseline-v1",2}, Shape{"unit","mcpp-unit-v1",4}, + Shape{"controller","mcpp-controller-v1",3}, + Shape{"reservation","mcpp-reservation-pointer-v1",1}, Shape{"admitted","mcpp-admitted-v1",0}, + Shape{"published","mcpp-publication-v1",0}, Shape{"retired","mcpp-retired-v1",3}, + Shape{"result","mcpp-result-v1",1}}) { + auto record=read(attempt.directory/shape.file,shape.tag); + if(!record)return std::unexpected(record.error()); + if(!*record)continue; + const auto& fields=**record; + if(fields.size()!=shape.fields)return std::unexpected("invalid compiler transaction record fields"); + const std::string_view file=shape.file;present.insert(file); + bool valid=true; + if(file=="retired")valid=decimal(fields[0])&&output_stamp(fields[1],fields[2]); + else if(file=="result")valid=decimal(fields[0]); + else if(file=="controller")valid=decimal(fields[0])&&fields[0]!="0" + &&decimal(fields[1])&&fields[1]!="0"&&named(fields[2]).is_absolute(); + else if(file=="unit")valid=!fields[0].empty()&&decimal(fields[1])&&fields[1]!="0" + &&decimal(fields[2])&&fields[2]!="0"&&named(fields[3]).is_absolute(); + else if(file=="baseline")valid=fields[0]==spelling(attempt.bmi)&&digest(fields[1]); + else if(file=="reservation")valid=named(fields[0]).is_absolute(); + if(!valid)return std::unexpected("invalid compiler transaction record value"); + } + const auto has=[&](std::string_view file){return present.contains(file);}; + if((has("admitted")&&(!has("unit")||!has("controller")||!has("reservation"))) + ||((has("published")||has("retired"))&&!has("admitted")) + ||(has("result")&&(!attempt.ordinary||!has("retired")))) + return std::unexpected("inconsistent compiler transaction stages"); + return {}; +} +std::expected certified_quiet(const Attempt& attempt) { + if(auto valid=validate(attempt);!valid)return std::unexpected(valid.error()); + auto retired=read(attempt.directory/"retired","mcpp-retired-v1"); + if(!retired)return std::unexpected(retired.error()); + if(*retired && (**retired).size()!=3)return std::unexpected("invalid compiler retirement certificate"); + if(*retired)return true; + auto admitted=read(attempt.directory/"admitted","mcpp-admitted-v1"); + if(!admitted)return std::unexpected(admitted.error()); + if(*admitted && !(**admitted).empty())return std::unexpected("invalid compiler admission record"); + return !*admitted; // publication/ACK precede admission; no compiler otherwise +} +std::expected observed(const Attempt& attempt,ObserverMap* held=nullptr) { + const auto quiet=certified_quiet(attempt);if(!quiet)return std::unexpected(quiet.error()); + if(*quiet)return os::process::unit_state::empty; + auto unit=read(attempt.directory/"unit","mcpp-unit-v1");if(!unit)return std::unexpected(unit.error()); + if(!*unit||(**unit).size()!=4)return std::unexpected("missing admitted compiler unit reference"); + const auto key=spelling(attempt.directory); + if(held)if(const auto found=held->find(key);found!=held->end()) { + auto value=found->second.observe(); + return value ? *value : os::process::unit_state::unknown; + } + auto observer=os::process::unit_observer::open((**unit)[0]); + if(!observer)return os::process::unit_state::unknown; + auto value=observer->observe();return value ? *value : os::process::unit_state::unknown; +} +std::expected reservation_available(const fs::path& reservation) { + auto previous=read(reservation,"mcpp-reservation-v1");if(!previous)return std::unexpected(previous.error()); + if(!*previous)return true; + if((**previous).size()!=1)return std::unexpected("invalid compiler reservation"); + auto attempt=Attempt::load(named((**previous)[0]));if(!attempt)return std::unexpected(attempt.error()); + const auto state=observed(*attempt);if(!state)return std::unexpected(state.error()); + return *state==os::process::unit_state::empty; +} +bool release_reservation(const fs::path& reservation,const fs::path& attempt) { + auto current=read(reservation,"mcpp-reservation-v1"); + if(!current)return false; + if(!*current)return true; + if((**current).size()!=1)return false; + if((**current)[0]!=spelling(attempt))return true; // a later, separately locked slot + std::error_code ec;fs::remove(reservation,ec);return !ec; +} +bool controller_alive(const Attempt& attempt) { + auto controller=read(attempt.directory/"controller","mcpp-controller-v1"); + if(!controller)return false; + if(!*controller)return true; // the existing startup bound applies + if((**controller).size()!=3)return false; + os::process::program who{};const auto& fields=**controller; + const auto id=std::from_chars(fields[0].data(),fields[0].data()+fields[0].size(),who.id); + const auto birth=std::from_chars(fields[1].data(),fields[1].data()+fields[1].size(),who.started_ns); + return id.ec==std::errc{}&&id.ptr==fields[0].data()+fields[0].size() + &&birth.ec==std::errc{}&&birth.ptr==fields[1].data()+fields[1].size() + &&who.id&&who.started_ns&&os::process::alive(who); +} + +std::expected receive(os::ipc::connection& connection,std::chrono::milliseconds bound) { + const auto limit=std::chrono::steady_clock::now()+bound; + auto exact=[&](std::size_t n)->std::expected { + std::string bytes; + while(bytes.size()(limit-std::chrono::steady_clock::now()); + if(left.count()<=0)return std::unexpected("compiler registration acknowledgement timed out"); + auto got=connection.read_for(left,n-bytes.size()); + if(!got||!*got||(**got).empty())return std::unexpected("compiler registration channel closed or timed out"); + bytes+=**got; + } + return bytes; + }; + auto prefix=exact(8);if(!prefix)return std::unexpected(prefix.error()); + std::size_t size=0;const auto parsed=std::from_chars(prefix->data(),prefix->data()+prefix->size(),size,16); + if(parsed.ec!=std::errc{}||parsed.ptr!=prefix->data()+prefix->size()||size>65536)return std::unexpected("invalid compiler registration frame"); + return exact(size); +} +bool send(os::ipc::connection& connection,std::string_view bytes) { + return bool(connection.write(std::format("{:08x}",bytes.size())+std::string(bytes))); +} +std::expected Attempt::register_unit(os::process::owned_unit& unit, + const fs::path& actor,const fs::path& reservation) const { + const auto who=os::process::self();if(!who||!who->id||!who->started_ns)return std::unexpected("cannot identify a compiler controller"); + if(!publish(directory/"unit","mcpp-unit-v1",{unit.reference(),std::to_string(who->id),std::to_string(who->started_ns),spelling(actor)})) + return std::unexpected("cannot record the compiler unit before admission"); + if(!publish(directory/"reservation","mcpp-reservation-pointer-v1",{spelling(reservation)}) + ||!publish(reservation,"mcpp-reservation-v1",{spelling(directory)})) + return std::unexpected("cannot reserve compiler capacity before admission"); + const auto generation=directory.parent_path().parent_path(); + auto transport=read(generation/"transport","mcpp-unit-transport-v1"); + if(!transport||!*transport||(**transport).size()!=2)return std::unexpected("missing compiler-run registration channel"); + auto connection=os::ipc::connection::connect(named((**transport)[0]),(**transport)[1]); + if(!connection||!send(*connection,spelling(directory)))return std::unexpected("cannot register the compiler unit"); + auto ack=receive(*connection,std::chrono::seconds(2)); + if(!ack||*ack!="ACK")return std::unexpected(ack ? "compiler-run owner refused unit admission" : ack.error()); + return {}; +} + +class Witnesses { +public: + static std::expected,std::string> start(const fs::path& generation) { + auto runtime=os::dirs::runtime();if(!runtime)return std::unexpected("no local directory for compiler registration"); + auto result=std::unique_ptr(new Witnesses);result->generation_=generation; + std::error_code ec;result->directory_=fs::weakly_canonical(*runtime,ec); + if(ec)return std::unexpected("cannot identify the compiler registration directory"); + result->name_="mc-"+mcpp::toolchain::hash_string(spelling(generation)); + auto listener=os::ipc::listener::listen(result->directory_,result->name_); + if(!listener)return std::unexpected("cannot create a compiler registration endpoint: "+listener.error().message()); + result->listener_.emplace(std::move(*listener)); + if(!publish(generation/"transport","mcpp-unit-transport-v1",{spelling(result->directory_),result->name_})) + return std::unexpected("cannot publish compiler registration transport"); + result->thread_=std::jthread([self=result.get()] {self->serve();}); + return result; + } + ~Witnesses() { stop(); } + void stop() { + { std::lock_guard guard(admission_);if(stopping_.exchange(true))return; } + // An ordinary local connection wakes the blocking accept. No periodic + // filesystem scan, extra process, loopback service or native PID code. + if(auto wake=os::ipc::connection::connect(directory_,name_))send(*wake,"STOP"); + if(thread_.joinable())thread_.join(); + listener_.reset(); + } + ObserverMap& observers() {return held_;} + bool failed() const {return failed_.load();} +private: + fs::path generation_,directory_; + std::string name_; + std::optional listener_; + std::jthread thread_; + std::atomic stopping_{false},failed_{false}; + std::mutex admission_; + ObserverMap held_; + void audit() { + for(auto it=held_.begin();it!=held_.end();) { + const auto directory=named(it->first); + auto retired=read(directory/"retired","mcpp-retired-v1"); + if(!retired||(*retired&&(**retired).size()!=3)) {failed_.store(true);++it;continue;} + if(*retired) {it=held_.erase(it);continue;} + auto attempt=Attempt::load(directory); + if(!attempt||!controller_alive(*attempt))failed_.store(true); + ++it; + } + } + void serve() { + auto next_audit=std::chrono::steady_clock::now()+std::chrono::milliseconds(500); + while(!stopping_.load()) { + if(std::chrono::steady_clock::now()>=next_audit) { + audit();next_audit=std::chrono::steady_clock::now()+std::chrono::milliseconds(500); + } + auto accepted=listener_->accept_for(std::chrono::milliseconds(500)); + if(!accepted||!*accepted)continue; + auto connection=std::move(**accepted); + auto request=receive(connection,std::chrono::milliseconds(500)); + if(!request||stopping_.load())continue; + const auto attempt=named(*request); + if(attempt.parent_path()!=generation_/"attempts") {send(connection,"REFUSED");continue;} + auto prepared=Attempt::load(attempt); + auto unit=read(attempt/"unit","mcpp-unit-v1"); + if(!prepared||!unit||!*unit||(**unit).size()!=4) {send(connection,"REFUSED");continue;} + const auto actor=named((**unit)[3]); + if(actor.parent_path()!=generation_/"actors") {send(connection,"REFUSED");continue;} + auto lease=os::fs::lock::acquire(actor,os::fs::lock::kind::exclusive,false); + if(lease||lease.error().code!=os::errc::again||!controller_alive(*prepared)) {send(connection,"REFUSED");continue;} + auto retained=os::process::unit_observer::open((**unit)[0]); + if(!retained) {send(connection,"REFUSED");continue;} + std::lock_guard guard(admission_); + if(stopping_.load()||failed_.load()) {send(connection,"REFUSED");continue;} + held_.insert_or_assign(*request,std::move(*retained)); + send(connection,"ACK"); + } + } +}; + +std::expected recover(const Attempt& attempt,ObserverMap* held=nullptr) { + const auto state=observed(attempt,held);if(!state)return std::unexpected(state.error()); + if(*state!=os::process::unit_state::empty)return false; + auto retired=read(attempt.directory/"retired","mcpp-retired-v1"); + auto result=attempt.ordinary ? read(attempt.directory/"result","mcpp-result-v1") : Record{std::optional{}}; + auto published=read(attempt.directory/"published","mcpp-publication-v1"); + auto admitted=read(attempt.directory/"admitted","mcpp-admitted-v1"); + if(!retired||!result||!published||!admitted)return std::unexpected("cannot validate compiler transaction completion"); + if(*published&&!(**published).empty())return std::unexpected("invalid BMI publication certificate"); + if(*published&&!*admitted)return std::unexpected("BMI publication without registered compiler admission"); + bool committed=false; + if(*retired) { + if((**retired).size()!=3)return std::unexpected("invalid compiler retirement certificate"); + const auto& status=(**retired)[0];int rc=0; + const auto parsed=std::from_chars(status.data(),status.data()+status.size(),rc); + if(parsed.ec!=std::errc{}||parsed.ptr!=status.data()+status.size())return std::unexpected("invalid retired compiler status"); + if(attempt.ordinary) { + if(*result&&((**result).size()!=1||(**result)[0]!=status))return std::unexpected("compiler result does not match retirement"); + committed=rc==0&&result->has_value(); + } else { + std::ifstream completed(suffix(attempt.slot,".rc"));int value=0;std::string extra; + committed=rc==0&&bool(completed>>value)&&!(completed>>extra)&&value==rc; + } + const auto output=identity(attempt.object); + committed=committed&&output&&(*output)[0]==(**retired)[1]&&(*output)[1]==(**retired)[2]; + if(!attempt.bmi.empty()) { + const auto bmi=identity(attempt.bmi); + const bool changed=bmi&&(!attempt.before_present||(*bmi)[0]!=attempt.before_size||(*bmi)[1]!=attempt.before_time); + committed=committed&&bmi&&(published->has_value()||changed); + } + } + std::error_code ec; + if(!committed) { + const auto baseline=attempt.baseline();if(!baseline)return std::unexpected(baseline.error()); + if(*baseline) { + if(*published) { + // An early BMI edge may already be in Ninja's log. Make it + // absent to force retry, while retaining a certified baseline + // for that retry's own failure/semantic-equivalence handling. + auto certificate=read(attempt.directory/"baseline","mcpp-baseline-v1"); + if(!certificate||!*certificate)return std::unexpected("lost compiler baseline certificate"); + if(!copy_atomic(attempt.backup(),suffix(attempt.bmi,".bak")) + ||!publish(suffix(attempt.bmi,".bak.cert"),"mcpp-carried-baseline-v1",{spelling(attempt.bmi),(**certificate)[1]})) + return std::unexpected("cannot retain the compiler baseline before invalidation"); + fs::remove(attempt.bmi,ec);if(ec)return std::unexpected("cannot invalidate an abandoned BMI"); + } else if(!copy_atomic(attempt.backup(),attempt.bmi))return std::unexpected("cannot restore the previous verified BMI"); + } else if(attempt.before_present) { + // No certified copy can precede admission when a previous BMI + // existed. Before admission, the original is still the baseline. + if(*admitted)return std::unexpected("an admitted compiler has no certified previous BMI"); + const auto old=identity(attempt.bmi); + if(!old||(*old)[0]!=attempt.before_size||(*old)[1]!=attempt.before_time + ||mcpp::toolchain::hash_file(attempt.bmi)!=attempt.before_hash) + return std::unexpected("the unmodified compiler baseline cannot be verified"); + } else if(!attempt.bmi.empty()) {fs::remove(attempt.bmi,ec);if(ec)return std::unexpected("cannot invalidate an abandoned BMI");} + if(*admitted)for(const auto& path:{attempt.object,attempt.dep})if(!path.empty()) { + fs::remove(path,ec);if(ec)return std::unexpected("cannot invalidate an abandoned compiler output"); + } + if(!attempt.slot.empty())for(std::string_view tail:{".rc",".rc.tmp"}) { + fs::remove(suffix(attempt.slot,tail),ec);if(ec)return std::unexpected("cannot invalidate an abandoned compiler status"); + } + } else if(!attempt.bmi.empty()) { + const auto object=identity(attempt.object),bmi=identity(attempt.bmi); + if(!object||!bmi||!publish(suffix(attempt.slot,".completed"),"mcpp-completed-unit-v1",{ + spelling(attempt.bmi),spelling(attempt.object),(*object)[0],(*object)[1],(*bmi)[0],(*bmi)[1]})) + return std::unexpected("cannot retain the verified compiler cache result"); + // Carried baselines are obsolete only after a verified successful + // result. Normal completion keeps every actual output untouched. + fs::remove(suffix(attempt.bmi,".bak.cert"),ec);if(ec)return std::unexpected("cannot retire the carried baseline certificate"); + fs::remove(suffix(attempt.bmi,".bak"),ec);if(ec)return std::unexpected("cannot retire the carried baseline"); + } + auto reservation=read(attempt.directory/"reservation","mcpp-reservation-pointer-v1"); + if(!reservation)return std::unexpected(reservation.error()); + if(*reservation) { + if((**reservation).size()!=1||!release_reservation(named((**reservation)[0]),attempt.directory)) + return std::unexpected("cannot retire the compiler reservation"); + } + return true; +} +// A completed generation leaves this quiet-unit proof beside the cached +// status. A later missing object can invalidate its BMI without guessing that +// an old controller's absence means no writer, even after journals are retired. +std::expected,std::string> cached_result(const fs::path& slot,const fs::path& requested_object) { + auto completed=read(suffix(slot,".completed"),"mcpp-completed-unit-v1"); + if(!completed)return std::unexpected(completed.error()); + if(!*completed)return std::optional{}; + const auto& v=**completed; + auto object=output_identity(requested_object); + if(v.size()!=6||!object||v[1]!=spelling(*object)||!named(v[0]).is_absolute()) + return std::unexpected("invalid completed compiler output identity"); + const auto current=identity(*object); + if(current&&(*current)[0]==v[2]&&(*current)[1]==v[3])return std::optional{0}; + const auto bmi=named(v[0]);const auto current_bmi=identity(bmi); + if(current_bmi) { + if((*current_bmi)[0]!=v[4]||(*current_bmi)[1]!=v[5])return std::unexpected("cached BMI identity changed outside its completed attempt"); + if(!copy_atomic(bmi,suffix(bmi,".bak")) + ||!publish(suffix(bmi,".bak.cert"),"mcpp-carried-baseline-v1",{spelling(bmi),mcpp::toolchain::hash_file(bmi)})) + return std::unexpected("cannot retain the cached BMI before retry"); + } + std::error_code ec; + for(const auto& path:{bmi,suffix(slot,".rc"),suffix(slot,".completed")}) { + fs::remove(path,ec);if(ec)return std::unexpected("cannot invalidate a missing cached compiler output"); + } + return std::optional{1}; +} + +std::expected recover_generation(const fs::path& generation,ObserverMap* held=nullptr) { + const auto limit=std::chrono::steady_clock::now()+std::chrono::seconds(5); + for(;;) { + bool pending=false;std::error_code ec; + const auto directory=generation/"attempts"; + if(!fs::exists(directory,ec))return ec ? std::unexpected("cannot inspect compiler transactions") : std::expected{}; + fs::directory_iterator entries(directory,ec);if(ec)return std::unexpected("cannot enumerate compiler transactions"); + for(const auto& entry:entries) { + auto prepared=read(entry.path()/"prepared","mcpp-attempt-v1"); + if(!prepared)return std::unexpected(prepared.error()); + if(!*prepared) { + // Preparation's atomic record precedes every modification. + // An empty, unreferenced staging directory has no outputs. + if(fs::exists(entry.path()/"unit",ec)||fs::exists(entry.path()/"admitted",ec))return std::unexpected("compiler unit without prepared transaction"); + continue; + } + auto attempt=Attempt::load(entry.path());if(!attempt)return std::unexpected(attempt.error()); + const auto done=recover(*attempt,held);if(!done)return std::unexpected(done.error()); + if(!*done)pending=true; + } + if(!pending)return {}; + if(std::chrono::steady_clock::now()>=limit)return std::unexpected("previous compiler unit is occupied or unknown; reservation and outputs were retained"); + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } +} +// Admit Ninja only after certified cached outputs have been checked. +// This reads existing completion certificates; it creates no source-dirty index. +std::expected admit_cached_outputs(const fs::path& output_directory); +} // namespace mcpp::build::schedule::transaction diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 55d3f982c..487c9e881 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -74,6 +74,7 @@ import mcpplibs.os.process; import mcpp.build.command_line; import mcpp.run; import mcpp.build.stage; +import mcpp.build.schedule.compiler_transaction; import mcpp.home; import mcpp.home.files; import mcpp.invocation; @@ -94,6 +95,7 @@ public: void environment(std::vector>& env) const; std::expected finish(); const std::filesystem::path& directory() const; + bool failed() const; private: struct State; std::unique_ptr state_; @@ -168,7 +170,8 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o // Their original argv and streams are retained; no shell is introduced. int run_capped(const std::vector& argv, const std::filesystem::path& semaphore, int maxCompilers, - const std::filesystem::path& output = {}); + const std::filesystem::path& output = {}, + const std::filesystem::path& depfile = {}); } // namespace mcpp::build::schedule::detach @@ -176,6 +179,7 @@ int run_capped(const std::vector& argv, namespace mcpp::build::schedule::detach { namespace { +namespace transactions = mcpp::build::schedule::transaction; std::filesystem::path suffixed(const std::filesystem::path& base, std::string_view s) { return std::filesystem::path{base.string() + std::string(s)}; @@ -308,23 +312,24 @@ std::optional read_rc(const std::filesystem::path& slot) { // Temp file + rename, so a reader never observes half a number. The same // guarantee, for the same reason, that makes watching the BMI path sound. -void write_rc(const std::filesystem::path& slot, int rc) { - const auto tmp = suffixed(slot, ".rc.tmp"); - { std::ofstream out(tmp, std::ios::trunc); out << rc << '\n'; } - std::error_code ec; - std::filesystem::rename(tmp, suffixed(slot, ".rc"), ec); +bool write_rc(const std::filesystem::path& slot, int rc) { + return transactions::replace_record(suffixed(slot,".rc"),std::to_string(rc)+"\n"); } // Stable kernel lock slots, shared by ordinary and detached compilers. The // lock ends with its holder even after SIGKILL; its file is never unlinked as // a way of reclaiming a live permit. No holder acquires a second permit. -std::expected, int> +struct Permit { + std::optional lock; + std::filesystem::path reservation; +}; +std::expected acquire_permit(const Actor& actor, const std::filesystem::path& requested, int cap) { const auto dir = actor.owner.permits(requested); if (!actor.ready) return std::unexpected(130); if (dir.empty() || cap <= 0) { if (actor.owner.managed) return std::unexpected(2); - return std::optional{}; + return Permit{}; } std::error_code ec; std::filesystem::create_directories(dir, ec); @@ -336,15 +341,28 @@ acquire_permit(const Actor& actor, const std::filesystem::path& requested, int c constexpr auto kLimit = std::chrono::hours(2); for (;;) { if (!actor.owner.active()) return std::unexpected(130); + bool orphan = false; for (int i = 0; i < cap; ++i) { auto permit = mcpplibs::os::fs::lock::acquire(dir / std::format("slot-{}.lock", i), mcpplibs::os::fs::lock::kind::exclusive, false); - if (permit) return std::optional{std::move(*permit)}; + if (permit) { + const auto reservation = dir / std::format("slot-{}.reservation", i); + if (actor.owner.managed) { + const auto available = transactions::reservation_available(reservation); + if (!available) { std::println(std::cerr,"mcpp: {}",available.error()); return std::unexpected(1); } + if (!*available) { orphan = true; continue; } + } + return Permit{std::optional{std::move(*permit)},reservation}; + } if (permit.error().code != mcpplibs::os::errc::again) { std::println(std::cerr, "mcpp: {}", permit.error().message()); return std::unexpected(1); } } + if (orphan && std::chrono::steady_clock::now()-started > std::chrono::seconds(5)) { + std::println(std::cerr,"mcpp: a previous compiler unit is occupied or unknown; its slot remains reserved"); + return std::unexpected(1); + } if (std::chrono::steady_clock::now() - started > kLimit) { std::println(std::cerr, "mcpp: no compiler slot became free in 2h — all {} tokens in {} " @@ -470,15 +488,20 @@ void settle_bmi(const std::filesystem::path& bmi) { // would keep the edge alive long after this process exits), and it runs in a // unit of its own, so the terminal's interrupt of the build is not delivered // to it mid-write. -bool spawn_detached(const std::vector& argv) { +bool spawn_detached(const std::vector& argv, bool declare_inside, + std::optional* identity = nullptr) { namespace process = mcpplibs::os::process; process::spawn_options o; o.program = argv.front(); o.args.assign(argv.begin() + 1, argv.end()); o.in = o.out = o.err = process::stdio::null; - o.own_unit = true; + o.own_unit = !declare_inside; auto p = process::process::spawn(o); if (!p) return false; + if(identity) { + const auto described=process::describe(p->id()); + if(described&&described->who.id&&described->who.started_ns)*identity=described->who; + } p->detach(); return true; } @@ -488,29 +511,53 @@ bool spawn_detached(const std::vector& argv) { // the one pair of quotes `/s` removes, #425 #832), its output and errors into // `logPath`. int run_to_completion(std::string_view command, - const std::filesystem::path& logPath, const Owner& owner) { + const std::filesystem::path& logPath, const Owner& owner, + mcpplibs::os::process::owned_unit* unit=nullptr) { // Created before the compiler starts: its existence is what tells the // waiters that a supervisor started. std::ofstream log(logPath, std::ios::binary | std::ios::trunc); log.flush(); + if(!log)return 2; + if(!owner.active())return 130; auto sh = mcpp::run::shell(command); auto r = mcpp::run::execute(sh.argv, { - .should_stop = [&] { return !owner.active(); }, .one_string = sh.one_string }); + .should_stop = [&] { return !owner.active(); }, .one_string = sh.one_string, + .declared_unit = unit }); // A failure carries the command it ran. Phase 2 replays this file, so the // failure arrives with the exact invocation attached -- without it a // remote failure can only be guessed at, which is how #425 cost a CI cycle // to even localise. if (r.exit_code != 0) log << "+ " << command << '\n'; log << r.output; + if(!log)return 2; return owner.active() ? r.exit_code : 130; } +std::expected retire_owned(mcpplibs::os::process::owned_unit& unit,const Owner& owner) { + const auto limit=std::chrono::steady_clock::now()+std::chrono::seconds(5); + for(;;) { + if(!owner.active()||mcpp::run::ending()) { + unit.terminate(); + return std::unexpected("compiler run ended before unit retirement"); + } + auto state=unit.retire(owner.program); + if(!state)return std::unexpected(state.error().message()); + if(*state==mcpplibs::os::process::unit_state::empty)return {}; + if(std::chrono::steady_clock::now()>=limit) { + unit.terminate(); + return std::unexpected("compiler descendants did not retire; outputs and reservation retained"); + } + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + } +} + } // namespace struct RunScope::State { std::filesystem::path root, generation; std::optional gate; std::optional owner; + std::unique_ptr witnesses; bool closed = false; std::string failure; std::expected close() { @@ -518,13 +565,21 @@ struct RunScope::State { : std::unexpected(failure); closed = true; if (generation.empty()) return {}; + if (witnesses) witnesses->stop(); owner.reset(); if (!drain_actors(generation)) { failure = "previous compiler helpers have not stopped; refusing to start another generation"; return std::unexpected(failure); } + auto recovered = transactions::recover_generation(generation,witnesses ? &witnesses->observers() : nullptr); + if (!recovered) { failure=recovered.error(); return std::unexpected(failure); } std::error_code ec; std::filesystem::remove_all(generation, ec); + if(ec) {failure="cannot retire completed compiler runtime state";return std::unexpected(failure);} + if(witnesses&&witnesses->failed()) { + failure="a compiler controller ended before completion; incomplete outputs were recovered"; + return std::unexpected(failure); + } return {}; } ~State() { (void)close(); } @@ -572,9 +627,14 @@ std::expected RunScope::begin(const std::filesystem::path std::this_thread::sleep_for(std::chrono::milliseconds(10)); } while (std::chrono::steady_clock::now() < limit); if (!free) return std::unexpected("previous compiler helpers are still active; their state was retained"); + if(auto recovered=transactions::recover_generation(entry.path());!recovered)return std::unexpected(recovered.error()); std::filesystem::remove_all(entry.path(), ec); if (ec) return std::unexpected("cannot retire previous compiler-run state"); } + // Old owners, actors and units are quiet before cached output recovery; + // no Ninja reader from this generation has started yet. + if(auto admitted=transactions::admit_cached_outputs(canonical);!admitted) + return std::unexpected(admitted.error()); static std::atomic sequence{0}; state->generation = state->root / std::format("run-{}-{}-{}", self->id, self->started_ns, sequence++); std::filesystem::create_directories(state->generation, ec); @@ -583,8 +643,11 @@ std::expected RunScope::begin(const std::filesystem::path mcpplibs::os::fs::lock::kind::exclusive, false); if (!owner) return std::unexpected("cannot lock the compiler-run owner: " + owner.error().message()); state->owner.emplace(std::move(*owner)); + auto witnesses = transactions::Witnesses::start(state->generation); + if (!witnesses) return std::unexpected(witnesses.error()); + state->witnesses = std::move(*witnesses); const auto record = std::format("mcpp-compiler-run-v1\n{} {}\n", self->id, self->started_ns); - if (!mcpplibs::os::fs::write_atomic(state->generation / "owner", record)) + if (!transactions::replace_record(state->generation / "owner", record)) return std::unexpected("cannot publish compiler-run ownership"); return RunScope(std::move(state)); } @@ -595,6 +658,7 @@ void RunScope::environment(std::vector>& env } std::expected RunScope::finish() { return state_->close(); } const std::filesystem::path& RunScope::directory() const { return state_->generation; } +bool RunScope::failed() const {return state_->witnesses&&state_->witnesses->failed();} int compile_release_at_bmi(const CompileRequest& req) { if (req.command.empty() || req.self.empty()) return 2; @@ -602,23 +666,39 @@ int compile_release_at_bmi(const CompileRequest& req) { if (!actor.ready) return 130; std::error_code ec; - if (!req.slot.parent_path().empty()) - std::filesystem::create_directories(req.slot.parent_path(), ec); - std::filesystem::remove(suffixed(req.slot, ".rc"), ec); - std::filesystem::remove(suffixed(req.slot, ".rc.tmp"), ec); - std::filesystem::remove(suffixed(req.slot, ".log"), ec); + if(!req.slot.parent_path().empty())std::filesystem::create_directories(req.slot.parent_path(),ec); + if(ec)return 1; + CompileRequest actual = req; + std::optional attempt; + if(actor.owner.managed) { + auto prepared=transactions::Attempt::prepare(actor.owner.generation,false,req.bmi,req.object,req.slot,req.depTo); + if(!prepared) {std::println(std::cerr,"mcpp: {}",prepared.error());return 1;} + attempt.emplace(std::move(*prepared)); + actual.bmi=attempt->bmi;actual.object=attempt->object;actual.slot=attempt->slot;actual.depTo=attempt->dep; + if(auto copied=attempt->prepare_backup();!copied) {std::println(std::cerr,"mcpp: {}",copied.error());return 1;} + } + std::filesystem::remove(suffixed(actual.slot, ".rc"), ec); + if(attempt&&ec)return 1; + std::filesystem::remove(suffixed(actual.slot, ".rc.tmp"), ec); + if(attempt&&ec)return 1; + std::filesystem::remove(suffixed(actual.slot, ".log"), ec); + if(attempt&&ec)return 1; + if(attempt) { + std::filesystem::remove(suffixed(actual.slot,".completed"),ec); + if(ec)return 1; + } // Keep the previous BMI for the equivalence check, and LEAVE THE ORIGINAL // IN PLACE — see BmiIdentity for why moving it away is what broke importers. // Snapshot through the SAME function in both cases, so "no previous BMI" // and "the BMI as it is now" are directly comparable. - const BmiIdentity before = bmi_identity(req.bmi); - if (!req.bmi.empty()) { - const auto backup = suffixed(req.bmi, ".bak"); + const BmiIdentity before = bmi_identity(actual.bmi); + if (!attempt && !actual.bmi.empty()) { + const auto backup = suffixed(actual.bmi, ".bak"); std::filesystem::remove(backup, ec); if (before.present) { std::filesystem::copy_file( - req.bmi, backup, std::filesystem::copy_options::overwrite_existing, ec); + actual.bmi, backup, std::filesystem::copy_options::overwrite_existing, ec); // AND CARRY THE MTIME ACROSS. `copy_file` stamps the copy with // the time of the copy, and `settle_bmi` restores this file when the // new BMI turns out equivalent — precisely so the mtime does NOT @@ -633,20 +713,21 @@ int compile_release_at_bmi(const CompileRequest& req) { // The BMI this slot publishes, for phase 2 to withdraw if the compile // fails (see `await_unit`). - if (!req.bmi.empty()) std::ofstream(suffixed(req.slot, ".bmi"), std::ios::trunc) << req.bmi.string(); + if (!actual.bmi.empty()) std::ofstream(suffixed(actual.slot, ".bmi"), std::ios::trunc) << actual.bmi.string(); // The supervisor reads the SAME argv file rather than receiving the command // on its own command line: one representation, no re-quoting, and no limit // on how long a compiler command may be. - std::vector sup{req.self.string(), "bmi-supervise", - "--slot", req.slot.string(), - "--command-file", req.commandFile.string()}; - if (!req.semaphore.empty()) { sup.push_back("--sem"); sup.push_back(req.semaphore.string()); } - if (req.maxCompilers > 0) { sup.push_back("--cap"); sup.push_back(std::to_string(req.maxCompilers)); } - if (!req.object.empty()) { sup.push_back("--object"); sup.push_back(req.object.string()); } + std::vector sup{actual.self.string(), "bmi-supervise", + "--slot", actual.slot.string(), + "--command-file", actual.commandFile.string()}; + if (!actual.semaphore.empty()) { sup.push_back("--sem"); sup.push_back(actual.semaphore.string()); } + if (actual.maxCompilers > 0) { sup.push_back("--cap"); sup.push_back(std::to_string(actual.maxCompilers)); } + if (!actual.object.empty()) { sup.push_back("--object"); sup.push_back(actual.object.string()); } if (!actor.owner.active()) return 130; - if (!spawn_detached(sup)) { - restore_backup(req.bmi); + std::optional supervisor; + if (!spawn_detached(sup,actor.owner.managed,&supervisor)) { + if(!attempt)restore_backup(actual.bmi); return 2; } @@ -664,32 +745,37 @@ int compile_release_at_bmi(const CompileRequest& req) { const auto started = clock::now(); constexpr auto kNoSupervisorGrace = std::chrono::seconds(10); constexpr auto kHardLimit = std::chrono::hours(2); + auto nextControllerCheck=started; for (;;) { if (!actor.owner.active()) return 130; // GCC unlinks an existing BMI immediately before renaming its new // temporary file. Absence is a transition, never publication: treating // it as a changed identity releases importers inside that unlink gap. - const auto published = bmi_identity(req.bmi); + const auto published = bmi_identity(actual.bmi); if (published.present && published != before) { - settle_bmi(req.bmi); - copy_first_rule(req.depFrom, req.depTo, - req.bmiTarget.empty() ? req.bmi.string() : req.bmiTarget); + if(attempt) { + if(!attempt->publication())return 1; + if(auto settled=attempt->settle();!settled) {std::println(std::cerr,"mcpp: {}",settled.error());return 1;} + } else settle_bmi(actual.bmi); + copy_first_rule(actual.depFrom, actual.depTo, + actual.bmiTarget.empty() ? actual.bmi.string() : actual.bmiTarget); return actor.owner.active() ? 0 : 130; // importers may proceed only in this run } - if (const auto rc = read_rc(req.slot)) { + if (const auto rc = read_rc(actual.slot)) { if (*rc != 0) { // failed before publishing a BMI - std::ifstream in(suffixed(req.slot, ".log")); + std::ifstream in(suffixed(actual.slot, ".log")); if (in) std::cerr << in.rdbuf(); // Nothing was published, so the previous BMI is the truth. Put // it back rather than leaving the unit with no BMI at all. - restore_backup(req.bmi); + if(!attempt)restore_backup(actual.bmi); } else { - if (!req.bmi.empty() && !file_exists(req.bmi)) { + if (!actual.bmi.empty() && (!file_exists(actual.bmi) + || (attempt && bmi_identity(actual.bmi)==before))) { std::println(std::cerr, "mcpp: compiler reported success but {} is missing", - req.bmi.string()); - restore_backup(req.bmi); - if (!req.object.empty()) std::filesystem::remove(req.object, ec); + actual.bmi.string()); + if(!attempt)restore_backup(actual.bmi); + if (!attempt && !actual.object.empty()) std::filesystem::remove(actual.object, ec); return 1; } // THE COMPILER CAN FINISH BETWEEN THE TWO CHECKS ABOVE. @@ -710,30 +796,41 @@ int compile_release_at_bmi(const CompileRequest& req) { // // in an importer — reproducible at `-j1`, so it was never a // race between compilers, only between this loop's two checks. - settle_bmi(req.bmi); - copy_first_rule(req.depFrom, req.depTo, - req.bmiTarget.empty() ? req.bmi.string() : req.bmiTarget); + if(attempt) { + if(!attempt->publication())return 1; + if(auto settled=attempt->settle();!settled) {std::println(std::cerr,"mcpp: {}",settled.error());return 1;} + } else settle_bmi(actual.bmi); + copy_first_rule(actual.depFrom, actual.depTo, + actual.bmiTarget.empty() ? actual.bmi.string() : actual.bmiTarget); } return actor.owner.active() ? *rc : 130; } + if(attempt && clock::now()>=nextControllerCheck) { + nextControllerCheck=clock::now()+std::chrono::milliseconds(50); + if(!transactions::controller_alive(*attempt) + ||(supervisor&&!mcpplibs::os::process::alive(*supervisor))) { + std::println(std::cerr,"mcpp: compiler controller ended without a final status; retaining its outputs until unit recovery"); + return 1; + } + } const auto waited = clock::now() - started; // No log file means no supervisor ever opened one. Distinguished from // the hard limit because the two need different advice: this one is // "the process is not there", not "it is taking too long". - if (!file_exists(suffixed(req.slot, ".log")) && waited > kNoSupervisorGrace) { + if (!file_exists(suffixed(actual.slot, ".log")) && waited > kNoSupervisorGrace) { std::println(std::cerr, "mcpp: the compiler supervisor for {} never started " "(no log after {}s) — restoring the previous BMI", - req.slot.string(), + actual.slot.string(), std::chrono::duration_cast(waited).count()); - restore_backup(req.bmi); + if(!attempt)restore_backup(actual.bmi); return 1; } if (waited > kHardLimit) { std::println(std::cerr, "mcpp: timed out waiting for {} to publish its BMI", - req.slot.string()); - restore_backup(req.bmi); + actual.slot.string()); + if(!attempt)restore_backup(actual.bmi); return 1; } std::this_thread::sleep_for(std::chrono::milliseconds(2)); @@ -745,6 +842,39 @@ int supervise(const std::filesystem::path& slot, const std::filesystem::path& object, std::string_view command) { auto actor = std::make_shared(); if (!actor->ready) return 130; + if(actor->owner.managed) { + auto loaded=transactions::for_slot(actor->owner.generation,slot); + if(!loaded||!*loaded) {std::println(std::cerr,"mcpp: missing or invalid prepared compiler attempt");return 2;} + auto attempt=std::move(**loaded); + const auto who=mcpplibs::os::process::self(); + if(!who||!who->id||!who->started_ns + ||!transactions::publish(attempt.directory/"controller","mcpp-controller-v1",{ + std::to_string(who->id),std::to_string(who->started_ns),transactions::spelling(actor->path)})) { + (void)write_rc(slot,2);return 2; + } + {std::ofstream started(suffixed(slot,".log"),std::ios::trunc);if(!started){(void)write_rc(slot,2);return 2;}} + if(command.empty()) {(void)write_rc(slot,2);return 0;} + auto permit=acquire_permit(*actor,semaphore,maxCompilers); + if(!permit) {(void)write_rc(slot,permit.error());return permit.error();} + auto created=mcpplibs::os::process::owned_unit::create(); + if(!created) {(void)write_rc(slot,2);return 2;} + auto unit=std::make_shared(std::move(*created)); + InterruptCleanup cleanup{mcpp::run::add_cleanup([actor,unit] {unit->terminate();})}; + if(auto registered=attempt.register_unit(*unit,actor->path,permit->reservation);!registered) { + std::println(std::cerr,"mcpp: {}",registered.error());(void)write_rc(slot,2);return 2; + } + if(!actor->owner.active())return 130; + if(!transactions::publish(attempt.directory/"admitted","mcpp-admitted-v1",{})) { + (void)write_rc(slot,2);return 2; + } + const int rc=run_to_completion(command,suffixed(slot,".log"),actor->owner,unit.get()); + if(auto retired=retire_owned(*unit,actor->owner);!retired) { + std::println(std::cerr,"mcpp: {}",retired.error());return 1; + } + if(!attempt.retired(rc)||!write_rc(slot,rc))return 1; + if(!transactions::release_reservation(permit->reservation,attempt.directory))return 1; + return 0; + } auto settled = std::make_shared>(false); auto settle = [actor, settled, slot, object](int rc) { if (settled->exchange(true)) return; @@ -783,6 +913,27 @@ int supervise(const std::filesystem::path& slot, int await_unit(const std::filesystem::path& slot, const std::filesystem::path& object) { Actor actor; if (!actor.ready) return 130; + std::optional attempt; + if(actor.owner.managed) { + auto pointer=transactions::read(suffixed(slot,".transaction"),"mcpp-attempt-pointer-v1"); + if(!pointer) {std::println(std::cerr,"mcpp: {}",pointer.error());return 1;} + bool current=false; + if(*pointer) { + if((**pointer).size()!=1)return 1; + current=transactions::named((**pointer)[0]).parent_path()==actor.owner.generation/"attempts"; + } + if(!current) { + auto cached=transactions::cached_result(slot,object); + if(!cached) {std::println(std::cerr,"mcpp: {}",cached.error());return 1;} + if(*cached)return **cached; + std::println(std::cerr,"mcpp: no verified completed compiler state for this cached object"); + return 1; + } + auto loaded=transactions::for_slot(actor.owner.generation,slot); + if(!loaded||!*loaded)return 1; + attempt.emplace(std::move(**loaded)); + } + // BOUNDED. An unbounded wait here turns "phase 1 never started a compiler" // into a build that hangs forever with no output — which is strictly worse // than a failure, because nothing says what to look at. If no supervisor @@ -793,8 +944,16 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o constexpr auto kHardLimit = std::chrono::hours(2); std::optional rc; + auto nextControllerCheck=started; while (!(rc = read_rc(slot))) { if (!actor.owner.active()) return 130; + if(attempt&&clock::now()>=nextControllerCheck) { + nextControllerCheck=clock::now()+std::chrono::milliseconds(50); + if(!transactions::controller_alive(*attempt)) { + std::println(std::cerr,"mcpp: compiler controller ended without a status; unit recovery will retain or invalidate its outputs"); + return 1; + } + } const auto waited = clock::now() - started; if (!file_exists(suffixed(slot, ".log")) && waited > kNoSupervisorGrace) { std::println(std::cerr, @@ -816,6 +975,18 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o std::ifstream in(suffixed(slot, ".log")); if (in) std::cerr << in.rdbuf(); } + if(attempt) { + if(*rc!=0)return *rc; + auto retired=transactions::read(attempt->directory/"retired","mcpp-retired-v1"); + const auto output=transactions::identity(attempt->object); + if(!retired||!*retired||(**retired).size()!=3||(**retired)[0]!="0" + ||!output||(*output)[0]!=(**retired)[1]||(*output)[1]!=(**retired)[2]) { + std::println(std::cerr,"mcpp: compiler result has no matching quiet-unit output proof"); + return 1; + } + return 0; + } + // A FAILURE IS NOT KEPT. Phase 1 reported success when the BMI was // published, so ninja holds that edge done; were the record of the failed // compile left in place, every later build would replay it here without @@ -842,10 +1013,36 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o int run_capped(const std::vector& argv, const std::filesystem::path& semaphore, int maxCompilers, - const std::filesystem::path& output) { + const std::filesystem::path& output, const std::filesystem::path& depfile) { if (argv.empty()) return 2; auto actor = std::make_shared(); if (!actor->ready) return 130; + if(actor->owner.managed) { + auto prepared=transactions::Attempt::prepare(actor->owner.generation,true,{},output,{},depfile); + if(!prepared) {std::println(std::cerr,"mcpp: {}",prepared.error());return 1;} + auto attempt=std::move(*prepared); + const auto who=mcpplibs::os::process::self(); + if(!who||!who->id||!who->started_ns + ||!transactions::publish(attempt.directory/"controller","mcpp-controller-v1",{ + std::to_string(who->id),std::to_string(who->started_ns),transactions::spelling(actor->path)}))return 1; + auto permit=acquire_permit(*actor,semaphore,maxCompilers);if(!permit)return permit.error(); + auto created=mcpplibs::os::process::owned_unit::create(); + if(!created)return 2; + auto unit=std::make_shared(std::move(*created)); + InterruptCleanup cleanup{mcpp::run::add_cleanup([actor,unit] {unit->terminate();})}; + if(auto registered=attempt.register_unit(*unit,actor->path,permit->reservation);!registered) { + std::println(std::cerr,"mcpp: {}",registered.error());return 2; + } + if(!actor->owner.active())return 130; + if(!transactions::publish(attempt.directory/"admitted","mcpp-admitted-v1",{}))return 2; + const auto result=mcpp::run::execute(argv,{ + .input=mcpp::run::Input::inherit,.streams=mcpp::run::Streams::inherit, + .should_stop=[&] {return !actor->owner.active();},.declared_unit=unit.get()}); + if(auto retired=retire_owned(*unit,actor->owner);!retired) {std::println(std::cerr,"mcpp: {}",retired.error());return 1;} + if(!attempt.retired(result.exit_code)||!attempt.result(result.exit_code))return 1; + if(!transactions::release_reservation(permit->reservation,attempt.directory))return 1; + return result.exit_code; + } InterruptCleanup cleanup{mcpp::run::add_cleanup([actor, output] { if (!output.empty()) { std::error_code ec; diff --git a/src/cli.cppm b/src/cli.cppm index ef0bc1868..f3ecc1957 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -281,16 +281,18 @@ int run(int argc, char** argv) { if (argc > 1 && std::string_view(argv[1]) == "__compiler-cap") { std::filesystem::path semaphore; std::filesystem::path output; + std::filesystem::path depfile; int cap = 0, i = 2; for (; i < argc && std::string_view(argv[i]) != "--"; ++i) { const std::string_view option = argv[i]; - if ((option != "--sem" && option != "--cap" && option != "--output") || i + 1 >= argc) { + if ((option != "--sem" && option != "--cap" && option != "--output" && option != "--depfile") || i + 1 >= argc) { std::println(stderr, "error: __compiler-cap: expected --sem --cap -- ..."); return 2; } const std::string_view value = argv[++i]; if (option == "--sem") semaphore = value; else if (option == "--output") output = value; + else if (option == "--depfile") depfile = value; else { const auto [end, ec] = std::from_chars(value.data(), value.data() + value.size(), cap); if (ec != std::errc{} || end != value.data() + value.size()) return 2; @@ -299,7 +301,7 @@ int run(int argc, char** argv) { if (semaphore.empty() || cap <= 0 || i + 1 >= argc) return 2; std::vector command; for (++i; i < argc; ++i) command.emplace_back(argv[i]); - return mcpp::build::schedule::detach::run_capped(command, semaphore, cap, output); + return mcpp::build::schedule::detach::run_capped(command, semaphore, cap, output, depfile); } ReportUnnarrowablePaths reportUnnarrowable_; diff --git a/src/run.cppm b/src/run.cppm index 64ec6e5a6..98637f8da 100644 --- a/src/run.cppm +++ b/src/run.cppm @@ -17,6 +17,7 @@ export module mcpp.run; import std; +import mcpplibs.os.process; export namespace mcpp::run { @@ -158,6 +159,9 @@ struct Options { // The arguments given as one string after the program's name (Windows): // `Shell::one_string`, for the command interpreter. bool one_string = false; + // An explicitly declared unit whose registration precedes this start. + // Its owner retains the unit until retirement and final bookkeeping. + mcpplibs::os::process::owned_unit* declared_unit = nullptr; }; RunResult execute(const std::vector& argv, const Options& options = {}); diff --git a/src/run/run.cpp b/src/run/run.cpp index a77e26a6c..d0b2900b6 100644 --- a/src/run/run.cpp +++ b/src/run/run.cpp @@ -104,6 +104,7 @@ struct Spec { bool splitOnCr = false; const std::function* shouldStop = nullptr; bool oneString = false; + os::process::owned_unit* declaredUnit = nullptr; }; struct Outcome { @@ -225,7 +226,8 @@ Outcome launch(const Spec& s) { case Output::stdout_err_kept: o.out = os::process::stdio::pipe; o.err = os::process::stdio::pipe; break; case Output::discard: o.out = os::process::stdio::null; o.err = os::process::stdio::null; break; } - o.own_unit = true; + o.own_unit = s.declaredUnit == nullptr; + o.declared_unit = s.declaredUnit; o.bound_lifetime = true; o.one_string = s.oneString; auto p = Process::spawn(o); @@ -243,10 +245,10 @@ Outcome launch(const Spec& s) { const bool keepErr = s.out == Output::stdout_err_kept; const bool captured = s.out == Output::merged || s.out == Output::stdout_only || s.out == Output::stdout_err_shown || keepErr; - auto should_stop = [&] { + auto should_stop = [&](bool check_idle = true) { const auto now = clock::now(); return (s.total.count() > 0 && now - start >= s.total) - || (captured && s.idle.count() > 0 && now - lastOutput >= s.idle) + || (check_idle && captured && s.idle.count() > 0 && now - lastOutput >= s.idle) || (s.shouldStop && *s.shouldStop && (*s.shouldStop)()); }; bool errDone = !keepErr; @@ -264,38 +266,51 @@ Outcome launch(const Spec& s) { if (captured) { std::string acc; std::optional endedAt; + auto consume_output = [&](std::string_view chunk) { + lastOutput = clock::now(); + if (s.onLine) { + acc += chunk; + deliver_lines(acc, *s.onLine, s.splitOnCr, false); + } else { + out.output += chunk; + } + }; for (;;) { const auto now = clock::now(); - if (should_stop()) { - stop = true; - break; - } + if (should_stop(false)) { stop = true; break; } // A program that has ended may leave one behind holding its // output open; what arrives in the next moment is read, then the - // rest of its unit is ended. + // rest of its unit is ended. Check even when output stays ready. if (!endedAt) { if (auto w = p->try_wait(); w && w->has_value()) endedAt = now; } else if (now - *endedAt > std::chrono::milliseconds(500)) { p->end_unit(); } + // A slow line handler or delayed reader may leave progress queued. + // Observe both streams before classifying their silence as idle. drain_error(std::chrono::milliseconds(0)); + auto ready = p->read_output_for(std::chrono::milliseconds(0)); + if (!ready) break; + if (ready->has_value()) { + if ((*ready)->empty()) break; + consume_output(**ready); + continue; + } + if (should_stop()) { stop = true; break; } auto chunk = p->read_output_for(std::chrono::milliseconds(keepErr ? 20 : 100)); if (!chunk) break; if (!chunk->has_value()) continue; - if ((*chunk)->empty()) break; // the end of the output - lastOutput = clock::now(); - if (s.onLine) { - acc += **chunk; - deliver_lines(acc, *s.onLine, s.splitOnCr, false); - } else { - out.output += **chunk; - } + if ((*chunk)->empty()) break; + consume_output(**chunk); } if (s.onLine && !stop) deliver_lines(acc, *s.onLine, s.splitOnCr, true); // The rest of the error stream, for as long as the program and the - // bounds allow. + // bounds allow. Ready stderr also precedes an idle classification. while (!stop && !errDone) { const auto now = clock::now(); + if (should_stop(false)) { stop = true; break; } + drain_error(std::chrono::milliseconds(0)); + if (errDone) break; if (should_stop()) { stop = true; break; } if (!endedAt) { if (auto w = p->try_wait(); w && w->has_value()) endedAt = now; @@ -508,7 +523,8 @@ RunResult execute(const std::vector& argv, const Options& opt) { Spec s{ .argv = argv, .env = opt.env, .unset = opt.unset, .hostTool = opt.host_tool, .cwd = opt.cwd, .nullInput = opt.input == Input::none, .total = opt.total, .idle = opt.idle, .onLine = opt.on_line ? &opt.on_line : nullptr, .splitOnCr = opt.split_on_cr, - .shouldStop = opt.should_stop ? &opt.should_stop : nullptr, .oneString = opt.one_string }; + .shouldStop = opt.should_stop ? &opt.should_stop : nullptr, .oneString = opt.one_string, + .declaredUnit = opt.declared_unit }; switch (opt.streams) { case Streams::inherit: s.out = Output::inherit; break; case Streams::merged: s.out = Output::merged; break; diff --git a/tests/e2e/234_bmi_schedule_on.sh b/tests/e2e/234_bmi_schedule_on.sh index a58690688..4d6336a31 100755 --- a/tests/e2e/234_bmi_schedule_on.sh +++ b/tests/e2e/234_bmi_schedule_on.sh @@ -260,4 +260,8 @@ assert b".mcpp/compiler-runs" not in graph_before[1] print("PASS: normal/quiet cached replay retains graph, objects and BMIs without persisting generations") PY_REPLAY + +echo "== 8. missing cached objects recover before Ninja admission ==" +python3 "$_root/tests/e2e/helpers/check_cached_object_recovery.py" "$MCPP" "$TMP" "$TMP/$BUILD_DIR" + echo "234 bmi_schedule OK (detach-codegen, tokens reclaimed, publication checked)" diff --git a/tests/e2e/helpers/check_cached_object_recovery.py b/tests/e2e/helpers/check_cached_object_recovery.py new file mode 100644 index 000000000..f6a190eee --- /dev/null +++ b/tests/e2e/helpers/check_cached_object_recovery.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""Recover completed module objects in one actual build, before Ninja reads BMIs.""" +import argparse +import hashlib +import json +from pathlib import Path +import subprocess + + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('mcpp',type=Path) + parser.add_argument('project',type=Path) + parser.add_argument('output',type=Path) + args=parser.parse_args() + mcpp=args.mcpp.resolve();project=args.project.resolve();output=args.output.resolve() + rows=json.loads((output/'compile_commands.json').read_text()) + objects=[Path(row['output']) for row in rows if row['file'].endswith(('src/core.cppm','src/mid.cppm'))] + assert len(objects)==2,'two actual module objects are required' + def build(success=True): + result=subprocess.run([str(mcpp),'build','-j2'],cwd=project,capture_output=True,text=True,timeout=60) + assert (result.returncode==0)==success,result.stdout+result.stderr + return result.stdout+result.stderr + def identity(): + return {str(p.relative_to(output)):(p.stat().st_mtime_ns,hashlib.sha256(p.read_bytes()).hexdigest()) + for pattern in ('obj/**/*.o','gcm.cache/**/*.gcm') for p in output.glob(pattern)} + graph=(output/'build.ninja').read_bytes() + for missing in (objects[:1],objects): + for path in missing:path.unlink() + build() + assert all(path.is_file() for path in objects),'one invocation did not restore missing objects' + before=identity();build();assert identity()==before,'recovery changed the following no-op' + assert (output/'build.ninja').read_bytes()==graph,'recovery replaced its build graph' + print(f'PASS: {len(missing)} missing completed objects recover in one build; following no-op unchanged',flush=True) + certificate=Path(str(objects[0])+'.sched.completed');original=certificate.read_bytes() + bmi_before={p:(p.stat().st_mtime_ns,p.read_bytes()) for p in (output/'gcm.cache').rglob('*.gcm')} + try: + certificate.write_bytes(b'X'+original[1:]);objects[0].unlink() + build(False) + assert all(p.exists() and (p.stat().st_mtime_ns,p.read_bytes())==value for p,value in bmi_before.items()),'corrupt proof authorized BMI deletion' + assert not objects[0].exists(),'Ninja ran despite corrupt completion evidence' + finally: + certificate.write_bytes(original) + build();assert objects[0].exists() + print('PASS: corrupt completion evidence refuses admission without deleting any BMI',flush=True) + outside=project/'unrelated-output';outside.mkdir();marker=outside/'unrelated.gcm';marker.write_text('unrelated') + alias=output/'obj/unrelated-cache-alias' + alias.symlink_to(outside,target_is_directory=True) + try: + log=build(False) + assert 'symlinked cached compiler state' in log,log + assert marker.read_text()=='unrelated','directory symlink authorized outside output modification' + finally: + alias.unlink() + build();before=identity();build();assert identity()==before + print('PASS: directory symlink refuses admission and preserves unrelated output',flush=True) + + +if __name__=='__main__':main() diff --git a/tests/e2e/helpers/check_compiler_transaction_faults.py b/tests/e2e/helpers/check_compiler_transaction_faults.py new file mode 100644 index 000000000..873419bda --- /dev/null +++ b/tests/e2e/helpers/check_compiler_transaction_faults.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +"""Linux x86-64 syscall-boundary controls for the actual compiler helpers.""" +import argparse +import ctypes +import importlib.util +import json +import os +from pathlib import Path +import platform +import signal +import subprocess +import time + +libc = ctypes.CDLL(None, use_errno=True) +libc.ptrace.restype = ctypes.c_long +register_names = 'r15 r14 r13 r12 rbp rbx r11 r10 r9 r8 rax rcx rdx rsi rdi orig_rax rip cs eflags rsp ss fs_base gs_base ds es fs gs'.split() + + +class Registers(ctypes.Structure): + _fields_ = [(name, ctypes.c_ulonglong) for name in register_names] + + +def ptrace(operation, pid, address=0, data=0): + result = libc.ptrace(operation, pid, ctypes.c_void_p(address), + data if isinstance(data, ctypes.c_void_p) else ctypes.c_void_p(data)) + if result == -1 and ctypes.get_errno(): + raise OSError(ctypes.get_errno(), str((operation, pid))) + return result + + +def string_at(pid, address): + result = bytearray() + for offset in range(0, 4096, 8): + ctypes.set_errno(0) + value = ptrace(2, pid, address + offset) & ((1 << 64) - 1) + word = value.to_bytes(8, 'little') + result.extend(word.split(b'\0', 1)[0]) + if b'\0' in word: + return result.decode(errors='replace') + return '' + + +def syscall_path(pid, regs): + if regs.orig_rax == 82: # rename + return string_at(pid, regs.rsi) + if regs.orig_rax in (264, 316): # renameat, renameat2 + return string_at(pid, regs.r10) + if regs.orig_rax == 257: # openat + return string_at(pid, regs.rsi) + return '' + + +def trace_fault(case, boundary): + log = (case.out / 'fault.log').open('w') + child = os.fork() + if child == 0: + os.dup2(log.fileno(), 1) + os.dup2(log.fileno(), 2) + os.chdir(case.out) + ptrace(0, 0) + os.kill(os.getpid(), signal.SIGSTOP) + os.execve(str(case.binary), [str(case.binary)], dict(case.env, MCPP_TEST_LIFETIME_DELAY='60')) + tracees = {child} + entering = {} + paths = {} + hit = None + try: + os.waitpid(child, 0) + # Trace native descendants and threads without modifying product code. + ptrace(0x4200, child, 0, 1 | 2 | 4 | 8 | 16) + ptrace(24, child) + limit = time.monotonic() + 25 + while time.monotonic() < limit: + pid, status = os.waitpid(-1, os.WNOHANG | 0x40000000) + if pid == 0: + time.sleep(.0005) + continue + if os.WIFEXITED(status) or os.WIFSIGNALED(status): + tracees.discard(pid) + if pid == child: + raise AssertionError('owner ended before fault boundary: ' + boundary) + continue + event = status >> 16 + stop = os.WSTOPSIG(status) + if event in (1, 2, 3): + spawned = ctypes.c_ulonglong() + ptrace(0x4201, pid, 0, ctypes.cast(ctypes.byref(spawned), ctypes.c_void_p)) + tracees.add(spawned.value) + if event == 4 and boundary == 'after-spawn': + environment = Path(f'/proc/{pid}/environ').read_bytes().split(b'\0') + if b'MCPP_TEST_LIFETIME_ROLE=compiler' in environment: + hit = {'boundary': boundary, 'native_event': 'exec', 'writer': pid} + if stop == (signal.SIGTRAP | 0x80): + regs = Registers() + ptrace(12, pid, 0, ctypes.cast(ctypes.byref(regs), ctypes.c_void_p)) + info=ctypes.create_string_buffer(128) + ptrace(0x420e,pid,128,ctypes.cast(info,ctypes.c_void_p)) + entry=info.raw[0]==1 + if entry: + paths[pid] = syscall_path(pid, regs) + name = Path(paths.get(pid, '')).name + desired, side = { + 'before-prepared': ('prepared', True), 'after-prepared': ('prepared', False), + 'before-backup': ('baseline.bmi', True), 'after-backup': ('baseline', False), + 'before-spawn': ('admitted', False), 'after-spawn': ('', True), + 'bmi-published': ('published.gcm', False), + 'before-rc': ('unit.rc', True), 'after-rc': ('unit.rc', False), + }[boundary] + rename = regs.orig_rax in (82, 264, 316) + backup_open = boundary == 'before-backup' and regs.orig_rax == 257 + if name == desired and entry == side and (rename or backup_open): + if not entry: + assert regs.rax == 0, (boundary, regs.rax) + hit = {'boundary': boundary, 'native_event': 'syscall-entry' if entry else 'syscall-exit', 'writer': pid} + if hit: + # The actual state writer and owner disappear at this boundary. + os.kill(pid, signal.SIGKILL) + if pid != child: + os.kill(child, signal.SIGKILL) + break + ptrace(24, pid, 0, 0 if stop in (signal.SIGSTOP, signal.SIGTRAP, signal.SIGTRAP | 0x80) else stop) + assert hit, 'fault boundary not reached: ' + boundary + finally: + for pid in list(tracees): + try: + ptrace(17, pid) + except OSError: + try: + ptrace(7, pid) + except OSError: + pass + try: + os.kill(child, signal.SIGKILL) + except ProcessLookupError: + pass + # Reap traced tasks, including threads, without retaining a stopped writer. + deadline = time.monotonic() + 2 + while time.monotonic() < deadline: + try: + pid, status = os.waitpid(-1, os.WNOHANG | 0x40000000) + except ChildProcessError: + break + if pid == 0: + time.sleep(.005) + continue + if os.WIFSTOPPED(status): + try: + ptrace(17, pid) + except OSError: + ptrace(7, pid) + log.close() + return hit + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('binary', type=Path) + parser.add_argument('fixture', type=Path) + args = parser.parse_args() + assert platform.system() == 'Linux' and platform.machine() == 'x86_64' + spec = importlib.util.spec_from_file_location('lifetime', Path(__file__).with_name('check_owned_schedule_lifetime.py')) + lifetime = importlib.util.module_from_spec(spec) + spec.loader.exec_module(lifetime) + rows = [] + for boundary in ('before-prepared', 'after-prepared', 'before-backup', 'after-backup', + 'before-spawn', 'after-spawn', 'bmi-published', 'before-rc', 'after-rc'): + case = lifetime.Case(args.fixture, boundary, args.binary) + try: + (case.out / 'published.gcm').write_text('same BMI') + time.sleep(.01) + (case.out / 'source').touch() + row = trace_fault(case, boundary) + new = case.start(delay=30) + case.ready(new) + case.success(new) + before = {name: (case.out / name).stat().st_mtime_ns for name in ('published.gcm', 'module.o')} + calls = (case.out / 'compiler.runs').read_bytes() + noop = case.start(delay=30) + case.success(noop) + assert calls == (case.out / 'compiler.runs').read_bytes(), 'recovery caused a no-op compile' + assert before == {name: (case.out / name).stat().st_mtime_ns for name in before} + row.update(recovery='success', unchanged_noop=True) + rows.append(row) + print('PASS:', boundary, 'recovery and unchanged no-op', flush=True) + finally: + case.close() + (args.fixture / 'transaction-faults.json').write_text(json.dumps(rows, indent=2) + '\n') + + +if __name__ == '__main__': + main() diff --git a/tests/e2e/helpers/check_owned_schedule_lifetime.py b/tests/e2e/helpers/check_owned_schedule_lifetime.py index a07383042..9652014e7 100644 --- a/tests/e2e/helpers/check_owned_schedule_lifetime.py +++ b/tests/e2e/helpers/check_owned_schedule_lifetime.py @@ -53,7 +53,10 @@ def __init__(self, root, name, binary, kind="module", linked=False, **extra): self.roots = [] self.logs = [] word = shlex.quote(str(binary)) - prefix = "setsid --wait " if extra.get("MCPP_TEST_LIFETIME_BLOCK") else "" + # A separate group protects the deliberate state-writer stall from + # Ninja cancellation, while preserving the owner's native session. + prefix = ("python3 -c 'import os,sys;os.setpgid(0,0);os.execvp(sys.argv[1],sys.argv[1:])' " + if extra.get("MCPP_TEST_LIFETIME_BLOCK") else "") (self.out / "compile.command").write_text(f"env MCPP_TEST_LIFETIME_ROLE=compiler {word}") if kind == "module": graph = (f"rule phase\n command = {prefix}env MCPP_TEST_LIFETIME_ROLE=phase {word}\n restat = 1\n" @@ -98,7 +101,7 @@ def close(self): if child.poll() is None: child.kill() child.wait(timeout=5) - for name in ("compiler.ready", "phase.pid"): + for name in ("compiler.ready", "writer.ready", "phase.pid"): pid = integer(self.out / name) if pid and alive(pid): os.kill(pid, signal.SIGKILL) @@ -150,31 +153,52 @@ def owner_closure(root, binary): case.close() -def blocked_writer(root, binary, library, restore): +def blocked_writer(root, binary, restore): case = Case(root, "restore-barrier" if restore else "settle-barrier", binary, - MCPP_TEST_LIFETIME_BLOCK="1", LD_PRELOAD=str(library)) + MCPP_TEST_LIFETIME_BLOCK="1", MCPP_TEST_LIFETIME_PUBLICATION_GATE="publish.release") + descriptor=None + lease_held=False + previous_handler=signal.getsignal(signal.SIGIO) + breaks=[] + signal.signal(signal.SIGIO,lambda *_:breaks.append(time.monotonic())) try: (case.out / "published.gcm").write_text("same BMI") - # Make the source newer than the previously published BMI. time.sleep(0.02) (case.out / "source").touch() old = case.start(delay=20, MCPP_TEST_LIFETIME_FAIL="19" if restore else "0") case.ready(old) - until(lambda: (case.out / "writer.entered").exists(), "phase1 did not reach controlled rename") - compiler = case.compiler() - until(lambda: not alive(compiler), "compiler still alive at state-writer control") - old.send_signal(signal.SIGINT) - new = case.start(delay=30, LD_PRELOAD="", MCPP_TEST_LIFETIME_BLOCK="") - time.sleep(0.15) - assert integer(case.out / "root.ready") != new.pid, "new generation bypassed old phase1 writer" - (case.out / "writer.release").write_text("go") + compiler=case.compiler() + baseline=next((case.out.parents[2]/".mcpp/compiler-runs").glob("*/run-*/attempts/*/baseline.bmi")) + # A kernel write lease delays the real reader, including direct-syscall + # OS I/O. No product hook or libc interception changes the state writer. + descriptor=os.open(baseline,os.O_RDWR) + fcntl.fcntl(descriptor,fcntl.F_SETOWN,os.getpid()) + fcntl.fcntl(descriptor,fcntl.F_SETLEASE,fcntl.F_WRLCK) + lease_held=True + (case.out/"publish.release").write_text("go") + until(lambda:bool(breaks),"state writer did not reach the certified baseline") + until(lambda:not alive(compiler),"compiler still alive at state-writer control") + if not restore: + old.send_signal(signal.SIGINT) + new=case.start(delay=30,MCPP_TEST_LIFETIME_BLOCK="",MCPP_TEST_LIFETIME_PUBLICATION_GATE="") + time.sleep(.15) + assert integer(case.out/"root.ready")!=new.pid,"new generation bypassed a pending state writer" + fcntl.fcntl(descriptor,fcntl.F_SETLEASE,fcntl.F_UNLCK) + lease_held=False case.ready(new) case.success(new) - assert not (case.out / "published.gcm.bak").exists() - assert (case.out / "unit.rc").read_text().strip() == "0" - print("PASS: old phase1 " + ("restore" if restore else "settle") + " writes block next generation after compiler permit release") + assert old.wait(timeout=5)!=0 + assert not (case.out/"published.gcm.bak").exists() + assert (case.out/"unit.rc").read_text().strip()=="0" + print("PASS: prior "+("generation restore" if restore else "phase1 settle") + +" is serialized before the next generation after compiler permit release",flush=True) finally: - (case.out / "writer.release").write_text("go") + if descriptor is not None: + try: + if lease_held:fcntl.fcntl(descriptor,fcntl.F_SETLEASE,fcntl.F_UNLCK) + finally: + os.close(descriptor) + signal.signal(signal.SIGIO,previous_handler) case.close() @@ -239,39 +263,86 @@ def late_registration(root, binary, supervisor): case.close() + +def record(path, tag): + data=path.read_bytes() + body,checksum=data[:-18],data[-17:-1] + value=0xcbf29ce484222325 + for byte in body:value=((value^byte)*0x100000001b3)&((1<<64)-1) + assert checksum==f"{value:016x}".encode() + schema,count,body=body.split(b"\n",2) + assert schema.decode()==tag + fields=[] + for _ in range(int(count)): + size,body=body.split(b"\n",1);size=int(size) + fields.append(body[:size].decode());body=body[size:] + assert not body + return fields + + +def hard_controller_loss(root, binary, kind): + case=Case(root,"hard-controller-"+kind,binary,kind, + MCPP_TEST_LIFETIME_DESCENDANT="1") + try: + if kind=="ordinary": + word=shlex.quote(str(binary)) + (case.out/"start-second.py").write_text("from pathlib import Path\nimport time\nwhile not Path('writer.ready').exists():time.sleep(.01)\nPath('second.gate').touch()\n") + with (case.out/"build.ninja").open("a") as graph: + graph.write("rule gate\n command = python3 start-second.py\n" + "build second.gate: gate\n" + "rule second\n command = env MCPP_TEST_LIFETIME_ROLE=ordinary " + f"MCPP_TEST_LIFETIME_OBJECT=second.o {word}\n" + "build second.o: second source | second.gate\ndefault ordinary.o second.o\n") + old=case.start(delay=1800);case.ready(old);case.compiler() + until(lambda:integer(case.out/"writer.ready")!=0,"descendant did not start") + writer=integer(case.out/"writer.ready") + runtime=case.out.parents[2]/".mcpp/compiler-runs" + def admitted(): + found=list(runtime.glob("*/run-*/attempts/*/admitted")) + return next((p.parent for p in found if (p.parent/"unit").exists()),None) + until(lambda:admitted() is not None,"unit was not admitted") + attempt=admitted();unit=record(attempt/"unit","mcpp-unit-v1") + controller=int(unit[1]);os.kill(controller,signal.SIGKILL) + reservation=Path(record(attempt/"reservation","mcpp-reservation-pointer-v1")[0]) + lock=reservation.with_suffix(".lock") + time.sleep(.15) + with lock.open("a+") as handle: + fcntl.lockf(handle,fcntl.LOCK_EX|fcntl.LOCK_NB) + assert alive(writer),"descendant control ended too early" + assert reservation.exists(),"live descendant lost its logical reservation" + obj=case.out/("ordinary.o" if kind=="ordinary" else "module.o") + assert obj.read_text()=="partial","output changed before writer completion" + assert len((case.out/"compiler.runs").read_text().splitlines())==1,"cap1 admitted a replacement compiler" + assert old.wait(timeout=8)!=0,"hard controller loss reported success" + assert (case.out/"writer.finished").exists(),"counterexample descendant did not perform its late write" + assert not obj.exists(),"quiet orphan result was not invalidated" + assert not reservation.exists(),"quiet orphan kept its reservation" + if kind=="module":assert not(case.out/"published.gcm").exists() + # The actual helper is normal on retry; admission remains capped and + # the late writer cannot overwrite this generation's result. + case.env.pop("MCPP_TEST_LIFETIME_DESCENDANT") + new=case.start(delay=30);case.ready(new);case.success(new) + print("PASS:",kind,"hard controller loss retains cap1 while descendant writes; quiet recovery and retry",flush=True) + finally: + case.close() + + def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("binary", type=Path) parser.add_argument("fixture", type=Path) args = parser.parse_args() - assert shutil.which("ninja") and shutil.which("setsid"), "Ninja and setsid required" - # Extend only the phase1 rename window; the actual C++ helper remains intact. - source = args.fixture / "delay-writer.c" - source.write_text(r'''#define _GNU_SOURCE -#include -#include -#include -#include -#include -int rename(const char *from,const char *to) { - static int (*next)(const char*,const char*); if(!next) next=dlsym(RTLD_NEXT,"rename"); - if(getenv("MCPP_TEST_LIFETIME_BLOCK") && strstr(from,"published.gcm.bak")) { - FILE *f=fopen("writer.entered","w"); if(f){fputs("entered",f);fclose(f);} - for(int i=0;i<3000 && access("writer.release",F_OK);++i) usleep(10000); - } - return next(from,to); -} -''') - library = args.fixture / "delay-writer.so" - subprocess.run(["/usr/bin/cc", "-shared", "-fPIC", str(source), "-o", str(library), "-ldl"], check=True, timeout=20) + assert shutil.which("ninja"), "Ninja required" invalid_owner(args.fixture, args.binary) + for kind in ("module","ordinary"): + hard_controller_loss(args.fixture,args.binary,kind) for kind in ("module", "ordinary"): for sig in (signal.SIGINT, signal.SIGKILL): cancellation(args.fixture, args.binary, sig, kind) cancellation(args.fixture, args.binary, signal.SIGKILL, "module", linked=True) owner_closure(args.fixture, args.binary) for restore in (False, True): - blocked_writer(args.fixture, args.binary, library, restore) + blocked_writer(args.fixture, args.binary, restore) for supervisor in (False, True): late_registration(args.fixture, args.binary, supervisor) diff --git a/tests/e2e/helpers/check_shared_compiler_cap.py b/tests/e2e/helpers/check_shared_compiler_cap.py index 7a547ed82..3f65758b1 100644 --- a/tests/e2e/helpers/check_shared_compiler_cap.py +++ b/tests/e2e/helpers/check_shared_compiler_cap.py @@ -26,9 +26,24 @@ def state(path): return json.loads(path.read_text()) if path.exists() else {"active": [], "peak": 0, "calls": 0} +def process_birth(pid): + try: + fields=Path(f"/proc/{pid}/stat").read_text().split(")",1)[1].split() + return int(fields[19]) if fields[0]!="Z" else None + except (FileNotFoundError,ProcessLookupError): + return None + + +def active_originals(record): + births=record.get("births",{}) + if any(str(pid) not in births for pid in record["active"]): + raise AssertionError("counter has a PID without its original birth identity") + return [pid for pid in record["active"] if process_birth(pid)==births[str(pid)]] + + def permits_free(semaphore): for entry in semaphore.iterdir(): - if not entry.is_file(): + if not entry.is_file() or entry.name.endswith(".reservation"): return False try: with entry.open("r+b") as lock: @@ -41,13 +56,16 @@ def permits_free(semaphore): def settled(path, semaphore): deadline = time.monotonic() + 5 while time.monotonic() < deadline: - if not state(path)["active"] and permits_free(semaphore): + if not active_originals(state(path)) and permits_free(semaphore): return time.sleep(0.02) - raise AssertionError("compilers or permits survived the completed workload") + details={"counter":state(path),"pid_alive":{str(pid):alive(pid) for pid in state(path)["active"]}, + "permit_files":[entry.name for entry in semaphore.iterdir()],"kernel_locks_free":permits_free(semaphore)} + path.with_name("unsettled-evidence.json").write_text(json.dumps(details,indent=2)+"\n") + raise AssertionError("compilers or permits survived the completed workload: "+json.dumps(details)) -def check_mixed(mcpp, root, cap, ninja): +def check_mixed(mcpp, root, cap, ninja, owner_driver=None): project = root / f"mixed-{cap}" (project / "src").mkdir(parents=True) (project / "mcpp.toml").write_text('''[package] @@ -76,11 +94,22 @@ def check_mixed(mcpp, root, cap, ninja): import fcntl,json,os,subprocess,sys,time from pathlib import Path counter=Path(os.environ["MCPP_CAP_COUNTER"]) +def born(pid): + try: + fields=Path(f"/proc/{pid}/stat").read_text().split(")",1)[1].split() + return int(fields[19]) if fields[0]!="Z" else None + except (FileNotFoundError,ProcessLookupError):return None def update(start): with counter.with_suffix(".lock").open("a") as lock: fcntl.flock(lock,fcntl.LOCK_EX) record=json.loads(counter.read_text()) if counter.exists() else {"active":[],"peak":0,"calls":0} + births=record.setdefault("births",{}) + if any(str(pid) not in births for pid in record["active"]):raise RuntimeError("counter birth missing") + expired=[pid for pid in record["active"] if born(pid)!=births[str(pid)]] + record.setdefault("ended_without_finally",[]).extend({"pid":pid,"birth":births[str(pid)]} for pid in expired) + record["active"]=[pid for pid in record["active"] if pid not in expired] if start: + births[str(os.getpid())]=born(os.getpid()) record["active"].append(os.getpid());record["calls"]+=1 record["peak"]=max(record["peak"],len(record["active"])) else:record["active"].remove(os.getpid()) @@ -104,8 +133,13 @@ def update(start): env = os.environ.copy() env["MCPP_CAP_COUNTER"] = str(counter) command = [ninja, "-j", str(cap * 6)] + if owner_driver: + env.update(GTEST_FILTER="DetachLifetime.SubprocessDriver", + MCPP_TEST_LIFETIME_ROLE="root", MCPP_TEST_LIFETIME_OUTPUT=str(build), + MCPP_TEST_LIFETIME_NINJA_JOBS=str(cap*6)) + command=[str(owner_driver)] checked(command, build, log, env) - semaphore = build / ".mcpp-sched" + semaphore = next((project/".mcpp/compiler-runs").glob("*/slots")) if owner_driver else build/".mcpp-sched" settled(counter, semaphore) measured = state(counter) assert measured["peak"] == cap, f"actual compiler driver peak {measured['peak']} differs from cap {cap}" @@ -124,7 +158,11 @@ def update(start): checked(command, build, log, env) settled(counter, semaphore) assert state(counter)["peak"] <= cap, "failure recovery exceeded the compiler cap" - print(f"PASS: mixed GCC modules/C++/C cap={cap}, joined without deadlock, no-op and both failure recoveries") + mode="owned-unit generation" if owner_driver else "standalone permits" + (root/f"cap-{cap}.json").write_text(json.dumps({"mode":mode,"cap":cap,"ninja_jobs":cap*6, + "peak":state(counter)["peak"],"calls":state(counter)["calls"],"active":state(counter)["active"], + "normal_and_late_failure_recovery":True,"unchanged_noop":True},indent=2)+"\n") + print(f"PASS: mixed GCC modules/C++/C cap={cap} ({mode}), joined without deadlock, no-op and both failure recoveries") def alive(pid): @@ -143,7 +181,10 @@ def check_cancellation(mcpp, root): command = [mcpp, "__compiler-cap", "--sem", str(semaphore), "--cap", "1", "--"] child = [sys.executable, "-c", "import os,time;from pathlib import Path;" f"Path({str(pid_file)!r}).write_text(str(os.getpid()));time.sleep(60)"] - process = subprocess.Popen([*command, *child], cwd=root) + wrapper=root/"mixed-1/compiler-driver" + counter=root/"cancel-counter.json" + env=dict(os.environ,MCPP_CAP_COUNTER=str(counter)) + process = subprocess.Popen([*command,str(wrapper),*child],cwd=root,env=env) pid = None try: deadline = time.monotonic() + 5 @@ -151,15 +192,22 @@ def check_cancellation(mcpp, root): time.sleep(0.01) assert pid_file.exists(), "controlled compiler did not start" pid = int(pid_file.read_text()) + original=state(counter) + assert len(original["active"])==1 + wrapper_pid=original["active"][0];wrapper_birth=original["births"][str(wrapper_pid)] process.send_signal(signal.SIGINT) assert process.wait(timeout=5) != 0 deadline = time.monotonic() + 5 while alive(pid) and time.monotonic() < deadline: time.sleep(0.01) assert not alive(pid), "cancellation left the compiler alive" + settled(counter,semaphore) assert permits_free(semaphore), "cancellation retained its compiler permit" - result = subprocess.run([*command, sys.executable, "-c", "raise SystemExit(7)"], cwd=root) + result = subprocess.run([*command,str(wrapper),sys.executable,"-c","raise SystemExit(7)"],cwd=root,env=env) assert result.returncode == 7 and permits_free(semaphore), "failure lost its status or leaked a permit" + record=state(counter) + assert {"pid":wrapper_pid,"birth":wrapper_birth} in record["ended_without_finally"], "hard-killed counter entry was not retained as history" + assert not active_originals(record) and record["peak"]==1, "dead counter history inflated live compiler capacity" words = ["space value", 'literal "quote"', "back\\slash", "$literal", "", "Ω", "--system", "--home", "--deny", "--toolchain", "-j200", "--sem", "--cap"] result = subprocess.run([*command, sys.executable, "-c", "import json,sys;print(json.dumps(sys.argv[1:]))", *words], @@ -178,12 +226,15 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("mcpp", type=Path) parser.add_argument("fixture", type=Path) + parser.add_argument("--owner-driver",type=Path, + help="fresh schedule unit-test driver for owned-generation coverage") args = parser.parse_args() args.fixture.mkdir(parents=True) ninja = shutil.which("ninja") assert ninja, "Ninja is required" for cap in (1, 2): - check_mixed(str(args.mcpp.resolve()), args.fixture.resolve(), cap, ninja) + check_mixed(str(args.mcpp.resolve()), args.fixture.resolve(), cap, ninja, + args.owner_driver.resolve() if args.owner_driver else None) check_cancellation(str(args.mcpp.resolve()), args.fixture.resolve()) diff --git a/tests/e2e/helpers/record_close_failure.c b/tests/e2e/helpers/record_close_failure.c new file mode 100644 index 000000000..abb342a78 --- /dev/null +++ b/tests/e2e/helpers/record_close_failure.c @@ -0,0 +1,27 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include + +/* Fail only the final close of the real runtime-record temporary. The native + * close still happens, so the caller must reject publication rather than rely + * on a later destructor or a leaked stream. */ +int fclose(FILE *stream) { + static int (*original)(FILE *); + if (!original) original = dlsym(RTLD_NEXT, "fclose"); + char descriptor[64], name[4096]; + const int fd = fileno(stream); + snprintf(descriptor, sizeof descriptor, "/proc/self/fd/%d", fd); + const ssize_t size = readlink(descriptor, name, sizeof name - 1); + if (size >= 0) name[size] = 0; + const int selected = size >= 0 && strstr(name, "/.mcpp-record-") != NULL; + const int result = original(stream); + if (!selected) return result; + const int marker = open("record-close-refused", O_WRONLY | O_CREAT | O_TRUNC, 0600); + if (marker >= 0) { (void)write(marker, "injected", 8); close(marker); } + errno = EIO; + return EOF; +} diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp new file mode 100644 index 000000000..4654b6165 --- /dev/null +++ b/tests/unit/test_compiler_transaction.cpp @@ -0,0 +1,222 @@ +#include +import std; +import mcpp.build.schedule.compiler_transaction; +import mcpp.toolchain.fingerprint; +import mcpp.run; +import mcpp.invocation; +import mcpp.runtime.elf; +import mcpplibs.os.platform; + +namespace tx = mcpp::build::schedule::transaction; +namespace fs = std::filesystem; +namespace { +struct Transaction { + fs::path root, generation, bmi, object, slot, dep, reservation; + Transaction() { + root=fs::temp_directory_path()/std::format("mcpp-attempt-{:08x}",std::random_device{}()); + fs::create_directories(root); + generation=root/"run";bmi=root/"unit.gcm";object=root/"unit.o"; + slot=root/"unit.sched";dep=root/"unit.d";reservation=root/"slot.reservation"; + } + ~Transaction(){std::error_code ec;fs::remove_all(root,ec);} + tx::Attempt prepare(bool previous=true) { + if(previous)std::ofstream(bmi)<<"verified old BMI"; + auto a=tx::Attempt::prepare(generation,false,bmi,object,slot,dep); + if(!a)throw std::runtime_error(a.error()); + return *a; + } + void admitted(const tx::Attempt& a) { + ASSERT_TRUE(tx::publish(a.directory/"controller","mcpp-controller-v1",{"1","1",tx::spelling(generation/"actors/helper.lock")})); + ASSERT_TRUE(tx::publish(a.directory/"unit","mcpp-unit-v1",{"opaque-test-reference","1","1",tx::spelling(generation/"actors/helper.lock")})); + ASSERT_TRUE(tx::publish(a.directory/"reservation","mcpp-reservation-pointer-v1",{tx::spelling(reservation)})); + ASSERT_TRUE(tx::publish(reservation,"mcpp-reservation-v1",{tx::spelling(a.directory)})); + ASSERT_TRUE(tx::publish(a.directory/"admitted","mcpp-admitted-v1",{})); + } + void completed(const tx::Attempt& a,int rc=0) { + admitted(a); + std::ofstream(bmi)<<"new compiler BMI";std::ofstream(object)<<"complete object"; + ASSERT_TRUE(a.publication());ASSERT_TRUE(a.retired(rc)); + std::ofstream(tx::suffix(slot,".rc"))<(in),{}}; + } +}; +} + +TEST(CompilerTransaction, RecordsAreBoundedChecksummedAndSchemaChecked) { + Transaction t;const auto record=t.root/"record"; + ASSERT_TRUE(tx::publish(record,"schema",{"a\nb", "", "unicode \xc3\xa9"})); + auto good=tx::read(record,"schema");ASSERT_TRUE(good&&*good);EXPECT_EQ((**good)[0],"a\nb"); + EXPECT_FALSE(tx::read(record,"another-schema")); + auto bytes=Transaction::contents(record);bytes[0]='X';std::ofstream(record)<prepare_backup());fs::remove(t.root/"alias");fs::create_directory_symlink(t.root/"second",t.root/"alias"); + std::ofstream(t.root/"first/unit.gcm")<<"changed";std::ofstream(t.root/"second/unit.gcm")<<"unrelated"; + ASSERT_TRUE(tx::recover(*a).value_or(false));EXPECT_EQ(Transaction::contents(t.root/"first/unit.gcm"),"original"); + EXPECT_EQ(Transaction::contents(t.root/"second/unit.gcm"),"unrelated"); +} + +namespace { +void configuration_outputs(Transaction& t,std::string_view name="unit") { + fs::create_directories(t.root/"obj");fs::create_directories(t.root/"gcm.cache"); + t.bmi=t.root/"gcm.cache"/(std::string(name)+".gcm"); + t.object=t.root/"obj"/(std::string(name)+".m.o"); + t.slot=tx::suffix(t.object,".sched"); +} +} +TEST(CompilerTransaction, MissingCertifiedObjectIsInvalidatedBeforeNinjaAdmission) { + Transaction t;configuration_outputs(t);auto a=t.prepare();ASSERT_TRUE(a.prepare_backup());t.completed(a); + ASSERT_TRUE(tx::recover(a).value_or(false));fs::remove(t.object); + ASSERT_TRUE(tx::admit_cached_outputs(t.root));EXPECT_FALSE(fs::exists(t.bmi)); + EXPECT_EQ(Transaction::contents(tx::suffix(t.bmi,".bak")),"new compiler BMI"); + EXPECT_FALSE(fs::exists(tx::suffix(t.slot,".completed"))); + EXPECT_TRUE(tx::admit_cached_outputs(t.root)); +} +TEST(CompilerTransaction, TwoMissingCertifiedObjectsAreInvalidatedInOneAdmission) { + Transaction t;configuration_outputs(t,"first");auto first=t.prepare();ASSERT_TRUE(first.prepare_backup());t.completed(first); + ASSERT_TRUE(tx::recover(first).value_or(false)); + configuration_outputs(t,"second");auto second=t.prepare();ASSERT_TRUE(second.prepare_backup());t.completed(second); + ASSERT_TRUE(tx::recover(second).value_or(false));fs::remove(first.object);fs::remove(second.object); + ASSERT_TRUE(tx::admit_cached_outputs(t.root));EXPECT_FALSE(fs::exists(first.bmi));EXPECT_FALSE(fs::exists(second.bmi)); +} +TEST(CompilerTransaction, CorruptOrForeignCertificateCannotAuthorizeAdmissionRecovery) { + Transaction t;configuration_outputs(t);auto a=t.prepare();ASSERT_TRUE(a.prepare_backup());t.completed(a); + ASSERT_TRUE(tx::recover(a).value_or(false));fs::remove(t.object); + const auto path=tx::suffix(t.slot,".completed");auto bytes=Transaction::contents(path);bytes[0]='X';std::ofstream(path)<interp.find("ld-linux")==std::string::npos) + GTEST_SKIP()<<"close interposition requires the dynamic glibc loader"; + Transaction t; + const auto library=t.root/"record-close-failure.so"; + const auto source=fs::current_path()/"tests/e2e/helpers/record_close_failure.c"; + auto compiled=mcpp::run::execute({"/usr/bin/cc","-shared","-fPIC",source.string(), + "-o",library.string(),"-ldl"},{.total=std::chrono::seconds(10)}); + ASSERT_EQ(compiled.exit_code,0)<failed(); }}); + std::cout<finish(); return done ? result.exit_code : 2; @@ -137,15 +139,28 @@ int lifetime_driver() { if (role == "await") return detach::await_unit("unit", "module.o"); if (role == "ordinary") return detach::run_capped({"env", "MCPP_TEST_LIFETIME_ROLE=compiler", self.string()}, - "permits", 1, "ordinary.o"); - if (role == "compiler") { + "permits", 1, lifetime_env("MCPP_TEST_LIFETIME_OBJECT","ordinary.o")); + if (role == "compiler" && lifetime_env("MCPP_TEST_LIFETIME_DESCENDANT") == "1") { + const auto who=mcpplibs::os::process::self(); + std::ofstream("compiler.ready")<id<<' '<started_ns; + mcpplibs::os::process::spawn_options options; + options.program=self;options.env={{"MCPP_TEST_LIFETIME_ROLE","writer"}}; + options.out=options.err=mcpplibs::os::process::stdio::inherit; + auto writer=mcpplibs::os::process::process::spawn(options); + if(!writer)return 2; + auto result=writer->wait();return result?result->shell_code():2; + } + if (role == "compiler" || role == "writer") { const auto process = mcpplibs::os::process::self(); - std::ofstream("compiler.ready") << process->id << ' ' << process->started_ns; + std::ofstream(role=="writer"?"writer.ready":"compiler.ready") << process->id << ' ' << process->started_ns; std::ofstream("compiler.runs", std::ios::app) << process->id << '\n'; const bool ordinary = lifetime_env("MCPP_TEST_LIFETIME_KIND") == "ordinary"; - const auto object = ordinary ? "ordinary.o" : "module.o"; + const auto object = ordinary ? lifetime_env("MCPP_TEST_LIFETIME_OBJECT","ordinary.o") : "module.o"; std::ofstream(object) << "partial"; const int fail = std::stoi(lifetime_env("MCPP_TEST_LIFETIME_FAIL", "0")); + const auto publicationGate=lifetime_env("MCPP_TEST_LIFETIME_PUBLICATION_GATE"); + while(!publicationGate.empty()&&!fs::exists(publicationGate)) + std::this_thread::sleep_for(std::chrono::milliseconds(5)); if (!ordinary && fail != 19) { std::ofstream("published.gcm.new") << "same BMI"; fs::rename("published.gcm.new", "published.gcm"); @@ -153,6 +168,7 @@ int lifetime_driver() { std::this_thread::sleep_for(std::chrono::milliseconds( std::stoi(lifetime_env("MCPP_TEST_LIFETIME_DELAY", "20000")))); if (!fail) std::ofstream(object) << "complete"; + if(role=="writer")std::ofstream("writer.finished")<<"committed"; return fail; } return 2; @@ -178,3 +194,29 @@ TEST(DetachLifetime, OwnerCancellationAndImmediateRestart) { mcpp::invocation::self_exe_path().string(), temporary.dir.string()}); EXPECT_EQ(result.exit_code, 0) << result.output; } + +TEST(DetachLifetime, TransactionRecoveryAtNativeSyscallBoundaries) { + if constexpr (!mcpplibs::os::is_linux + || mcpplibs::os::ARCH != mcpplibs::os::architecture::x86_64) + GTEST_SKIP() << "Linux x86-64 syscall boundary control"; + Slot temporary; + const auto helper=fs::current_path()/"tests/e2e/helpers/check_compiler_transaction_faults.py"; + ASSERT_TRUE(fs::exists(helper)); + const auto result=mcpp::run::execute({"python3",helper.string(), + mcpp::invocation::self_exe_path().string(),temporary.dir.string()}, + {.total=std::chrono::seconds(120)}); + EXPECT_EQ(result.exit_code,0)< #include +#include #include import std; @@ -19,6 +20,64 @@ namespace process = mcpp::run; // change mcpp's own environment — that mutation is exactly what leaked into // /bin/sh and crashed it on newer-glibc hosts. The addition is the started // program's alone; this invocation's statement is what it was. +TEST(RunExec, OutputProducer) { + if (mcpp::invocation::get("MCPP_TEST_OUTPUT_PRODUCER") != "1") + GTEST_SKIP() << "subprocess producer"; + std::fputs("mcpp-stderr-ready\n", stdout); + std::fflush(stdout); + const bool error = mcpp::invocation::get("MCPP_TEST_OUTPUT_STREAM") != "stdout"; + if (error) std::fclose(stdout); + FILE* stream = error ? stderr : stdout; + const int count = std::stoi(mcpp::invocation::get("MCPP_TEST_OUTPUT_TICKS").value_or("24")); + auto next = std::chrono::steady_clock::now(); + for (int i = 0; i < count; ++i) { + const auto tick = std::chrono::duration_cast( + std::chrono::steady_clock::now().time_since_epoch()).count(); + std::fprintf(stream, "%lld\n", static_cast(tick)); + std::fflush(stream); + next += std::chrono::milliseconds(30); + std::this_thread::sleep_until(next); + } + std::_Exit(7); +} + +TEST(RunExec, QueuedOutputRenewsIdleAfterALineHandler) { + std::vector ticks; + const auto result = process::execute({mcpp::invocation::self_exe_path().string(), + "--gtest_filter=RunExec.OutputProducer"}, { + .env = {{"MCPP_TEST_OUTPUT_PRODUCER", "1"}, {"MCPP_TEST_OUTPUT_STREAM", "stdout"}, {"MCPP_TEST_OUTPUT_TICKS", "48"}}, + .streams = process::Streams::merged, + .total = std::chrono::seconds(3), .idle = std::chrono::milliseconds(500), + .on_line = [&](std::string_view line) { + if (line == "mcpp-stderr-ready") + std::this_thread::sleep_for(std::chrono::milliseconds(750)); + long long tick; + if (std::from_chars(line.data(), line.data() + line.size(), tick).ec == std::errc{}) + ticks.push_back(tick); + }}); + EXPECT_EQ(result.exit_code, 7); + EXPECT_FALSE(result.timed_out); + EXPECT_EQ(ticks.size(), 48u); +} + +TEST(RunExec, QueuedErrorOutputRenewsIdleAfterALineHandler) { + const auto result = process::execute({mcpp::invocation::self_exe_path().string(), + "--gtest_filter=RunExec.OutputProducer"}, { + .env = {{"MCPP_TEST_OUTPUT_PRODUCER", "1"}, {"MCPP_TEST_OUTPUT_TICKS", "48"}}, + .streams = process::Streams::output_error_kept, + .total = std::chrono::seconds(3), .idle = std::chrono::milliseconds(500), + .on_line = [](std::string_view line) { + if (line == "mcpp-stderr-ready") + std::this_thread::sleep_for(std::chrono::milliseconds(750)); + }}); + EXPECT_EQ(result.exit_code, 7) << result.error_output; + EXPECT_FALSE(result.timed_out); + std::istringstream input(result.error_output); + int ticks = 0; + for (long long tick; input >> tick;) ++ticks; + EXPECT_EQ(ticks, 48); +} + TEST(RunExec, StreamCompletionAndChildWaitKeepTheirBounds) { struct Case { const char* name; @@ -48,7 +107,13 @@ TEST(RunExec, StreamCompletionAndChildWaitKeepTheirBounds) { if (test.stop) options.should_stop = [&] { return std::chrono::steady_clock::now() - started > std::chrono::milliseconds(50); }; - const auto result = process::execute({"/bin/sh", "-c", test.command}, options); + const bool producer = std::string_view(test.command) == "@stderr-producer"; + if (producer) options.env.emplace_back("MCPP_TEST_OUTPUT_PRODUCER", "1"); + const std::vector argv = producer + ? std::vector{mcpp::invocation::self_exe_path().string(), + "--gtest_filter=RunExec.OutputProducer"} + : std::vector{"/bin/sh", "-c", test.command}; + const auto result = process::execute(argv, options); if (test.exit >= 0) { EXPECT_EQ(result.exit_code, test.exit); EXPECT_FALSE(result.timed_out); @@ -56,7 +121,22 @@ TEST(RunExec, StreamCompletionAndChildWaitKeepTheirBounds) { EXPECT_NE(result.exit_code, 0); EXPECT_TRUE(result.timed_out); } - EXPECT_LT(std::chrono::steady_clock::now() - started, std::chrono::seconds(1)); + const auto elapsed = std::chrono::steady_clock::now() - started; + if (producer) { + // One process emits across the idle window. Forking an external + // sleep for every tick would add platform-dependent launch gaps. + SCOPED_TRACE(result.error_output); + std::istringstream input(result.error_output); + std::vector ticks; + for (long long tick; input >> tick;) ticks.push_back(tick); + ASSERT_EQ(ticks.size(), 24u); + for (std::size_t i = 1; i < ticks.size(); ++i) + EXPECT_LT(std::chrono::nanoseconds(ticks[i] - ticks[i - 1]), options.idle); + EXPECT_GT(elapsed, options.idle); + EXPECT_LT(elapsed, std::chrono::seconds(3)); + } else { + EXPECT_LT(elapsed, std::chrono::seconds(1)); + } }; for (const auto streams : {process::Streams::inherit, process::Streams::discard, process::Streams::merged, process::Streams::output, @@ -66,8 +146,7 @@ TEST(RunExec, StreamCompletionAndChildWaitKeepTheirBounds) { exercise(streams, {"both-EOF-idle", "exec 1>&- 2>&-; sleep 2", 0, 50}); } exercise(process::Streams::output_error_kept, - {"stderr-activity", "exec 1>&-; for i in 1 2 3 4 5 6 7 8; do printf x >&2; sleep .03; done; exit 7", - 0, 100, false, 7}); + {"stderr-activity", "@stderr-producer", 3000, 500, false, 7}); } TEST(RunExec, StopPredicateAppliesAfterCapturedStreamsClose) { From 268f6f28097eaba4b48e02a87f2b024d1763b4bc Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 07:57:13 +0900 Subject: [PATCH 13/40] docs: record compiler recovery publication gates and remaining path --- .../2026-11-06-compiler-attempt-recovery.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md index fd254d24c..d5f52942f 100644 --- a/.agents/docs/2026-11-06-compiler-attempt-recovery.md +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -187,3 +187,54 @@ A deterministic cancellation control kills a real wrapper, retains that history, then starts a command returning seven without inflating peak one. The prior false failure and its dead-process/empty-reservation evidence remain part of the experiment record. + +## 2026-11-06 publication dependencies and remaining qualification + +The source group carries the same four additive C operations through the +native implementation, C library, C++ runtime, image runner and platform. +The core consumes the OS composition rather than native process identifiers. +No runtime generation becomes a dependency version or build-cache identity. + +| Package | Candidate version | Review | +|---|---|---| +| OpenKal | 0.24.0 | [OpenKal 62](https://github.com/mcpplibs/openkal/pull/62) | +| Linux implementation | 0.25.0 | [Linux 51](https://github.com/mcpplibs/openkal-linux/pull/51) | +| macOS implementation | 0.22.0 | [macOS 43](https://github.com/mcpplibs/openkal-macos/pull/43) | +| Windows implementation | 0.20.0 | [Windows 44](https://github.com/mcpplibs/openkal-windows/pull/44) | +| Emscripten implementation | 0.13.0 | [Emscripten 15](https://github.com/mcpplibs/openkal-emscripten/pull/15) | +| Image loader, runner, tools and binding | 0.5.0 | [Image 29](https://github.com/mcpplibs/openkal-image/pull/29) | +| C library | 0.28.0 | [musl 86](https://github.com/mcpplibs/openkal-musl/pull/86) | +| C++ runtime | 0.19.0 | [Runtime 66](https://github.com/mcpplibs/openkal-llvm-runtime/pull/66) | +| Platform | 0.24.0 | Publication follows its public lower dependencies | +| OS composition | 0.2.10 | [OS 17](https://github.com/mcpplibs/os/pull/17) | +| Core | 2026.11.5.2 | [mcpp 889](https://github.com/mcpp-community/mcpp/pull/889) | + +Emscripten receives dependency metadata alignment; it does not acquire process +support. The image runner retains its existing generated binding mechanism, +format, prefetch and cache policies. Its generated table retains all 153 prior +C names and adds exactly the four declared-unit names, for 157 total. The new +image consumer calls all four operations through the image convention, +including the fifth retirement argument and an invalid continuation that +changes the state output from `empty` to `unknown`. + +Source-group controls establish implementation behaviour. They do not establish +that public versions are installable. Release gates remain full repository CI, +canonical tag archives, byte-identical CN mirrors, published index entries and +ordinary GLOBAL/CN consumers. Temporary image source overrides are removed +before its final exact-head public-dependency CI. Platform and OS qualification +follow the lower public group; core then updates its OS and default platform +pins and runs full exact-head CI before independent ecosystem review. Plugins +PR 52 retains the public 2026.11.5.1 pin until core 2026.11.5.2 is published, +then runs its actual sidecar integration and final CI. + +## Separate Windows build-program defect + +The released 2026.11.5.1 engine can try to install `msvc@system` as an indexed +`xim` package when a Windows host with the MSVC default executes a `build.mcpp` +for a bare cross target. The specification's ABI-layout fixture exposed this +path because its build program only supplied an include directory. Declaring +that directory through the existing `[build].include_dirs` preserves the real +32/64-bit target-layout assertions and removes that unnecessary host build. +It does not fix the engine's MSVC host-toolchain selection. This is a core +build-program/toolchain issue, separate from unit lifetime and recovery, and +remains an explicit review limitation of this source group. From 05a8fda30ab034df5158b281e1a01c4f7f0a16d6 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 08:18:48 +0900 Subject: [PATCH 14/40] chore: bind recovery to the isolated OS 0.3 platform group --- .agents/docs/2026-11-06-compiler-attempt-recovery.md | 6 +++++- docs/24-openkal-cross.md | 10 ++++++++-- docs/zh/24-openkal-cross.md | 8 ++++++-- mcpp.toml | 2 +- modules/invocation/mcpp.toml | 2 +- modules/manifest/mcpp.toml | 2 +- modules/toolchain-model/mcpp.toml | 2 +- modules/toolchain-model/src/triple.cppm | 4 ++-- 8 files changed, 25 insertions(+), 11 deletions(-) diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md index d5f52942f..dc30669d5 100644 --- a/.agents/docs/2026-11-06-compiler-attempt-recovery.md +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -206,9 +206,13 @@ No runtime generation becomes a dependency version or build-cache identity. | C library | 0.28.0 | [musl 86](https://github.com/mcpplibs/openkal-musl/pull/86) | | C++ runtime | 0.19.0 | [Runtime 66](https://github.com/mcpplibs/openkal-llvm-runtime/pull/66) | | Platform | 0.24.0 | Publication follows its public lower dependencies | -| OS composition | 0.2.10 | [OS 17](https://github.com/mcpplibs/os/pull/17) | +| OS composition | 0.3.0 | [OS 17](https://github.com/mcpplibs/os/pull/17) | | Core | 2026.11.5.2 | [mcpp 889](https://github.com/mcpp-community/mcpp/pull/889) | +OS advances to 0.3.0 so an existing 0.2 constraint cannot silently absorb a +different OpenKal minor group. The core updates all four OS requirements +together. Platform 0.24 is the corresponding explicit image/hosted boundary. + Emscripten receives dependency metadata alignment; it does not acquire process support. The image runner retains its existing generated binding mechanism, format, prefetch and cache policies. Its generated table retains all 153 prior diff --git a/docs/24-openkal-cross.md b/docs/24-openkal-cross.md index fd7df1b35..6fbce239e 100644 --- a/docs/24-openkal-cross.md +++ b/docs/24-openkal-cross.md @@ -111,7 +111,7 @@ practice. ```toml [target.'cfg(any(linux, macos, windows))'] -platform = "openkal-platform@0.23.6" +platform = "openkal-platform@0.24.0" ``` One table (2026.10.27.1+). It selects the openkal platform for the host triples, @@ -411,7 +411,13 @@ Nothing else is declared (2026.10.27.1+): the target table selects `openkal-platform` for both rows, and the platform states the toolchain, the layers, the tools, the conventions below and the runner ([35](35-platforms.md)). -In mcpp 2026.11.5.1, both built-in openkal targets select platform 0.23.6 by default. An explicit `platform` configuration overrides that selection. This platform preserves the emulated TLS convention through the final ThinLTO link. +Both built-in openkal targets select platform 0.24.0 by default (2026.11.5.2+). +An explicit `platform` configuration overrides that selection. This platform +preserves the emulated TLS convention through the final ThinLTO link. + +OS 0.3 uses the OpenKal 0.24 dependency group. A project retaining OS 0.2 +selects a platform from the matching 0.23 group explicitly; incompatible +OpenKal minor requirements are rejected during dependency resolution. | | the platform's statement | the reason | |---|---|---| diff --git a/docs/zh/24-openkal-cross.md b/docs/zh/24-openkal-cross.md index 353683d76..e5fdf039c 100644 --- a/docs/zh/24-openkal-cross.md +++ b/docs/zh/24-openkal-cross.md @@ -93,7 +93,7 @@ SDK，或者区分 `linux`/`windows`/`macos`。那些是 C 环境层或平台层 ```toml [target.'cfg(any(linux, macos, windows))'] -platform = "openkal-platform@0.23.6" +platform = "openkal-platform@0.24.0" ``` 一张表（2026.10.27.1+）。它为宿主三元组选择 openkal 平台，平台的宿主部分命名编译器与目标侧三个层的包； @@ -361,7 +361,11 @@ mcpp run --target x86_64-openkal # okrun bin/app.oke 无需其他声明（2026.10.27.1+）：目标表为两行选择 `openkal-platform`，由平台陈述工具链、各层、工具、下表的约定 与 runner（[35](35-platforms.md)）。 -mcpp 2026.11.5.1 的两个内置 openkal 目标默认选择平台 0.23.6。显式 `platform` 配置可覆盖默认选择；该平台的最终 ThinLTO 链接保持 emulated TLS 约定。 +两个内置 openkal 目标默认选择平台 0.24.0（2026.11.5.2+）。显式 `platform` 配置可覆盖默认选择； +该平台的最终 ThinLTO 链接保持 emulated TLS 约定。 + +OS 0.3 使用 OpenKal 0.24 依赖组。保留 OS 0.2 的工程显式选择匹配的 0.23 平台组； +不兼容的 OpenKal 次版本要求在依赖解析阶段被拒绝。 | | 平台的陈述 | 原因 | |---|---|---| diff --git a/mcpp.toml b/mcpp.toml index 8d04a856b..312633a23 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -91,7 +91,7 @@ mcpplibs.cmdline = "0.0.1" # The operating system is mcpplibs.os's (design 2026-10-26 §25): mcpp keeps no # layer of its own. [dependencies.mcpplibs] -os = "0.2.3" +os = "0.3.0" # The codecs of a package's container (SPEC-014 §5): gzip and raw DEFLATE, # zstd, xz. The reference libraries, so an archive is written the same way on diff --git a/modules/invocation/mcpp.toml b/modules/invocation/mcpp.toml index 84fa40b7b..a69997f91 100644 --- a/modules/invocation/mcpp.toml +++ b/modules/invocation/mcpp.toml @@ -13,4 +13,4 @@ description = "The environment of one invocation of mcpp: statements over the st license = "Apache-2.0" [dependencies.mcpplibs] -os = "0.2.3" +os = "0.3.0" diff --git a/modules/manifest/mcpp.toml b/modules/manifest/mcpp.toml index 6376dc060..b249e1e80 100644 --- a/modules/manifest/mcpp.toml +++ b/modules/manifest/mcpp.toml @@ -28,7 +28,7 @@ versioning = { path = "../versioning" } # state its contract in isolation; the root `tests/unit/` states the facts two # layers must agree on. [dependencies.mcpplibs] -os = "0.2.3" +os = "0.3.0" [dev-dependencies.compat] gtest = "1.15.2" diff --git a/modules/toolchain-model/mcpp.toml b/modules/toolchain-model/mcpp.toml index ff0598951..a288b4276 100644 --- a/modules/toolchain-model/mcpp.toml +++ b/modules/toolchain-model/mcpp.toml @@ -27,7 +27,7 @@ versioning = { path = "../versioning" } # state its contract in isolation; the root `tests/unit/` states the facts two # layers must agree on. [dependencies.mcpplibs] -os = "0.2.3" +os = "0.3.0" [dev-dependencies.compat] gtest = "1.15.2" diff --git a/modules/toolchain-model/src/triple.cppm b/modules/toolchain-model/src/triple.cppm index 25f1af073..c12d0267f 100644 --- a/modules/toolchain-model/src/triple.cppm +++ b/modules/toolchain-model/src/triple.cppm @@ -906,9 +906,9 @@ inline constexpr TargetInfo kKnownTargets[] = { // `openkal-platform` (SPEC-013), pinned here. The two architectures the // loader relocates. { "x86_64-openkal", "preview", "ok", "llvm@23.1.3","", true, - "mcpplibs.openkal-platform@0.23.6" }, + "mcpplibs.openkal-platform@0.24.0" }, { "aarch64-openkal", "preview", "ok", "llvm@23.1.3","", true, - "mcpplibs.openkal-platform@0.23.6" }, + "mcpplibs.openkal-platform@0.24.0" }, }; inline std::span known_targets() { return kKnownTargets; } From 0ee2ce403d8fac82ddfe9058692f8ba5772b078f Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 08:34:26 +0900 Subject: [PATCH 15/40] docs: distinguish exact pins from compatible OS ranges --- .agents/docs/2026-11-06-compiler-attempt-recovery.md | 5 +++-- docs/24-openkal-cross.md | 2 +- docs/zh/24-openkal-cross.md | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md index dc30669d5..c242ddbdf 100644 --- a/.agents/docs/2026-11-06-compiler-attempt-recovery.md +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -209,8 +209,9 @@ No runtime generation becomes a dependency version or build-cache identity. | OS composition | 0.3.0 | [OS 17](https://github.com/mcpplibs/os/pull/17) | | Core | 2026.11.5.2 | [mcpp 889](https://github.com/mcpp-community/mcpp/pull/889) | -OS advances to 0.3.0 so an existing 0.2 constraint cannot silently absorb a -different OpenKal minor group. The core updates all four OS requirements +OS advances to 0.3.0 so an explicit range such as `^0.2.3` cannot silently +absorb a different OpenKal minor group. Bare package versions name exact wire +addresses; `os = "0.2.3"` itself does not automatically upgrade. The core updates all four OS requirements together. Platform 0.24 is the corresponding explicit image/hosted boundary. Emscripten receives dependency metadata alignment; it does not acquire process diff --git a/docs/24-openkal-cross.md b/docs/24-openkal-cross.md index 6fbce239e..ef495dac3 100644 --- a/docs/24-openkal-cross.md +++ b/docs/24-openkal-cross.md @@ -415,7 +415,7 @@ Both built-in openkal targets select platform 0.24.0 by default (2026.11.5.2+). An explicit `platform` configuration overrides that selection. This platform preserves the emulated TLS convention through the final ThinLTO link. -OS 0.3 uses the OpenKal 0.24 dependency group. A project retaining OS 0.2 +OS 0.3 uses the OpenKal 0.24 dependency group. A project retaining OS 0.2.9 selects a platform from the matching 0.23 group explicitly; incompatible OpenKal minor requirements are rejected during dependency resolution. diff --git a/docs/zh/24-openkal-cross.md b/docs/zh/24-openkal-cross.md index e5fdf039c..63cab31d4 100644 --- a/docs/zh/24-openkal-cross.md +++ b/docs/zh/24-openkal-cross.md @@ -364,7 +364,7 @@ mcpp run --target x86_64-openkal # okrun bin/app.oke 两个内置 openkal 目标默认选择平台 0.24.0（2026.11.5.2+）。显式 `platform` 配置可覆盖默认选择； 该平台的最终 ThinLTO 链接保持 emulated TLS 约定。 -OS 0.3 使用 OpenKal 0.24 依赖组。保留 OS 0.2 的工程显式选择匹配的 0.23 平台组； +OS 0.3 使用 OpenKal 0.24 依赖组。保留 OS 0.2.9 的工程显式选择匹配的 0.23 平台组； 不兼容的 OpenKal 次版本要求在依赖解析阶段被拒绝。 | | 平台的陈述 | 原因 | From 5279ab7573231e0352ec35e31410d1854f279d1c Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 12:58:29 +0900 Subject: [PATCH 16/40] feat: describe selected launch configurations and named test runners --- ...ch-configuration-and-named-test-runners.md | 39 ++++++++++++++ CHANGELOG.md | 8 ++- docs/04-mcpp-toml.md | 2 +- docs/08-testing.md | 12 +++++ docs/24-openkal-cross.md | 2 +- docs/25-home-indices-and-store.md | 2 +- docs/30-build-mcpp.md | 2 +- docs/50-machine-output.md | 7 +++ docs/specs/concurrency.md | 4 +- docs/zh/04-mcpp-toml.md | 2 +- docs/zh/08-testing.md | 11 ++++ docs/zh/24-openkal-cross.md | 2 +- docs/zh/25-home-indices-and-store.md | 2 +- docs/zh/30-build-mcpp.md | 2 +- docs/zh/50-machine-output.md | 5 ++ mcpp.toml | 2 +- modules/versioning/src/version.cppm | 2 +- src/build/execute.cppm | 21 +++++++- src/cli.cppm | 2 + src/cli/cmd_build.cppm | 5 ++ .../e2e/333_named_runners_locked_and_sbom.sh | 19 +++++++ tests/e2e/936_the_control_contract.sh | 5 ++ ...ery_member_and_runs_where_it_is_reached.sh | 11 ++++ tests/e2e/993_a_test_describes_its_launch.sh | 53 +++++++++++++++++++ 24 files changed, 208 insertions(+), 14 deletions(-) create mode 100644 .agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md diff --git a/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md b/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md new file mode 100644 index 000000000..c33ff4ac1 --- /dev/null +++ b/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md @@ -0,0 +1,39 @@ +# Launch configuration and named test runners + +Status, 2026-11-06: implemented for candidate 2026.11.6.1. Focused local +controls pass; final public-dependency and cross-platform CI remain pending. + +A consumer choosing a debugger runner needs the configuration of the program +selected to run. A separate `self env` query may describe a different profile +or the first leg of a target set instead of its reachable launch leg. The +launch producer already holds the selected build context and can describe it +without another planning protocol. + +`mcpp.launch.data.configuration` therefore adds `triple`, `profile`, +`platform` and `runners` to both program and test launch descriptions. +The first three fields match the build result's configuration. `runners` +lists nonempty effective named-runner declarations even when no platform is +active. Older consumers can ignore the additive object; newer consumers test +for its presence. No device or debugger name acquires core semantics. + +`mcpp test --runner ` uses the existing named-runner resolver. The same +choice applies to ordinary test execution, launch description and execution +of tests from an archive. A missing name is refused, rather than running the +test directly. `--runner` and `--no-runner` are contradictory and refused. +Timeouts, output capture, execution-host routing and test verdicts retain +their existing behavior. The platform or project owns the runner's command. + +The IDE can use the additive configuration to choose a platform's `debuggee` +runner while preserving the selected target and profile, and reject a +different configuration in the second result. OpenKal platform 0.24 declares +that runner using existing platform fields. The core contains no OpenKal +debugger flag, loader path or JIT policy. + +The fresh local candidate passes E2E 01, 333, 993 and 936. E2E 993 exercises a +real named runner, its returned command, ordinary and archived test execution, +argument preservation and refusals. E2E 936 passes with the inherited +`NO_COLOR=1` removed; its assertions remain unchanged. The build uses the OS +0.3 candidate source and does not establish final public installation. +An actual IDE-backend/core integration additionally passes eleven program +and test launch checks without a mock. The modified target-set E2E 963 and +the final packaged editor/runtime checks still require their full scopes. diff --git a/CHANGELOG.md b/CHANGELOG.md index 49c4e4474..904b26e51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,16 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. -## [2026.11.5.2] - 2026-11-05 +## [2026.11.6.1] - 2026-11-06 ### Fixed +- Launch descriptions include the selected target, profile, resolved platform + and available named runners, including the reachable leg of a target set. + Editor debug launches can retain the configuration that actually built the program. +- Tests accept `--runner ` with the same runner resolution as `run`, + including `--print-launch` and archive execution. Missing names and a + simultaneous `--no-runner` are refused. - Cached builds load action ownership when an action starts, so internal progress appears while Ninja is still buffering stdout. Idle builds retain lazy loading; command results and output checks remain authoritative. diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index b02ca68f1..207b3da8f 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1049,7 +1049,7 @@ with two behaviours has to be. `bmi_schedule` decides when importers are unblocked. With GCC and `bmi_schedule = "on"`, ordinary compilation and detached code -generation share the same total `jobs` limit (2026.11.5.2+). Additional +generation share the same total `jobs` limit (2026.11.6.1+). Additional scheduling and join tasks do not increase the number of compiler processes. Cancelling the build ends its ordinary and detached compilers. An immediately following build waits for their cleanup before starting another compiler; diff --git a/docs/08-testing.md b/docs/08-testing.md index c76813369..45418b758 100644 --- a/docs/08-testing.md +++ b/docs/08-testing.md @@ -170,6 +170,18 @@ or a QEMU exit code is what a bare-metal runner is chosen to produce. `--no-runner` exists for a host that can execute the binaries natively and should not pay for an emulator. +`--runner ` (2026.11.6.1+) selects a named runner instead of the default, +using the same declarations and argument substitution as `mcpp run --runner`. +It also applies to `--print-launch` and tests read from an archive. An absent +runner name is refused; `--runner` and `--no-runner` cannot be combined. + +For a platform that declares `debuggee`, this builds one test and prints the +command a debugger starts, without starting the test: + +```bash +mcpp test unit/math --runner debuggee --print-launch +``` + `--no-run` makes the narrower claim, and it has to be asked for. Without it, a target this host can neither execute nor reach through a runner leaves every test not run and the command exits 2: mcpp did not establish whether the tests diff --git a/docs/24-openkal-cross.md b/docs/24-openkal-cross.md index ef495dac3..4210a02db 100644 --- a/docs/24-openkal-cross.md +++ b/docs/24-openkal-cross.md @@ -411,7 +411,7 @@ Nothing else is declared (2026.10.27.1+): the target table selects `openkal-platform` for both rows, and the platform states the toolchain, the layers, the tools, the conventions below and the runner ([35](35-platforms.md)). -Both built-in openkal targets select platform 0.24.0 by default (2026.11.5.2+). +Both built-in openkal targets select platform 0.24.0 by default (2026.11.6.1+). An explicit `platform` configuration overrides that selection. This platform preserves the emulated TLS convention through the final ThinLTO link. diff --git a/docs/25-home-indices-and-store.md b/docs/25-home-indices-and-store.md index 998d25276..dd4814da8 100644 --- a/docs/25-home-indices-and-store.md +++ b/docs/25-home-indices-and-store.md @@ -90,7 +90,7 @@ configuration, so it remains usable while the network is broken. An install tracks byte progress and source changes separately for each active download. Time spent waiting for a download slot does not consume that budget. Progress on another file does not renew a stalled file's budget. -From 2026.11.5.2, a stalled-download diagnostic names the file, elapsed interval +From 2026.11.6.1, a stalled-download diagnostic names the file, elapsed interval and source attempt. Provider preparation and process timeouts are reported separately. Mirror selection uses `mcpp self config --mirror auto`, `CN` or `GLOBAL`. diff --git a/docs/30-build-mcpp.md b/docs/30-build-mcpp.md index bf197dccd..2ba194da1 100644 --- a/docs/30-build-mcpp.md +++ b/docs/30-build-mcpp.md @@ -1097,7 +1097,7 @@ none to rely on), and the only interpolations are a closed set: | `${mcpp.out_dir}` | the build output directory | | `${mcpp.bin_dir}` | where produced binaries land: `bin/`, or, for an action a workspace member's build program declares, that member's product directory (2026.9.29.4+) | | `${mcpp.compile_db}` | path to `compile_commands.json` (what clang-tidy's `-p` wants) | -| `${mcpp.contract}` *(2026.10.30.1+)* | path to the configuration's contract, `mcpp-contract.json`, beside its compile database: the compiler of each language with the words every unit is compiled with (target, sysroot, target-side headers, platform and profile flags), and the link driver, its words and the archiver. A foreign build system reads it to compile for the target as the engine does; a package's own include directories are not in it, nor the engine's C++ dialect and module words (`-std=`, `-fmodules`), which are its choice for its own units. Since 2026.11.5.2, link arguments describe the target configuration; the consuming package's link flags and libraries are excluded, since they can name outputs the foreign build has yet to produce | +| `${mcpp.contract}` *(2026.10.30.1+)* | path to the configuration's contract, `mcpp-contract.json`, beside its compile database: the compiler of each language with the words every unit is compiled with (target, sysroot, target-side headers, platform and profile flags), and the link driver, its words and the archiver. A foreign build system reads it to compile for the target as the engine does; a package's own include directories are not in it, nor the engine's C++ dialect and module words (`-std=`, `-fmodules`), which are its choice for its own units. Since 2026.11.6.1, link arguments describe the target configuration; the consuming package's link flags and libraries are excluded, since they can name outputs the foreign build has yet to produce | | `${mcpp.target_file:}` | the built file of target `` | | `${mcpp.stage_dir}` *(2026.9.11.1+)* | the tree `mcpp pack` staged, absolute. `artifact` role only, and only under `mcpp pack --pkg-format ` | | `${mcpp.self}` *(2026.9.13.1+)* | the engine's own executable, absolute. An action's command is an argv with no shell, so a build program has no portable way to copy a file; the engine is present wherever a build runs, and `${mcpp.self} stage --verify content --output ` copies one file, creates the destination's parent, and writes only when the bytes differ. That argument shape is a contract from 2026.9.13.1 on; a build program that names it declares that release as its floor | diff --git a/docs/50-machine-output.md b/docs/50-machine-output.md index 9962a8256..ee3e9a903 100644 --- a/docs/50-machine-output.md +++ b/docs/50-machine-output.md @@ -1002,6 +1002,13 @@ starting it. `data` is `program`, `argv`, `cwd`, `env` and `via`, as `launch` above. `--print-launch` takes no `--format`: on `mcpp run`, `--pkg-format` names a distributable. +`data.configuration` (2026.11.6.1+) contains the selected program's `triple`, +`profile` and resolved `platform`, using the same fields as the build record's +configuration. For a target set it describes the leg selected for this launch. +`configuration.runners` lists the available named runners, including project +declarations when `platform` is null. Older engines omit `configuration`; +consumers check for its presence. + ``` mcpp test --print-launch [-- ] ``` diff --git a/docs/specs/concurrency.md b/docs/specs/concurrency.md index 3030c33d6..46039caf5 100644 --- a/docs/specs/concurrency.md +++ b/docs/specs/concurrency.md @@ -7,7 +7,7 @@ | 状态 | 评审中 v1.1 | | 版本 | 1.1 | | 最后修改 | 2026-11-06 | -| 对应实现 | 基础保证: mcpp >= 2026.10.26.1; §5.1: mcpp >= 2026.11.5.2 | +| 对应实现 | 基础保证: mcpp >= 2026.10.26.1; §5.1: mcpp >= 2026.11.6.1 | | 相关设计文档 | `.agents/docs/2026-10-26-control-contract-runtime-and-ide-ecosystem-design.md` §3、§4、§5.4 | | 相关 issue | #839 | | 使用文档 | [50 — 机器可读输出](../zh/50-machine-output.md) §9、§11 | @@ -88,7 +88,7 @@ CI。在 2026.10.24.1 上,同一配置的两次冷构建相隔 0.1–0.5 s 启 ### 5.1 分离编译的执行单元与恢复 -以下保证适用于 GCC 的 `bmi_schedule = "on"`（2026.11.5.2+）: +以下保证适用于 GCC 的 `bmi_schedule = "on"`（2026.11.6.1+）: 1. 普通编译与分离代码生成**必须**共用实际的 `jobs` 总上限。等待任务与调度任务 **禁止**作为编译名额的替代。内核锁已释放而原编译单元仍被占用或状态未知时, diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 30009a59b..d59b03da0 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -973,7 +973,7 @@ bmi_schedule = "off" # auto (default, = off) | on | off `bmi_schedule` 决定 importer 何时被解除阻塞。 使用 GCC 且 `bmi_schedule = "on"` 时，普通编译与分离的代码生成共用同一个 `jobs` 总上限 -（2026.11.5.2+）。额外的调度和等待任务不增加编译器进程数量。 +（2026.11.6.1+）。额外的调度和等待任务不增加编译器进程数量。 取消构建会结束本轮普通编译器和分离的编译器。紧接着的构建等待它们完成清理后 才启动新的编译器；未完成的输出会重新构建。已完成模块的 object 缺失时， 该次构建会重新生成它。输入和输出均不变的构建仍不进行编译。 diff --git a/docs/zh/08-testing.md b/docs/zh/08-testing.md index c842c522c..6f9aca1c7 100644 --- a/docs/zh/08-testing.md +++ b/docs/zh/08-testing.md @@ -151,6 +151,17 @@ mcpp test --target aarch64-macos --no-run # build the tests for the target an `--no-runner` 是为「本机能原生执行这些二进制、不该为模拟器付代价」的宿主准备的。 +`--runner `（2026.11.6.1+）选择具名运行器代替默认运行器，声明与参数替换 +规则与 `mcpp run --runner` 相同。该选项同样适用于 `--print-launch` 与从归档读取的 +测试。不存在的运行器名称会被拒绝；`--runner` 与 `--no-runner` 不能同时使用。 + +对于声明了 `debuggee` 的平台，以下命令构建一个测试，打印调试器要启动的命令， +但不启动测试： + +```bash +mcpp test unit/math --runner debuggee --print-launch +``` + `--no-run` 做出的是更窄的断言，且必须显式要求。不给它时，一个本机既无法执行、也没有 runner 可达的目标，会让每个测试都停在「未运行」，命令退出 2：mcpp 没有查明这些测试 是否通过，把这种情况报成成功，是本仓记录得最多的一种假读数。但 2 同样是 runner diff --git a/docs/zh/24-openkal-cross.md b/docs/zh/24-openkal-cross.md index 63cab31d4..056c47987 100644 --- a/docs/zh/24-openkal-cross.md +++ b/docs/zh/24-openkal-cross.md @@ -361,7 +361,7 @@ mcpp run --target x86_64-openkal # okrun bin/app.oke 无需其他声明（2026.10.27.1+）：目标表为两行选择 `openkal-platform`，由平台陈述工具链、各层、工具、下表的约定 与 runner（[35](35-platforms.md)）。 -两个内置 openkal 目标默认选择平台 0.24.0（2026.11.5.2+）。显式 `platform` 配置可覆盖默认选择； +两个内置 openkal 目标默认选择平台 0.24.0（2026.11.6.1+）。显式 `platform` 配置可覆盖默认选择； 该平台的最终 ThinLTO 链接保持 emulated TLS 约定。 OS 0.3 使用 OpenKal 0.24 依赖组。保留 OS 0.2.9 的工程显式选择匹配的 0.23 平台组； diff --git a/docs/zh/25-home-indices-and-store.md b/docs/zh/25-home-indices-and-store.md index cb8e25aeb..2079c863b 100644 --- a/docs/zh/25-home-indices-and-store.md +++ b/docs/zh/25-home-indices-and-store.md @@ -73,7 +73,7 @@ Windows 首次显式 `self init` 时，若 D 是可写的本地固定磁盘， 安装过程分别跟踪每个活动下载的字节进度与来源切换，等待下载槽位的时间不消耗该预算。 其他文件取得进度不会延长停滞文件的预算。 -从 2026.11.5.2 起，下载停滞诊断会标明文件、停滞时长与来源尝试次数；provider 准备阶段与进程超时单独报告。 +从 2026.11.6.1 起，下载停滞诊断会标明文件、停滞时长与来源尝试次数；provider 准备阶段与进程超时单独报告。 镜像选择使用 `mcpp self config --mirror auto`、`CN` 或 `GLOBAL`。 ### 构建产物的保留期（2026.10.31.3+） diff --git a/docs/zh/30-build-mcpp.md b/docs/zh/30-build-mcpp.md index c128ec11f..1f6dff233 100644 --- a/docs/zh/30-build-mcpp.md +++ b/docs/zh/30-build-mcpp.md @@ -933,7 +933,7 @@ mcpp 会写出 `<暂存树>.stage-manifest` —— 一个兄弟文件，永不 | `${mcpp.out_dir}` | 构建输出目录 | | `${mcpp.bin_dir}` | 产出的二进制所在目录:`bin/`;工作区成员的构建程序声明的 action 中为该成员的产物目录(2026.9.29.4+) | | `${mcpp.compile_db}` | `compile_commands.json` 的路径（clang-tidy 的 `-p` 要的就是它） | -| `${mcpp.contract}` *(2026.10.30.1+)* | 配置的契约 `mcpp-contract.json` 的路径，与该配置的编译数据库同目录：每种语言的编译器与每个单元都带的词（目标、sysroot、目标侧的头文件、平台与 profile 的标志），以及链接驱动、它的词与归档工具。外部构建系统读它，以引擎的方式为目标编译；包自己的包含目录不在其中，引擎为自己的单元选择的 C++ 方言与模块词（`-std=`、`-fmodules`）也不在其中。自 2026.11.5.2 起，链接参数描述目标配置，不含消费包自己的链接标志和库；这些库可能正是外部构建尚待生成的产物 | +| `${mcpp.contract}` *(2026.10.30.1+)* | 配置的契约 `mcpp-contract.json` 的路径，与该配置的编译数据库同目录：每种语言的编译器与每个单元都带的词（目标、sysroot、目标侧的头文件、平台与 profile 的标志），以及链接驱动、它的词与归档工具。外部构建系统读它，以引擎的方式为目标编译；包自己的包含目录不在其中，引擎为自己的单元选择的 C++ 方言与模块词（`-std=`、`-fmodules`）也不在其中。自 2026.11.6.1 起，链接参数描述目标配置，不含消费包自己的链接标志和库；这些库可能正是外部构建尚待生成的产物 | | `${mcpp.target_file:}` | target `` 构建出的文件 | | `${mcpp.stage_dir}` *(2026.9.11.1+)* | `mcpp pack` 暂存出的那棵树，绝对路径。仅 `artifact` role 可用，且仅在 `mcpp pack --pkg-format ` 下可用 | | `${mcpp.self}` *(2026.9.13.1+)* | 引擎自己的可执行文件，绝对路径。action 的命令是没有 shell 的 argv，构建程序因此没有可移植的拷贝手段；引擎在构建运行的每台机器上都在，`${mcpp.self} stage --verify content --output ` 拷贝一个文件、创建目标的父目录、只在字节不同时写入。这个参数形状自 2026.9.13.1 起是契约；写下它的构建程序即以该版本为下限 | diff --git a/docs/zh/50-machine-output.md b/docs/zh/50-machine-output.md index d2e80b1d0..495beb3e2 100644 --- a/docs/zh/50-machine-output.md +++ b/docs/zh/50-machine-output.md @@ -890,6 +890,11 @@ mcpp run [target] --print-launch `argv`、`cwd`、`env` 与 `via`，同上文的 `launch`。`--print-launch` 不接受 `--format`：在 `mcpp run` 上，`--pkg-format` 表示可分发物的格式。 +`data.configuration`（2026.11.6.1+）包含所选程序的 `triple`、`profile` 与解析后的 +`platform`，字段与构建记录的 configuration 相同。目标集合中，它描述本次启动所选的 +目标。`configuration.runners` 列出可用的具名运行器，包括 platform 为 null 时 +项目声明的运行器。旧引擎不提供 `configuration`；消费者检查该字段是否存在。 + ``` mcpp test --print-launch [-- <参数>] ``` diff --git a/mcpp.toml b/mcpp.toml index 312633a23..c752c16ae 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.11.5.2" +version = "2026.11.6.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index 582a25ec6..ae049ca57 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.11.5.2"; +inline constexpr std::string_view MCPP_VERSION = "2026.11.6.1"; } // namespace mcpp diff --git a/src/build/execute.cppm b/src/build/execute.cppm index b2ae52b3a..410748143 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -1519,6 +1519,15 @@ export struct RunnerOnThisMachine { std::string refusal; // non-empty: runs are refused here, and why }; +nlohmann::json launch_configuration(const BuildContext& ctx) { + auto runners = nlohmann::json::array(); + for (const auto& [name, entry] : ctx.plan.platform.fields.runners) + if (!entry.value.empty()) runners.push_back(name); + return {{"triple", ctx.tc.targetTriple}, {"profile", ctx.profile}, + {"platform", nlohmann::json::parse(mcpp::platform::to_json(ctx.plan.platform), nullptr, false)}, + {"runners", std::move(runners)}}; +} + export RunnerOnThisMachine runner_on_this_machine(const BuildContext& ctx) { RunnerOnThisMachine out; const auto choice = choose_runner(ctx); @@ -3314,6 +3323,7 @@ int run_artifact_via_runner(mcpp::build::BuildContext& ctx, mcpp::wire::Effect::WriteProject}}; env.data = mcpp::build::result::to_json(launch); env.data["program"] = exe.generic_string(); + env.data["configuration"] = launch_configuration(ctx); mcpp::wire::emit(env); return 0; } @@ -3734,6 +3744,7 @@ export struct TestOptions { // runner — the operator on this host stating that the triple is native // here, a fact the manifest has no axis for (#544, D3). bool noRunner = false; + std::string runnerName; // empty: the default runner, as in `mcpp run` // `--no-run`: build the tests for `--target` and stop. THE CLAIM IT MAKES // IS NARROWER THAN A PASS, AND IT IS STATED RATHER THAN INFERRED. // @@ -4171,8 +4182,15 @@ static void test_bulk(TestBuild& tb, const TestOptions& testOpts, Keep&& keep) { // holds under a runner exactly as it does on the host. static void test_resolve_runner(TestBuild& tb, const TestOptions& testOpts, bool json) { auto* ctx = &*tb.ctx; - tb.runnerChoice = choose_runner(*ctx, testOpts.noRunner); + tb.runnerChoice = choose_device_action(*ctx, testOpts.runnerName, testOpts.noRunner); auto& runnerChoice = tb.runnerChoice; + if (!testOpts.runnerName.empty() && runnerChoice.tmpl.empty()) { + tb.invocationNotRunReason = std::format("no runner named '{}' for '{}'", + testOpts.runnerName, runnerChoice.tripleKey); + std::println(stderr, "error: {}", tb.invocationNotRunReason); + tb.runnerFatal = 2; + return; + } if (runnerChoice.ignored && !json) mcpp::ui::info("note", std::format( "--no-runner: ignoring the runner declared for {}", runnerChoice.tripleKey)); @@ -4640,6 +4658,7 @@ static int test_run_member(TestBuild& tb, const TestOptions& testOpts, env.data = mcpp::build::result::to_json(launch); env.data["program"] = chosen->program.generic_string(); env.data["test"] = chosen->name; + env.data["configuration"] = launch_configuration(*ctx); mcpp::wire::emit(env); return 0; } diff --git a/src/cli.cppm b/src/cli.cppm index f3ecc1957..11d250006 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -865,6 +865,8 @@ int run(int argc, char** argv) { .help("List (filtered) tests without building or running them")) .option(cl::Option("no-runner") .help("Run test binaries directly, ignoring any [target.].runner (a host that runs them natively)")) + .option(cl::Option("runner").takes_value().value_name("NAME") + .help("Run tests through this named runner instead of the default runner")) .option(cl::Option("print-launch") .help("Build the one test PATTERN names, then print how it would be started (the mcpp.launch envelope, as `mcpp run --print-launch` prints) instead of running it; arguments after `--` are in its argv")) .option(cl::Option("no-run") diff --git a/src/cli/cmd_build.cppm b/src/cli/cmd_build.cppm index fb7496683..c7d1aa2d9 100644 --- a/src/cli/cmd_build.cppm +++ b/src/cli/cmd_build.cppm @@ -1186,6 +1186,11 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed, if (parsed.positional_count() > 0) to.filter = parsed.positional(0); to.list = parsed.is_flag_set("list"); to.noRunner = parsed.is_flag_set("no-runner"); // see cmd_run + if (auto runner = parsed.value("runner")) to.runnerName = *runner; + if (to.noRunner && !to.runnerName.empty()) { + mcpp::ui::error("--runner and --no-runner cannot be combined"); + return 2; + } to.noRun = parsed.is_flag_set("no-run"); // The two read alike and mean opposite things: `--no-runner` says to run // the binaries WITHOUT the declared runner, `--no-run` says not to run diff --git a/tests/e2e/333_named_runners_locked_and_sbom.sh b/tests/e2e/333_named_runners_locked_and_sbom.sh index 2b658805d..57c1b70b8 100755 --- a/tests/e2e/333_named_runners_locked_and_sbom.sh +++ b/tests/e2e/333_named_runners_locked_and_sbom.sh @@ -58,6 +58,25 @@ case "$out" in esac echo " ok an undeclared name is refused, listing what this project does have" +# A client receives the actual profile and effective runner declarations of +# the built program, rather than configuring the default profile separately. +cat >> mcpp.toml <<'TOML' + +[profile.inspect] +opt = 0 +debug = true +TOML +"$MCPP" run --profile inspect --print-launch > inspect-launch.json 2> inspect-launch.err \ + || { cat inspect-launch.err; echo "FAIL: custom-profile launch"; exit 1; } +python3 - <<'PY' || exit 1 +import json +data=json.load(open("inspect-launch.json"))["data"] +configuration=data["configuration"] +assert configuration["profile"] == "inspect", configuration +assert configuration["runners"] == ["flash", "monitor"], configuration +PY +echo " ok launch names the actual custom profile and effective named runners" + # ── D: --locked asserts the recorded resolution ──────────────────────────── mkdir -p "$work/q/src" cd "$work/q" diff --git a/tests/e2e/936_the_control_contract.sh b/tests/e2e/936_the_control_contract.sh index 22dfd0a57..88f7aa024 100644 --- a/tests/e2e/936_the_control_contract.sh +++ b/tests/e2e/936_the_control_contract.sh @@ -72,6 +72,11 @@ d = json.load(open(sys.argv[1])) assert d["kind"] == "mcpp.launch", d["kind"] assert d["data"]["argv"] and same(d["data"]["argv"][0], d["data"]["program"]), d["data"] assert "env" in d["data"] and "cwd" in d["data"] +last = json.load(open("target/.last-result.json"))["data"] +record = json.load(open(last["result"]))["data"]["configuration"] +configuration = d["data"]["configuration"] +assert all(configuration[key] == record[key] for key in ("triple", "profile", "platform")), configuration +assert isinstance(configuration["runners"], list) and all(isinstance(n,str) for n in configuration["runners"]), configuration EOF # C diff --git a/tests/e2e/963_a_target_set_builds_every_member_and_runs_where_it_is_reached.sh b/tests/e2e/963_a_target_set_builds_every_member_and_runs_where_it_is_reached.sh index 718b557fa..f0ab22719 100755 --- a/tests/e2e/963_a_target_set_builds_every_member_and_runs_where_it_is_reached.sh +++ b/tests/e2e/963_a_target_set_builds_every_member_and_runs_where_it_is_reached.sh @@ -82,6 +82,17 @@ grep -q "image of the set" c.log || fail "C: the program did not run" c.log grep -q "Leg.*$HT" c.log || fail "C: the run does not say which leg" c.log echo "ok: run runs this machine's leg" +# The launch describes the leg run selected, not the first self-env report. +"$MCPP" run --target "$OT" --target "$HT" --print-launch > c-launch.json 2> c-launch.err \ + || fail "C: selected leg launch" c-launch.err +python3 - c-launch.json "$HT" <<'PY' || fail "C: launch configuration is not the selected leg" c-launch.json +import json,sys +data=json.load(open(sys.argv[1]))["data"] +assert data["configuration"]["triple"] == sys.argv[2], data +assert data["configuration"]["profile"] == "dev", data +assert data["configuration"]["platform"], data +PY + # D "$MCPP" test --target openkal > d.log 2>&1 || fail "D: mcpp test --target openkal" d.log grep -q "smoke ... ok" d.log || fail "D: this machine's leg did not pass" d.log diff --git a/tests/e2e/993_a_test_describes_its_launch.sh b/tests/e2e/993_a_test_describes_its_launch.sh index 767be9dc8..4aa5ebc3d 100755 --- a/tests/e2e/993_a_test_describes_its_launch.sh +++ b/tests/e2e/993_a_test_describes_its_launch.sh @@ -52,6 +52,10 @@ argv = data["argv"] assert any(same(a, data["program"]) for a in argv), data assert argv[-2:] == ["--flag", "x"], argv assert "env" in data and "cwd" in data +last = json.load(open("target/.last-result.json"))["data"] +record = json.load(open(last["result"]))["data"]["configuration"] +assert all(data["configuration"][key] == record[key] for key in ("triple", "profile", "platform")), data +assert isinstance(data["configuration"]["runners"], list), data EOF echo "ok: the envelope describes the test" @@ -90,4 +94,53 @@ echo "ok: the refusals" grep -q '"test-launch"' e.out || fail "E: test-launch is not stated" e.out echo "ok: the feature is stated" +# A named runner is the same executable transformation for a test and a run. +# Python is a real host runner here, so the generated argv can also be executed. +$PY - <<'PY' +import json,sys +from pathlib import Path +Path('runner.py').write_text('import json,subprocess,sys\nwith open("named-invocations.txt","a") as f: f.write(json.dumps(sys.argv[1:])+"\\n")\nprint("NAMED RUNNER", flush=True)\nsys.exit(subprocess.call(sys.argv[1:]))\n') +with Path('mcpp.toml').open('a') as f: + os_name={'linux':'linux','darwin':'macos','win32':'windows'}[sys.platform] + f.write('\n[target.\'cfg(os = "'+os_name+'")\'.runners]\ndebuggee = '+json.dumps([sys.executable,str(Path('runner.py').resolve()),'{}'])+'\n') +PY +"$MCPP" test unit/math --runner debuggee --print-launch -- 'with spaces' > named.out 2> named.err \ + || fail "named test launch" named.err +$PY - named.out > named-run.out 2> named-run.err <<'PY' || fail "named launch execution" named-run.out named-run.err +import json,sys,os,subprocess +d=json.load(open(sys.argv[1]))['data'] +assert d['via']=='runner', d +assert d['configuration']['runners']==['debuggee'], d +assert d['argv'][-1]=='with spaces', d +assert d['argv'][1].endswith('runner.py'), d +sys.exit(subprocess.call(d['argv'],cwd=d['cwd'],env={**os.environ,**d['env']})) +PY +grep -q 'NAMED RUNNER' named-run.out || fail "named runner did not run" named-run.out +grep -q 'last \[with spaces\]' named-run.out || fail "named runner changed arguments" named-run.out +rm named-invocations.txt +"$MCPP" test unit/math2 --runner debuggee -- direct > named-direct.out 2> named-direct.err \ + || fail "named runner direct test" named-direct.out named-direct.err +$PY - <<'PY' || fail "direct test skipped named runner" named-direct.out +import json +rows=[json.loads(line) for line in open('named-invocations.txt')] +assert len(rows)==1 and 'math2' in rows[0][0] and rows[0][-1]=='direct', rows +PY +"$MCPP" test unit/math2 --no-run --archive named-tests.tar.gz > named-archive-build.out 2> named-archive-build.err \ + || fail "test archive build" named-archive-build.out named-archive-build.err +rm named-invocations.txt +"$MCPP" test --archive named-tests.tar.gz --runner debuggee -- archived > named-archive.out 2> named-archive.err \ + || fail "named archive runner" named-archive.out named-archive.err +$PY - <<'PY' || fail "archive skipped named runner" named-archive.out +import json +rows=[json.loads(line) for line in open('named-invocations.txt')] +assert len(rows)==1 and 'math2' in rows[0][0] and rows[0][-1]=='archived', rows +PY +set +e +"$MCPP" test unit/math --runner absent --print-launch > named-missing.out 2> named-missing.err; rc1=$? +"$MCPP" test unit/math --runner debuggee --no-runner > named-conflict.out 2> named-conflict.err; rc2=$? +set -e +[ "$rc1" = 2 ] && grep -q "no runner named 'absent'" named-missing.err || fail "missing test runner ($rc1)" named-missing.err +[ "$rc2" = 2 ] && grep -q -- '--runner and --no-runner' named-conflict.err || fail "contradictory test runners ($rc2)" named-conflict.err +echo "ok: named test launch, execution, archive, argument preservation and refusal" + echo "OK" From fd4e2ce576daf5c579c504659f9776ec2eacd2f8 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 14:17:30 +0900 Subject: [PATCH 17/40] fix: close compiler status races and align release group --- .../2026-11-06-compiler-attempt-recovery.md | 53 +++++++++--- ...ch-configuration-and-named-test-runners.md | 5 ++ .agents/docs/README.md | 9 ++- CHANGELOG.md | 6 ++ docs/04-mcpp-toml.md | 2 +- docs/10-pack-and-release.md | 2 +- docs/35-platforms.md | 53 ++++++------ docs/50-machine-output.md | 2 +- docs/specs/concurrency.md | 1 - docs/specs/packaging.md | 2 +- docs/zh/04-mcpp-toml.md | 2 +- docs/zh/10-pack-and-release.md | 2 +- docs/zh/35-platforms.md | 53 ++++++------ docs/zh/50-machine-output.md | 2 +- src/build/schedule/compiler_transaction.cppm | 28 ++++++- src/build/schedule/detach_codegen.cppm | 6 ++ src/pack/distribution.cppm | 7 +- tests/e2e/234_bmi_schedule_on.sh | 3 + .../helpers/check_cached_object_recovery.py | 7 +- .../helpers/check_completion_observation.py | 54 +++++++++++++ tests/e2e/helpers/compiler_completion_race.c | 81 +++++++++++++++++++ tests/unit/test_compiler_transaction.cpp | 29 +++++++ tests/unit/test_pack_distribution.cpp | 2 +- 23 files changed, 331 insertions(+), 80 deletions(-) create mode 100644 tests/e2e/helpers/check_completion_observation.py create mode 100644 tests/e2e/helpers/compiler_completion_race.c diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md index c242ddbdf..ce6aebc62 100644 --- a/.agents/docs/2026-11-06-compiler-attempt-recovery.md +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -1,3 +1,8 @@ +--- +subject: compiler-scheduling +status: active +--- + # Compiler attempt lifetime and output recovery Status: source implementation and focused acceptance in progress. This record @@ -207,7 +212,7 @@ No runtime generation becomes a dependency version or build-cache identity. | C++ runtime | 0.19.0 | [Runtime 66](https://github.com/mcpplibs/openkal-llvm-runtime/pull/66) | | Platform | 0.24.0 | Publication follows its public lower dependencies | | OS composition | 0.3.0 | [OS 17](https://github.com/mcpplibs/os/pull/17) | -| Core | 2026.11.5.2 | [mcpp 889](https://github.com/mcpp-community/mcpp/pull/889) | +| Core | 2026.11.6.1 | [mcpp 889](https://github.com/mcpp-community/mcpp/pull/889) | OS advances to 0.3.0 so an explicit range such as `^0.2.3` cannot silently absorb a different OpenKal minor group. Bare package versions name exact wire @@ -222,15 +227,43 @@ image consumer calls all four operations through the image convention, including the fifth retirement argument and an invalid continuation that changes the state output from `empty` to `unknown`. -Source-group controls establish implementation behaviour. They do not establish -that public versions are installable. Release gates remain full repository CI, -canonical tag archives, byte-identical CN mirrors, published index entries and -ordinary GLOBAL/CN consumers. Temporary image source overrides are removed -before its final exact-head public-dependency CI. Platform and OS qualification -follow the lower public group; core then updates its OS and default platform -pins and runs full exact-head CI before independent ecosystem review. Plugins -PR 52 retains the public 2026.11.5.1 pin until core 2026.11.5.2 is published, -then runs its actual sidecar integration and final CI. +Source-group controls establish implementation behaviour. The lower versions, +including platform 0.24.0 and OS 0.3.0, are now publicly released and indexed; +their ordinary consumers have been checked. Core 2026.11.6.1 uses those pins +and still requires full exact-head CI, an independent ecosystem review, +canonical tag archives, byte-identical CN mirrors, a published index entry and +ordinary GLOBAL/CN consumers. Plugins PR 52 retains the published +2026.11.5.1 pin until core 2026.11.6.1 is published, then runs its actual +sidecar integration and final CI. + +## 2026-11-06 final-head CI failure controls + +The first 2026.11.6.1 CI run exposed four distinct boundaries. Clang/libc++ +uses a wider file-clock tick representation than `std::to_string` accepts; +the journal now rejects values outside its signed 64-bit schema and preserves +in-range ticks. A direct exact-toolchain control compiled the failing expression +and the corrected one, including a real file identity. Native Clang, ARM and +bootstrap jobs still require the next full CI run. + +An actual split-compiler control held a previously absent `.rc` observation +until the controller had published status and exited. The earlier engine failed +with `compiler controller ended without a status`; the corrected engine rereads +status before treating controller loss as failure. Phase-one, quiet-unit and +owner-witness observations use the same ordering rule. A real GCC build then +passed its compiler-cap, cancellation, cached replay and same-invocation +missing-object recovery controls. A missing status remains a bounded failure. + +On Windows, the unit fixture can derive its root from an 8.3 TEMP alias while +the MSVC filesystem implementation records canonical long paths. Production +`RunScope` already canonicalizes its root before admission. The fixture now +does the same before deriving BMI, object and certificate paths; the native +Windows suite must confirm the negative symlink controls remain intact. + +The generated Windows package launcher also named OS 0.2.9, whose OpenKal +0.23 group conflicts with the current default platform 0.24. It now names +OS 0.3.0. The older consumer platform selected by the application's own +target remains independent of the launcher's build platform. The actual +cross-platform packing control and complete CI remain release gates. ## Separate Windows build-program defect diff --git a/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md b/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md index c33ff4ac1..74b143dc4 100644 --- a/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md +++ b/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md @@ -1,3 +1,8 @@ +--- +subject: heterogeneous-build +status: active +--- + # Launch configuration and named test runners Status, 2026-11-06: implemented for candidate 2026.11.6.1. Focused local diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 977f265a6..c9efb5146 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -350 records. +352 records. ## By subject @@ -38,6 +38,10 @@ Records that declare one. Everything else is listed by date below. - [BMI publication, compiler capacity and active download budgets](2026-11-06-bmi-publication-and-compiler-cap-closeout.md) — active +### compiler-scheduling + +- [Compiler attempt lifetime and output recovery](2026-11-06-compiler-attempt-recovery.md) — active + ### design - [升级的连续性：打包修订不在使用路径上替换载荷、替换失败不拆掉可用状态、home 中的悬空记录自愈、真实的升级 lane](2026-10-17-upgrade-continuity-design.md) — active @@ -86,6 +90,7 @@ Records that declare one. Everything else is listed by date below. ### heterogeneous-build +- [Launch configuration and named test runners](2026-11-06-launch-configuration-and-named-test-runners.md) — active - [异构构建语义与插件边界补充方案（mcpp#885、mcpp#887）](2026-11-05-heterogeneous-build-semantics-and-plugin-boundaries-supplement.md) — active - [异构构建补充方案的第二轮调研与实验](2026-11-05-heterogeneous-build-research-and-experiments.md) — active - [生态 885 与异构构建补充方案的实现依赖](2026-11-05-heterogeneous-build-implementation-plan.md) — active @@ -171,6 +176,8 @@ Records that declare one. Everything else is listed by date below. ### 2026-11 +- [Launch configuration and named test runners](2026-11-06-launch-configuration-and-named-test-runners.md) — active +- [Compiler attempt lifetime and output recovery](2026-11-06-compiler-attempt-recovery.md) — active - [BMI publication, compiler capacity and active download budgets](2026-11-06-bmi-publication-and-compiler-cap-closeout.md) — active - [异构构建语义与插件边界补充方案（mcpp#885、mcpp#887）](2026-11-05-heterogeneous-build-semantics-and-plugin-boundaries-supplement.md) — active - [异构构建补充方案的第二轮调研与实验](2026-11-05-heterogeneous-build-research-and-experiments.md) — active diff --git a/CHANGELOG.md b/CHANGELOG.md index 904b26e51..1aa97dcd6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,12 @@ ### Fixed +- A finished detached compiler is recognized when its final status appears + between status and liveness checks. Recovery still validates its output proof. +- Windows launcher packages use the OS release required by the current default + platform, so their generated dependency resolves alongside openkal-platform 0.24. +- Compiler attempt journals preserve file timestamps on Clang/libc++ when the + file clock uses a wider integer representation. - Launch descriptions include the selected target, profile, resolved platform and available named runners, including the reachable leg of a target set. Editor debug launches can retain the configuration that actually built the program. diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index 207b3da8f..ef4c358c1 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1518,7 +1518,7 @@ the platform's own key names ([35 — Platforms](35-platforms.md)): ```toml [target.'cfg(os = "openkal")'] -platform = "openkal-platform@0.23.3" # or { path = "…" }, or "none" +platform = "openkal-platform@0.24.0" # or { path = "…" }, or "none" [target.'cfg(platform = "openkal")'.layers] c-abi = { path = "../openkal-musl" } diff --git a/docs/10-pack-and-release.md b/docs/10-pack-and-release.md index 3f79e8307..58806c802 100644 --- a/docs/10-pack-and-release.md +++ b/docs/10-pack-and-release.md @@ -956,7 +956,7 @@ and sha256. A provided format, `--merge` and `mcpp emit xpkg` read it ([SPEC-014](specs/packaging.md) §4.5): ```json -{"schema": 1, "name": "hello", "version": "0.1.0", "platform": "openkal-platform@0.23.3", +{"schema": 1, "name": "hello", "version": "0.1.0", "platform": "openkal-platform@0.24.0", "linkage": "shared", "mode": "vendored", "legs": ["x86_64-openkal", "aarch64-openkal"], "systems": [], "launch": {"runner": ["okrun", "{}"], "entry": {"x86_64": "lib/x86_64/hello.oke", "aarch64": "lib/aarch64/hello.oke"}}, "files": [{"path": "lib/x86_64/hello.oke", "role": "image", "leg": "x86_64-openkal", "sha256": "…"}]} diff --git a/docs/35-platforms.md b/docs/35-platforms.md index 4dc50c9f8..8db4f947e 100644 --- a/docs/35-platforms.md +++ b/docs/35-platforms.md @@ -40,13 +40,13 @@ mcpp run --target x86_64-openkal ``` Resolved llvm@23.1.3 → x86_64-openkal → …/xim-x-llvm/23.1.3/bin/clang++ - the platform's (openkal-platform 0.18.0), replacing your gcc@16.1.0 — override with `[target.x86_64-openkal] toolchain` + the platform's (openkal-platform 0.24.0), replacing your gcc@16.1.0 — override with `[target.x86_64-openkal] toolchain` Target x86_64-openkal → x86_64-unknown-none-elf - compiler-runtime compiler-rt (openkal-llvm-runtime@0.16.6, platform) - c-abi musl (openkal-musl@0.22.6, platform) - c++-abi libc++ (openkal-llvm-runtime@0.16.6, platform) - Platform openkal-platform 0.18.0 (built-in) - tools okld, okrun (openkal-tools@0.1.3, platform) + compiler-runtime compiler-rt (openkal-llvm-runtime@0.19.0, platform) + c-abi musl (openkal-musl@0.28.0, platform) + c++-abi libc++ (openkal-llvm-runtime@0.19.0, platform) + Platform openkal-platform 0.24.0 (built-in) + tools okld, okprof, okrun (openkal-tools@0.5.0, platform) runner okrun (platform) link lld, start ok_main (platform) Compiling app v0.1.0 (.) @@ -113,9 +113,9 @@ The first of these that answers selects the target's platform: | Source | Spelling | |---|---| -| the command line | `--platform openkal-platform@0.23.3`, `--platform path:../openkal-platform`, `--platform none` | +| the command line | `--platform openkal-platform@0.24.0`, `--platform path:../openkal-platform`, `--platform none` | | the environment | `MCPP_TARGET_X86_64_OPENKAL_PLATFORM=…` | -| the project | `[target.] platform = "openkal-platform@0.23.3"`, or `{ path = "…" }` | +| the project | `[target.] platform = "openkal-platform@0.24.0"`, or `{ path = "…" }` | | this machine | the same key in `~/.mcpp/config.toml` | | the target table | the platform column (`mcpp toolchain list`) | @@ -129,11 +129,11 @@ its own conventions. `[host] platform` selects a platform for them instead: ```toml [host] -platform = "openkal-platform@0.23.3" +platform = "openkal-platform@0.24.0" ``` ``` - Host platform openkal-platform 0.18.0 (mcpp.toml:2): build programs are x86_64-openkal images run by okrun + Host platform openkal-platform 0.24.0 (mcpp.toml:2): build programs are x86_64-openkal images run by okrun ``` Every build program then compiles as an ok image of this machine's @@ -151,7 +151,7 @@ implementation: ```toml [target.'cfg(any(linux, macos, windows))'] -platform = "openkal-platform@0.23.3" +platform = "openkal-platform@0.24.0" ``` On Windows the arrangement is the GNU ABI's row (`--target x86_64-windows-gnu`); @@ -165,12 +165,12 @@ packages pin one another exactly. A graph that holds another version of one of them is refused before anything compiles: ``` -error: openkal-musl 0.22.2 is in this build, and the platform openkal-platform 0.18.0 is a version group that names openkal-musl 0.22.9. +error: openkal-musl 0.22.2 is in this build, and the platform openkal-platform 0.24.0 is a version group that names openkal-musl 0.28.0. requested by: own The family's packages pin one another exactly, so two versions of one of them cannot be linked into one program. Either select the platform release of that version ([target.] platform = "openkal-platform@"), or write - openkal-musl = "0.22.6" where it is requested. + openkal-musl = "0.28.0" where it is requested. ``` A package taken from a directory or a repository is not bound by the group. @@ -184,39 +184,40 @@ abridged: [package] namespace = "mcpplibs" name = "openkal-platform" -version = "0.18.0" -mcpp = ">=2026.10.30.1" +version = "0.24.0" +mcpp = ">=2026.11.3.1" [platform] targets = ["x86_64-openkal", "aarch64-openkal"] toolchain = "llvm@23.1.3" -runner = { cmd = ["okrun", "{}"], host = "same-arch" } -carrier = { package = "openkal-run@0.1.3", section = ".oke", main = "carrier/main.cpp" } +runner = { cmd = ["okrun", "{}"], host = "same-arch", hosts = ["linux-x86_64", "linux-aarch64", "macos-aarch64", "windows-x86_64"] } +carrier = { package = "openkal-run@0.5.0", section = ".oke", main = "carrier/main.cpp" } plugin = "build.mcpp" [platform.layers] -compiler-runtime = "openkal-llvm-runtime@0.16.6" -"c++-abi" = "openkal-llvm-runtime@0.16.6" -c-abi = { name = "openkal-musl", version = "0.22.6", default-features = false, features = ["posix"] } +compiler-runtime = "openkal-llvm-runtime@0.19.0" +"c++-abi" = "openkal-llvm-runtime@0.19.0" +c-abi = { name = "openkal-musl", version = "0.28.0", default-features = false, features = ["posix", "malloc-mimalloc"] } [platform.tools] -okld = "openkal-tools@0.1.3" -okrun = "openkal-tools@0.1.3" +okld = "openkal-tools@0.5.0" +okprof = "openkal-tools@0.5.0" +okrun = "openkal-tools@0.5.0" [platform.build] -cxxflags = ["-fPIC", "-femulated-tls", "-fvisibility=hidden", "-fasynchronous-unwind-tables"] +cxxflags = ["-fPIC", "-femulated-tls", "-fvisibility=hidden", "-fasynchronous-unwind-tables", "-Xclang", "-fstack-clash-protection", "-ffunction-sections", "-fdata-sections", "-fno-omit-frame-pointer", "-mno-omit-leaf-frame-pointer", "-fstack-protector-strong", "-mstack-protector-guard=global", "-ftrivial-auto-var-init=zero", "-D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_FAST"] [platform.build.'cfg(arch = "aarch64")'] -cxxflags = ["-ffixed-x18", "-mno-outline-atomics"] +cxxflags = ["-ffixed-x18", "-mno-outline-atomics", "-mbranch-protection=pac-ret+bti"] [platform.link] driver = "lld" start = "ok_main" -flags = ["-shared", "-z", "max-page-size=65536", "-z", "now", "--gc-sections", "--icf=safe", "--eh-frame-hdr"] +flags = ["-shared", "-z", "max-page-size=65536", "-z", "pack-relative-relocs", "-z", "now", "-z", "noexecstack", "--hash-style=both", "--gc-sections", "--icf=safe", "--eh-frame-hdr", "--build-id=sha1", "--emit-relocs", "-mllvm", "-emulated-tls"] inputs = [{ from = ["okld", "note-source"], as = "ok-notes.S" }] exe = "{name}.oke" shared = "lib{name}.okso" -seal = [["okld", "note", "{}"], ["okld", "check", "{}", "--with", "{needed}"]] +seal = [["okld", "note", "{}"], ["okld", "place", "{}"], ["okld", "check", "{}", "--with", "{needed}"]] ``` - **Programs are named, not located.** `okld` in `inputs` and `seal` is found among diff --git a/docs/50-machine-output.md b/docs/50-machine-output.md index ee3e9a903..a44c21cd3 100644 --- a/docs/50-machine-output.md +++ b/docs/50-machine-output.md @@ -527,7 +527,7 @@ installation, which is located on the machine rather than installed by mcpp. A target row is `{target, note, toolchain, pin, status, default}`, and `status` is one of `installed` / `available` / `via dependency graph` / `planned`. A target row also carries `platform` (2026.10.27.1+): the platform package the -target table selects for it (`mcpplibs.openkal-platform@0.23.3`), or `null`. +target table selects for it (`mcpplibs.openkal-platform@0.24.0`), or `null`. **`toolchain` and `pin` are not the same field twice.** `toolchain` is what the row is associated with — the installed payload on an installed row, the diff --git a/docs/specs/concurrency.md b/docs/specs/concurrency.md index 46039caf5..20dcabacf 100644 --- a/docs/specs/concurrency.md +++ b/docs/specs/concurrency.md @@ -127,5 +127,4 @@ CI。在 2026.10.24.1 上,同一配置的两次冷构建相隔 0.1–0.5 s 启 | 版本 | 日期 | 变更 | |---|---|---| | 1.0 | 2026-10-26 | 首版:三段、构建目录锁、原子发布、`mcpp.lock` 的乐观写入、取消(#839) | - | 1.1 | 2026-11-06 | 分离编译的单元观察、共享名额、绝对输出事务与有限恢复边界;对齐取消请求的 OS 接收方式。 | diff --git a/docs/specs/packaging.md b/docs/specs/packaging.md index afef45c99..9eee1dae1 100644 --- a/docs/specs/packaging.md +++ b/docs/specs/packaging.md @@ -237,7 +237,7 @@ openkal 目标上 `static` 为一个 `.oke`、无 `.okso`,`shared` 为 `.oke` "schema": 1, "name": "hello", "version": "0.1.0", - "platform": "openkal-platform@0.23.3", + "platform": "openkal-platform@0.24.0", "linkage": "declared", "mode": "vendored", "legs": ["x86_64-openkal", "aarch64-openkal"], diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index d59b03da0..7e0d1c3a4 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -1397,7 +1397,7 @@ runner 与 `run_under`。 ```toml [target.'cfg(os = "openkal")'] -platform = "openkal-platform@0.23.3" # or { path = "…" }, or "none" +platform = "openkal-platform@0.24.0" # or { path = "…" }, or "none" [target.'cfg(platform = "openkal")'.layers] c-abi = { path = "../openkal-musl" } diff --git a/docs/zh/10-pack-and-release.md b/docs/zh/10-pack-and-release.md index 6835c8f71..0e640535b 100644 --- a/docs/zh/10-pack-and-release.md +++ b/docs/zh/10-pack-and-release.md @@ -801,7 +801,7 @@ Authenticode 可以签名它。Linux 与 macOS 经 shell 启动它：`sh pg-0.1. 提供的格式、`--merge` 与 `mcpp emit xpkg` 读取它（[SPEC-014](../specs/packaging.md) §4.5）： ```json -{"schema": 1, "name": "hello", "version": "0.1.0", "platform": "openkal-platform@0.23.3", +{"schema": 1, "name": "hello", "version": "0.1.0", "platform": "openkal-platform@0.24.0", "linkage": "shared", "mode": "vendored", "legs": ["x86_64-openkal", "aarch64-openkal"], "systems": [], "launch": {"runner": ["okrun", "{}"], "entry": {"x86_64": "lib/x86_64/hello.oke", "aarch64": "lib/aarch64/hello.oke"}}, "files": [{"path": "lib/x86_64/hello.oke", "role": "image", "leg": "x86_64-openkal", "sha256": "…"}]} diff --git a/docs/zh/35-platforms.md b/docs/zh/35-platforms.md index 12166dc74..5f363c247 100644 --- a/docs/zh/35-platforms.md +++ b/docs/zh/35-platforms.md @@ -32,13 +32,13 @@ mcpp run --target x86_64-openkal ``` Resolved llvm@23.1.3 → x86_64-openkal → …/xim-x-llvm/23.1.3/bin/clang++ - the platform's (openkal-platform 0.18.0), replacing your gcc@16.1.0 — override with `[target.x86_64-openkal] toolchain` + the platform's (openkal-platform 0.24.0), replacing your gcc@16.1.0 — override with `[target.x86_64-openkal] toolchain` Target x86_64-openkal → x86_64-unknown-none-elf - compiler-runtime compiler-rt (openkal-llvm-runtime@0.16.6, platform) - c-abi musl (openkal-musl@0.22.6, platform) - c++-abi libc++ (openkal-llvm-runtime@0.16.6, platform) - Platform openkal-platform 0.18.0 (built-in) - tools okld, okrun (openkal-tools@0.1.3, platform) + compiler-runtime compiler-rt (openkal-llvm-runtime@0.19.0, platform) + c-abi musl (openkal-musl@0.28.0, platform) + c++-abi libc++ (openkal-llvm-runtime@0.19.0, platform) + Platform openkal-platform 0.24.0 (built-in) + tools okld, okprof, okrun (openkal-tools@0.5.0, platform) runner okrun (platform) link lld, start ok_main (platform) Compiling app v0.1.0 (.) @@ -98,9 +98,9 @@ C++ 运行时的包同时提供编译器运行时,因此 `"c++-abi" = false` 只 | 来源 | 写法 | |---|---| -| 命令行 | `--platform openkal-platform@0.23.3`、`--platform path:../openkal-platform`、`--platform none` | +| 命令行 | `--platform openkal-platform@0.24.0`、`--platform path:../openkal-platform`、`--platform none` | | 环境变量 | `MCPP_TARGET_X86_64_OPENKAL_PLATFORM=…` | -| 工程 | `[target.] platform = "openkal-platform@0.23.3"`,或 `{ path = "…" }` | +| 工程 | `[target.] platform = "openkal-platform@0.24.0"`,或 `{ path = "…" }` | | 本机 | `~/.mcpp/config.toml` 中的同一个键 | | 目标表 | 平台列(`mcpp toolchain list`) | @@ -112,11 +112,11 @@ C++ 运行时的包同时提供编译器运行时,因此 `"c++-abi" = false` 只 ```toml [host] -platform = "openkal-platform@0.23.3" +platform = "openkal-platform@0.24.0" ``` ``` - Host platform openkal-platform 0.18.0 (mcpp.toml:2): build programs are x86_64-openkal images run by okrun + Host platform openkal-platform 0.24.0 (mcpp.toml:2): build programs are x86_64-openkal images run by okrun ``` 此后每个构建程序都编译为本机架构的 ok 映像,在 Linux、macOS 与 Windows 上使用同一个工具链,经平台的运行器 @@ -130,7 +130,7 @@ platform = "openkal-platform@0.23.3" ```toml [target.'cfg(any(linux, macos, windows))'] -platform = "openkal-platform@0.23.3" +platform = "openkal-platform@0.24.0" ``` Windows 上这一安排对应 GNU ABI 的行(`--target x86_64-windows-gnu`);`[platform.hosted] targets` 之外的 @@ -142,12 +142,12 @@ Windows 上这一安排对应 GNU ABI 的行(`--target x86_64-windows-gnu`);`[pl 另一个版本时,在任何编译之前拒绝: ``` -error: openkal-musl 0.22.2 is in this build, and the platform openkal-platform 0.18.0 is a version group that names openkal-musl 0.22.9. +error: openkal-musl 0.22.2 is in this build, and the platform openkal-platform 0.24.0 is a version group that names openkal-musl 0.28.0. requested by: own The family's packages pin one another exactly, so two versions of one of them cannot be linked into one program. Either select the platform release of that version ([target.] platform = "openkal-platform@"), or write - openkal-musl = "0.22.6" where it is requested. + openkal-musl = "0.28.0" where it is requested. ``` 从目录或仓库取得的包不受版本组约束。 @@ -160,39 +160,40 @@ error: openkal-musl 0.22.2 is in this build, and the platform openkal-platform 0 [package] namespace = "mcpplibs" name = "openkal-platform" -version = "0.18.0" -mcpp = ">=2026.10.30.1" +version = "0.24.0" +mcpp = ">=2026.11.3.1" [platform] targets = ["x86_64-openkal", "aarch64-openkal"] toolchain = "llvm@23.1.3" -runner = { cmd = ["okrun", "{}"], host = "same-arch" } -carrier = { package = "openkal-run@0.1.3", section = ".oke", main = "carrier/main.cpp" } +runner = { cmd = ["okrun", "{}"], host = "same-arch", hosts = ["linux-x86_64", "linux-aarch64", "macos-aarch64", "windows-x86_64"] } +carrier = { package = "openkal-run@0.5.0", section = ".oke", main = "carrier/main.cpp" } plugin = "build.mcpp" [platform.layers] -compiler-runtime = "openkal-llvm-runtime@0.16.6" -"c++-abi" = "openkal-llvm-runtime@0.16.6" -c-abi = { name = "openkal-musl", version = "0.22.6", default-features = false, features = ["posix"] } +compiler-runtime = "openkal-llvm-runtime@0.19.0" +"c++-abi" = "openkal-llvm-runtime@0.19.0" +c-abi = { name = "openkal-musl", version = "0.28.0", default-features = false, features = ["posix", "malloc-mimalloc"] } [platform.tools] -okld = "openkal-tools@0.1.3" -okrun = "openkal-tools@0.1.3" +okld = "openkal-tools@0.5.0" +okprof = "openkal-tools@0.5.0" +okrun = "openkal-tools@0.5.0" [platform.build] -cxxflags = ["-fPIC", "-femulated-tls", "-fvisibility=hidden", "-fasynchronous-unwind-tables"] +cxxflags = ["-fPIC", "-femulated-tls", "-fvisibility=hidden", "-fasynchronous-unwind-tables", "-Xclang", "-fstack-clash-protection", "-ffunction-sections", "-fdata-sections", "-fno-omit-frame-pointer", "-mno-omit-leaf-frame-pointer", "-fstack-protector-strong", "-mstack-protector-guard=global", "-ftrivial-auto-var-init=zero", "-D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_FAST"] [platform.build.'cfg(arch = "aarch64")'] -cxxflags = ["-ffixed-x18", "-mno-outline-atomics"] +cxxflags = ["-ffixed-x18", "-mno-outline-atomics", "-mbranch-protection=pac-ret+bti"] [platform.link] driver = "lld" start = "ok_main" -flags = ["-shared", "-z", "max-page-size=65536", "-z", "now", "--gc-sections", "--icf=safe", "--eh-frame-hdr"] +flags = ["-shared", "-z", "max-page-size=65536", "-z", "pack-relative-relocs", "-z", "now", "-z", "noexecstack", "--hash-style=both", "--gc-sections", "--icf=safe", "--eh-frame-hdr", "--build-id=sha1", "--emit-relocs", "-mllvm", "-emulated-tls"] inputs = [{ from = ["okld", "note-source"], as = "ok-notes.S" }] exe = "{name}.oke" shared = "lib{name}.okso" -seal = [["okld", "note", "{}"], ["okld", "check", "{}", "--with", "{needed}"]] +seal = [["okld", "note", "{}"], ["okld", "place", "{}"], ["okld", "check", "{}", "--with", "{needed}"]] ``` - **程序写名字,不写位置。** `inputs` 与 `seal` 中的 `okld` 依次在图为本机构建的程序、声明的载荷、`PATH` 中查找。 diff --git a/docs/zh/50-machine-output.md b/docs/zh/50-machine-output.md index 495beb3e2..37fc7a997 100644 --- a/docs/zh/50-machine-output.md +++ b/docs/zh/50-machine-output.md @@ -463,7 +463,7 @@ mcpp toolchain list --format json 的形状是 `{target, note, toolchain, pin, status, default}`，`status` 取 `installed` / `available` / `via dependency graph` / `planned` 之一。 目标行另有 `platform`（2026.10.27.1+）：目标表为该目标选择的平台包 -（`mcpplibs.openkal-platform@0.23.3`），或 `null`。 +（`mcpplibs.openkal-platform@0.24.0`），或 `null`。 **`toolchain` 与 `pin` 不是把同一个字段写了两遍。** `toolchain` 表示这一行 关联到什么 —— 已装的行是装好的载荷，词表行是它自己的约定。`pin` 只承载目标 diff --git a/src/build/schedule/compiler_transaction.cppm b/src/build/schedule/compiler_transaction.cppm index 17431459c..d7bafb982 100644 --- a/src/build/schedule/compiler_transaction.cppm +++ b/src/build/schedule/compiler_transaction.cppm @@ -85,11 +85,24 @@ std::expected output_identity(const fs::path& path) { std::string spelling(const fs::path& path) { return mcpp::home::spelling(path); } fs::path named(std::string_view path) { return mcpp::home::from_utf8(path); } fs::path suffix(const fs::path& path,std::string_view tail) { return named(spelling(path)+std::string(tail)); } +std::optional journal_time(fs::file_time_type time) { + // libc++ uses a wider file-clock representation. The v1 journal accepts + // signed 64-bit ticks; never truncate a timestamp into a different identity. + using Rep=fs::file_time_type::duration::rep; + const Rep ticks=time.time_since_epoch().count(); + if constexpr(std::numeric_limits::digits>std::numeric_limits::digits) { + if(ticks>static_cast(std::numeric_limits::max()))return {}; + if constexpr(std::numeric_limits::is_signed) + if(ticks(std::numeric_limits::min()))return {}; + } + return static_cast(ticks); +} std::optional identity(const fs::path& path) { if(path.empty())return Fields{"", ""}; std::error_code ec;const auto size=fs::file_size(path,ec);if(ec)return {}; const auto time=fs::last_write_time(path,ec);if(ec)return {}; - return Fields{std::to_string(size),std::to_string(time.time_since_epoch().count())}; + const auto ticks=journal_time(time);if(!ticks)return {}; + return Fields{std::to_string(size),std::to_string(*ticks)}; } bool copy_atomic(const fs::path& from,const fs::path& to) { std::error_code ec;const auto temp=suffix(to,".mcpp-restore"); @@ -127,6 +140,9 @@ struct Attempt { return std::unexpected("missing compiler transaction output identity"); if(!a.bmi.empty())if(auto old=identity(a.bmi)) { a.before_present=true;a.before_size=(*old)[0];a.before_time=(*old)[1];a.before_hash=mcpp::toolchain::hash_file(a.bmi); + } else { + std::error_code ec; + if(fs::exists(a.bmi,ec)||ec)return std::unexpected("cannot identify the previous compiler BMI"); } const auto who=os::process::self();if(!who||!who->id||!who->started_ns)return std::unexpected("cannot identify compiler transaction writer"); static std::atomic sequence{0}; @@ -368,7 +384,15 @@ private: if(!retired||(*retired&&(**retired).size()!=3)) {failed_.store(true);++it;continue;} if(*retired) {it=held_.erase(it);continue;} auto attempt=Attempt::load(directory); - if(!attempt||!controller_alive(*attempt))failed_.store(true); + if(!attempt) {failed_.store(true);++it;continue;} + if(!controller_alive(*attempt)) { + // Retirement can be published between the first read and the + // controller's exit. Observe it again before reporting loss. + retired=read(directory/"retired","mcpp-retired-v1"); + if(!retired||(*retired&&(**retired).size()!=3))failed_.store(true); + else if(*retired) {it=held_.erase(it);continue;} + else failed_.store(true); + } ++it; } } diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 487c9e881..9818d957b 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -809,6 +809,9 @@ int compile_release_at_bmi(const CompileRequest& req) { nextControllerCheck=clock::now()+std::chrono::milliseconds(50); if(!transactions::controller_alive(*attempt) ||(supervisor&&!mcpplibs::os::process::alive(*supervisor))) { + // The final status can be published between the first read and + // this liveness observation. The next iteration validates it. + if(read_rc(actual.slot))continue; std::println(std::cerr,"mcpp: compiler controller ended without a final status; retaining its outputs until unit recovery"); return 1; } @@ -950,6 +953,9 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o if(attempt&&clock::now()>=nextControllerCheck) { nextControllerCheck=clock::now()+std::chrono::milliseconds(50); if(!transactions::controller_alive(*attempt)) { + // A controller may publish its status and exit after the read + // in the loop condition. Let the normal result path inspect it. + if((rc=read_rc(slot)))break; std::println(std::cerr,"mcpp: compiler controller ended without a status; unit recovery will retain or invalidate its outputs"); return 1; } diff --git a/src/pack/distribution.cppm b/src/pack/distribution.cppm index d23ee31a2..72ae107c0 100644 --- a/src/pack/distribution.cppm +++ b/src/pack/distribution.cppm @@ -1074,10 +1074,9 @@ std::string exe_launcher_manifest(std::string_view name, std::string_view versio "# Generated by `mcpp pack`: the Windows launcher of {0} (design 2026-10-29 §4.3).\n" "[package]\nname = \"{0}\"\nversion = \"{1}\"\n\n" "[targets.{0}]\nkind = \"bin\"\nmain = \"src/main.cpp\"\n\n" - "# 0.2.9: the release written against openkal 0.23, the openkal the\n" - "# platform of mcpp's openkal rows names (openkal-platform 0.23.3), which\n" - "# the launcher is built over.\n" - "[dependencies.mcpplibs]\nos = \"0.2.9\"\n", + "# The launcher uses the OS release selected by the current default\n" + "# platform (openkal-platform 0.24).\n" + "[dependencies.mcpplibs]\nos = \"0.3.0\"\n", name, version.empty() ? std::string_view("0.0.0") : version); } diff --git a/tests/e2e/234_bmi_schedule_on.sh b/tests/e2e/234_bmi_schedule_on.sh index 4d6336a31..0295e0d86 100755 --- a/tests/e2e/234_bmi_schedule_on.sh +++ b/tests/e2e/234_bmi_schedule_on.sh @@ -264,4 +264,7 @@ PY_REPLAY echo "== 8. missing cached objects recover before Ninja admission ==" python3 "$_root/tests/e2e/helpers/check_cached_object_recovery.py" "$MCPP" "$TMP" "$TMP/$BUILD_DIR" +echo "== 9. final status is reread after controller exit ==" +python3 "$_root/tests/e2e/helpers/check_completion_observation.py" "$MCPP" "$TMP/completion-observation" + echo "234 bmi_schedule OK (detach-codegen, tokens reclaimed, publication checked)" diff --git a/tests/e2e/helpers/check_cached_object_recovery.py b/tests/e2e/helpers/check_cached_object_recovery.py index f6a190eee..d77644ff0 100644 --- a/tests/e2e/helpers/check_cached_object_recovery.py +++ b/tests/e2e/helpers/check_cached_object_recovery.py @@ -2,6 +2,7 @@ """Recover completed module objects in one actual build, before Ninja reads BMIs.""" import argparse import hashlib +import difflib import json from pathlib import Path import subprocess @@ -18,7 +19,9 @@ def main(): objects=[Path(row['output']) for row in rows if row['file'].endswith(('src/core.cppm','src/mid.cppm'))] assert len(objects)==2,'two actual module objects are required' def build(success=True): - result=subprocess.run([str(mcpp),'build','-j2'],cwd=project,capture_output=True,text=True,timeout=60) + # Match the graph's original configuration, including MCPP_JOBS. + # Changing the compiler cap changes the generated Ninja commands. + result=subprocess.run([str(mcpp),'build'],cwd=project,capture_output=True,text=True,timeout=60) assert (result.returncode==0)==success,result.stdout+result.stderr return result.stdout+result.stderr def identity(): @@ -30,7 +33,7 @@ def identity(): build() assert all(path.is_file() for path in objects),'one invocation did not restore missing objects' before=identity();build();assert identity()==before,'recovery changed the following no-op' - assert (output/'build.ninja').read_bytes()==graph,'recovery replaced its build graph' + assert (output/'build.ninja').read_bytes()==graph,'recovery replaced its build graph\n'+''.join(difflib.unified_diff(graph.decode().splitlines(True),(output/'build.ninja').read_text().splitlines(True))) print(f'PASS: {len(missing)} missing completed objects recover in one build; following no-op unchanged',flush=True) certificate=Path(str(objects[0])+'.sched.completed');original=certificate.read_bytes() bmi_before={p:(p.stat().st_mtime_ns,p.read_bytes()) for p in (output/'gcm.cache').rglob('*.gcm')} diff --git a/tests/e2e/helpers/check_completion_observation.py b/tests/e2e/helpers/check_completion_observation.py new file mode 100644 index 000000000..f8568c849 --- /dev/null +++ b/tests/e2e/helpers/check_completion_observation.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""A completed compiler cannot fail because status and liveness were sampled apart.""" +import argparse +import os +from pathlib import Path +import subprocess + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('mcpp', type=Path) + parser.add_argument('project', type=Path) + args = parser.parse_args() + mcpp = args.mcpp.resolve() + if not Path('/proc/self/stat').exists() or not Path('/usr/bin/cc').exists(): + print('SKIP: completion observation requires Linux libc interposition') + return + elf = subprocess.run(['readelf', '-l', str(mcpp)], capture_output=True, text=True, check=True) + if 'ld-linux' not in elf.stdout: + print('SKIP: completion observation requires a dynamic glibc mcpp') + return + root = args.project.resolve() + (root / 'src').mkdir(parents=True) + (root / 'mcpp.toml').write_text('''[package] +name = "completion-observation" +version = "0.1.0" +[build] +bmi_schedule = "on" +default-profile = "release" +[toolchain] +default = "gcc@16.1.0" +''') + (root / 'src/value.cppm').write_text('export module race.value;\nexport int value(){return 42;}\n') + (root / 'src/main.cpp').write_text('import race.value;\nint main(){return value()==42?0:1;}\n') + library = root / 'completion-race.so' + subprocess.run(['/usr/bin/cc', '-shared', '-fPIC', '-Wall', '-Wextra', '-Werror', + str(Path(__file__).with_name('compiler_completion_race.c')), + '-o', str(library), '-ldl'], check=True) + env = dict(os.environ, LD_PRELOAD=str(library)) + result = subprocess.run([str(mcpp), 'build'], cwd=root, env=env, + capture_output=True, text=True, timeout=60) + assert result.returncode == 0, result.stdout + result.stderr + assert list((root / 'target').rglob('*.completion-race')), 'the completion race was not exercised' + programs = list((root / 'target').glob('*/*/bin/completion-observation')) + assert len(programs) == 1, programs + subprocess.run([str(programs[0])], check=True, timeout=10) + result = subprocess.run([str(mcpp), 'build'], cwd=root, + capture_output=True, text=True, timeout=60) + assert result.returncode == 0, result.stdout + result.stderr + print('PASS: a final status published before controller exit survives an earlier absent observation') + + +if __name__ == '__main__': + main() diff --git a/tests/e2e/helpers/compiler_completion_race.c b/tests/e2e/helpers/compiler_completion_race.c new file mode 100644 index 000000000..ff958a5b5 --- /dev/null +++ b/tests/e2e/helpers/compiler_completion_race.c @@ -0,0 +1,81 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include + +/* Model an absent status sampled immediately before the controller publishes + * it and exits. Only a real split-compiler waiter is affected, once per process. + * No certificate, status, output or process lifetime is changed by this hook. */ +static FILE *(*native_open)(const char *, const char *); +static int injected; + +static int waiter(void) { + char command[8192]; + FILE *in = native_open("/proc/self/cmdline", "rb"); + if (!in) return 0; + size_t n = fread(command, 1, sizeof command - 1, in); + fclose(in); command[n] = 0; + char *arg = memchr(command, 0, n); + return arg && (!strcmp(arg + 1, "bmi-await") || !strcmp(arg + 1, "bmi-compile")); +} + +static int first_field(const char *path, char *value, size_t capacity) { + FILE *in = native_open(path, "rb"); + if (!in) return 0; + char line[128]; size_t n = 0; + int ok = fgets(line, sizeof line, in) && fgets(line, sizeof line, in) + && fgets(line, sizeof line, in) && sscanf(line, "%zu", &n) == 1 + && n > 0 && n < capacity && fread(value, 1, n, in) == n; + fclose(in); + if (ok) value[n] = 0; + return ok; +} + +static int ended(long pid) { + char path[64], line[4096]; + snprintf(path, sizeof path, "/proc/%ld/stat", pid); + FILE *in = native_open(path, "rb"); + if (!in) return errno == ENOENT; + int ok = fgets(line, sizeof line, in) != NULL; + fclose(in); + char *end = ok ? strrchr(line, ')') : NULL; + return end && end[1] == ' ' && (end[2] == 'Z' || end[2] == 'X'); +} + +FILE *fopen64(const char *path, const char *mode) { + if (!native_open) native_open = dlsym(RTLD_NEXT, "fopen64"); + FILE *result = native_open(path, mode); + size_t n = strlen(path); + if (!result || injected || mode[0] != 'r' || n < 3 + || strcmp(path + n - 3, ".rc") || !waiter()) return result; + char pointer[4096], directory[4096], controller[4096], pid_text[64]; + if (n + sizeof ".transaction" >= sizeof pointer) return result; + memcpy(pointer, path, n - 3); strcpy(pointer + n - 3, ".transaction"); + if (!first_field(pointer, directory, sizeof directory)) return result; + int length = snprintf(controller, sizeof controller, "%s/controller", directory); + if (length < 0 || (size_t)length >= sizeof controller + || !first_field(controller, pid_text, sizeof pid_text)) return result; + long pid = strtol(pid_text, NULL, 10); + if (pid <= 1) return result; + for (int tries = 0; tries < 600 && !ended(pid); ++tries) + nanosleep(&(struct timespec){ .tv_nsec = 5000000 }, NULL); + if (!ended(pid)) return result; + /* Make the waiter's next bounded controller observation due. */ + nanosleep(&(struct timespec){ .tv_nsec = 80000000 }, NULL); + injected = 1; + fclose(result); + char marker[4096]; + if (snprintf(marker, sizeof marker, "%s.completion-race", path) >= (int)sizeof marker) abort(); + FILE *receipt = native_open(marker, "wb"); + if (!receipt) abort(); + fputs("status exists; controller ended; one earlier absent observation replayed\n", receipt); + if (fclose(receipt)) abort(); + errno = ENOENT; + return NULL; +} + +FILE *fopen(const char *path, const char *mode) { return fopen64(path, mode); } diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp index 4654b6165..035670d92 100644 --- a/tests/unit/test_compiler_transaction.cpp +++ b/tests/unit/test_compiler_transaction.cpp @@ -15,6 +15,7 @@ struct Transaction { Transaction() { root=fs::temp_directory_path()/std::format("mcpp-attempt-{:08x}",std::random_device{}()); fs::create_directories(root); + root=fs::weakly_canonical(root); generation=root/"run";bmi=root/"unit.gcm";object=root/"unit.o"; slot=root/"unit.sched";dep=root/"unit.d";reservation=root/"slot.reservation"; } @@ -44,6 +45,34 @@ struct Transaction { }; } +TEST(CompilerTransaction, JournalTimePreservesSignedTicksAtBothBoundaries) { + using Duration=fs::file_time_type::duration; + using Rep=Duration::rep; + for(const auto value:{std::numeric_limits::min(),-1LL,0LL,1LL,std::numeric_limits::max()}) { + const auto stamp=tx::journal_time(fs::file_time_type{Duration{static_cast(value)}}); + ASSERT_TRUE(stamp);EXPECT_EQ(*stamp,value); + EXPECT_TRUE(tx::output_stamp("1",std::to_string(*stamp))); + } + if constexpr(std::numeric_limits::digits>std::numeric_limits::digits) { + auto above=static_cast(std::numeric_limits::max());++above; + auto below=static_cast(std::numeric_limits::min());--below; + EXPECT_FALSE(tx::journal_time(fs::file_time_type{Duration{above}})); + EXPECT_FALSE(tx::journal_time(fs::file_time_type{Duration{below}})); + } +} +TEST(CompilerTransaction, PreparedIdentityRoundTripsTheActualFileTimestamp) { + Transaction t;const auto prepared=t.prepare(); + const auto ticks=tx::journal_time(fs::last_write_time(t.bmi));ASSERT_TRUE(ticks); + EXPECT_EQ(prepared.before_time,std::to_string(*ticks)); + const auto loaded=tx::Attempt::load(prepared.directory);ASSERT_TRUE(loaded); + EXPECT_TRUE(loaded->before_present);EXPECT_EQ(loaded->before_time,prepared.before_time); + EXPECT_TRUE(tx::validate(*loaded)); +} +TEST(CompilerTransaction, UnidentifiableExistingBmiIsNotRecordedAsMissing) { + Transaction t;fs::create_directory(t.bmi); + EXPECT_FALSE(tx::Attempt::prepare(t.generation,false,t.bmi,t.object,t.slot,t.dep)); + EXPECT_TRUE(fs::is_directory(t.bmi));EXPECT_FALSE(fs::exists(t.generation)); +} TEST(CompilerTransaction, RecordsAreBoundedChecksummedAndSchemaChecked) { Transaction t;const auto record=t.root/"record"; ASSERT_TRUE(tx::publish(record,"schema",{"a\nb", "", "unicode \xc3\xa9"})); diff --git a/tests/unit/test_pack_distribution.cpp b/tests/unit/test_pack_distribution.cpp index cfb34240d..9447c4df1 100644 --- a/tests/unit/test_pack_distribution.cpp +++ b/tests/unit/test_pack_distribution.cpp @@ -261,7 +261,7 @@ TEST(PackDistribution, TheWindowsLauncherIsAProgram) { EXPECT_NE(src.find("(\"windows-\" + arch)"), std::string::npos) << src; EXPECT_NE(src.find("os::process::exec"), std::string::npos) << src; const auto manifest = d::exe_launcher_manifest("hello", "1.0.0"); - EXPECT_NE(manifest.find("os = \"0.2.9\""), std::string::npos) << manifest; + EXPECT_NE(manifest.find("os = \"0.3.0\""), std::string::npos) << manifest; EXPECT_NE(manifest.find("[targets.hello]"), std::string::npos) << manifest; } From ab4f4cbaa7ae97a297d8dc9d4080261e12f200d3 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 14:57:49 +0900 Subject: [PATCH 18/40] fix: retry atomic journal opens and preserve supervisor diagnostics --- .../2026-11-06-compiler-attempt-recovery.md | 23 ++++++++++++++++ CHANGELOG.md | 4 +++ src/build/schedule/compiler_transaction.cppm | 26 ++++++++++++++----- src/build/schedule/detach_codegen.cppm | 23 +++++++++++++--- tests/e2e/helpers/record_open_appearance.c | 24 +++++++++++++++++ tests/unit/test_compiler_transaction.cpp | 26 +++++++++++++++++++ 6 files changed, 117 insertions(+), 9 deletions(-) create mode 100644 tests/e2e/helpers/record_open_appearance.c diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md index ce6aebc62..5a08fba50 100644 --- a/.agents/docs/2026-11-06-compiler-attempt-recovery.md +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -245,6 +245,20 @@ in-range ticks. A direct exact-toolchain control compiled the failing expression and the corrected one, including a real file identity. Native Clang, ARM and bootstrap jobs still require the next full CI run. +The next cold GCC run failed at the owner witness without a branch-specific +diagnostic. Code review found a separate record-publication observation race: +a reader's first open can report absence while an atomic publisher installs +the record before the following existence probe. Treating this pair as a +corrupt record can falsely fail the owner witness. A libc open-boundary control +reproduced the old failure and the corrected retry with identical published +bytes. The reader retries one open; a persistent refusal still fails. Witness +diagnostics now distinguish an invalid record, an unreadable attempt and a +controller without retirement so the next CI can test this attribution. The +native Windows suite also showed that +`file_size` can report a directory's size; output identity now requires a +regular leaf before reading size and timestamp. The Windows negative test and +the cold GCC build remain final-CI gates. + An actual split-compiler control held a previously absent `.rc` observation until the controller had published status and exited. The earlier engine failed with `compiler controller ended without a status`; the corrected engine rereads @@ -265,6 +279,15 @@ OS 0.3.0. The older consumer platform selected by the application's own target remains independent of the launcher's build platform. The actual cross-platform packing control and complete CI remain release gates. +The cross-host Windows→Linux job still reached a missing final status after +the controller exited. Because detached supervisor stdio was discarded, the +first CI did not retain whether unit retirement, certificate publication or +status publication failed. The supervisor now appends a bounded reason to its +existing compiler log at each of those exits; a waiter replays that log only +when the controller is gone and status remains absent. This changes the +diagnostic boundary, not the success criteria. The next Windows cross job must +identify the cause before any protocol or OS change is attributed to it. + ## Separate Windows build-program defect The released 2026.11.5.1 engine can try to install `msvc@system` as an indexed diff --git a/CHANGELOG.md b/CHANGELOG.md index 1aa97dcd6..34718b306 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,10 @@ platform, so their generated dependency resolves alongside openkal-platform 0.24. - Compiler attempt journals preserve file timestamps on Clang/libc++ when the file clock uses a wider integer representation. +- Compiler journals retry an open that raced with atomic record publication, + and BMI/object identity checks require a regular output file on every host. + A witness failure identifies its failed observation boundary, and a detached + supervisor's failure is available to the waiter even though its stdio is closed. - Launch descriptions include the selected target, profile, resolved platform and available named runners, including the reachable leg of a target set. Editor debug launches can retain the configuration that actually built the program. diff --git a/src/build/schedule/compiler_transaction.cppm b/src/build/schedule/compiler_transaction.cppm index d7bafb982..a0536d7ed 100644 --- a/src/build/schedule/compiler_transaction.cppm +++ b/src/build/schedule/compiler_transaction.cppm @@ -44,7 +44,12 @@ Record read(const fs::path& path, std::string_view tag) { if (!input) { std::error_code ec; if (!fs::exists(path,ec) && !ec) return std::optional{}; - return std::unexpected("cannot read compiler transaction record"); + if(ec)return std::unexpected("cannot read compiler transaction record"); + // An atomic publisher may install the record between the failed open + // and the existence probe. Retry once; a persistent refusal still + // fails rather than being mistaken for an absent record. + input.open(path,std::ios::binary); + if(!input)return std::unexpected("cannot read compiler transaction record"); } std::string bytes(65537,'\0'); input.read(bytes.data(),bytes.size());bytes.resize(static_cast(input.gcount())); @@ -99,7 +104,10 @@ std::optional journal_time(fs::file_time_type time) { } std::optional identity(const fs::path& path) { if(path.empty())return Fields{"", ""}; - std::error_code ec;const auto size=fs::file_size(path,ec);if(ec)return {}; + std::error_code ec; + const auto status=fs::symlink_status(path,ec); + if(ec||status.type()!=fs::file_type::regular)return {}; + const auto size=fs::file_size(path,ec);if(ec)return {}; const auto time=fs::last_write_time(path,ec);if(ec)return {}; const auto ticks=journal_time(time);if(!ticks)return {}; return Fields{std::to_string(size),std::to_string(*ticks)}; @@ -377,21 +385,27 @@ private: std::atomic stopping_{false},failed_{false}; std::mutex admission_; ObserverMap held_; + void fail(std::string_view reason) { + if(!failed_.exchange(true)) + std::println(std::cerr,"mcpp: compiler witness: {}",reason); + } void audit() { for(auto it=held_.begin();it!=held_.end();) { const auto directory=named(it->first); auto retired=read(directory/"retired","mcpp-retired-v1"); - if(!retired||(*retired&&(**retired).size()!=3)) {failed_.store(true);++it;continue;} + if(!retired) {fail(retired.error());++it;continue;} + if(*retired&&(**retired).size()!=3) {fail("invalid retirement certificate");++it;continue;} if(*retired) {it=held_.erase(it);continue;} auto attempt=Attempt::load(directory); - if(!attempt) {failed_.store(true);++it;continue;} + if(!attempt) {fail(attempt.error());++it;continue;} if(!controller_alive(*attempt)) { // Retirement can be published between the first read and the // controller's exit. Observe it again before reporting loss. retired=read(directory/"retired","mcpp-retired-v1"); - if(!retired||(*retired&&(**retired).size()!=3))failed_.store(true); + if(!retired)fail(retired.error()); + else if(*retired&&(**retired).size()!=3)fail("invalid retirement certificate"); else if(*retired) {it=held_.erase(it);continue;} - else failed_.store(true); + else fail("controller ended without retirement certificate"); } ++it; } diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 9818d957b..dcf7f4a14 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -533,6 +533,18 @@ int run_to_completion(std::string_view command, return owner.active() ? r.exit_code : 130; } +void record_supervisor_error(const std::filesystem::path& slot,std::string_view reason) { + // Detached supervisors have null stdio. Keep the failure beside the + // compiler's ordinary log so either waiter can report why status is absent. + std::ofstream log(suffixed(slot,".log"),std::ios::app); + if(log)log << "mcpp: " << reason << '\n'; +} + +void replay_supervisor_log(const std::filesystem::path& slot) { + std::ifstream log(suffixed(slot,".log")); + if(log)std::cerr << log.rdbuf(); +} + std::expected retire_owned(mcpplibs::os::process::owned_unit& unit,const Owner& owner) { const auto limit=std::chrono::steady_clock::now()+std::chrono::seconds(5); for(;;) { @@ -812,6 +824,7 @@ int compile_release_at_bmi(const CompileRequest& req) { // The final status can be published between the first read and // this liveness observation. The next iteration validates it. if(read_rc(actual.slot))continue; + replay_supervisor_log(actual.slot); std::println(std::cerr,"mcpp: compiler controller ended without a final status; retaining its outputs until unit recovery"); return 1; } @@ -872,10 +885,13 @@ int supervise(const std::filesystem::path& slot, } const int rc=run_to_completion(command,suffixed(slot,".log"),actor->owner,unit.get()); if(auto retired=retire_owned(*unit,actor->owner);!retired) { - std::println(std::cerr,"mcpp: {}",retired.error());return 1; + record_supervisor_error(slot,retired.error());return 1; + } + if(!attempt.retired(rc)) {record_supervisor_error(slot,"cannot publish compiler retirement certificate");return 1;} + if(!write_rc(slot,rc)) {record_supervisor_error(slot,"cannot publish compiler final status");return 1;} + if(!transactions::release_reservation(permit->reservation,attempt.directory)) { + record_supervisor_error(slot,"cannot release compiler reservation");return 1; } - if(!attempt.retired(rc)||!write_rc(slot,rc))return 1; - if(!transactions::release_reservation(permit->reservation,attempt.directory))return 1; return 0; } auto settled = std::make_shared>(false); @@ -956,6 +972,7 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o // A controller may publish its status and exit after the read // in the loop condition. Let the normal result path inspect it. if((rc=read_rc(slot)))break; + replay_supervisor_log(slot); std::println(std::cerr,"mcpp: compiler controller ended without a status; unit recovery will retain or invalidate its outputs"); return 1; } diff --git a/tests/e2e/helpers/record_open_appearance.c b/tests/e2e/helpers/record_open_appearance.c new file mode 100644 index 000000000..3532624aa --- /dev/null +++ b/tests/e2e/helpers/record_open_appearance.c @@ -0,0 +1,24 @@ +#define _GNU_SOURCE +#include +#include +#include +#include + +/* Simulate an atomic publish between the reader's first open and its + * existence probe. The real completed record remains present throughout. */ +static FILE *(*native_open)(const char *, const char *); +static int injected; + +FILE *fopen64(const char *path, const char *mode) { + if (!native_open) native_open = dlsym(RTLD_NEXT, "fopen64"); + if (!injected && strcmp(path, "record") == 0 && mode[0] == 'r') { + injected = 1; + FILE *marker = native_open("record-open-interposed", "wb"); + if (marker) fclose(marker); + errno = ENOENT; + return NULL; + } + return native_open(path, mode); +} + +FILE *fopen(const char *path, const char *mode) { return fopen64(path, mode); } diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp index 035670d92..f8c191524 100644 --- a/tests/unit/test_compiler_transaction.cpp +++ b/tests/unit/test_compiler_transaction.cpp @@ -227,6 +227,32 @@ TEST(CompilerTransaction, RecordCloseFailureDriver) { EXPECT_EQ(Transaction::contents("record"),"previous"); EXPECT_TRUE(fs::exists("record-close-refused")); } +TEST(CompilerTransaction, RecordOpenAppearanceDriver) { + if(!mcpp::invocation::get("MCPP_TEST_RECORD_OPEN_DRIVER"))GTEST_SKIP()<<"subprocess driver"; + auto record=tx::read("record","schema"); + EXPECT_TRUE(fs::exists("record-open-interposed")); + ASSERT_TRUE(record&&*record); + EXPECT_EQ((**record)[0],"published"); +} +TEST(CompilerTransaction, RecordAppearanceBetweenOpenAndExistsIsNotCorruption) { + if constexpr (!mcpplibs::os::is_linux)GTEST_SKIP()<<"Linux libc open interposition"; + const auto runtime=mcpp::runtime::elf::inspect_elf_runtime(mcpp::invocation::self_exe_path()); + if(!runtime || runtime->interp.find("ld-linux")==std::string::npos) + GTEST_SKIP()<<"open interposition requires the dynamic glibc loader"; + Transaction t; + const auto library=t.root/"record-open-appearance.so"; + const auto source=fs::current_path()/"tests/e2e/helpers/record_open_appearance.c"; + auto compiled=mcpp::run::execute({"/usr/bin/cc","-shared","-fPIC","-Wall","-Wextra","-Werror", + source.string(),"-o",library.string(),"-ldl"},{.total=std::chrono::seconds(10)}); + ASSERT_EQ(compiled.exit_code,0)< Date: Fri, 6 Nov 2026 15:38:19 +0900 Subject: [PATCH 19/40] fix: diagnose compiler status publication and retry release lookup --- .../2026-11-06-compiler-attempt-recovery.md | 8 +++++ .github/tools/upgrade_lane.sh | 33 ++++++++++++++----- CHANGELOG.md | 2 ++ src/build/schedule/compiler_transaction.cpp | 21 +++++++++--- src/build/schedule/compiler_transaction.cppm | 2 +- src/build/schedule/detach_codegen.cppm | 10 ++++-- tests/unit/test_compiler_transaction.cpp | 12 ++++++- 7 files changed, 70 insertions(+), 18 deletions(-) diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md index 5a08fba50..91d0c8ddb 100644 --- a/.agents/docs/2026-11-06-compiler-attempt-recovery.md +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -287,6 +287,14 @@ existing compiler log at each of those exits; a waiter replays that log only when the controller is gone and status remains absent. This changes the diagnostic boundary, not the success criteria. The next Windows cross job must identify the cause before any protocol or OS change is attributed to it. +The first diagnostic run narrowed this to final-status publication after unit +retirement. The publisher now reports whether its temporary open, write, close +or atomic rename failed, with the OS wrapper's OpenKal condition code for the +rename and without embedding a path or user name. This wrapper does not expose +the original Windows NTSTATUS. A controlled close failure and a +directory-target rename refusal exercise the stage classification. The next +cross-host run must determine the actual condition before any retry policy is +chosen. ## Separate Windows build-program defect diff --git a/.github/tools/upgrade_lane.sh b/.github/tools/upgrade_lane.sh index b079d65d0..0bb3491ac 100644 --- a/.github/tools/upgrade_lane.sh +++ b/.github/tools/upgrade_lane.sh @@ -59,17 +59,32 @@ IS_LINUX=0; [[ $PLAT == linux-* ]] && IS_LINUX=1 # ── the previous release ─────────────────────────────────────────────── CURRENT=$(sed -n 's/^version *= *"\([^"]*\)".*/\1/p' mcpp.toml | head -1) -OLD_VER=$(curl -fsSL ${GH_TOKEN:+-H "Authorization: Bearer $GH_TOKEN"} \ - "https://api.github.com/repos/mcpp-community/mcpp/releases?per_page=20" \ - | "$PY" -c ' +release_headers=() +if [ -n "${GH_TOKEN:-}" ]; then release_headers=(-H "Authorization: Bearer $GH_TOKEN"); fi +OLD_VER='' +for attempt in 1 2 3; do + # A hosted Windows run once received an empty release list before it could + # exercise any upgrade. Retry this read-only input, never a product action. + if curl -fsSL --connect-timeout 10 --max-time 30 "${release_headers[@]}" \ + "https://api.github.com/repos/mcpp-community/mcpp/releases?per_page=20" \ + -o "$W/releases.json"; then + OLD_VER=$("$PY" -c ' import json, sys current = sys.argv[1] -for r in json.load(sys.stdin): - v = r["tag_name"].lstrip("v") - if not r["draft"] and not r["prerelease"] and v != current: - print(v); break -' "$CURRENT") -case "$OLD_VER" in ''|*[!0-9.]*) echo "::error::no previous release resolved (got '$OLD_VER')"; exit 1 ;; esac +with open(sys.argv[2], encoding="utf-8") as source: + releases = json.load(source) +if isinstance(releases, list): + for release in releases: + version = release["tag_name"].lstrip("v") + if not release["draft"] and not release["prerelease"] and version != current: + print(version) + break +' "$CURRENT" "$W/releases.json") || OLD_VER='' + case "$OLD_VER" in ''|*[!0-9.]*) ;; *) break ;; esac + fi + [ "$attempt" = 3 ] || sleep 2 +done +case "$OLD_VER" in ''|*[!0-9.]*) echo "::error::no previous release resolved after three metadata reads (got '$OLD_VER')"; exit 1 ;; esac echo "previous release: $OLD_VER; this commit: $CURRENT ($PLAT, variant $VARIANT)" name="mcpp-$OLD_VER-$PLAT.$EXT" diff --git a/CHANGELOG.md b/CHANGELOG.md index 34718b306..7db928668 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ and BMI/object identity checks require a regular output file on every host. A witness failure identifies its failed observation boundary, and a detached supervisor's failure is available to the waiter even though its stdio is closed. + Status publication now distinguishes temporary open, write, close and rename + failures, including the OpenKal condition of a failed rename. - Launch descriptions include the selected target, profile, resolved platform and available named runners, including the reachable leg of a target set. Editor debug launches can retain the configuration that actually built the program. diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index 6ccacafe4..8d4f6524c 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -3,22 +3,35 @@ import std; namespace mcpp::build::schedule::transaction { -bool replace_record(const fs::path& path,std::string_view bytes) { +bool replace_record(const fs::path& path,std::string_view bytes,std::string* diagnostic) { // No process may observe a partial record. This protects process loss, // not power loss; do not change the OS library's durable write_atomic. static const auto who=os::process::self(); static std::atomic sequence{0}; - if(!who||!who->id||!who->started_ns)return false; + auto fail=[&](std::string_view stage) { + if(diagnostic)*diagnostic=stage; + return false; + }; + if(!who||!who->id||!who->started_ns)return fail("identify writer"); const auto temporary=path.parent_path()/named(std::format(".mcpp-record-{}-{}-{}", who->id,who->started_ns,sequence++)); std::ofstream output(temporary,std::ios::binary|std::ios::out|std::ios::noreplace); - if(!output)return false; // Never remove a name this invocation did not create. + if(!output)return fail("open temporary"); // Never remove a name this invocation did not create. output.write(bytes.data(),static_cast(bytes.size())); output.flush(); const bool written=bool(output); output.close(); const bool closed=bool(output); - if(written&&closed&&os::fs::rename(temporary,path,true))return true; + if(written&&closed) { + auto renamed=os::fs::rename(temporary,path,true); + if(renamed)return true; + if(diagnostic)*diagnostic=std::format("rename temporary (condition {}, native {})", + static_cast(renamed.error().code),renamed.error().native); + } else if(!written) { + fail("write/flush temporary"); + } else { + fail("close temporary"); + } std::error_code ec;fs::remove(temporary,ec);return false; } bool publish(const fs::path& path,std::string_view tag,const Fields& fields) { diff --git a/src/build/schedule/compiler_transaction.cppm b/src/build/schedule/compiler_transaction.cppm index a0536d7ed..70977162c 100644 --- a/src/build/schedule/compiler_transaction.cppm +++ b/src/build/schedule/compiler_transaction.cppm @@ -37,7 +37,7 @@ std::string pack(std::string_view tag, const Fields& fields) { return body + "\n" + mcpp::toolchain::hash_string(body) + "\n"; } // Atomic visibility across process loss; storage durability is not promised. -bool replace_record(const fs::path& path,std::string_view bytes); +bool replace_record(const fs::path& path,std::string_view bytes,std::string* diagnostic=nullptr); bool publish(const fs::path& path,std::string_view tag,const Fields& fields); Record read(const fs::path& path, std::string_view tag) { std::ifstream input(path,std::ios::binary); diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index dcf7f4a14..16d002bb0 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -312,8 +312,8 @@ std::optional read_rc(const std::filesystem::path& slot) { // Temp file + rename, so a reader never observes half a number. The same // guarantee, for the same reason, that makes watching the BMI path sound. -bool write_rc(const std::filesystem::path& slot, int rc) { - return transactions::replace_record(suffixed(slot,".rc"),std::to_string(rc)+"\n"); +bool write_rc(const std::filesystem::path& slot, int rc,std::string* diagnostic=nullptr) { + return transactions::replace_record(suffixed(slot,".rc"),std::to_string(rc)+"\n",diagnostic); } // Stable kernel lock slots, shared by ordinary and detached compilers. The @@ -888,7 +888,11 @@ int supervise(const std::filesystem::path& slot, record_supervisor_error(slot,retired.error());return 1; } if(!attempt.retired(rc)) {record_supervisor_error(slot,"cannot publish compiler retirement certificate");return 1;} - if(!write_rc(slot,rc)) {record_supervisor_error(slot,"cannot publish compiler final status");return 1;} + std::string status_failure; + if(!write_rc(slot,rc,&status_failure)) { + record_supervisor_error(slot,std::format("cannot publish compiler final status: {}",status_failure)); + return 1; + } if(!transactions::release_reservation(permit->reservation,attempt.directory)) { record_supervisor_error(slot,"cannot release compiler reservation");return 1; } diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp index f8c191524..22ac0cd57 100644 --- a/tests/unit/test_compiler_transaction.cpp +++ b/tests/unit/test_compiler_transaction.cpp @@ -223,10 +223,20 @@ TEST(CompilerTransaction, ChangedBmiParentCannotRedirectCertifiedRecovery) { TEST(CompilerTransaction, RecordCloseFailureDriver) { if(!mcpp::invocation::get("MCPP_TEST_RECORD_CLOSE_DRIVER"))GTEST_SKIP()<<"subprocess driver"; - EXPECT_FALSE(tx::replace_record("record","replacement")); + std::string diagnostic; + EXPECT_FALSE(tx::replace_record("record","replacement",&diagnostic)); + EXPECT_EQ(diagnostic,"close temporary"); EXPECT_EQ(Transaction::contents("record"),"previous"); EXPECT_TRUE(fs::exists("record-close-refused")); } +TEST(CompilerTransaction, FailedRenameReportsTheStageAndPreservesItsTarget) { + Transaction t; + fs::create_directory(t.root/"record"); + std::string diagnostic; + EXPECT_FALSE(tx::replace_record(t.root/"record","replacement",&diagnostic)); + EXPECT_TRUE(diagnostic.starts_with("rename temporary (condition "))< Date: Fri, 6 Nov 2026 16:55:28 +0900 Subject: [PATCH 20/40] test(ci): isolate macOS Android install and diagnose Windows record rename --- .github/workflows/ci-macos-e2e.yml | 3 +++ src/build/schedule/compiler_transaction.cpp | 13 +++++++++++-- tests/e2e/run_all.sh | 17 +++++++++++++++-- tests/unit/test_compiler_transaction.cpp | 13 +++++++++++++ 4 files changed, 42 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci-macos-e2e.yml b/.github/workflows/ci-macos-e2e.yml index 56f28dcd5..0d552eb39 100644 --- a/.github/workflows/ci-macos-e2e.yml +++ b/.github/workflows/ci-macos-e2e.yml @@ -52,6 +52,9 @@ jobs: timeout-minutes: 60 env: E2E_SHARD: ${{ matrix.shard }}/${{ matrix.shards }} + # 721 installs the large NDK in its own bounded step below. Running it in + # the ordinary shard duplicates the download and can fail before linking. + E2E_EXCLUDE: 721_*.sh E2E_TIMINGS: tests/e2e/timings/macos.tsv E2E_REPORT: ${{ github.workspace }}/e2e-report-${{ matrix.image == 'macos-15' && 'macos' || matrix.image }}-${{ matrix.shard }}.tsv # NOTE: no MCPP_VERBOSE — the e2e suite asserts mcpp's default quiet diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index 8d4f6524c..c1ac35047 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -25,8 +25,17 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia if(written&&closed) { auto renamed=os::fs::rename(temporary,path,true); if(renamed)return true; - if(diagnostic)*diagnostic=std::format("rename temporary (condition {}, native {})", - static_cast(renamed.error().code),renamed.error().native); + if(diagnostic) { + // The OpenKal condition does not retain the Windows NTSTATUS. These + // observations distinguish a blocked replacement from a source or + // parent failure without exposing private build paths in CI logs. + std::error_code sourceError, targetError; + const auto source=fs::symlink_status(temporary,sourceError); + const auto target=fs::symlink_status(path,targetError); + *diagnostic=std::format("rename temporary (condition {}, native {}; source regular {} ec {}; target exists {} ec {})", + static_cast(renamed.error().code),renamed.error().native, + fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value()); + } } else if(!written) { fail("write/flush temporary"); } else { diff --git a/tests/e2e/run_all.sh b/tests/e2e/run_all.sh index ef1468774..b76f535e3 100755 --- a/tests/e2e/run_all.sh +++ b/tests/e2e/run_all.sh @@ -468,8 +468,8 @@ report() { } if [[ -n "${E2E_REPORT:-}" ]]; then : > "$E2E_REPORT"; fi -# E2E_LIST=1 prints the tests this invocation would run (after E2E_ONLY and the -# shard, before the capability check) and exits, so a shard's membership can be +# E2E_LIST=1 prints the tests this invocation would run (after E2E_ONLY, +# E2E_EXCLUDE and the shard, before the capability check) and exits, so a shard's membership can be # read without running it. if [[ "${E2E_LIST:-}" == "1" ]]; then for test in "$HERE"/[0-9]*.sh; do @@ -478,6 +478,10 @@ if [[ "${E2E_LIST:-}" == "1" ]]; then # shellcheck disable=SC2053 -- glob match is the point [[ "$name" == $E2E_ONLY ]] || continue fi + if [[ -n "${E2E_EXCLUDE:-}" ]]; then + # shellcheck disable=SC2053 -- glob match is the point + [[ "$name" == $E2E_EXCLUDE ]] && continue + fi if (( SHARD_TOTAL > 1 )); then in_shard "$name" || continue; fi echo "$name" done @@ -496,6 +500,15 @@ for test in "$HERE"/[0-9]*.sh; do if (( SHARD_TOTAL > 1 )); then in_shard "$name" || continue fi + if [[ -n "${E2E_EXCLUDE:-}" ]]; then + # shellcheck disable=SC2053 -- glob match is the point + if [[ "$name" == $E2E_EXCLUDE ]]; then + echo "SKIP: $name (owned by a dedicated workflow step)" + ((SKIP++)) + report skip "$name" 0 "owned by a dedicated workflow step" + continue + fi + fi echo missing_cap="$(check_requires "$test")" if [[ -n "$missing_cap" ]]; then diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp index 22ac0cd57..641bdc548 100644 --- a/tests/unit/test_compiler_transaction.cpp +++ b/tests/unit/test_compiler_transaction.cpp @@ -237,6 +237,19 @@ TEST(CompilerTransaction, FailedRenameReportsTheStageAndPreservesItsTarget) { EXPECT_TRUE(diagnostic.starts_with("rename temporary (condition "))<(held),std::istreambuf_iterator()),"old"); +} TEST(CompilerTransaction, RecordOpenAppearanceDriver) { if(!mcpp::invocation::get("MCPP_TEST_RECORD_OPEN_DRIVER"))GTEST_SKIP()<<"subprocess driver"; auto record=tx::read("record","schema"); From e5bb7270c0e504754f99d05fa9f08b63aa3f209b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 17:29:42 +0900 Subject: [PATCH 21/40] fix(build): try atomic no-replace for absent status records --- src/build/schedule/compiler_transaction.cpp | 20 ++++++++++++++++++-- src/build/schedule/compiler_transaction.cppm | 1 + src/build/schedule/detach_codegen.cppm | 2 ++ tests/unit/test_compiler_transaction.cpp | 12 ++++++++++++ 4 files changed, 33 insertions(+), 2 deletions(-) diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index c1ac35047..a5f88040c 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -32,9 +32,25 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia std::error_code sourceError, targetError; const auto source=fs::symlink_status(temporary,sourceError); const auto target=fs::symlink_status(path,targetError); - *diagnostic=std::format("rename temporary (condition {}, native {}; source regular {} ec {}; target exists {} ec {})", + // A final status is a new name. If the replace form itself was + // refused as invalid while that name is absent, try the atomic + // no-replace form on the exact same source and destination. Its + // refusal is preserved as evidence rather than hidden by a loop. + int noReplaceCode=0; + const bool targetAbsent=target.type()==fs::file_type::not_found + || targetError==std::errc::no_such_file_or_directory; + if(renamed.error().code==os::errc::invalid && fs::is_regular_file(source) + && targetAbsent) { + auto noReplace=os::fs::rename(temporary,path,false); + if(noReplace) { + *diagnostic="replace form refused absent target; no-replace published"; + return true; + } + noReplaceCode=static_cast(noReplace.error().code); + } + *diagnostic=std::format("rename temporary (condition {}, native {}; source regular {} ec {}; target exists {} ec {}; no-replace condition {})", static_cast(renamed.error().code),renamed.error().native, - fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value()); + fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value(),noReplaceCode); } } else if(!written) { fail("write/flush temporary"); diff --git a/src/build/schedule/compiler_transaction.cppm b/src/build/schedule/compiler_transaction.cppm index 70977162c..c9ecb4b66 100644 --- a/src/build/schedule/compiler_transaction.cppm +++ b/src/build/schedule/compiler_transaction.cppm @@ -3,6 +3,7 @@ export module mcpp.build.schedule.compiler_transaction; import std; import mcpplibs.os.fs; +import mcpplibs.os.error; import mcpplibs.os.process; import mcpplibs.os.ipc; import mcpplibs.os.dirs; diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 16d002bb0..d3044adff 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -893,6 +893,8 @@ int supervise(const std::filesystem::path& slot, record_supervisor_error(slot,std::format("cannot publish compiler final status: {}",status_failure)); return 1; } + if(!status_failure.empty()) + std::println(std::cerr,"mcpp: {}",status_failure); if(!transactions::release_reservation(permit->reservation,attempt.directory)) { record_supervisor_error(slot,"cannot release compiler reservation");return 1; } diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp index 641bdc548..5ae129439 100644 --- a/tests/unit/test_compiler_transaction.cpp +++ b/tests/unit/test_compiler_transaction.cpp @@ -6,6 +6,7 @@ import mcpp.run; import mcpp.invocation; import mcpp.runtime.elf; import mcpplibs.os.platform; +import mcpplibs.os.fs; namespace tx = mcpp::build::schedule::transaction; namespace fs = std::filesystem; @@ -250,6 +251,17 @@ TEST(CompilerTransaction, ReaderHeldTargetKeepsAnAtomicRecord) { EXPECT_EQ(Transaction::contents(record),replaced?"new":"old"); EXPECT_EQ(std::string(std::istreambuf_iterator(held),std::istreambuf_iterator()),"old"); } +TEST(CompilerTransaction, NoReplacePublishesOnlyAnAbsentRecord) { + Transaction t; + const auto source=t.root/"new"; + const auto target=t.root/"status"; + std::ofstream(source)<<"ready"; + ASSERT_TRUE(mcpplibs::os::fs::rename(source,target,false)); + EXPECT_EQ(Transaction::contents(target),"ready"); + std::ofstream(source)<<"later"; + EXPECT_FALSE(mcpplibs::os::fs::rename(source,target,false)); + EXPECT_EQ(Transaction::contents(target),"ready"); +} TEST(CompilerTransaction, RecordOpenAppearanceDriver) { if(!mcpp::invocation::get("MCPP_TEST_RECORD_OPEN_DRIVER"))GTEST_SKIP()<<"subprocess driver"; auto record=tx::read("record","schema"); From efe2e029fdf21d2ca9dd261e49fcd8f29cfcaf52 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 17:59:39 +0900 Subject: [PATCH 22/40] test(ci): isolate Windows status rename failure --- .github/workflows/cross-build-test.yml | 1 + src/build/schedule/compiler_transaction.cpp | 22 +++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 709cd4dc6..68a3fb7dc 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -529,6 +529,7 @@ jobs: shell: bash run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + export MCPP_TEST_NATIVE_RENAME=1 # CI-only diagnosis; remove before release "$MCPP_SELF" build --target x86_64-linux-musl # Scope the search to the TARGET's output tree — target/ also holds the # host build from the previous step, and both are named "mcpp*". diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index a5f88040c..f94a51984 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -48,6 +48,28 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia } noReplaceCode=static_cast(noReplace.error().code); } +#if defined(_WIN32) + // CI-only root-cause probe. Remove before merge: it intentionally + // compares the standard library's rename of this exact source to + // a distinct diagnostic name, without publishing the status. + if(noReplaceCode==static_cast(os::errc::invalid) + && std::getenv("MCPP_TEST_NATIVE_RENAME")!=nullptr) { + const auto sourceLocation=os::fs::locate(temporary); + const auto targetLocation=os::fs::locate(path); + const auto readable=os::fs::read(temporary); + auto probe=temporary;probe+=".diagnostic"; + std::error_code nativeError; + fs::rename(temporary,probe,nativeError); + if(!nativeError) {std::error_code ignored;fs::remove(probe,ignored);} + std::println(std::cerr, + "mcpp: CI rename probe source-rel {} target-rel {} same-root {} KAL-read {} std-rename-code {} category {}", + sourceLocation?sourceLocation->relative.size():0, + targetLocation?targetLocation->relative.size():0, + sourceLocation&&targetLocation&&sourceLocation->directory==targetLocation->directory, + readable?0:static_cast(readable.error().code), + nativeError.value(),nativeError.category().name()); + } +#endif *diagnostic=std::format("rename temporary (condition {}, native {}; source regular {} ec {}; target exists {} ec {}; no-replace condition {})", static_cast(renamed.error().code),renamed.error().native, fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value(),noReplaceCode); From ebff0ac85d4617fc8356647767539d85eafd19a6 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 18:21:00 +0900 Subject: [PATCH 23/40] test(ci): trace early Windows status publication --- src/build/schedule/detach_codegen.cppm | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index d3044adff..36aee42b4 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -313,7 +313,16 @@ std::optional read_rc(const std::filesystem::path& slot) { // Temp file + rename, so a reader never observes half a number. The same // guarantee, for the same reason, that makes watching the BMI path sound. bool write_rc(const std::filesystem::path& slot, int rc,std::string* diagnostic=nullptr) { - return transactions::replace_record(suffixed(slot,".rc"),std::to_string(rc)+"\n",diagnostic); + std::string localDiagnostic; + auto* detail=diagnostic?diagnostic:&localDiagnostic; + const bool written=transactions::replace_record(suffixed(slot,".rc"),std::to_string(rc)+"\n",detail); +#if defined(_WIN32) + // CI-only: early supervisor failures have no caller-owned diagnostic. + // Report their status-publication failure without printing private paths. + if(!written&&!diagnostic&&std::getenv("MCPP_TEST_NATIVE_RENAME")!=nullptr) + std::println(std::cerr,"mcpp: CI early status publication: {}",localDiagnostic); +#endif + return written; } // Stable kernel lock slots, shared by ordinary and detached compilers. The From d210c9cd40d1f97907ee3b909aeae563b6087bf7 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 18:41:29 +0900 Subject: [PATCH 24/40] test(ci): trace detached Windows status writes directly --- .github/workflows/cross-build-test.yml | 1 - src/build/schedule/compiler_transaction.cpp | 3 +-- src/build/schedule/detach_codegen.cppm | 2 +- 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 68a3fb7dc..709cd4dc6 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -529,7 +529,6 @@ jobs: shell: bash run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - export MCPP_TEST_NATIVE_RENAME=1 # CI-only diagnosis; remove before release "$MCPP_SELF" build --target x86_64-linux-musl # Scope the search to the TARGET's output tree — target/ also holds the # host build from the previous step, and both are named "mcpp*". diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index f94a51984..3868aec6c 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -52,8 +52,7 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia // CI-only root-cause probe. Remove before merge: it intentionally // compares the standard library's rename of this exact source to // a distinct diagnostic name, without publishing the status. - if(noReplaceCode==static_cast(os::errc::invalid) - && std::getenv("MCPP_TEST_NATIVE_RENAME")!=nullptr) { + if(noReplaceCode==static_cast(os::errc::invalid)) { const auto sourceLocation=os::fs::locate(temporary); const auto targetLocation=os::fs::locate(path); const auto readable=os::fs::read(temporary); diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 36aee42b4..c22d2a278 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -319,7 +319,7 @@ bool write_rc(const std::filesystem::path& slot, int rc,std::string* diagnostic= #if defined(_WIN32) // CI-only: early supervisor failures have no caller-owned diagnostic. // Report their status-publication failure without printing private paths. - if(!written&&!diagnostic&&std::getenv("MCPP_TEST_NATIVE_RENAME")!=nullptr) + if(!written&&!diagnostic) std::println(std::cerr,"mcpp: CI early status publication: {}",localDiagnostic); #endif return written; From 0cc3968ed9b2d4e1adbc528c15cabee01c9ceb54 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 19:19:06 +0900 Subject: [PATCH 25/40] test(ci): carry Windows rename probe through supervisor log --- src/build/schedule/compiler_transaction.cpp | 9 +++++---- src/build/schedule/detach_codegen.cppm | 3 ++- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index 3868aec6c..c9e12a75e 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -32,6 +32,7 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia std::error_code sourceError, targetError; const auto source=fs::symlink_status(temporary,sourceError); const auto target=fs::symlink_status(path,targetError); + std::string probeReport; // A final status is a new name. If the replace form itself was // refused as invalid while that name is absent, try the atomic // no-replace form on the exact same source and destination. Its @@ -60,8 +61,8 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia std::error_code nativeError; fs::rename(temporary,probe,nativeError); if(!nativeError) {std::error_code ignored;fs::remove(probe,ignored);} - std::println(std::cerr, - "mcpp: CI rename probe source-rel {} target-rel {} same-root {} KAL-read {} std-rename-code {} category {}", + probeReport=std::format( + "; CI probe source-rel {} target-rel {} same-root {} KAL-read {} std-rename-code {} category {}", sourceLocation?sourceLocation->relative.size():0, targetLocation?targetLocation->relative.size():0, sourceLocation&&targetLocation&&sourceLocation->directory==targetLocation->directory, @@ -69,9 +70,9 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia nativeError.value(),nativeError.category().name()); } #endif - *diagnostic=std::format("rename temporary (condition {}, native {}; source regular {} ec {}; target exists {} ec {}; no-replace condition {})", + *diagnostic=std::format("rename temporary (condition {}, native {}; source regular {} ec {}; target exists {} ec {}; no-replace condition {}{})", static_cast(renamed.error().code),renamed.error().native, - fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value(),noReplaceCode); + fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value(),noReplaceCode,probeReport); } } else if(!written) { fail("write/flush temporary"); diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index c22d2a278..060691e1f 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -312,6 +312,7 @@ std::optional read_rc(const std::filesystem::path& slot) { // Temp file + rename, so a reader never observes half a number. The same // guarantee, for the same reason, that makes watching the BMI path sound. +void record_supervisor_error(const std::filesystem::path& slot,std::string_view reason); bool write_rc(const std::filesystem::path& slot, int rc,std::string* diagnostic=nullptr) { std::string localDiagnostic; auto* detail=diagnostic?diagnostic:&localDiagnostic; @@ -320,7 +321,7 @@ bool write_rc(const std::filesystem::path& slot, int rc,std::string* diagnostic= // CI-only: early supervisor failures have no caller-owned diagnostic. // Report their status-publication failure without printing private paths. if(!written&&!diagnostic) - std::println(std::cerr,"mcpp: CI early status publication: {}",localDiagnostic); + record_supervisor_error(slot,std::format("CI early status publication: {}",localDiagnostic)); #endif return written; } From 8791c2ca5f568087b34012b61153e8d1c65a3185 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Fri, 6 Nov 2026 19:55:04 +0900 Subject: [PATCH 26/40] fix(build): publish compiler records in host filesystem namespace --- src/build/schedule/compiler_transaction.cpp | 56 ++++----------------- src/build/schedule/detach_codegen.cppm | 8 ++- tests/unit/test_compiler_transaction.cpp | 2 +- 3 files changed, 14 insertions(+), 52 deletions(-) diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index c9e12a75e..e19902fe4 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -23,56 +23,20 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia output.close(); const bool closed=bool(output); if(written&&closed) { - auto renamed=os::fs::rename(temporary,path,true); - if(renamed)return true; + // The temporary was created through std::filesystem's host namespace. + // Publish it through the same namespace, in this same directory. A + // The detached controller's OpenKal lookup rejected this exact file + // on Windows even though the host filesystem could rename it. + std::error_code renamed; + fs::rename(temporary,path,renamed); + if(!renamed)return true; if(diagnostic) { - // The OpenKal condition does not retain the Windows NTSTATUS. These - // observations distinguish a blocked replacement from a source or - // parent failure without exposing private build paths in CI logs. std::error_code sourceError, targetError; const auto source=fs::symlink_status(temporary,sourceError); const auto target=fs::symlink_status(path,targetError); - std::string probeReport; - // A final status is a new name. If the replace form itself was - // refused as invalid while that name is absent, try the atomic - // no-replace form on the exact same source and destination. Its - // refusal is preserved as evidence rather than hidden by a loop. - int noReplaceCode=0; - const bool targetAbsent=target.type()==fs::file_type::not_found - || targetError==std::errc::no_such_file_or_directory; - if(renamed.error().code==os::errc::invalid && fs::is_regular_file(source) - && targetAbsent) { - auto noReplace=os::fs::rename(temporary,path,false); - if(noReplace) { - *diagnostic="replace form refused absent target; no-replace published"; - return true; - } - noReplaceCode=static_cast(noReplace.error().code); - } -#if defined(_WIN32) - // CI-only root-cause probe. Remove before merge: it intentionally - // compares the standard library's rename of this exact source to - // a distinct diagnostic name, without publishing the status. - if(noReplaceCode==static_cast(os::errc::invalid)) { - const auto sourceLocation=os::fs::locate(temporary); - const auto targetLocation=os::fs::locate(path); - const auto readable=os::fs::read(temporary); - auto probe=temporary;probe+=".diagnostic"; - std::error_code nativeError; - fs::rename(temporary,probe,nativeError); - if(!nativeError) {std::error_code ignored;fs::remove(probe,ignored);} - probeReport=std::format( - "; CI probe source-rel {} target-rel {} same-root {} KAL-read {} std-rename-code {} category {}", - sourceLocation?sourceLocation->relative.size():0, - targetLocation?targetLocation->relative.size():0, - sourceLocation&&targetLocation&&sourceLocation->directory==targetLocation->directory, - readable?0:static_cast(readable.error().code), - nativeError.value(),nativeError.category().name()); - } -#endif - *diagnostic=std::format("rename temporary (condition {}, native {}; source regular {} ec {}; target exists {} ec {}; no-replace condition {}{})", - static_cast(renamed.error().code),renamed.error().native, - fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value(),noReplaceCode,probeReport); + *diagnostic=std::format("rename temporary (system {} {}; source regular {} ec {}; target exists {} ec {})", + renamed.value(),renamed.category().name(), + fs::is_regular_file(source),sourceError.value(),fs::exists(target),targetError.value()); } } else if(!written) { fail("write/flush temporary"); diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 060691e1f..e886c7b8e 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -317,12 +317,10 @@ bool write_rc(const std::filesystem::path& slot, int rc,std::string* diagnostic= std::string localDiagnostic; auto* detail=diagnostic?diagnostic:&localDiagnostic; const bool written=transactions::replace_record(suffixed(slot,".rc"),std::to_string(rc)+"\n",detail); -#if defined(_WIN32) - // CI-only: early supervisor failures have no caller-owned diagnostic. - // Report their status-publication failure without printing private paths. + // An early supervisor failure has no caller-owned diagnostic. Keep its + // status-publication error beside the compiler log for the waiter. if(!written&&!diagnostic) - record_supervisor_error(slot,std::format("CI early status publication: {}",localDiagnostic)); -#endif + record_supervisor_error(slot,std::format("cannot publish compiler status: {}",localDiagnostic)); return written; } diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp index 5ae129439..0006aea98 100644 --- a/tests/unit/test_compiler_transaction.cpp +++ b/tests/unit/test_compiler_transaction.cpp @@ -235,7 +235,7 @@ TEST(CompilerTransaction, FailedRenameReportsTheStageAndPreservesItsTarget) { fs::create_directory(t.root/"record"); std::string diagnostic; EXPECT_FALSE(tx::replace_record(t.root/"record","replacement",&diagnostic)); - EXPECT_TRUE(diagnostic.starts_with("rename temporary (condition "))< Date: Fri, 6 Nov 2026 20:56:16 +0900 Subject: [PATCH 27/40] docs: remove private diagnostic path from design record --- .../docs/2026-11-05-heterogeneous-build-implementation-plan.md | 2 +- src/build/schedule/compiler_transaction.cpp | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md index 16dcf4174..301121940 100644 --- a/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md +++ b/.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md @@ -307,7 +307,7 @@ platform 0.23.7 的一致版本组已在本地准备，待两个前置公开后 发布准备补齐 CHANGELOG 的 2026.11.5.1 条目；Release 按该版本标题提取说明， 缺失条目会拒绝发布。包含本次说明的最终提交仍须完成核心 CI，随后再独立进行 生态 review。新增证据已逐文件校验并验证归档往返： -`/home/speak/workspace/mcpp-885-realpath-and-final-core-diagnostic-evidence.tar.gz`， +`mcpp-885-realpath-and-final-core-diagnostic-evidence.tar.gz`， 152 个文件，SHA256 为 `4d7b783c27bb25dc75c354af72e98dc4b47037ab75043f792a28dceeeb10083a`。 diff --git a/src/build/schedule/compiler_transaction.cpp b/src/build/schedule/compiler_transaction.cpp index e19902fe4..edd634091 100644 --- a/src/build/schedule/compiler_transaction.cpp +++ b/src/build/schedule/compiler_transaction.cpp @@ -24,7 +24,7 @@ bool replace_record(const fs::path& path,std::string_view bytes,std::string* dia const bool closed=bool(output); if(written&&closed) { // The temporary was created through std::filesystem's host namespace. - // Publish it through the same namespace, in this same directory. A + // Publish it through the same namespace, in this same directory. // The detached controller's OpenKal lookup rejected this exact file // on Windows even though the host filesystem could rename it. std::error_code renamed; From 5aaf5873eaad769eb61dc7119ee30adaad0c7cfe Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 04:55:00 +0900 Subject: [PATCH 28/40] docs: record the part-4 implementation evidence of the openkal 0.23 round Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- ...art3-ok-image-performance-parity-design.md | 20 + ...03-ecosystem-part4-beyond-native-design.md | 1522 +++++++++++++++++ docs/20-toolchains.md | 3 +- docs/zh/20-toolchains.md | 2 - 4 files changed, 1543 insertions(+), 4 deletions(-) diff --git a/.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md b/.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md index 44c107dbe..f995c6c40 100644 --- a/.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md +++ b/.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md @@ -685,3 +685,23 @@ mcpp 在首次初始化时已把 Ninja 装进自己的 home（`ensure_ninja`，` （memchr/4096 1.46、memcpy/4096 1.34、strcmp/64 1.29），另有 clock、stat 与线程私有数据；同一份源码以 Clang 编成 hosted 时为 9 行，映像多出的部分大半来自编译器而不是映像。这些差距在 L2 的工作负载中不显现。 +## 更正：L3 缺页统计的边界（2026-11-04，Sunrisepeak） + +本块更正第 9.4 节及后续记录中的缺页口径。周期进程树采样会漏计两次采样之间 +退出并被回收的 worker。冻结构件的 macOS 对照中，原生与映像就绪采样分别为 +126581 与 294066，而不含 Python 探针的完整引擎内核总账分别为 506753 与 +692680。两条路径漏计程度不同；该对照不证明新引擎达标。 + +测量版本二将缺页门禁明确为完整引擎生命期的内核总账。每次引擎启动前读取 +RUSAGE_CHILDREN，完成 shutdown、exit 与 wait 后再取差；minor 与 major 相加。 +范围包含引擎自身、它等待回收的后代及关闭阶段，不含 Python 探针和重挂至其他 +父进程的后代。Linux 与 macOS 统一采用该范围，标识为 +complete-engine-and-waited-descendants-v1。原 page_faults 字段保留就绪时诊断， +新增 complete_page_faults 承担门禁，阈值仍为原生的两倍。Windows 缺少该 +POSIX 接口，完整总账显式为空，继续只作两条不同构建路径的对照记录。 + +就绪时间与就绪时处理器门禁仍为原生的 1.15 倍，三百零四单元、三个打开文件、 +一次冷会话与三次暖会话保持。验收拒绝旧版本、范围不符、原始计数缺失、负数或 +与原始计数不一致的合计及暖中位数。旧口径的结果继续保留，不能改名后充作 +新口径验收；新版本组的完整 CI 重新取得两边原始记录。 + diff --git a/.agents/docs/2026-11-03-ecosystem-part4-beyond-native-design.md b/.agents/docs/2026-11-03-ecosystem-part4-beyond-native-design.md index 43da9e8a6..eff99a47c 100644 --- a/.agents/docs/2026-11-03-ecosystem-part4-beyond-native-design.md +++ b/.agents/docs/2026-11-03-ecosystem-part4-beyond-native-design.md @@ -818,3 +818,1525 @@ launcher 固定 os 0.2.9 后，平台 CI 还须完整重跑。mcpp 的本地新 根包与 toolchain-model 共 164 个测试程序通过、0 失败。公开版本组上的 E2E 942、951 完整通过，覆盖两种架构的映像、std、拒绝路径、self-contained、single-file 与工作区。编译器完整 CI、发布、编辑器交付及最终资源实测尚未全部完成；这里不以生产者 或本地结果替代这些门禁。 + +## 21. 已发布编译器的验收记录（2026-11-03） + +本节记录 §20 之后完成的编译器交付及其验证范围。引擎、语言服务端和 IDE 的发布、 +最终资源实测仍按各自的门禁推进；本节不将它们计作完成。 + +### 21.1 发布产物与安装索引 + +[mcpp PR #884](https://github.com/mcpp-community/mcpp/pull/884) 的最终检查为 78 项通过、 +2 项跳过；合入提交为 `6028cced5c961e908a12d275b806c34db9d6ff48`。 +[2026.11.3.1 发布流程](https://github.com/mcpp-community/mcpp/actions/runs/41650681477) +的九项检查全部通过。独立下载核对了 21 个发布文件、版本别名的字节一致性、封存清单 +及 16 份资源镜像。两个 Linux 载荷中的 mcpp 与 xlings 均为静态链接且已剥离符号。 + +| 平台 | 压缩包字节数 | SHA-256 | +| --- | ---: | --- | +| Linux x86_64 | 16,521,040 | `898d8447532a0d181e584d69fc286d225210a4ecaf5b4af354cb62773a8ab7ca` | +| Linux aarch64 | 13,772,546 | `d7b4767529d90fa0ac5fe252db9027e8a07413bba0a4c6b0314393ce9d09bc8e` | +| macOS arm64 | 9,804,182 | `75c7ad910a080c9ee75b56b29408879e8e1031745dc67050036d1dadd70b61d2` | +| Windows x86_64 | 10,839,480 | `e38678187fb42e9572104d384163625f7953f104f1d972c54a915b3652b1e81e` | + +Linux 两个包比前一版本分别增加 11,665 字节、11,911 字节，约为 0.07%、0.09%。 +首次发布作业在镜像上传期间被取消；补齐缺失的镜像后只重跑发布作业。重跑前后 21 个 +GitHub 资产的标识、大小、摘要和更新时间全部相同，未重写已发布归档。 + +普通安装索引的 [PR #1105](https://github.com/openxlings/xim-pkgindex/pull/1105) 已合入为 +`7cd7002eba5fb12242310ed48640a0f0d399cc98`。三个系统条目的 latest 均指向 +2026.11.3.1，旧版本条目保持不变。索引主分支 CI 的 macOS 作业未取得运行机， +因此该轮 CI 不计作通过；维护者明确要求直接发布，不等待该排队作业。发布使用合入 +提交上的同一组脚本，未修改 CI 的常规验证策略。 + +GitHub 与 GitCode 的公开指针均指向 `xim-index-7cd7002.tar.gz`，大小为 1,756,327 字节， +SHA-256 为 `5f7912c3fc84dd75bcdb82b108cd2a88d440e6918733f8df1fb1b6c5df33bed7`。 +两个站点的归档、其中的 mcpp 配方和 GitHub 兼容指针均经匿名下载核对。 + +### 21.2 已发布二进制的实际执行 + +在中立工作目录、独立的 xlings/mcpp 与 XDG 状态目录中，用 xlings 的实际可执行文件 +刷新公开索引，再分别安装精确版本和 latest。两次均解析到 mcpp 2026.11.3.1。 +实际执行的 Linux x86_64 ELF 摘要为 +`e1fb3ece147c7511cc447c4b4e2499cc6f51a15c276cf771772f27e4539c1799`， +与发布包内的文件相同；未通过用户已有的 shim 执行。 + +一个没有平台覆盖、没有已有锁文件的新工程，以全新安装的 `llvm@23.1.3` 构建 +`import std` 的 openkal 程序，实际输出 `released default image: 42`。 +目标锁文件选择平台 0.23.3、musl 0.27.3、C++ 运行时 0.18.4、加载器及工具 0.4.3。 +该锁文件证明映像目标的依赖闭包；原生运行器中的 Linux 实现由工具发布与源码组的 +独立验证证明，不能从映像目标锁文件推断。 + +同一发布 ELF 的两项工具链边界检查实际执行如下： + +- `[toolchain] default = "system"` 以状态码 2 拒绝，报出仅支持受管理工具链。 +- 默认工具链显式指向已有 LLVM 23.1.3 目录，另写 `bootstrap = "llvm@23.1.3"`， + 能构建并运行默认 openkal `import std` 程序，输出 `released custom bootstrap image: 42`。 + +不显式指定 bootstrap 的外部路径配置，在本次涉及平台 `build.mcpp` 的构建中被主机 +工具链解析拒绝；上面的显式 bootstrap 路径已验证可用。 + +另一个全新隔离环境显式选择 CN，配置查询实际报告 `mirror CN`，索引更新实际取得 +`xim-index-7cd7002.tar.gz`。精确版本和 latest 两次安装均成功，实际 ELF 的版本与摘要 +和 GLOBAL 安装相同。此项证明 CN 配置下的真实安装；产品保留镜像后备，普通下载日志 +未记录最终资源主机，因此这里不声明每次网络请求均只经过 GitCode。 + +### 21.3 发布后的生产者验证 + +四个仓库的临时 `MCPP_SOURCE_REF` 变量已删除，删除前确认它们均为本轮候选分支。 +使用已发布 2026.11.3.1 的主分支检查中， +[platform 的四项检查](https://github.com/mcpplibs/openkal-platform/actions/runs/41675003365)、 +[os 的二十项检查](https://github.com/mcpplibs/os/actions/runs/41675019377) 和 +[运行时的十项检查](https://github.com/mcpplibs/openkal-llvm-runtime/actions/runs/41675052794) +均已通过；对应构建日志实际报告 mcpp 2026.11.3.1。 +os 的 hosted macOS 作业实际执行七个测试程序，包括只读拒绝和可写创建断言。 + +image 的主分支检查 41675067975 虽然五项全部通过，日志却显示安装了 xlings 安装包 +自带旧索引中的 mcpp 2026.10.24.1，因此不计作 2026.11.3.1 的验证。 +[PR #26](https://github.com/mcpplibs/openkal-image/pull/26) 在安装前刷新索引，精确固定 +2026.11.3.1，并在映像构建和四个原生运行器构建入口校验实际版本。修正后的五项检查 +全部通过，已合入为 `df212eaf3f56cfd84a4aeb19bb43bb9fcba34c3b`，其 +[主分支的五项检查](https://github.com/mcpplibs/openkal-image/actions/runs/41681148927) 也已通过。 +musl 的 macOS 映像作业因未取得运行机而单独重跑，随后 +[主分支的十项检查](https://github.com/mcpplibs/openkal-musl/actions/runs/41675036253) 全部通过。 +五个生产者主分支的 49 项检查至此全部通过；索引 CI 固定版本、自举固定版本和编译器 +最终主分支的完整检查分别验收。 + +### 21.4 索引的验证版本固定 + +[mcpp-index PR #699](https://github.com/mcpplibs/mcpp-index/pull/699) 的十七项最终检查 +全部通过或按工作流条件跳过，包括 Linux 与 Wine 中 Windows 的兼容性测量，已合入为 +`0b65b1f18269467e102058925e77930403d55448`。合入提交与审查头的树相同。 +三个工作流与测量 pin 统一使用 mcpp 2026.11.3.1，hosted 平台测试成员选择 0.23.3。 +客户端版本下界 2026.9.18.3、兼容性运行时基线 0.15.2 和工具链基线 LLVM 22.1.8 +保持不变。 + +## 22. 候选测量有效性与 Windows 分段映射修正（2026-11-03） + +### 22.1 布局作业的成功状态与测量结果分开验收 + +引擎候选 `a5a075b` 的两种架构布局作业均报告成功，但顺序与触页两次测量的 +会话 JSON 均为 `ok: false`，在二十分钟超时后仅建成一个模块。这些作业结论不作为 +有效测量，也不支持把 openkal 台账项 0002、0003 改为 steady。 + +原始映像不在安装包的 `bin/../lib/clang` 布局中。相同 x86_64 映像、相同工作区在 +系统 clang 18 的工具路径下，未指定资源目录时报告 `stddef.h` 缺失；显式指定本次 +构建的 LLVM 23 资源目录后，304 个翻译单元的冷会话实际建成 176 个模块并就绪。 +这项对照是资源目录问题的诊断，不是已发布产物的性能测量。 + +[引擎修正 `994f1be`](https://github.com/Sunrisepeak/mcppls-clangd/commit/994f1be9c56e685b2fd0d7230799df917d349d8a) +为两次原始映像测量显式传递资源目录，要求会话集合非空且每个会话的 `ok` 为真， +并在失败时保留顺序、触页和会话日志。 +[ARM 候选的诊断归档](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41689675773/artifacts/13256979913) +实际记录两次就绪，均建成 176 个模块。完整四宿主验收与最终发布测量仍分别执行。 + +### 22.2 Windows 启动异常的原生取证 + +旧候选的 Windows 映像验收在 LSP 初始化回复前发生访问异常。 +[原生复现](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41691553925) +覆盖原始 `.oke` 和 `.okb`，各自使用默认、关闭快照、关闭放置三种设置；六次均以 +`0xC0000005` 退出。该异常不依赖快照或放置，不能解释为缺失名字记录的语义错误。 + +进程早于调试器附加而结束。随后由 CDB 从创建时启动同一映像， +[六次均捕获首个访问异常](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41692290011)， +保存匹配的运行器、寄存器、指令和六份有效 minidump。指令在把 ELF 头写入已具有该 +内容的目标页时被拒绝。调试器报告的实际指令位置是运行器的 `+0x51834`；产品故障 +报告所示 `+0x50c34` 不能直接用作 PE 的 RVA。此处未声明具有匹配 PDB 的源码栈。 + +### 22.3 部分映射成功后的复制范围 + +加载器按触页信息把一个只读段分成多个映射请求。早先请求成功后，后续请求可能被 +环境拒绝，例如 Windows 不接受最后不足 64 KiB 的文件视图。旧实现随即复制整个段， +写入先前已经映为只读的页。 + +[openkal-image PR #27](https://github.com/mcpplibs/openkal-image/pull/27) 只复制被拒绝的 +那一段请求；其中包含指令时，单独发布该复制范围。发布被拒绝则释放本次区域并拒绝 +加载，后续不再进入整段复制。成功的文件映射保留。 + +同一源码提取的受保护映射对照中，旧路径因写入只读页收到 `SIGSEGV`；修正后字节 +相同，恰有两次映射请求和一次复制范围的发布。新增真实映像回归验证首个只读段的 +长度和未对齐尾段，再由四个原生宿主分别运行其 `.oke` 和通用 `.okb`。 +[五项生产者 CI](https://github.com/mcpplibs/openkal-image/actions/runs/41693546622) 已全部通过。 +原始 clangd 映像还须由修正后的 Windows 运行器独立复验；新不可变版本、消费者闭包、 +缺失名字与 warm-prime 的完整验收尚未由本节宣告完成。已发布标签与归档未被改写。 + +## 23. IDE 发布渠道与真实索引消费（2026-11-03） + +### 23.1 不可变发布文件与编辑器分发 + +[mcpp-ide 0.2.0](https://github.com/mcpp-community/mcpp-ide/releases/tag/v0.2.0) +来自合入提交 `421613a98e1868fc295262a67f322e49c6f90a99`。发布检查的 23 项构建与验收 +全部通过。五个 GitHub 资源与五个 GitCode 镜像分别下载后摘要相同；发布清单内的 +四个文件均与 `SHA256SUMS` 一致。通用后端归档为 5,083,938 字节，SHA-256 为 +`3d692b433f85588327b7aa06e158e97db61490498870af84ace731a633c1ebe0`。 + +原发布作业在创建 GitHub release 后因缺少 Open VSX 凭据失败。配置凭据后的 +[补发作业](https://github.com/mcpp-community/mcpp-ide/actions/runs/41696583381) +直接使用已有 VSIX，验证摘要后上传，没有重新构建或覆盖 release。 +[Open VSX 公开版本 1.1.0](https://open-vsx.org/extension/mcpp-community/mcpp-vscode) +已可下载；实取文件为 6,171,177 字节，SHA-256 为 +`7383a9f21565685a0a15cd2ea49dac5dbfa2ed479e5db412986b7c81bb880034`， +与 GitHub VSIX 相同。插件版本 1.1.0 与项目发布版本 0.2.0 分别核验。 + +[JetBrains 上传作业](https://github.com/mcpp-community/mcpp-ide/actions/runs/41698011098) +提交同一 release 的 `mcpp-clion-0.2.0.zip`，输入文件为 5,574,062 字节，SHA-256 为 +`6daccbaa5b52eb30c784da4ebc14fdaa333d6a769e00e4707b9ac74f8b6a3c8b`。 +平台返回 HTTP 201，更新 ID 为 1211284，回执的 `approve` 和 `listed` 均为 false。 +此项结论是已受理、等待审核；公开上架另行验收。Marketplace 回执中的资源大小 +为 5,579,978 字节，不能据此声明平台处理后的下载文件与上传 ZIP 字节相同。 + +### 23.2 索引启动器适配与双配置安装 + +新通用包的 Windows 入口是 `bin/mcpp-ide.exe`，旧包使用 `.cmd`。 +[xim-pkgindex PR #1106](https://github.com/openxlings/xim-pkgindex/pull/1106) +增加 0.2.0 的三个系统版本条目，并使安装配置优先选择实际存在的 `.exe`，保留旧包的 +`.cmd` 回退。旧版本条目保持原文。回归检查覆盖五个场景；CI 的 Linux、macOS 和 +Windows 安装作业全部通过，Windows 作业实际执行新入口。 + +PR 合入为 `114761a4332cb92eff20022c94a5f8d0540c66d7`。按本轮直接发布授权，使用 +同一提交的索引构建、发布和指针脚本生成并发布 `xim-index-114761a.tar.gz`。 +GitHub 和 GitCode 两个实际公开归档均为 1,756,998 字节，SHA-256 为 +`da5c97e8fb1115d1ceb6592fc1c3eb4f84aed919fc7d6e48aefb5b64d5538b43`。 +两份归档内的 IDE 描述符逐字等于合入版本，mcpp latest 三个条目仍指向 2026.11.3.1。 +GitHub 原始指针和兼容 release 指针已验证；GitCode 原始文件 HTTP 403，改由公开 Git +读取指针，读取方法与对应提交单独记录，不记作原始 URL 下载成功。 + +GLOBAL 与 CN 两个既有隔离环境分别刷新真实公开索引，精确安装 `mcpp-ide@0.2.0`， +再安装 latest。两边实际安装的 11 个文件逐个等于已验证公开通用归档内的对应文件， +运行安装后的入口均输出 `mcpp-ide 0.2.0 (protocol 2)`。CN 测试允许产品原有镜像 +后备，因此证明 CN 配置下的安装，不声明每个下载请求只经过 GitCode。 + +## 24. 已发布 Windows 运行器的独立复验（2026-11-03） + +openkal-image 0.4.4 的不可变标签指向 +`fda22ddd9af3c03ebd3996a385bdb0382bbf62c7`；发布版本的五项源码 CI 全部通过。 +源码归档在 GitHub 与 GitCode 的摘要相同，mcpp-index PR #700 已合入为 `7e2b542`， +其发布作业成功，两个公开索引归档均验证了四个组件各三个系统条目。 +读取时 GitHub 原始文件 CDN 仍返回旧指针，公开仓库 contents API 已返回新指针； +记录保留这一区别，没有把旧 CDN 响应记作新版本可见。 + +[Windows 原生复验](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41699690007) +直接下载公开的 `okrun-0.4.4-windows-x86_64.zip`，其 SHA-256 为 +`1adc1bfcef6a90f9edabf4b265ae547ff8663a522758a949c175d4079aa2c5ca`。 +归档中实际 `okrun.exe` 的 SHA-256 为 +`22455616bf0907dbd6a17169e4623c941c6e7bcf283f2b945d3dcb17f802076e`， +与取证归档保留的运行器逐字相同。 + +原始 `.oke` 和 `.okb` 各在默认、关闭快照、关闭放置三种设置下运行。六组版本查询 +均报告 `clangd version 23.1.0-mcppls.9`，六组语义会话均完成初始化、查询得到名为 +`probe` 的文档符号和正常关闭；没有首个访问异常或调试器强制结束。工作流和独立读取 +原始回复的检查均通过。取证 ZIP 为 4,428,615 字节，SHA-256 为 +`22320181d885f2eeda26f799a5d8f35cac24308782795cba2ec9aad817243cf0`。 + +这里仍使用同一旧引擎映像隔离验证已发布加载器的修正。新的 musl、C++ 运行时与平台 +依赖闭包、引擎的四宿主缺失名字及 warm-prime 验收、最终发布性能测量分别验收， +不由这次启动与语义复验代替。 + +## 25. JetBrains 公开版本的验收（2026-11-03） + +第 23.1 节提交时尚在审核的 mcpp-ide 0.2.0 已通过审核。 +[更新 1211284](https://plugins.jetbrains.com/plugin/35632-mcpp/versions/stable/1211284) +的公开 API 实际报告 `approve: true`、`listed: true`、`hidden: false`。 +从 Marketplace 公开地址下载的 ZIP 为 5,579,978 字节，SHA-256 为 +`7e3fec81186d897e4185413ffbf3b69b08f9185f0da78842f4826f2408d274ba`； +包内插件 XML 的 ID 为 `io.github.mcpp-community.mcpp-clion`，版本为 0.2.0。 +该公开归档与原上传文件分别记录摘要，没有把平台处理后的文件记作原文件字节相同， +也没有声明独立完成 Marketplace 签名验证。 + +## 26. 0.4.4 工具分发与隔离安装（2026-11-03） + +[0.4.4 二进制发布](https://github.com/mcpplibs/openkal-image/actions/runs/41698431841) +的四个宿主全部通过。macOS 首次请求在十五分钟内未分配到运行机，没有执行步骤； +只重跑该未成功作业后通过，其他三宿主的成功结果保留。13 个 GitHub 归档已下载并 +校验摘要，12 个程序包内的清单身份与所列文件摘要均通过，15 份 GitCode 镜像分别 +下载后与原归档一致，包括三个工具仓库各自的调试脚本副本。 + +[xim-pkgindex PR #1107](https://github.com/openxlings/xim-pkgindex/pull/1107) +的完整 CI 和 31 项检查全部通过或按条件跳过，包括三个系统的实际安装检查，已合入为 +`43781b8389f09662df4d2f81f1f70351798d2460`。三个描述符各增加新版本和四个归档摘要， +更新各自三个系统的 latest 引用；撤销这些新增与 latest 更新后，描述符逐字等于旧版。 +发布使用同一合入树的原有构建、上传和指针脚本。两个公开索引归档均为 1,757,443 +字节，SHA-256 为 `6e27ff754e6fc2b99db95624feb30519a98d896038e20762632c3468d6278084`。 +三份工具描述符逐字等于合入版本，mcpp 2026.11.3.1 和 IDE 0.2.0 的 latest 引用保留。 + +GLOBAL 与 CN 的既有隔离环境均成功安装三种工具。每个环境中九个安装文件逐个等于 +已验证公开 ZIP 内的对应文件；实际 ELF 分别通过 `okld check`、`okrun --prof` 与 +`okprof report`。这些是工具安装与功能检查，不作为引擎最终验收或性能数字。 + +GLOBAL 环境此前显式选择 0.4.3。不带版本的安装保留该选择；`@latest` 实际解析为 +0.4.4，执行 `xlings use @0.4.4` 后，不带版本的安装仍选择 0.4.4。 +CN 环境此前未安装这三种工具；本轮精确安装后，不带版本的安装均选择 0.4.4。 +这里没有声明裸安装会自动替换已有的显式选择。Linux 0.4.4 的 okprof ELF 与先前实测的 0.4.3 字节相同； +独立草稿 PR #25 的 `record` 修正未包含在本次发布，`record` 不计作通过。 + +## 27. 最终主分支验证（2026-11-03） + +[mcpp 最终主分支运行 41686943730](https://github.com/mcpp-community/mcpp/actions/runs/41686943730) +在提交 `e0aee5cb7dfa5b13d1a5248d447947534d3cfa8b` 上成功结束，64 项成功，2 项按条件 +跳过。跳过项为 `openkal` 汇总入口和 `macos-ios / iOS - what this runner actually +provides`。本轮只重跑了未成功的基础设施作业；重跑后的 ARM musl 交叉构建、Android +模拟器和两个架构的 iOS 模拟器作业均通过。 + +该提交的直接父提交为 `a86d09bc8bff016f14f05c6b9ff8003256d6e094`，包含用户 PR #878 +对 glob 的 `**/` 语义修正，祖先关系已验证。自举 PR #886 仅更新 `.xlings.json`， +没有撤回该用户改动。结合第 21.3 节五个生产者的 49 项验证和索引固定版本验证， +T6 已完成；新加载器依赖组与引擎、服务端发布继续分别验收。 + +## 28. musl 0.27.4 的发布（2026-11-03） + +[openkal-musl PR #80](https://github.com/mcpplibs/openkal-musl/pull/80) 的 +[十项生产者检查](https://github.com/mcpplibs/openkal-musl/actions/runs/41700027602) +全部通过。原生 macOS 和交叉构建程序的 macOS 运行请求先后因十五分钟内没有取得 +运行机而取消，没有执行测试步骤；每次均待运行结束后只重跑未成功项。最终包括 +两个 Linux 原生工具链、原生 macOS、交叉链接及其 macOS 执行、一次构建的映像与 +四个宿主对这些相同映像的执行，均有实际成功结果。 + +合入提交为 `657fa1609b9aba10527501e47e98c0bd5ddc347e`，树为 +`9e2f7afe49178f555e5ee372b8615e569b15acba`，与已审查并通过 CI 的 `fc64463` 相同。 +不可变标签 0.27.4 指向该合入提交，GitHub release 已创建；家族分支通过旧引用的 +精确租约由 `91aa89e` 更新到合入提交。同树的主分支重复构建 41706351111 已取消， +本节的十项成功结果来自前述 PR 运行。 + +[GitHub 源码归档](https://github.com/mcpplibs/openkal-musl/archive/refs/tags/0.27.4.tar.gz) +与 GitCode 镜像均为 1,266,813 字节，SHA-256 为 +`6620ec32d8a3058153d392094edae6345c0c47f60e9ca0f871b9a01f6217eb89`。 +归档根 manifest 逐字等于合入源码：包版本 0.27.4，映像的加载器依赖为 0.4.4。 +C 库实现和各原生内核实现的固定版本保持不变。 + +[mcpp-index PR #701](https://github.com/mcpplibs/mcpp-index/pull/701) +新增三个宿主条目并更新 Linux 测试成员，旧版本条目保留。其完整索引检查与公开索引 +指针验收另行完成，后续运行时 0.18.5 和平台 0.23.4 在此前不启动消费者闭包验收。 + +## 29. musl 公开索引与运行时接续（2026-11-03） + +第 28 节的索引 PR #701 已完成 22 项检查，全部通过或按条件跳过，包括两个 Linux +候选工具链、镜像摘要检查和 macOS 工作区，合入为 +`a9615dbda0aabe62330c93253d0dae7f8ecb9e5e`。 +[索引发布运行 41708174098](https://github.com/mcpplibs/mcpp-index/actions/runs/41708174098) +成功。普通 GitHub 原始指针已报告该提交；两个公开索引归档均为 664,441 字节， +SHA-256 为 `feff22f2ad123d7d2b48041ec54357d314b8ec60f6b34e08f5fb2fbe2d36237e`。 +归档内的 musl 0.27.4 三个宿主条目及其源码摘要均已检查。 + +在这些公开字节可用后，才推送运行时 0.18.5 的 +[PR #62](https://github.com/mcpplibs/openkal-llvm-runtime/pull/62)，完整生产者 CI +41708741056 已启动。另用已发布 mcpp 2026.11.3.1 的实际 ELF 和 LLVM 23.1.3，运行 +该候选源码原有的 `cxx` 与 `import-std` 两个 Linux musl 示例。前者报告零个失败观察， +后者输出 `import std above openkal: 2 4 7`。 + +两个示例的实际锁文件均只含四项索引依赖：musl 0.27.4、Linux 0.24.1、openkal +0.23.0、mimalloc 3.6.1，目标均为 `x86_64-linux-musl`；运行时自身是待测本地源码。 +这些依赖没有被改成工作树路径。此项使用已有工具链状态，验证公开依赖的实际消费； +运行时四宿主发布验收、完整平台闭包和最终性能测量分别完成。 + +## 30. C++ 运行时 0.18.5 的发布（2026-11-03） + +[运行时 PR #62](https://github.com/mcpplibs/openkal-llvm-runtime/pull/62) 的 +[完整生产者运行 41708741056](https://github.com/mcpplibs/openkal-llvm-runtime/actions/runs/41708741056) +十项检查全部通过，包括原生与交叉目标行为、原生 macOS 的开发和发布程序、交叉 +构建程序的实际执行，以及同一组 C++ 映像在四个宿主上的运行。 + +合入提交为 `e7a7b8ea019dbbcb68bc96480abaae6835f9694d`，树为 +`22db06238d26db2490969ce9491e76f61dc5b591`，与审查头 `3e65aa7` 的树相同。 +不可变标签 0.18.5 指向该合入提交，GitHub release 已创建，家族分支通过旧引用的 +精确租约更新。同树的主分支重复构建 41712011165 已取消，本节的成功验证来自 +前述完整 PR 运行。 + +[GitHub 源码归档](https://github.com/mcpplibs/openkal-llvm-runtime/archive/refs/tags/0.18.5.tar.gz) +和 GitCode 镜像均为 15,643,016 字节，SHA-256 为 +`45414755cc65755ca05fb5f21f8862151cad484eb6f6b6ad52ae7a3fcf6e6eb0`。 +归档根 manifest 逐字等于合入源码；两种分配器安排均选择 musl 0.27.4，其映像加载器 +为 0.4.4。C++ 运行时实现保持不变。 + +[mcpp-index PR #702](https://github.com/mcpplibs/mcpp-index/pull/702) 增加三个宿主 +条目并更新 openkal 测试成员。撤销新增版本块后，描述符逐字等于先前版本。 +使用已发布 mcpp 2026.11.3.1 和该候选索引，在 `x86_64-linux-musl` 上构建并运行 +原有 `kal` 测试，通过一项测试、零项失败。实际锁文件记录运行时 0.18.5、musl +0.27.4 和 Linux 0.24.1，均为索引来源。此项使用既有工具链缓存，不作为隔离安装或 +最终性能数字；公开索引验收和平台 0.23.4 的完整闭包验证分别完成。 + +## 31. 运行时索引与普通客户端的传播边界（2026-11-03） + +第 30 节的索引 PR #702 完成四个工作流和二十项检查：十五项成功，五项按条件 +跳过。合入提交为 `ed1348cc6cd9959e026138ca28d3171e3b984446`。 +[索引发布 41713458419](https://github.com/mcpplibs/mcpp-index/actions/runs/41713458419) +成功；GitHub 和 GitCode 两个索引归档均为 664,489 字节，SHA-256 为 +`eaf45f962593a12a2e3783e3c761ad54c1a1cc97fb4f7ba82c4414d64949de83`。 +归档内三个运行时 0.18.5 条目的源码摘要均已检查。 + +首次验收通过仓库内容 API 的退路确认了新指针与两个归档，但普通原始文件地址仍 +由 CDN 返回旧提交 `a9615db`。据此启动平台 CI 过早：两个 Linux 作业和一次本地 +检查均因旧索引没有运行时 0.18.5 而失败，尚未进入平台行为验证。初次失败日志保留。 +本轮检查脚本已改为默认要求普通原始指针跟进；API 退路须显式选择，只用于观测 +仓库发布，不能作为普通客户端已可安装的证明。 + +20:39 UTC，普通原始指针已返回 `ed1348c`；严格检查再次下载两个归档并验证摘要和 +三个版本条目。随后本地实际刷新索引开始下载运行时 0.18.5。平台 PR #17 的同一 +源码继续验证；其初次失败项在传播完成后分别重试，成功结果另行记录。 + +## 32. 平台 0.23.4 的发布与索引候选（2026-11-03） + +[平台 PR #17](https://github.com/mcpplibs/openkal-platform/pull/17) 的 +[完整四宿主运行 41713909104](https://github.com/mcpplibs/openkal-platform/actions/runs/41713909104) +全部通过。macOS 和 Windows 的初次运行成功；第 31 节中因旧索引而失败的两个 +Linux 作业，在普通指针传播完成后单独重跑，通过平台测试、打包、快照、hosted +程序和以映像运行的构建程序。没有更改源码或放宽检查。 + +合入提交为 `575eb44261f70de6db7e08af3da49f3eccdd5424`，树为 +`5dcbc2351eb299e39129bf495d5d2c8506014664`，与审查头 `395ddae` 相同。 +不可变标签 0.23.4 指向该合入提交，GitHub release 已创建，家族分支通过旧引用的 +精确租约由 `4de4285` 更新。同树的主分支重复构建 41717608555 已取消，本节的 +四宿主成功结果来自前述完整 PR 运行。 + +[GitHub 源码归档](https://github.com/mcpplibs/openkal-platform/archive/refs/tags/0.23.4.tar.gz) +与 GitCode 镜像均为 18,573 字节，SHA-256 为 +`dd1ea3ae41fd0607ccb02f1dbf2fe7aced073107b8442a81f33bfe74c96454d7`。 +归档根 manifest 逐字等于合入源码。映像和 hosted 两种安排均选择 musl 0.27.4、 +运行时 0.18.5 和映像加载器及工具 0.4.4；内核实现固定版本与编译、链接约定保持不变。 +已有 mcpp 2026.11.3.1 和 IDE 0.2.0 发布保留其固定版本；新消费者显式选择平台 0.23.4。 + +[mcpp-index PR #703](https://github.com/mcpplibs/mcpp-index/pull/703) 增加三个宿主 +条目并更新 hosted 测试成员，兼容性检查的运行时与工具链基线保持不变。撤销新增 +版本块后，描述符逐字等于先前版本。使用已发布编译器和候选索引运行原有 hosted +测试，通过一项、零项失败；实际锁文件记录平台 0.23.4、运行时 0.18.5 和 musl +0.27.4，均为索引来源。此项使用已有工具链缓存，不作为隔离安装或最终性能数字。 +索引完整检查、公开指针和归档验收完成后，才启动引擎在新平台组上的完整验证。 + +## 33. 公开平台闭包与消费者接续（2026-11-03） + +索引 PR #703 最终仅改三个文件：版本描述符、hosted 测试成员固定版本和 CHANGELOG。 +初次补充的两行基线注释触发旧运行时 0.15.2 的全部 32 个兼容性成员；该注释随后 +撤回，固定版本文件逐字恢复为基线，选择器和工作流保持不变。按原有规则，新增版本 +不影响该旧的精确固定图，新版本候选及 hosted 检查继续执行。 + +合入提交为 `586a7e5293e695096eea9b79c3d03447f0b2e53f`，树为 +`3beaeccf8d43ad38c34fe6f464f77d9d339cfa52`，等于审查头 `4c93b5d` 的树。 +合入时两项重复 Mac 检查仍在排队，依照用户不等待该队列的发布要求，复核了前一头 +`b243b18` 两项实际 Mac 成功日志，均运行平台 0.23.4 的 hosted 测试。两个头仅相差 +上述两行注释，TOML 解析值相同，其他文件逐字相同；没有将排队作业记作已经执行。 +随后 `4c93b5d` 的四个工作流和二十项检查全部结束，十六项成功、四项按条件跳过。 +同树的重复主分支验证 41720309336 已取消，没有将其记作绿色主分支验证。 + +[索引发布 41720309193](https://github.com/mcpplibs/mcpp-index/actions/runs/41720309193) +成功。两个公开归档均为 664,515 字节，SHA-256 为 +`32b89034957232a7e301bfc3b5133185f0fb45782dab052f84228b382b5163f5`；归档内 +三个平台 0.23.4 条目与源码摘要已检查。21:23:39 UTC，普通原始指针实际跟进该提交， +随后严格检查再次下载并验证两个公开归档。 + +在原有 GLOBAL 隔离环境中新建显式选择平台 0.23.4 的项目，使用已发布 mcpp +2026.11.3.1 的实际 ELF 和该环境的托管 LLVM 23.1.3，刷新普通公开索引后构建、运行 +`import std` 程序，输出 `released explicit 0.23.4 image: 42`。实际锁文件的 +平台 0.23.4、运行时 0.18.5、musl 0.27.4、工具 0.4.4 均为索引来源，没有工作树 +依赖替换。此项使用既有隔离工具链状态，验证新平台公开消费；编译器内置默认仍为 +0.23.3，不作为最终引擎或服务端性能测量。 + +21:24:27 UTC 才推送引擎修订 `ca208ce`，启动 +[完整引擎运行 41721522543](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41721522543)。 +台账检查 41721522428 因 0002 仍为 stabilizing 而拒绝发布，符合当前阶段；在实际 +四宿主缺失名字验收及 warm-prime 完成前，不提升 0002、0003 的状态。 + +服务端以相同公开索引配置所有六个既有目标，二十二项依赖均为索引来源。 +原生构建通过，授权目录、模板与进程三组分别通过 8/65、7/110、28/131 个用例/断言， +合计四十三个用例、三百零六条断言，零失败。测试后的全部包字段、目标集合及锁文件 +逐字等于此前六目标配置，SHA-256 为 +`b0de59c62473e2967271657e644a1bd0639d19b588c51a7c1191aa3e44e66b31`。 +固定版本与锁文件整理为本地提交 `66078ca1a1514777567a30bb47fd286ba1de8d5d`， +尚未推送；新引擎候选的完整服务端 CI 仍先于引擎正式发布。 + + +## 34. Windows 工作进程路径与连续租约的修正（2026-11-03，Sunrisepeak） + +这是 §33 后续验收的实现记录，不将候选测量作为正式发布结果。 +[引擎运行 41721522543](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41721522543) +以十三项成功、一项失败、一项跳过结束。四个原生构建、两个兼容性底线包、两种架构 +的映像及 bundle 身份检查均成功。Linux x64、Linux arm64、macOS 的映像验收成功； +Windows 的模块用例返回 3，缺失名字 canary 在首次构建模块时失败。warm-prime +因依赖失败而跳过，不能据此提升补丁状态。 + +原始日志给出两项具体原因：缓存能力探针报告共享锁没有共享，工作进程报告 +`bounded worker outputs require its private unit`。Windows 会在同一文件句柄的独占锁 +旁增加共享锁，而非将前者替换；两个锁存在时，第一次相同范围的解锁释放独占锁。 +此行为见 [Microsoft LockFileEx 文档](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-lockfileex)。 +原有探针错误地将再次申请共享锁视为转换，实际独占锁仍阻止第二个句柄取得共享锁。 +工作进程的输出边界、私有缓存路径和隐式缓存清单还包含硬编码的 `/` 判断， +与 Windows 运行时使用的路径风格不一致。 + +修正提交 `02714d0ee7c7a25dc4102cb49db025e3206636d3` 增加主系列 0043 和 openkal +系列 0004。0043 按路径组成部分检查私有输出，保留外部目录、相邻前缀目录和 `..` +逃逸路径的拒绝，并以同一风格构造私有缓存路径、识别隐式缓存目录。 +0004 在 Windows 上先取得同一打开文件的共享锁，再释放独占锁，发布期间始终持有 +租约；能力探针检查转换后的独占排斥与共享准入。Windows 的共享锁也禁止持有者 +写入，探针因此在测试写入前释放两个共享锁。真实缓存的租约文件和载荷文件仍分开， +没有通过释放后重取租约来放宽发布协议，也没有修改底层已发布版本。 + +路径回归用例同时覆盖 POSIX 与 Windows 风格。抽取实际新增函数与用例体，链接 +本机 LLVM 18 的路径实现后通过；将缓存路径判断换回旧的拼接写法，同一个 Windows +正向用例触发断言。此项不替代 LLVM 23 的完整构建。十项发布门槛和包身份负向 +控制通过；八个相关文件从固定上游文件按全部四十三个主补丁、四个 openkal 补丁 +重建后逐字一致。 + +[Windows 原生锁探针 41730623432](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41730623432) +成功，十二项检查、零失败。实际检查旧的替换假设被拒绝、第一次解锁后共享租约仍 +排斥另一个独占持有者、两个共享持有者共存，以及共享锁持有期间写入被拒绝、释放后 +写入成功。探针使用系统文件锁接口，验证环境语义；完整引擎的实际 openkal 路径仍须 +由后续四宿主验收与 warm-prime 验证。 + +[完整修正运行 41730464895](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41730464895) +已启动。独立的补丁检查 41730464752 已将主系列及 openkal 系列完整应用于固定 +LLVM 源码，随后因 0043 为 stabilizing 而在发布台账检查处停止，符合当前阶段。 +0043、0002、0003、0004 保持 stabilizing。此处没有将该运行记作完整绿色验证。 + +修正前的 `ca208ce` 候选还完成了一次匹配版本的 balanced 预检：原生 `.9`、映像 +`.9` 与已发布运行器 0.4.4，原有 L2 探针每项五次、含标准库用例，验收结果为通过。 +这是修正前候选的预检，不是修正后提交或正式发布的最终数字。执行的双架构 bundle +为 150,706,147 字节，内存额度使用经其 manifest 摘要验证的 x86_64 原始映像 +73,307,928 字节，未使用两个架构合计大小。最终实测脚本已要求显式提供该原始映像， +记录执行 bundle 与选中映像各自的摘要和大小；测量场景及验收阈值未改动。 + + +## 35. Windows 账本排斥与定点读写的修正（2026-11-03，Sunrisepeak） + +§34 的完整运行 41730464895 最终以十三项成功、一项失败、一项跳过结束。四个 +原生构建、两个兼容性底线包、两个映像构建及 bundle 检查均成功，三个其他宿主的 +映像验收成功；Windows 映像的模块用例五次返回 3，缺失名字 canary 的首次模块 +构建报告 `Resource temporarily unavailable`。warm-prime 因依赖失败而跳过。 +此结果保留为失败，不能提升台账状态。 + +Windows 后端 0.19.0 的定点读写使用第二个文件句柄，以保留原句柄的流位置。 +账本原有整文件独占锁是强制排斥，会拒绝第二句柄的数据访问。 +[原生 Windows 验证 41738774174](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41738774174) +通过十六项检查，复现整文件锁下读写被拒绝，并验证数据范围之外的单字节锁仍排斥 +另一个账本写入者、允许定点读写、不增加文件长度，且新旧两种锁范围相互冲突。 +这验证系统接口的行为，不替代完整映像的实际运行。 + +提交 `82e33ba097eff3d14fd513778a9ed42c52725d88` 增加 openkal 补丁 0005。 +Windows 账本锁使用偏移 2,147,483,647 的单字节范围，编译期断言全部有限账本数据 +均在其前；POSIX 仍使用既有 flock 协议，载荷租约仍使用既有范围。能力探针在持锁 +期间实际执行定点写入及读回。三个修改文件从固定基线经四十三个主补丁、五个 +openkal 补丁重建后逐字一致。实际平台适配文件在本机 LLVM 18 下编译和运行通过， +仅将 scope-exit 工具的名字映射为该旧版本的拼写；十项发布门禁与包身份检查通过。 +底层已发布后端及平台版本没有变动。 + +[完整验证 41739179747](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41739179747) +已启动，LLVM 23 的四宿主映像验收和 warm-prime 尚未完成。0043 及 0002 至 0005 +保持 stabilizing，服务端提交 66078ca 仍在本地等待合格候选。补充澄清：原有 +PrerequisiteModulesTest.cpp 整体受非 Windows 条件保护，新增路径用例在 Linux 与 +macOS 实际通过，不能表述为该用例在原生 Windows 执行通过。 + +## 36. 嵌套构建的进度粒度与缓存诊断（2026-11-03，Sunrisepeak） + +用户补充的日志在总用时 6:04 至 30:06 之间保持 `1630/1636`，当前动作为 +`clangd: llvm:clangd-libraries`，动作耗时从 1:11 增至 25:13。mcpp 2026.11.3.1 +的该计数表示外层 Ninja 已完成和总动作数。这个 prepare 动作调用 okgen，再由 +CMake/Ninja 构建 LLVM 与 clangd 静态库；内部编译步骤不增加外层计数，后续链接 +等待该动作整体结束。末尾时间是动作已用时间，不是预计剩余时间。非终端输出的 +心跳间隔为六十秒；Ninja 在命令完成前缓冲其输出。该映像构建随后实际成功。 +因此可确认进度可见性的粒度限制，不能由此日志判定调度死锁。 + +同一完整运行的 x86 映像日志记录 ccache 4.9.1-1：2,592 次调用中 83 次可缓存， +2,509 次不可缓存。隔离的同版本 ccache 对最小程序及未修改的公开平台编译契约均 +取得首次未命中、第二次直接命中；样本不含 LLVM 的完整头文件负载，尚未复现其 +不可缓存原因。82e33ba 的必要重建同时保存详细统计原因，未为诊断单独重建 LLVM。 +本轮未修改 mcpp 的进度实现。后续显示改进可分别呈现外层动作和嵌套构建活动； +动作数量不能直接换算为剩余墙钟时间。 + + +## 37. 预编译头缓存统计的补充（2026-11-03，Sunrisepeak） + +§36 的原因待查状态由本轮 ARM 映像作业的详细统计补充。 +运行 41739179747、提交 82e33ba 的 ARM 布局 artifact 13281721622 摘要为 +`ccc0aafaa9abd0437a86ea5f344b557112374865947ad202ac10bcbeb2639540`。 +该作业最终成功。计数在映像构建前清零，2,592 次调用中 82 次可缓存、均未命中， +2,510 次不可缓存；其中 2,482 次为 `Could not use precompiled header`。 +另有十四次编译失败及十四次预处理失败计数，未逐一归因，不能将其解释为整个作业 +失败。这组统计来自 ARM，不是当时仍在运行的 x86 映像作业。 + +[ccache 4.9.1 文档](https://ccache.dev/manual/4.9.1.html#_precompiled_headers) +说明该计数表示 PCH 缓存前提未满足。固定 LLVM 源码的 CMake 配置默认在相应 Clang +版本下启用 PCH，并为 `LLVM_CCACHE_BUILD` 设置配套的缓存参数；映像工作流直接 +设置编译器 launcher，没有采用该参数配置。此项是构建缓存配置问题，与外层动作 +计数的可见性限制分别记录，不是调度死锁的证据。 + +本地以相同 ccache 4.9.1、Clang 23.1.3 和公开平台 0.23.4 的 Release 编译契约 +复核。最小程序及含 vector、cstdint 的头文件样本均可缓存，第二次为直接命中。 +随后生成真实 PCH：相同源码两次编译均成功，但缓存两次报告同一 PCH 拒绝计数； +改为文本包含后，两次均可缓存，第二次直接命中。日志明确给出缺少 PCH 配套参数。 +实验未放宽缓存策略，也未修改编译选项或正在运行的 CI；不据小样本推断关闭 PCH +后的完整 LLVM 首次构建或重复构建用时。 + +## 38. 统一载荷安装适配与临时缓存整理（2026-11-03，Sunrisepeak） + +XIM 配方的本地提交 bcc26a6 按载荷版本 5 的平台条目选择运行器和引擎路径， +保留旧布局分支，未添加尚未发布的资源或修改 latest。十项静态及隔离检查通过； +使用实际 xlings 2026.10.26.1 ELF 在既有隔离 GLOBAL 环境完成注册、搜索和移除， +覆盖条目已清除。这些检查只证明注册与静态条件，版本 0.0.20 的公开载荷、安装、 +执行、卸载及四宿主 CI 仍分别验收。 + +整理缓存前核对自有目录、活动进程及永久归档，保留旧诊断日志和原生 Windows +故障转储。随后仅移除六处过期目录：临时 Wine 前缀、旧运行时示例的 target， +以及旧索引验证工作树的工具与构建缓存。观察区间内可用空间增加约 1.759 GiB， +达到约 4.284 GiB。最新运行时及平台索引验证的 target、源文件和全局缓存均保留。 +该值是区间内观察值，不声明对整个文件系统活动具有排他控制。 + + +## 39. 联合候选验证的并行推进（2026-11-04，Sunrisepeak） + +重新核对服务端发布文档 §“新引擎”第一、二步后，调整先前将两边完整验证串行执行 +的推进安排：服务端候选 CI 依赖已打包且身份有效的引擎部件，可以与引擎尚未结束的 +映像验收及 warm-prime 并行。两边完整通过后才合入、发布引擎；补丁台账提升、正式 +发布归档的两次下载及摘要核对、服务端删除候选文件后的最终 CI 均保持原有要求。 +这是验证调度的调整，不将部分成功视为完整引擎验收。 + +引擎 82e33ba 的四个原生构建、两个兼容性底线包、两种架构的映像及布局探针， +以及 bundle 身份检查均通过。五个部件均已上传，并核对运行号、头提交、大小、 +摘要及未过期状态。实际 Linux 底线包的二进制摘要及主补丁系列摘要也已复算。 +其 engine.json 的 fork 为 GitHub 合成合并提交 +`df31cf255657f958c5128402980dff9718f1999b`；API 记录的父提交含 82e33ba，树 +逐字等于该 PR 头的树。此项不将合成提交误写为 PR 头。 + +随后推送服务端提交 `191dc4d1d82c399771483ff625cb2692251e5eed`，候选文件改为 +运行 41739179747 和引擎头 82e33ba。 +[服务端联合 CI 41744249054](https://github.com/Sunrisepeak/mcpp-language-server/actions/runs/41744249054) +已触发，与四宿主引擎映像验收并行。公开平台 0.23.4 的六目标锁文件保持原摘要， +载荷锁与 CLANGD_VERSION 仍指向已发布的 .7，待 .9 正式发布后在同一提交中更换 +并删除候选文件。PR 描述明确记录完整引擎验收仍在进行，没有宣称候选已完整合格。 + +PCH 缓存另按 LLVM 原生构建已经采用的参数完成受控验证：原始 PCH 为未命中、命中， +改变头文件并重建 PCH 后为未命中、命中，两组缓存对象分别与独立调用编译器生成的 +对象逐字相同。四次调用均可缓存。该配置采用上游对 PCH 和时间宏的假设，不承诺 +任意含时间宏源码的输出；未据此推断完整 LLVM 的缓存命中率或用时改善。配置仅在 +本地准备，待实际完整引擎验收后随必要的台账提交及其完整 CI 一同验证。 + + +## 40. 编译器预热与独立构建的依赖更正（2026-11-04，Sunrisepeak） + +服务端首次联合运行 41744249054 的八项工具及静态检查已通过，macOS 编译器预热 +仍排队，所有交叉构建因依赖整个预热矩阵而未开始。核对四个消费者后，它们均自行 +调用 setup-mcpp，且不读取预热矩阵的输出。提交 aec2630 仅移除 build、host-platform、 +server 和 clion-verify 对整个矩阵的控制依赖；任务名、预热矩阵、消费者矩阵、 +步骤及命令逐项保持一致。结构化 YAML 比较通过。 + +[后继 CI 41744974822](https://github.com/Sunrisepeak/mcpp-language-server/actions/runs/41744974822) +已触发，前一运行由原有 cancel-in-progress 策略取消，不能记作完整候选通过。 +macOS 编译器及功能检查仍为整轮验证的一部分，不适用索引发布的排队例外。 +新的拓扑使其他宿主的工作可以先进行，完整候选及发布检查仍须全部通过。 + + +## 41. Windows 载荷发布刷盘的访问权限（2026-11-04，Sunrisepeak） + +运行 41739179747 最终为十三项成功、一项失败及一项跳过。Windows 映像验收中， +worker 已成功构建模块：墙钟五十二毫秒、处理器四十六毫秒，随后四毫秒内的先决模块 +发布报告 Permission denied。五次模块导入检查均返回 3，并行模块检查也返回 3； +warm-prime 因前置失败跳过。此轮不满足完整引擎验收，主补丁 0043 及映像 +0002–0006 保持 stabilizing，不能依据其他宿主成功提升台账或发布引擎。 + +发布代码的两处常规文件刷盘路径以 O_RDONLY 重开载荷；公开 Windows 后端将该操作 +映射到 FlushFileBuffers。[Microsoft 接口文档](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-flushfilebuffers) +要求句柄具有 GENERIC_WRITE。映像补丁 0006 因此仅按原生 Windows 路径风格将这两处 +载荷重开改为 O_RDWR；目录、普通读者及 POSIX 载荷保持原模式。身份校验及实际刷盘 +失败传播保留，错误增加操作名。独立原生 Windows 对照运行 41746593503 已提交， +覆盖只读刷盘拒绝、可写刷盘、内容与大小保持；记录时尚在排队，不计作执行成功。 + +按顺序重建四十三个主补丁及六个映像补丁后，缓存实现及平台适配的三个目标文件与 +修正源码逐字相同。本地以 LLVM 18 编译从真实修正源码提取的辅助函数，确认 POSIX +只读刷盘、载荷内容与大小保持，及带操作名的错误保留 EACCES；十项发布门禁与包身份 +对照通过。这些小范围检查不能替代完整 LLVM 23 构建及四宿主映像执行。 + +§39 所述等待现轮通过后合并 PCH 配置的安排随该实际失败调整：已有的 PCH 配套配置 +与必要的 0006 修正合为下一轮完整验证，避免为缓存配置单独启动大型重建。完整 LLVM +命中率和耗时改善仍待实际统计。服务端 41744974822 的独立源码、构建及工具验证继续 +并行，但其候选仍指向已失败的 82e33ba，不能视为合格的联合发布验证；修正引擎的 +完整打包部件及身份检查就绪后，再更新候选。 + + +## 42. 载荷刷盘对照与完整重建（2026-11-04，Sunrisepeak） + +原生 Windows 运行 41746593503 的十四项检查全部通过：只读句柄可读，但刷盘返回 +ERROR_ACCESS_DENIED；以可读写且不截断的模式重开后刷盘成功，内容及长度保持。 +artifact 13283168509 的 ZIP 摘要为 +`276dfd0b26c747d97111ad09b672f32bd0351aa8c65aec6c438155cffd67a3af`， +回收文件中的检查数、零失败数及只读拒绝结果均已核对。 + +引擎提交 d87eea43ccd313e1a0069a4faed999c71a963f8b 已推送，将映像补丁 0006 +与 PCH 配套缓存配置合为必要的同次重建。 +[完整 CI 41747171954](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41747171954) +已开始；补丁状态仍为 stabilizing。原生内核对照仅确认访问权限条件，最终映像中的 +模块导入、缺失头文件失效及四宿主 warm-prime 仍须实际通过。 + + +## 43. 服务端发布脚本登记的修正（2026-11-04，Sunrisepeak） + +服务端联合运行 41744974822 的 Linux 开发构建在仓库不变式检查失败：新增 +.github/scripts/publish-jetbrains.py 尚未登记到 tools/devtools/scripts.allow。 +补充脚本路径及其 CI 发布用途后，使用实际已发布的 mcpp 2026.11.3.1、公开平台 +0.23.4 构建并运行原有 devtools check all，全部通过，七个受管脚本均有用途说明； +二十二包、六目标锁文件逐字保持。修正本地提交为 47eb181。 + +该登记修正待下一组完整引擎部件及身份检查就绪后，与服务端候选更新一并推送， +避免为仍指向已失败引擎的候选单独重建整个矩阵。现轮其余独立检查继续保留结果； +先前失败不改记为成功，候选及最终发布验证仍须重新实际通过。 + + +## 44. PCH 缓存配置的实际完整构建结果（2026-11-04，Sunrisepeak） + +运行 41747171954 的两个映像构建及布局检查均成功。ARM 的 2 592 次编译调用中， +2 562 次可缓存，占 98.84%；x86-64 为 2 563 次，占 98.88%。两者均不再出现 +PCH 前置条件未满足的不可缓存原因。此前两项构建各有 2 482 次该类拒绝。 +本轮两者仍均为冷缓存，所有可缓存调用均未命中，因此只能确认可缓存性改善， +不能据此报告重复构建的命中率、耗时改善或映像启动性能收益。 + +ARM 布局 artifact 13284439473 的 ZIP 摘要为 +`954aa86fa9b66a57d5be976f3634a99de834b2f34b5bb2350e5f5a6d4d9a3127`； +x86-64 artifact 13287260650 的 ZIP 摘要为 +`37a0d989fcaf47fa24622e9eb6b83ef83a9d2be85137aa34dbbcdba209debb8a`。 +两项仍使用既有的 304 单元、176 模块探针，工作进程受监督、缓存属于测量目录。 + + +## 45. 缺失头文件验收的语义加强（2026-11-04，Sunrisepeak） + +运行 41747171954 最终为十三项成功、一项失败及一项跳过。Windows 映像中的模块 +导入及缺失头文件检查已成功，表明 §41 的实际发布权限失败在最终映像中得到修正。 +本轮失败发生在 macOS：新增搜索路径前部的头文件后，日志记录模块实际重建、 +零错误，但验收要求特定的“先前不存在”重建理由，而本次走的是“命令及先决输入无 +现存 BMI”的重建路径。warm-prime 随前置失败跳过，本轮不能提升台账或发布。 + +补充对照保持模块、使用者及编译命令在三次调用中逐字相同。后部头文件导出的 +constexpr 值为 42；第三次调用前只新增前部头文件，其值为 7。使用者根据该文件 +是否存在对导入模块的值执行 static_assert，因此复用旧 BMI 会实际报错。 +在 macOS 运行 41754360840 中，原生引擎五次均复用旧值并在新增后的断言失败； +相同候选的映像五次均完成首次构建、未变复用及新增后重建，断言通过。 +Linux 原生的三次同型对照也均拒绝旧值。该负向对照确认探针没有绕过缓存失效问题。 + +正式验收提交 89d6a50 保留首次实际构建、未变复用及新增后实际重建要求，并用上述 +取值断言验证结果，不再要求唯一的诊断措辞。README 同时更正主补丁数为四十三， +并如实说明发布要求两个台账均为 steady。 +[完整后继 CI 41755260087](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41755260087) +已开始。主台账 0043 及映像 0002–0006 仍为 stabilizing；小范围对照不代替新一轮 +四宿主完整验收。macOS 对照原始 artifact 13287795264 的 ZIP 摘要为 +`f1adc50311482cb0f521f9f531af40b1721830e78686f044583a7f885ded22d8`。 + + +## 46. 服务端 Windows 开发测试的峰值磁盘占用（2026-11-04，Sunrisepeak） + +服务端候选提交 765ec10 的联合运行 41753047474 最终为二十一项成功、一项失败及 +十六项跳过。失败任务是在 Linux 上交叉构建 Windows 开发测试，LLVM 明确报告 +No space left on device；四十三个根测试中三十七个已构建，余下六个链接失败。 +这不是原生 Windows 的测试执行结果，也不能记作完整候选通过。 + +本地修正 37281b0 仅调整 Windows 开发行的测试归档顺序。由真实发现结果生成互斥 +且完整的子串筛选组，每组至多四个测试；每组完成既有配置的构建及归档后，核对 +归档中的目标、测试集合及可执行文件摘要，再回收已经归档的对应可执行文件。 +公共对象、模块、程序及其他七项构建矩阵的命令保持。原生测试任务原有的归档循环 +会执行所有分组归档；完整覆盖记录及各归档摘要随 artifact 保留。 + +本地按实际发现的六十二个测试验证归档控制逻辑，覆盖无遗漏、无重复、公共文件 +保持，以及遗漏、重复、错误载荷、缺少程序、截断归档、筛选重叠的先拒绝后回收 +边界。该检查未执行编译器或 Windows 测试。另在独立诊断分支以生产工作流的原有 +步骤执行 Linux 交叉构建与 Windows 原生归档测试；记录时仍在进行。正式联合候选 +将随新引擎部件就绪一并更新，仍须完整 CI 通过。 + + +## 47. 构建尾部等待与嵌套构建进度（2026-11-04，用户补充） + +用户报告 Building 1630/1636 长时间保持，而 llvm:clangd-libraries 的运行时间持续 +增加。源码确认 openkal/clangd/build.mcpp 将 LLVM 静态库构建声明为一个 prepare +动作；okgen 内部再启动 CMake 与 Ninja。因此分子和分母计外层构建步骤，整个内层 +LLVM 构建只占一步，并非时间百分比。完成大部分外层任务后，仍可剩余大量实际编译。 + +进度行中的第一个时间为本轮构建已经消耗的时间，动作后的时间为该动作已经运行的 +时间，均不是预计剩余时间。非交互日志每六十秒输出当前状态；定时输出只能确认 +状态显示线程仍在工作，不能单独证明编译器仍有处理器或 IO 进展。 + +上一轮 x86-64 映像任务 126302992405 也在同一外层计数等待，最终构建成功。 +构建总结为三十六分三十七秒，其中 llvm:clangd-libraries 为三十四分钟；该轮 +2 563 次可缓存编译调用全部未命中。这给出了真实冷编译而非死锁的已完成案例。 +用户所贴片段本身仍不足以判定其运行是否出现调度、内存或磁盘异常。 + +此处确认的局限是嵌套构建的可观测性：外层步骤计数不能表达内层编译的完成量。 +后续进度改进应提供真实子进度，或明确显示外部构建动作的阶段及等待；不能把定时 +心跳伪装成实际完成量。当前未为该显示局限改动构建调度或另行发布 mcpp 版本。 + + +## 48. warm-prime 运行器输入的一致性更正（2026-11-04，Sunrisepeak） + +逐项核对验收输入发现，build-test.yml 中 warm-prime 的辅助项目仍固定平台 +0.23.3，而映像、封装与四宿主验收使用 0.23.4。该辅助项目会安装旧工具组；随后 +按全局工具缓存排序选取 okrun 也不能证明取到所声明平台的工具。因此即使运行 +41755260087 其余步骤成功，也不能据此声称完成了当前工具组的四宿主 warm-prime。 + +提交 776c4d5 将该辅助项目固定为 0.23.4，并让 warm-prime、封装及四宿主验收均从 +mcpp self env 的结构化结果读取 runner 或 okld 路径。矩阵、探针工作量、会话次数 +及原验收阈值保持。新增发布输入回归从映像项目读取平台版本，检查两个工作流中的 +固定值一致，并拒绝按全局缓存通配符选取工具；原 89d6a50 工作流的负向对照失败， +修正后的六项发布门禁对照通过。 + +本地使用已发布的 mcpp 2026.11.3.1 和公共平台 0.23.4 构建最小映像，self env +返回工具组 0.4.4 的准确路径；所选运行器实际执行及所选 okld 的映像检查均成功。 +这是公共源包构建工具的本地控制，不证明这些工具与发布 ZIP 中的二进制逐字相同。 +最终发布性能测量仍使用独立验证的已发布二进制。 + +[新完整 CI 41758340462](https://github.com/Sunrisepeak/mcppls-clangd/actions/runs/41758340462) +已触发。资格收集器同时保存语义探针、warm-prime 源码及固定图，核对未缩减的 +304 单元图的摘要，并检查四宿主 warm-prime 的实际运行器来自 0.4.4 工具组。 +Windows 路径仅在核对版本时统一分隔符。主台账 0043 及映像 0002–0006 保持 +stabilizing，待准确输入的新完整运行提供证据后提升，再对齐服务端联合候选。 + + +## 49. 编译缓存清理与原始证据归档（2026-11-04，Sunrisepeak） + +按用户限定的相关项目范围，已清理引擎 PR16 的七个旧映像编译缓存快照，共 +187 134 137 字节。保留计划生成时两个架构各自最新的快照，删除后通过 GitHub API +回读核对；这两个快照均已被正在运行的后继构建读取。未将 sccache 的逐对象条目 +误当成独立整轮快照，也未删除其他引用的缓存。 + +第十四份永久检查点为 mcpp-885-semantic-canary-checkpoint-evidence.tar.gz， +大小 13 452 018 字节，包含 1 074 个逐项摘要核对的文件，压缩包摘要为 +`274d5f64621e13ad84d27fa6099e0252a4490c4f9c6789fa1b7c911f116e02dc`。 +它保存 macOS 原始对照 ZIP 及源码、两个映像布局原始 ZIP、引擎 d87 与服务端 765 +完整运行日志、语义对照、归档控制及本地运行器选择证据。首次引擎日志传输发生 +HTTP/2 流错误，其不完整 ZIP 未被用作证据；替代传输及全部 ZIP 的 CRC 已核对。 +该检查点不包含尚未完成的 CI 结论或最终发布性能数字，前十三份永久检查点保持。 + + +## 50. 分组归档校验的单遍读取及容器校验（2026-11-04，Sunrisepeak） + +服务端本地后继提交 5ab2909 将分组归档校验改为顺序读取：遍历时计算载荷摘要， +并读完 gzip 尾部后才回收已经匹配的可执行文件。此前先遍历成员、再按成员回读， +会在压缩流中回退，重复解压大型调试程序。六十二个真实发现名称、四十八个归档的 +固定对照中，回退次数由四十八次变为零；该数字描述对照的读取模式，不是 CI 用时。 + +原校验的负向对照还表明，损坏 gzip 校验尾部而不改变解压载荷时，程序仍可能 +接受归档并回收可执行文件。新读取器在任何回收前拒绝此输入。完整覆盖及七类 +负向输入对照通过，公共对象、模块与程序保持。独立运行 41756189512 仍验证原有 +分组构建与原生 Windows 执行流程；新读取器待随正式联合候选的完整 CI 实际运行， +本地控制不代替该门禁。 + +后继本地提交 5bf5919 为已经压缩的测试归档设置上传层 compression-level 为零。 +该输入已按 actions/upload-artifact v7 的定义核对，调整的是外层 ZIP 的存储方式， +测试归档内容、编译配置及测试命令保持。此调整同样待正式联合候选 CI 验证。 + + +## 51. 恢复编译缓存后的实际命中与身份对照（2026-11-04，Sunrisepeak） + +后继运行 41755260087 的 ARM 映像步骤明确恢复了前轮的 298 902 136 字节缓存。 +本次 2 562 / 2 592 次调用仍可缓存，但命中为零。构建用时二十三分五十九秒， +其中 LLVM 库动作二十二分二十五秒；宿主与负载未受控，不能将与前轮的用时差 +解释为缓存收益。布局原始 artifact 13289540925 的 ZIP 摘要为 +`900c48534a373652cbf9db543446a28429d787a650d467729087a26aebd57cde`。 +此前“重复命中尚未测量”的状态据此更新为“已观察到一次恢复后仍全部未命中”。 + +独立对照使用真实 Clang 23.1.3 的自有副本及 ccache 4.9.1：编译器字节不变、 +仅修改时间变化时，mtime 身份模式两次均未命中，content 模式为未命中、命中。 +随后仅改变 ELF 末尾不参与执行的一个字节，保持大小与修改时间，content 模式 +重新未命中，下次未变调用命中。六个对象均与独立调用原编译器的对象逐字相同， +原安装编译器的内容及修改时间保持。 + +该对照证明身份模式的行为，尚不能证明实际完整 CI 的全部未命中由何处造成， +也未证明完整 LLVM 或 PCH 的跨安装复用。不同安装前缀的编译器还可能因 ELF +运行时路径修正而具有不同字节；不能仅凭版本号假定编译器相同。生产缓存参数 +未因此更改，后续调查应保留实际编译器身份及 PCH 输入证据，不为推测另行启动 +完整 LLVM 重建。最终发布引擎与服务端的性能数字仍待 T10 的准确发布输入测量。 + + +## 52. 准确运行器输入下的性能门禁失败（2026-11-04，Sunrisepeak） + +运行 41758340462、引擎提交 776c4d5 最终为十五项成功、三项失败。四个原生构建、 +两个 Ubuntu 20.04 下限、两个映像及布局、封装身份、四宿主功能与缺失名字语义验收 +均通过；Windows warm-prime 按原有路径条件通过。其余三个宿主均在 warm-prime +的 Acceptance 步骤失败，而非模块无法构建：这三个宿主的每个首会话均实际构建 +一百七十六个模块，后续三个会话均复用三百四十一个模块且不再构建，语义就绪 +检查均成功。原生 Windows 使用进程内编译及非 owned 路径，其原有门禁记录比值， +但不将该比值作为同路径的性能通过条件。 + +Linux x64 的冷会话处理器时间为映像 15.144 秒、原生 11.696 秒，比值 1.295； +warm 中位处理器时间为 5.49 秒对 4.55 秒，比值 1.207，均超过 1.15。 +Linux arm64 冷会话为 13.726 秒对 11.319 秒，比值 1.213，超过 1.15。 +macOS 冷就绪时间为 14.922 秒对 11.619 秒，比值 1.284；缺页为 +320 382 对 117 855，比值 2.718，分别超过 1.15 与 2.0。 + +这些是实际未发布候选及当前源工具组的门禁结果，不是最终发布测量。旧输入运行 +41755260087 也已结束，十四项成功、四个 warm-prime 失败，不能作为替代资格。 +两个观察进程均已终止，不能将观察终止误当成仍在运行的完整验证。 +主台账 0043 与映像 0002–0006 继续保持 stabilizing，服务端候选仍未更新或发布。 +下一步使用准确候选的原始数据分析 Linux 处理器开销及 macOS 缺页来源；保留原 +探针、会话和阈值，不通过缩减工作量或改变标准使门禁通过。 + + +## 53. Windows 开发归档的实际覆盖与产物体积（2026-11-04，Sunrisepeak） + +独立诊断 41756189512 的 Linux 构建任务 126332740455 已成功。逐项解析原始日志中 +四十八条归档回执，与真实发现列表比较，六十二个测试均恰好归档一次，所有归档 +摘要保留，归档后回收的可执行文件逻辑大小合计 12 702 464 000 字节。 +原生 Windows 任务已执行完归档，三个测试程序失败，尚不能计为通过。清理后磁盘 +采样的最小空闲量为 +76 940 771 328 字节；这是清理后的采样，不是峰值，也未复现原失败机器的容量。 + +实际 artifact 13290692289 为 9 355 073 547 字节，摘要为 +`b0e7e89807d6fffec071b3cbc264b03601d5b82b3dd0b73c8d664649a46a01a0`。 +每个分组重复携带包的程序使体积过大，当前原型尚不作为最终联合候选推送。 +后继应将已验证的分组流式合并，每个包只保留一份公共程序及一个完整测试清单， +再核对最终容器的测试覆盖、文件摘要及元数据。中间采用 mcpp 支持的未压缩 tar， +避免对公共程序反复执行 gzip；同时保持一次只持有一个分组的可执行文件及中间包。 +该改进尚未实现，后续实际 Linux 构建及 Windows 执行仍为必须的验证。 + + +## 54. 原生 Windows 归档执行发现的失败（2026-11-04，Sunrisepeak） + +任务 126351796514 已结束。devtools 的 test_bench_review、根的 test_grants、 +pack 的 test_payload 失败。前者的两个快照案例及后者的十个载荷案例在文件系统 +weakly_canonical 中收到 Invalid argument；实际调用点使用 filesystem::relative， +不是测试代码直接调用 weakly_canonical。test_grants 的两项新安全案例有三条 +断言失败，分别涉及合并后父目录的 immutable 属性及过多可写授予的保留。 + +执行日志明确显示 Windows GNU 对象 ABI 下的实际 C 库为 musl 0.27.4、C++ 库为 +openkal-llvm-runtime 0.18.5，平台为 hosted 0.23.4。因此需核对 hosted 路径语义及 +具体调用目的，不能假定这是普通 MinGW 标准库，也不能仅以跳过测试处理。 +当前尚未确定根因或修改这些产品调用点；后继修正须保持符号链接、目录范围及 +授予边界，并在实际 Windows 上验证。此诊断不构成服务端完整候选通过。 + + +## 55. Windows 开发归档的合并实现与路径修正（2026-11-04，Sunrisepeak） + +服务端本地提交 723d633 将九处目录走查所得路径的相对名称改为词法计算，保留 +目录项在原树中的名称，避免为映射复制目标或快照键而解析符号链接的物理目标。 +授予数量测试以已解析的根目录构造，原断言保留；另增案例核对过多可写根目录的 +每个输入拼写及物理别名均被保留，未改为无限制授予。Windows 原失败的原因与 +修正效果仍需原生执行核实，尚未声明底层标准库路径处理已修复。 + +Linux 本机复测通过：test_grants 九个案例、一百零三条断言；test_payload +十五个案例、七十八条断言；test_bench_review 十一个案例、二十六条断言。 +这些结果仅证明本机对应程序的行为，不替代 Windows 执行。 + +提交 625a101 将分组运输改为每包一个最终归档。mcpp 先产生未压缩 tar，脚本 +前向校验并将载荷写入该包的 gzip 容器。同路径公共程序的摘要、大小及权限均 +一致时才去重；不同测试的路径不合并。各分组的元数据保持一致，测试清单取完整 +并集，重命名时去除旧 PAX path 与 linkpath，避免旧前缀覆盖新名称。 + +每组的可执行文件先与归档内容逐一比较，整组有效才回收匹配的文件；对象、模块 +及公共程序保留。最终容器关闭后重读全部载荷、清单与 gzip 尾部，覆盖与摘要 +一致后原子改名。控制测试覆盖真实发现的六十二个测试，并拒绝十三类分组或 +最终容器错误。仓库不变式检查通过。 + +专项分支 9ed9564 的运行 41770334975 复制生产构建和执行任务，仅选取 Windows +dev 矩阵行，全部测试及调试选项保留。运行已启动；此处尚无实际归档体积、构建 +耗时或 Windows 通过结论。服务端正式分支尚未推送这些本地提交，最终联合 +候选仍需完整 CI。 + + +## 56. 准确候选的本地性能归因记录（2026-11-04，Sunrisepeak） + +776c4d5 的原生 Linux 包与映像包均已下载并核对 artifact 摘要。原生包的三百一十 +个文件通过 SHA256SUMS 校验，二进制摘要为 +`ccaa8f510abb8a54eebd10381957413cb1a3a93dc6d4dc776ccff634bd9d2b68`。 +原生补丁序列、构建提交的合并父节点及树均已核对。映像束的两架构成员也与 +okb.json 中的大小及摘要一致；这些检查不是完整性能资格。 + +同一本地 Linux 环境分别使用隔离目录运行原探针的三百零四个单元。四次会话均 +就绪，冷会话构建一百七十六个模块，三个暖会话均未重建。原生暖会话处理器 +时间中位数为 2.74 秒，映像为 4.43 秒；就绪时间分别为 1.493 与 2.132 秒。 +这是候选的本地复现，不能用作已发布版本的最终数字或替代失败的 CI 门禁。 + +另行跟踪三个暖会话的文件系统调用，原生失败路径查询为 288848、289014、 +288930 次，映像为 289097、289029、289128 次。原始跟踪均已无损压缩并记录 +解压后摘要。两边的查询数量接近，额外处理器时间的原因仍未确定；带跟踪会话 +的耗时不参与验收，也尚未据此声明第三级性能通过。 + +映像的独立调用、阶段及处理器采样报告已保留。涉及调用计时本身的样本需与 +产品开销区分，阻塞等待的调用耗时不能累加为处理器时间。macOS 专项运行 +41770916075 复用相同候选并按平台 0.23.4 构建工具，尚待报告。原阈值、工作量 +及 stabilizing 台账状态均保持不变。 + + +## 57. 嵌套构建末尾长任务的进度诊断（2026-11-04，Sunrisepeak） + +llvm:clangd-libraries 在 mcpp 中是一项 prepare 动作，内部运行 CMake 与 Ninja +构建 clangd 的静态库。Building 1630/1636 计算外层已完成任务，不按预期耗时 +加权。任务右侧的时钟仅表示该动作已运行时间，其刷新不证明编译器仍有进展。 + +外层 Ninja 缓冲该命令的输出，mcpp 默认又不展示成功步骤的普通输出。使用真实 +Ninja 1.12.1 的三步嵌套控制，每步运行一秒，默认及 verbose 两种方式均在外层 +动作结束后才输出内层三行进度。历史 x86 映像任务曾保持同一外层计数超过 +三十三分钟，最终成功，该动作共三十四分钟。 + +另一次实际 ARM 构建恢复 298902136 字节的 ccache，但 2562 次可缓存调用均未 +命中，库动作耗时二十二分二十五秒。恢复缓存不等于复用了编译结果，未命中的 +具体原因尚待核实。已确认的进度可见性不足与缓存效率问题分别记录；现有日志 +不足以认定调度死锁。后继可独立显示嵌套动作的阶段及子进度，不把外层任务数 +表述为时间完成率。 + + +## 58. 原生归档构建及最终运输体积（2026-11-04，Sunrisepeak） + +运行 41770334975 的 Linux 交叉构建任务 126378300821 已成功。逐项比较实际日志中 +四十八个分组和四个最终回执，六十二个测试均恰好保留一次。每包最终清单的测试 +集合与发现列表一致；公共程序去重后，根包为四十三个测试、四十八个载荷文件， +其余三包分别为八个、十个、一个测试。 + +最终 artifact 13296016507 为 2 463 309 786 字节，摘要为 +`83c0c19a1f2f73ac49c8758bef1b917dae79d8b89fdd993f42ae32a9f6cd06cf`。 +相较此前 9 355 073 547 字节的分组产物，体积减少约 73.67%。本次归档步骤耗时 +1535 秒；机器与负载未控制，不据此声明性能加速比例。整轮回收可执行文件逻辑 +大小 12 702 354 944 字节，清理后最小空闲量 84 196 241 408 字节，仍不作为峰值。 + +原生 Windows 任务 126387978453 已开始执行四份归档，结果尚待核对。这份实际 +构建证明运输方案保留完整覆盖并减少重复，尚不能证明此前三个 Windows 程序的 +行为修正通过，也不替代服务端正式联合候选 CI。 + + +## 59. 缺页采样边界及 Linux 处理器计数修正（2026-11-04，Sunrisepeak） + +macOS 专项 41770916075 成功，未采样会话中，原生冷会话缺页为 126640，映像为 +309248。专项 41772175101 在 LSP initialize 之前取得两边的实际 vmmap。映像指令 +仍为映像束文件的映射，准备数据为 okseg 的写时复制映射；因此不能将差距归为 +装入器复制了全部指令。首次冷启动、预取及采样自身的开销仍需分别核对。 + +后继专项 41772570686 比较周期采样与内核完整进程树总账，并以短生命期孙进程 +触及六十四 MiB 的控制确认内核会累计被回收后代。原生就绪时采样缺页 135516， +内核完整运行总账为 508628；映像分别为 295915 与 699903。两种算法所得比值 +分别为 2.184 与 1.376。完整总账包含 Python 探针及关闭阶段，不能直接替换就绪 +时的统计；该证据说明采样漏计的程度在两条路径上不一致，尚未修改原缺页门禁。 + +源码审查另发现 Linux 冷会话报告在 stop 之后再读取 proc；此时进程已消失， +导致先前取得的内核处理器时间被采样及 worker 日志的回退值覆盖。本地提交 +f3ee948 按宿主选择原有算法，Linux 保留就绪时的内核值；另增加明确标注边界的 +waited_usage，记录引擎完整生命期及已回收后代的内核统计，不含 Python 探针。 +异步重挂至其他父进程的后代不计入该字段。原缺页指标、阈值与工作量均保持。 + +五项回归通过，包括直接驱动实际 session 报告函数，以及真实被回收孙进程。 +旧 776 报告函数将给定的内核值 12.345 改写为 2.01，修正后保持 12.345。 +复用原候选完成同一三百零四单元图的四次会话，两边均语义就绪且暖阶段无重建。 +修正后本机冷处理器比值为 7.83 / 6.61 = 1.185，暖阶段为 2.54 / 2.13 = 1.192， +仍超过 1.15；不得将计数修正写作引擎性能已通过。提交尚未推送到正式 PR， +原完整运行的失败状态及 stabilizing 台账保留。 + +第十五份永久检查点 mcpp-885-consumer-diagnostics-checkpoint-evidence.tar.gz 保存 +2193 个逐项核对文件，大小 101119999 字节，摘要为 +`396f81ae860bcb1f1dba551bdf730e6287fdefa51bc1ce614f6f8f97e74b86fa`。 +其中保存准确候选的原始 ZIP、本地双路径跟踪及独立 macOS 采样报告；后取得的 +映射、总账对照、计数修正及 Windows 构建终态另行保存，不追写既有检查点。 + +## 60. Windows 完整执行及归档回归终态（2026-11-04，Sunrisepeak） + +专项运行 41770334975 已成功结束。原生 Windows job 126387978453 实际执行 +六十二个程序，名称与发现清单逐项对应且各出现一次；四个包均无失败。 +先前失败的 test_bench_review、test_grants、test_payload 均通过。生产源码与 +625a101 相同，专项提交仅多一个诊断工作流；不得据此宣称底层标准库的 +weakly_canonical 已被修正。 + +归档构建的四个最终包共 2463309786 字节，对先前 9355073547 字节减少约 +73.67%。实际归档步骤为 1535 秒；它与旧原型不是受控速度对照。构建过程中 +回收 12702354944 字节可执行文件；清理后可用空间不是峰值占用证据。 +原生执行日志摘要为 +`69c05125001e5b53e3316a4ab96c2db04b302ca317aef807bdbbd6bffc345251`。 + +生产提交 625a101 已推入 server PR 75，完整回归 41775013004 正在运行。 +该回归仍使用先前冻结的 d87 引擎候选；其结果只能证明当前 server 源码回归， +最终合格引擎的联合资格及发布后的精确依赖仍需另行完成。 + +## 61. 准确映像的处理器采样与临时区初始化（2026-11-04，Sunrisepeak） + +宿主 perf 权限现已可用。对冻结 776 构件各执行一次冷阶段及三个暖阶段， +两边均保持三百零四单元图、四次会话、暖阶段零构建及语义就绪；采样只在暖 +阶段开启，使用 499 Hz cpu-clock 与帧指针，两边均限定处理器 0–7。 +该限定及采样开销使这组数据仅用于归因，不代替最终验收。 + +映像束内准确 ELF 成员偏移为 0x49d0000。解析器依据程序头与符号表转换 +文件偏移，调用栈另以准确进程的 perf map 校正加载基址。三个映像会话的 +4812 个叶样本中，214 个位于 memset，约 4.45%；其中 79 个来自 +__okm_syscall，24 个来自 okm_unlock。内核符号仍受限制，未解析的内核样本 +不能继续归到具体函数。 + +准确构件反汇编确认：__okm_syscall 在分派前清零 8224 字节，okm_unlock 在 +检查待关闭数量前清零 10752 字节。发布 0.27.4 源码中，这些自动临时区位于 +公共入口；映像启用的自动变量初始化使原本不用该区域的调用也承担清零。 +修正方向为限定临时区生命范围，保留自动清零、栈探测及其他保护。 + +## 62. 路由临时区的源码修正候选（2026-11-04，Sunrisepeak） + +musl PR 82 的 d583f24 将五组使用临时区的原有 case 原样移动到不内联的 +共用辅助函数，六个参数保持原次序传递。解锁在持锁状态检查空队列；非空 +队列仍按当前上下文取出资源，并在解锁后关闭。空队列不再初始化大数组。 +null 名称检查对普通末尾分量直接拒绝；null、点分量及歧义形式仍完整解析， +扫描长度不超过原解析器的路径上限。版本准备为 0.27.5，尚未发布。 + +本地使用实际映像编译命令重编两处源码，核对保留自动变量清零。x86_64 的 +分派入口栈帧在 O0/O2 为 2712/1112 字节，ARM 为 2704/1104；慢路径继续 +共用一个有界临时区。这是编译器报告的局部栈帧，不是完整调用链用量。 + +对照发布 0.27.4 与修正源码的实际映像，二十六项观测一致且全部通过，涵盖 +2916 种路径、十二个路径长度边界、复制内容与偏移、拒绝旗标、名称修改、 +时间与容量，以及四百次并发管道生命期。相同编译器、保护参数、运行器及 +处理器集合下，交替执行三次完整的六十八项原语测试。修正对旧源码的中位 +耗时比为 stat 0.840，单线程 open/close 0.845，四线程 0.852，单线程 pread +0.874，四线程 0.945。源码控制仅早于最终提交的包版本变更；最终提交另由 +完整 CI 41777022375 验证。这些数值不代表 clangd 的性能门禁已通过。 + +## 63. 此轮版本组的最短依赖路线（2026-11-04，Sunrisepeak） + +此轮修改仅在 musl，不改变后端、装入器或工具。所需发布依赖为 musl +0.27.5、runtime 0.18.6、platform 0.23.5，再构建合格 clangd 并完成 server +联合资格。os 直接依赖的后端保持原版本，故此轮不追加 os 发布。 + +runtime PR 63 使用现有同名分支图测试未发布的 musl，允许与上游 CI 同时 +验证。platform 0.23.5 的版本组已在独立工作树准备，待上游版本发布并进入 +公开索引后再启动其完整 CI。原有映像保护策略与验收阈值均保持。 + +## 64. musl 发布及 runtime 验证中的性能失败（2026-11-04，Sunrisepeak） + +musl 完整运行 41777022375 的十项检查全部通过。PR 82 以 c0430c0 合入， +合入树与受测 d583f24 完全一致；0.27.5 标签及 GitHub release 已建立，家族 +分支同步。GitHub 与 GitCode 归档均为 1268814 字节，摘要一致： +`458c9ef1f46446f27d0199118357ced065598f8aadf653a5a4c498a6d811b295`。 +索引 PR 704 保留既有版本行，仅前移直接 C 库测试成员；本地实际运行该成员 +通过。索引候选的 Linux default 与 LLVM 两项及兼容性工作流均通过。 + +runtime 0.18.6 的完整运行 41777349050 在 macOS 映像 L2 的 index/symbol +失败：281.13 对 212.69，比值约 1.32，超过 1.15；其功能观测通过。 +此失败尚未被判定为源码修正的性能回归。基准以散列表遍历次序选择查询， +并在若干表达式中多次调用可变随机数生成器，跨编译器的输入与查询需核对。 +本地 GCC 与 Clang 的原始符号集合摘要不同；本地 Clang 与公开映像的摘要 +相同，故这些观测还不能解释 macOS 的失败。专项 41779143805 在同一 macOS +宿主比较冻结的已发布与候选映像，并记录编译器及工作量摘要。原基准与阈值 +尚未修改，runtime 与 platform 均未发布。 + +## 65. Windows 标准库卷路径的独立复现（2026-11-04，Sunrisepeak） + +专项 41777888621 在原生 Windows 上，使用公开 platform 0.23.4、runtime +0.18.5、musl 0.27.4 及 runner 0.4.4，对同一源码分别使用宿主 MSVC 运行时 +与映像运行时。MSVC 的 current_path 为 D:/ 前缀路径且被识别为绝对路径， +canonical、weakly_canonical 及已遍历条目的 relative 均成功。映像返回同类 +路径却将其识别为相对路径，absolute 重复拼接当前目录，上述三项均返回 +EINVAL 22；文件的创建、遍历及删除完成，lexically_relative 返回 leaf。 + +该公开运行时缺陷登记为 openkal-llvm-runtime issue 64。专项成功表示两边 +的观测已取得，不表示错误的映像结果通过了功能断言。server 的词法遍历 +映射修正仅覆盖相应调用者，底层标准库在此轮尚无修正。 + +## 66. 不含探针的完整引擎内核总账对照（2026-11-04，Sunrisepeak） + +专项 41777772516 的 Linux 与 macOS 均成功，五项计数控制通过。两条引擎 +路径各完成原三百零四单元图的四次会话，均语义就绪且暖阶段无构建。 +与第 59 节包含 Python 的外层总账不同，这次的 waited_usage 直接在每个 +引擎启动前及其 wait 完成后取差，不含探针自身，包含关闭及已回收后代。 + +macOS 冷阶段原生就绪采样为 126581，完整引擎总账为 506753；映像分别为 +294066 与 692680。比值为 2.323 与 1.367；暖阶段分别为 1.109 与 1.107。 +Linux 冷阶段两种缺页比值均约 1.076，暖阶段为 1.028 与 0.878，显示关闭 +阶段的额外用量在两边也不相同。因此两种统计不能在同一名称下静默互换。 + +原有缺页门禁仍保持。若调整测量边界，应先明确指标的阶段、回收范围及旧 +记录的可比性，再统一所有支持该内核计数的宿主。此专项使用冻结的失败 +776 构件，仅验证计数：Linux 冷、暖处理器比值仍为 1.286 与 1.190，不能 +作为最终引擎资格，后续仍需用新 musl 版本组重建。 + +## 67. runtime 零点十八点六资格与发布（2026-11-04，Sunrisepeak） + +musl 零点二十七点五已完成全部十项生产检查、双端源码摘要及 GLOBAL、CN +普通客户端测试。runtime 的第一次 macOS L2 index 比值为 1.32，失败保留。 +专项 41780521076 在同一宿主交替执行准确冻结的旧、新映像及两个原生编译器， +每组五次，核对 bitstream、symbols 与 queries 输入指纹相同。五项新映像对原生 +比值均低于原阈值，index 为 0.948；具体调度或资源原因未获证明。 + +据此对相同提交及原阈值仅重新验证原失败 job 一次，生产 CI 41777349050 的 +十项检查均通过。PR 63 合入 f450b4d418a84a6b2b18184bc12663111fffce9f， +发布零点十八点六。GitHub、GitCode 源码均为 15642891 字节，SHA256 为 +811fbb8b59344b319ecf70880708fd7c97c3325b0e47f5a8f8ec3b2f2bb4a898。 +索引 PR 705 全部适用检查通过后合入 500054cd5100fdf7c40cd97b568551f033b64c83； +普通客户端指针及后继 platform 仍待核对，以上不构成最终引擎验收。 + +## 68. 模块 worker 的装入器只读授权（2026-11-04，Sunrisepeak） + +服务端候选 625 的 Neovim 三宿主及 Linux 清洁容器出现模块 worker 启动失败， +日志为 posix_spawn 的 ENOENT。实际授权包含引擎映像目录，未包含 payload 的 +libexec 装入器目录。worker 经 kal_env_loader 取得装入器，再经受授权限制的 +路径解析启动；该缺项使工作区位于 payload 外的编辑器路径失效。 + +专项 41782955938 保持准确候选 payload 与第一次原导航断言，仅向客户端初始化 +加入装入器父目录的只读授权。Linux、macOS、Windows 均在原九十秒截止之前取得 +非空跳转目标，报告中不再出现该启动失败。Linux 独立 token 对照没有增加授权， +不能据导航恢复宣称暖 token 已恢复。本地生产修正将装入器目录作为默认不可变 +payload 授权，九个单元用例的一百零五项断言通过；尚待最终候选的完整 CI。 + +## 69. L3 缺页口径版本二（2026-11-04，Sunrisepeak） + +依据第 59、66 节的冻结构件审计，采用第三部分末尾具名更正块的完整引擎生命期 +内核总账。page_faults 保留就绪诊断，complete_page_faults 包含关闭及等待回收 +后代；版本和范围显式记录。此更正没有调高二倍缺页上限，没有修改 1.15 倍就绪 +时间与处理器上限，也没有缩减三百零四单元及四次会话。 + +本地真实验收 CLI 的控制验证旧版本、错误范围、缺失或不一致计数、伪造暖中位数、 +缩短工作量及原有时间和 CPU 超标均遭拒绝。短生命期孙进程触及十六 MiB 的内核 +计数控制通过。最终新引擎仍需在所有宿主重新构建和测量，旧候选结果不提升状态。 + +## 70. 分离目录的冷暖 token 对照与公开 runtime 客户端（2026-11-04，Sunrisepeak） + +准确 625 服务端、stage 与语义 kit 的三个原始 artifact 均核对完整 SHA256、ZIP +CRC 并解包。最初将 payload 和工作区一起放入临时目录时，两条路径均通过，因为 +临时目录授权已覆盖装入器；该控制不检验目录缺项。分离 payload 至工作区目录、 +保留 fixture 于临时目录后，原默认授权的冷、暖会话均缺少 completion，日志复现 +worker 的 ENOENT。仅补充装入器父目录的只读授权，两次均通过原万行文件用例， +暖 compiler tokens 为 0.0378 秒，completion 为 0.1005 秒，编辑及 range 断言通过。 +这组数字用于根因对照，不作为最终发布性能数据。 + +runtime 索引发布 41784367275 成功，普通 raw 指针已观察到合入提交。GitHub 与 +GitCode 索引归档均为 664572 字节，摘要为 +37a902d184744ee8909d72e2036b2c457d9f8bdfb6e84564af8cc1e8a49bec7b。 +两个隔离客户端分别选择 GLOBAL、CN，均以真实 mcpp 三点一执行不改源码的 cxx +及 import-std 示例，版本选择来自普通公开索引，锁文件固定 runtime 零点十八点六 +及 musl 零点二十七点五。CN 所选客户端本次源码日志实际使用 GitHub，因此此项 +证明客户端可用，不宣称本次经 CN 源码链路下载；双端源码字节另已核验。 + +测量版本二还在冻结 776 原生及映像上实际运行三百零四单元、三个文件、四次会话， +均就绪，完整缺页与原始内核计数相符。原阈值验收仍拒绝该旧映像的冷时间、冷暖 +CPU；该结果验证报告及拒绝路径，没有提升旧候选的资格。 + +## 71. platform 零点二十三点五发布与默认授权实际验证（2026-11-04，Sunrisepeak） + +platform PR 18 的生产 CI 41784967446 四项均成功，合入提交为 +b9cc96d17a9595c341bd9e9b7029d86766f22456。零点二十三点五的 GitHub 与 +GitCode 源码均为 18575 字节，摘要为 +e533ce5c9c68ff10f8b8748312a7e25180ae691713498aa5378e9a38ff27b01e。 +版本组及映像层一致选择 musl 零点二十七点五与 runtime 零点十八点六；装入器、 +工具、后端及加固参数保持。索引 PR 706 的所有适用检查通过后合入 +b00684acb8e2784e86a4712ada9393b6e8f75f70，发布专项 41786984354 仍待核对。 +本地索引测试已用发布源码执行 hosted 程序通过，303 个既有描述符的保留检查通过。 + +服务端本地生产修正 4d591a6 的实际 hosted 可执行文件完成冷、暖 token 检查， +fixture 没有附加授权，仍采用原参数及预算。该控制使用冻结的旧引擎、装入器与 +语义 kit，证明默认授权代码可用，不构成新映像组的完整资格。旧服务端 CI +41775013004 在记录 49 项成功、十项实际失败及一项跳过后取消剩余任务，保留 +原失败。最终服务端将与新引擎构件一起重新验收。 + +第十八份永久证据归档为 mcpp-885-runtime-loader-checkpoint-evidence.tar.gz， +87541419 字节，1589 个文件逐项核对摘要，归档摘要为 +7fe7bf86327ac90e63249bc1f9329c048da4e18f4e778f444a0244590dce7611。 +该归档覆盖本轮冻结载荷、作用域控制及 runtime 发布；platform 发布与后续 +默认代码实际验证产生于其后，将由后继归档保留。 + +## 72. 新版本组实际消费与引擎重建（2026-11-04，Sunrisepeak） + +platform 索引发布 41786984354 成功，普通 raw 指针已到合入提交。两个端点的 +索引归档均为 664633 字节，摘要为 +7e82b3766b44738ea6670ccda8e3629f63932f20de4d32945f4c88a7d8a3076a。 +隔离 GLOBAL 客户端用真实 mcpp 三点一、显式 x86_64-openkal 目标构建并执行 +程序，输出四十二，锁文件确认 platform 零点二十三点五、runtime 零点十八点六、 +musl 零点二十七点五及 tools 零点四点四，均来自普通公开索引。 + +两项辅助验证曾并发共用归档输出路径，其中一次摘要断言失败；该次未作为镜像 +损坏的结论。辅助脚本现改为进程独立下载临时文件，验证后原子替换，后续双端 +摘要和实际消费均通过。未显式指定 openkal 目标的早期原生执行只作普通命令 +控制，不计入上述平台消费证明。 + +服务端本地提交 9e0c1d4 将 manifest、host graph CI 及原六目标依赖锁切换至 +新版本组。仍为二十二个索引包，仅 platform、musl、runtime 三个记录变化， +锁摘要为 fa8c751d8e119d8e79dad7a6829727945a8ad93d3b3dca65ad60b0a28021b939。 +本机构建及 grants、templates、process 共四十四个用例、三百四十六项断言均通过， +构建临时增加的 xxhash 单列保留后恢复上述规范锁文件。尚未推送，等待新引擎构件。 + +引擎 cfd2243 已推送，完整 build-test 为 41788130200。初步 Linux 原生构建通过， +其余宿主、映像及后继性能检查仍在进行。独立 verify-series 41788130169 已应用 +主系列及映像系列，随后发布账本门禁拒绝仍为 stabilizing 的 0043；普通账本 +一致性检查通过。该条目及映像条目二至六仍需在完整引擎、服务端联合资格满足后 +提升并完成最终检查，不能在此阶段发布。Windows 历史崩溃采集专项已通过。 + +## 73. macOS 规则容量与既存目录名缓存截断（2026-11-04，Sunrisepeak） + +专项 41790529996 的短路径可写、只读拒绝及十六条长路径对照均达到预期。 +混有反斜杠的转义用例在未隔离时已无法解析，因此不能证明规则容量。改为 +只含引号的路径后，41791186105 的六项隔离行为均达到预期，但 std::filesystem +清理目录返回目录非空，整体作业失败。两份原失败及原始 artifact 均保留。 + +专项 41792589035 在同一 macOS 宿主比较准确公开版本组零点二十三点四及五。 +两组的六项隔离结果一致：短路径可写成功、只读拒绝写入；十六条最长八百二十四 +字节的可写物理路径在未隔离时有效，隔离时返回 invalid，目标未执行；最长 +八百二十七字节、含引号的只读物理路径亦在未隔离时有效、隔离时因规则容量拒绝。 +实际后端均为 macOS 零点二十一点三，锁文件不存在工作树代入。该结果确认原 +十六 KiB 规则区不足时为拒绝启动，错误分类仍不准确；没有观察到扩大权限。 + +两个版本组的独立文件系统程序均能清理浅层目录，十层深目录及引号目录均返回 +可移植错误三十九，系统运行时对相同结构成功。Linux 同机对照复现相同差异， +且独立程序未使用进程授权。因此该清理问题在本轮路由优化之前已存在；此结论 +只覆盖已测用例，不扩展为所有文件系统行为无回归。专项成功只代表证据采集完成， +两个版本组的完整隔离程序仍保留因清理失败而返回的一，不能称为整体通过。 + +Linux 实际内核跟踪及两版公开源码进一步定位至 musl 的目录名缓存。目录名区 +为五百一十二字节，保存时未保留溢出状态，之后把最多五百一十一字节的部分名称 +当作完整绝对路径。实际 openat 把七十二字节的相对分量接在三十一字节的截断 +祖先分量后，所请求路径不存在。独立问题记录及原始跟踪已保留；尚未修改该实现， +不将其归为本轮新回归，也不宣称已经证明越过授予边界或覆写其他文件。 + +## 74. 新引擎构件准入与服务端联合验收（2026-11-04，Sunrisepeak） + +引擎 41788130200 的四个原生构建、两个 Ubuntu 20.04 兼容包、两架构映像及 +通用载荷身份检查均通过，五份构件的 run、head 与 artifact 摘要匹配 cfd2243。 +准入记录仅允许联合候选测试，不代替后继四宿主功能与性能门禁，也不允许发布。 + +服务端提交 99a27fe 将候选文件切换至该引擎构建，连同 4d591a6 的默认只读装入器 +授权及 9e0c1d4 的公开版本组、六目标依赖锁一起推送。完整 CI 为 41793927769。 +已发布载荷锁及编译时引擎常量保持原发布值，待双方完整资格通过、引擎发布字节 +独立下载校验后，再与删除候选文件在同一提交中切换，并执行最终完整服务端 CI。 + +## 75. Windows 归档执行与服务端首次计划时序（2026-11-04，Sunrisepeak） + +服务端完整 CI 41793927769 结束于六十一项成功、四项失败及两项跳过。 +Windows 开发归档执行残留上一个解包目录，后续两个包无法解包；暖缓存观察器的 +路径匹配另遗漏盘符，实际引擎已复用三个模块且没有重建。修正宿主目录清理及 +盘符、反斜杠识别后，41803299158 的 Windows 开发和发布归档、实际暖缓存 +一致性检查均通过。该轮整体为七十七项成功、四项失败及两项跳过，仍未合格。 + +该轮剩余问题包括 mcpp 冷启动首次正确作答超过一百五十秒、xlings 暖启动一次 +额外重启，以及两项真实 Zed 的进程观察器失败。xlings 的异步目标选择比首次 +引擎启动晚约二百毫秒，首份计划需要新增只读目录。服务端 4497dca 将受限映像 +首次启动延后至首份计划或显式无计划回退；单纯运行计时器不再提前启动，也不 +产生忙轮询。原生及无限制启动保持即时。既有模型回退仍能解除无计划等待。 + +本地服务端七十个用例、七百二十五项断言通过，准确旧引擎载荷与默认授权下， +原万行文件的冷、暖 token 用例均通过。原目标选择、切换、删除及重建场景的 +十九项检查在七点九秒通过。新增两个 Python 控制脚本最初未登记，完整构建被 +仓库脚本清单拒绝；5bdeddb 补充用途登记后，全部本地 devtools 不变量及二进制 +检查通过。该修正保留清单门禁，没有删去脚本检查。 + +## 76. 物理目录视图与冷启动 CPU 定位（2026-11-04，Sunrisepeak） + +引擎通用补丁 0044 直接在编译命令的绝对目录建立独立物理文件系统视图，避免 +查询随后立即被替换的进程工作目录；自定义视图及无效目录的回退保持。二十二项 +LLVM VFS 测试、原生与公开平台映像组件控制通过。组件构造测量不能代替完整 +引擎性能验收。完整候选 6fc 的 41807943117 仍为十五项成功、三项失败： +Linux x64 冷时间比一点一七九、CPU 比一点二五七，ARM 冷 CPU 比一点一七四， +macOS 冷时间比一点一九五，均存在超过一点一五的指标。POSIX 暖 CPU 和完整 +生命期缺页门禁通过，未据此提升该候选。 + +冻结 6fc 的 macOS 专项 41840386025 完成三百零四单元、一百七十六 worker 的 +同一图，采用命令行配置只采样父进程，实际仅生成一份父进程报告。八千四百 +四十七个 CPU 样本中，1491 个文件信息查询及 659 个打开样本经过拥有缓存的 +后备租约查询，约占父进程样本四分之一。缓存发布样本为八十三个，不能将此前 +观察到的发布日志间隙直接解释为主要 CPU 成本；并行区间之和也不是整体耗时。 + +Linux 同图 syscall 控制中，原生和映像的全部 stat 类调用分别为 614198 和 +1448442 次，不能只比较某一种 stat 拼法。准确映像栈解析保留进程、线程及 +映射事件，3063 个样本涉及 179 个映射进程，映像栈未解析帧为零。采样和跟踪 +用于定位，不作为带跟踪开销的性能资格数字。 + +## 77. 有界账本扫描与旧编译器兼容修正（2026-11-04，Sunrisepeak） + +通用补丁 0045 在持有账本锁时，每次读取十六个相邻槽位的双份记录，保留原 +校验和、序列、类型及字段检查。查询先筛出稳定且键匹配的槽位，释放账本锁后 +才申请候选租约；后续仍重读记录并检查命名空间、租约身份、代、载荷及完整清单。 +未增加持久缺失缓存，记录布局、配额、加固参数及持久化协议保持。准入复用同一 +有界读取，继续核算全部占用、溢出及独占租约。截断和不明确记录仍为错误。 + +原生与公开 platform 零点二十三点五映像的组件控制通过跨读取块、其他实例 +发布、提示、活跃租约、配额、载荷变化、截断及损坏记录检查。新增三个测试体在 +最小真实文件系统组件夹具通过；该组件仅排除无关的 CompilerInvocation 参数 +检查及两个 Clang 头文件，完整 ClangdTests 仍由 CI 验证。四十次交替组件调用 +中，缺失查询 CPU 中位数分别为原生基线的约百分之四十八、映像基线的约百分之 +十九。十六次缺失查询的映像 openat 从 65567 次降为 31 次，newfstatat 从 +65595 次降为 59 次。这些数字不表示完整引擎已达到性能门禁。 + +首轮 75b 的完整构建 41846781069 在四项 Linux 原生或兼容构建中失败。 +旧 GCC 及 Ubuntu 20.04 的 Clang 12 不接受依赖模板返回值的隐式移动；本地 +较新编译器此前接受该代码。77e4e2f 显式移动仅能移动的错误对象，本地 GCC 13、 +Clang 18 组件编译、原生控制及三个新增测试体通过。主补丁先于全部映像补丁的 +应用次序仍得到完全相同的受跟踪源码树。新完整 CI 为 41849712864；两项 +Ubuntu 20.04 兼容打包已经通过，整体功能及性能资格尚待完成。旧失败轮次取消 +剩余构建以释放队列，主账本 0043 至 0045 与映像二至六保持 stabilizing。 + +## 78. 首次计划修正的完整结果与真实 Zed 对照（2026-11-04，Sunrisepeak） + +服务端 5bdeddb 的完整 CI 41839033098 结束于六十二项成功、两项失败、 +一项取消及两项跳过。两项失败均为 U4 冷启动补全的引擎作答占比：mcpp 与 +xlings 各八次请求中七次来自引擎或其缓存，一次来自回退，占比零点八七五， +低于原零点九要求。两组各二十五项场景检查中的二十四项通过；暖启动、编辑、 +故障恢复阶段均通过，xlings 先前的一次暖启动额外重启降为零。mcpp 暖阶段 +连续五分钟零重启、一百七十六个模块零重建。mcpp 冷阶段 U1 为一百一十七点六 +秒，首次正确补全为一百二十一点五秒，仍不能代替占比检查的失败。 + +旧轮次各有一次冷补全回退，但冷阶段持续五或六轮，观测占比约百分之九十四或 +九十五；本轮各持续两轮。该原始分母差异不能直接证明产品退化，也不能作为 +降低门禁的理由。早期请求日志已被既有轮转淘汰，尚不能定位单个回退的精确 +原因。当前服务器仍采用冻结 cfd 引擎，新引擎构件的联合验收仍然必要。 + +取消项为 macOS 洁净安装，GitHub 记录十五分钟内多次未获得 hosted runner， +没有执行测试步骤。另一 macOS 一致性作业通过。该容量问题与两项真实冷占比 +失败分别保留，未合并为同一种原因，也不将缺失结果计为通过。 + +Zed 原观察器要求进程名以 clangd 开头，而实际映像使用 okrun。新观察器按安装 +清单核对可执行文件、准确映像参数及实际服务器的直接子进程，排除死亡进程并 +区分共用装入器的服务器与引擎。专项 41851760562 使用 41839033098 的准确 +payload、stage 构件，源码与 5bdeddb 相同，仅调度包装不同。两项真实 GUI +作业及进程控制均通过：推荐配置十一秒、默认配置八秒。协议初始化和默认配置 +下独立 clangd 的原断言保留；该专项不代替完整联合资格。 + +相关项目缓存清理保留每个宿主最近目录，删除已确认无活动进程使用的两个旧 +服务端目录，回收 1800843264 字节。无关项目和仍供验证使用的构建目录未删除。 +第二十五份永久证据归档为 mcpp-885-owned-ledger-scan-evidence.tar.gz， +12067999 字节，1827 个文件逐项核验，摘要为 +d8eeeac6aeb4f7541ad8348189111263d51194558d73475b4ca859c185e1888e。 +它覆盖首次账本扫描诊断，不含其后产生的显式移动修正及真实 Zed 专项结果。 + +## 79. 准确候选的冷 CPU 结果与后续优化（2026-11-05，Sunrisepeak） + +77e4e2f 的冻结构件在专项 41862659565 完成与正式门禁相同的三百零四单元、 +三个文件、每组一次冷启动及三次暖启动。两份原始构件均核对下载摘要、ZIP CRC +及准确源码，并以未修改的验收程序重算。Linux x64 冷 CPU 为八点四六秒， +原生七点零七秒，比值一点一九七，超过一点一五。该宿主的就绪时间、暖 CPU、 +完整生命期缺页指标通过；ARM 的全部原门禁通过。macOS 未获得 runner， +没有测量结果。这一候选不能进入联合载荷，不能提升发布账本。 + +准确 x64 原始日志的一百七十六个 worker 累计 CPU 为原生二四九八毫秒、 +映像三一五三毫秒；这是逐项日志之和，不是中位数乘以数量。本地同图冷采样 +同样显示 worker 成本差额，并将位流读取识别为热点。采样及 syscall 跟踪的 +CPU 和时间均带观察开销，不作为资格数字。 + +规范路径解析的属性预查询产生约六点七万次 fstatfs。七组交替局部对照表明， +省去查询可将同量成功符号链接解析的 CPU 从零点一二一零六二秒降为零点 +零八七八三六秒，节省约零点零三三秒；该收益不足以单独解决完整门禁，暂未 +增加 musl 发布链。账本读取块从十六槽扩大到六十四槽的映像对照没有收益， +同样未进入产品补丁。 + +新通用补丁 0046 仅要求 Read 与 ReadVBR64 在调用处内联，位提取、补充输入、 +溢出及错误处理的函数体保持。以准确候选产生的全部一百七十六个 PCM 为输入， +每次完整解码四遍，每组原生和映像各七对交替调用，所有解码累计值一致。 +映像 CPU 中位数由零点二四七七一一秒降为零点一六六八九九秒，比值约零点 +六七四；原生比值约零点九九六。该组件结果不等于完整冷启动降低同一比例。 +上游原有位流读写十八项测试通过，候选头文件与被测组件仅有注释及换行差异。 + +引擎新候选 6e3749f 的完整构建为 41876171347，新增补丁保持 stabilizing。 +此前仅在非活动构建脚本准备的 VFS 覆盖已移至实际 ci/ci_build.sh，构建并运行 +SupportTests 的 VFS 过滤、ClangdTests 的 FSTests，以及本轮 BitstreamTests。 +本地完整应用次序的准确源码树相同，仍须由这一轮实际 CI 验证集成与性能。 + +服务端 161d178 的 41860882938 已终止，三十项成功、一项准入失败、四项 +macOS 取消、十四项跳过。新的十二项准入对抗控制在实际 CI 通过；准入作业 +正确拒绝上游 macOS 取消，未组装任何联合载荷。原生构建可以先行，载荷仍以 +完整上游身份和结果为依赖；下一次准入必须改用修正后的准确引擎及构建轮次， +不能仅重试旧候选的取消项。 + +## 80. 位流候选的完整结果与冷启动采样（2026-11-05，Sunrisepeak） + +6e3749f 的完整构建 41876171347 已结束，十三项成功，一项 macOS 原生构建因 +runner 容量取消，依赖它的 warm-prime 矩阵跳过。Linux 两架构原生构建、 +Windows 原生构建、两架构映像、通用载荷及四宿主基础映像验收通过。准确 Linux +原生日志确认执行二百九十八项选定 ClangdTests、二十二项 VFS 测试及十八项 +位流测试，全部通过；这不是全部 ClangdTests 的覆盖声明。 + +同一候选的冻结专项 41884872035 完成十六次语义会话。Linux x64 冷 CPU 为 +映像十五点零七秒、原生十二点零八秒，比值一点二四八；ARM 为十一点五七秒、 +九点七零秒，比值一点一九三，均超过一点一五。其他就绪、暖 CPU 和缺页门禁 +通过。两份原始报告经摘要、ZIP CRC 和准确输入身份核对，并由原验收程序重算。 +完整生命期 CPU 仅供诊断，不能代替原就绪 CPU 边界；跨轮原生绝对时间不同， +也不足以据此断言源码退化。本候选保持不合格，不重跑抽取较好结果。 + +服务器 80e7c70 的 41876772794 为三十项成功、一项上游准入拒绝、三项 macOS +取消及十四项跳过。准入正确拒绝缺失上游资格，未组装联合载荷。引擎账本仍为 +stabilizing，编译常量与正式载荷锁没有提前切至未发布构件。 + +本地以准确构件重做三百零四单元、三个文件、一百七十六个 worker 的完整冷 +采样。所有映像帧均按准确 ZIP 成员偏移及 ELF 符号解析，无未解析映像帧； +位流 Read 和 ReadVBR64 自身已不再占据主要热点。输入重放在包含调用者的 +采样中占一百九十六个样本，分类及路径规范化分别占六十八和六十个，存在重叠， +不能相加为独立成本。采样与 syscall 跟踪均改变时间和 CPU，仅用于定位。 + +最新 x64 整体构建缓存 10014569779 已保存并核验后，删除旧快照 10006490473， +记录大小一三三九六七五六五五字节。同一 PR 的各架构保留最近快照，Windows +分散编译对象及其他项目、分支的缓存保持。该大小不等于已刷新的物理用量统计。 + +## 81. 操作内重放复用的修正与验证范围（2026-11-05，Sunrisepeak） + +新通用补丁 0047 将缓存存入时已经完成的全量输入重放记入当前操作 epoch， +同一次操作首次查询不再重复审计；下一次操作及构建后的 preamble 审计仍重新 +验证。命中快速路径前明确检查取消状态。对照在准确旧代码的原生及映像中均 +复现同一 epoch 的已取消查询仍返回命中，新代码均拒绝。 + +连续相同路径的观测共用一次规范化及来源分类，逐项文件系统操作、别名关系、 +包标记检查和计数保持。重复事实在重新分类前按原完整相等关系去重。最终被测 +两个头文件与补丁源码逐字节相同；四十七个通用补丁先于六个映像补丁的应用树， +与本地映像补丁先行的源码树相同。 + +局部测试使用真实记录器、重放器、缓存、Context、Cancellation 及 LLVM Support， +以五百个内存文件、三千条观测隔离元数据成本，不等于完整编译器或真实磁盘。 +原生和公开平台映像各七组交替对照，最终映像重放 CPU 比值零点九零九， +存入后首次查询零点七零四；原生分别零点八九一及零点七二六。单纯重放的 +三十六万项 project 计数、六万次 status、六万次 open 均相同。内容、负依赖、 +工作目录和取消控制通过，编译安全选项没有变更。 + +引擎提交 31e3513 的完整构建为 41891364456。服务端 605a532 的 41891564035 +仅更新这一准确候选引用，继续将独立编译与载荷准入分开。新增集成测试验证 +首次存入的操作范围及命中后取消；实际入口同时执行既有来源分类测试。 +完整性能及联合资格仍待当前源码 CI 验证，局部收益不提升发布账本。 + +## 82. 80e 服务器构建计数更正（2026-11-05，Sunrisepeak） + +逐项统计保留的 41876772794 终态快照，共四十九个作业，其中三十一项成功、 +一项准入失败、三项取消、十四项跳过。第八十节的“三十项成功”为计数错误， +更正为“三十一项成功”。上游准入仍被拒绝，未组装载荷，资格判断不变。 +第七十九节所述 161d178 的另一轮三十项成功不在本次更正范围内。 + +## 83. 重放修正的实际集成与链接优化后备（2026-11-05，Sunrisepeak） + +31e3513 的准确 Linux x64 原生作业 126789382644 通过三百项选定 ClangdTests， +其中新增的首次存入重放范围、命中后取消及既有来源分类用例均在原始日志确认。 +二十二项 VFS、十八项位流、五项命名空间及两项正向 memo 测试通过。 +Linux ARM 原生与两个 Ubuntu 20.04 兼容底线作业亦成功。准确 x64 包三百一十个 +文件校验、ELF 摘要、四十七补丁及 overlay 身份、PR 合成提交的父节点与源码树 +均核对。完整映像性能仍待构件齐备；macOS 原生仍为无 runner 的容量取消。 + +独立后备实验仅增加 ThinLTO，保留 O3、自动清零、栈保护、栈冲突保护、帧指针、 +libc++ hardening 及控制流保护。原始源码逐字相同，编译参数按顺序核对后仅多 +一项 -flto=thin。一百七十六个准确 PCM 每次完整解码四遍，九组交替对照的 +两个翻译单元 LTO CPU 比值为零点八四六；同时将真实 LLVM Support 以 LTO +编译后为零点八三五，累计解码值一致。实际 LLVM 库交叉构建一百八十九步完成， +归档成员确认为 bitcode，最终映像封装和执行通过。该结果不是完整 clangd 的 +LTO 性能或资格，当前运行的 31e3513 没有因此改变。 + +后备本地提交 03a3a8c 为映像 LLVM 启用 ThinLTO，链接限制两个后端作业； +优化模式同时写入生成工具链的身份，旧构建树因此重新配置。准确重建的 okgen +通过使用记录型 CMake、Ninja 适配器的模式切换控制，确认旧状态被清除、同模式 +不重复配置、归档次序保留。该控制验证生成器流程，不代替完整 LLVM 配置。 +实际 LLVM Support 的另一项交叉构建独立验证 LTO 参数及原安全选项。 +该提交仅在本地保存，没有推送，也没有提升任何发布账本。 + +第二十九份永久证据归档为 mcpp-885-replay-audit-evidence.tar.gz， +60510575 字节，3312 个文件逐项核验，摘要为 +3fa9a31dafd751abe63064c942982bc392d10a5f71ad734cbde6940dd56986e4。 +它保留 6e 冷 CPU 失败的原始 ZIP、准确 bundle、完整 perf 数据，以及重放修正 +的源码与组件构件；早于第八十二节的计数更正及其后的真实集成和 LTO 实验。 +核验第二十八、二十九份归档中的准确可执行文件后，删除两个过期组件 target， +记录逻辑大小 205366817 字节，保留最新重放候选构建目录及所有源码、控制结果。 + +## 84. 重放候选的性能边界与冷补全回退证据（2026-11-05，Sunrisepeak） + +31e3513 的完整构建 41891364456 已结束，十三项成功、一项 macOS 原生容量 +取消、一项依赖矩阵跳过；基础四宿主映像验收均通过。其冻结对照 41900280136 +的两份 ZIP 均校验摘要、CRC 和源码身份，并以原程序重算。x64 冷 CPU 为 +十六点二四秒对十三点六四秒，比值一点一九一。ARM 冷 CPU 为十一点六三秒 +对九点八五秒，比值一点一八一，暖 CPU 同为一点一八一；冷、暖就绪时间 +分别一点一五五、一点一五四，亦超过一点一五。十六次语义会话及缺页门禁通过。 +这一候选不合格；原生绝对时间随宿主变化，不据此声称源码性能退化。 + +前轮服务端 605a532 的 41891564035 逐项统计为三十三项成功、一项准入拒绝、 +一项 macOS 取消、十四项跳过，没有载荷组装。进入后备优化前，旧性能采集器 +与容量观察器均已关闭；因真实性能失败，没有重试 macOS 容量取消。 + +冻结日志专项 41897714970 使用准确 5bdeddb 服务器与 cfd2243 引擎的原载荷、 +测试工具及 kit，三个构件均核对准确源码、轮次、摘要及 ZIP CRC。只增加 +Python 日志观察：运行时以只读、不跟随链接的描述符保留有界日志 inode， +子进程退出后才复制内容。五项真实文件控制通过，命令、请求、预算和原 +conformance 可执行文件保持。mcpp 保留八个文件 40950189 字节，xlings +九个文件 45296797 字节，均无上限触发、错误或文件截断。早期请求因此可见。 + +该专项直接确认 execute.cppm 的首次补全：clangd 请求编号六在十九时三十四分 +五十七点三七零秒收到，三一四二毫秒后答复。请求内部计时三一二一毫秒，其中 +输入检查一六零八毫秒，一一五八项逐项检查、九十五次包标记检查覆盖三二一项 +观测，三二一项系统输入已检查，等待模块为零。服务器明确记录权威引擎结果在 +三一五一毫秒替换 native 回退。源码的常规补全预算为一千毫秒；这确认了先 +回退、后以引擎答案替换的路径，不是仅按请求总数作出的猜测。 + +这不是找回已轮转丢失的旧轮次日志，也不是新引擎资格。新专项共十四项冷阶段 +检查，十三项通过。mcpp 七轮的补全占比为二十二比二十三，xlings 三轮为十比 +十一，均通过；mcpp 首次跳转一五二点四六秒超过一百五十秒，仍失败。原完整 +轮次两轮的七比八占比失败保留，未按新分母改判，也未修改门禁。 + +映像 ThinLTO 后备提交以相同源码树进入主候选 39f8589，完整构建为 +41901404153。原生基线、四十七通用补丁、六映像补丁、安全选项及门槛保持。 +服务端 5392412 的 41901990586 仅更新准确候选引用，仍先完整准入再组装载荷。 +当前完整性能和联合资格待验证；没有发布、账本提升或正式常量、载荷锁提前迁移。 diff --git a/docs/20-toolchains.md b/docs/20-toolchains.md index db84919e4..f0119782b 100644 --- a/docs/20-toolchains.md +++ b/docs/20-toolchains.md @@ -370,8 +370,7 @@ Pinned toolsets coexist with each other and with a system Visual Studio. > refusing to use what is already installed would cost more than it buys: > Visual Studio is very often present and cannot always be redistributed. > `@system` for any other family is an error that names both things -> that may have been intended. (The family-less `[toolchain] … = "system"` — the PATH -> compiler — is a separate and deliberate escape hatch, and is unaffected.) +> that may have been intended. ### `[toolchain] … = "system"` — refused diff --git a/docs/zh/20-toolchains.md b/docs/zh/20-toolchains.md index b494ea157..a7dc2e063 100644 --- a/docs/zh/20-toolchains.md +++ b/docs/zh/20-toolchains.md @@ -341,8 +341,6 @@ toolset 之间、以及与系统 Visual Studio 之间都可以共存。 > 机器都用同一个编译器。Windows 是唯一一处「拒绝使用已经装好的东西」代价大于 > 收益的地方：Visual Studio 常常已经装了，又不总能重新分发。写 > `@system` 给其他任何一个族都会直接报错，并同时给出两种可能的意图。 -> （不带族的 `[toolchain] … = "system"`——即 PATH 上的编译器——是另一条、 -> 同样有意保留的逃生口，不受影响。） ### `[toolchain] … = "system"` —— 拒绝 From dedffc341c2603fb7f792b7f389b09e96632ca42 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 04:56:37 +0900 Subject: [PATCH 29/40] docs: record the #885 closeout review, measurements and design Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- ...026-11-07-ecosystem-885-closeout-design.md | 902 ++++++++++++++++++ .agents/docs/README.md | 4 +- 2 files changed, 905 insertions(+), 1 deletion(-) create mode 100644 .agents/docs/2026-11-07-ecosystem-885-closeout-design.md diff --git a/.agents/docs/2026-11-07-ecosystem-885-closeout-design.md b/.agents/docs/2026-11-07-ecosystem-885-closeout-design.md new file mode 100644 index 000000000..aaa0f455d --- /dev/null +++ b/.agents/docs/2026-11-07-ecosystem-885-closeout-design.md @@ -0,0 +1,902 @@ +--- +subject: ecosystem +status: active +--- + +# 生态 #885 收尾设计：核心、插件、openkal、语言服务与编辑器的一轮闭合 + +**读者：** 推进 mcpp、mcpp-plugins、openkal 家族与 mcpplibs.os、mcppls 与 mcppls-clangd、mcpp-ide、两个索引的维护者。 + +**本文回答的问题：** + +- 关闭 [mcpp#885](https://github.com/mcpp-community/mcpp/issues/885) 还差什么； +- 各仓库分别怎样改、按什么顺序发布； +- 怎样验收。 + +**范围之外：** + +- mcpp 本身以 ok 映像分发：用户已决定延后，分析保留在附录 B； +- mcppls #77 中与发布无关的性能优化实验； +- #887 之外新增的加速器厂商支持。 + +日期：2026-11-07。依据： + +- #885 正文与 2026-11-05、2026-11-06 两条补充； +- [实现依赖记录](2026-11-05-heterogeneous-build-implementation-plan.md)； +- [编译器尝试恢复](2026-11-06-compiler-attempt-recovery.md)、[BMI 发布与编译器上限](2026-11-06-bmi-publication-and-compiler-cap-closeout.md)（两份均位于 #889 分支）； +- [第 4 部分设计](2026-11-03-ecosystem-part4-beyond-native-design.md)； +- mcpp-index 的 `docs/mwrap.md`。 + +修订： + +- 第一版：质量审查与计划（原文件 `2026-11-07-ecosystem-885-closeout-review-and-plan.md`），提出 D1–D7，用户批准。 +- 第二版：测量 O1–O4，提出 D8、D9。 +- 第三版（取代并重命名原文件）：按用户决定把 mcpp 映像分发延后；把 mwrap 规范扩展为“头文件收敛到最小范围”； + 把审查、测量与计划整理为按仓库组织的设计。 +- 第四版：按用户决定，C 标准库的宏也收敛（新增 libc-m，4.4.1）；`msvc@system` 在本轮修复（C3）；级联架构 + 优化本轮实施（4.2，D12）；#889 按提交检查点组织（D11）；第 10 节重做多视角深度审查。 +- 第五版：mcpp-ide 统一版本语义，前两位为共享系列、末位留给各组件（D15）；补入运行入口图标与测试输出、 + 结果标识的设计（D16）；D10、D11、D13、D14 已批准；第 10 节按生态、架构、性能、交互、用户体验、agent 体验 + 重新审查。 +- 第六版（本文）：D12、D15、D16 批准；核心版本号改为 2026.11.7.1，继续使用 #889；第 11 节为实施计划与检查点记录。 + +## 1. 目标 + +本轮结束时同时满足： + +1. 核心 2026.11.7.1、插件 0.21.0、openkal 0.24.1 更正组、mcpplibs.os 0.3.1，以及新增的六个 mwrap 包装都已发布。 + 每一项都满足 D5 所说的“已发布”：GitHub、GitCode、索引、市场、中立环境安装。 +2. 已经在 GitHub 发布、但分发不完整的 mcppls 0.0.20 与 IDE 0.2.1 补齐分发。mcpp-ide 以 1.2.0 发布，内容包括： + - 统一的版本语义（D15）； + - 适配 mcppls 0.0.20 的载荷并兼容旧版（D13）； + - 运行入口图标与测试输出、结果标识（D16）。 +3. 头文件收敛到最小范围：第三方库与 C 标准库的头文件只出现在各自的 mwrap 包装里（D10）。 +4. 级联架构改为“精确版本只出现在平台版本组与锁文件”，并以一次演练证明叶子修复只需发布两个包（D12）。 +5. 所有已知缺陷要么已修复，要么有跟踪议题，并在用户文档中如实写为限制。 +6. #885 发布最终台账后关闭。 + +## 2. 现状（2026-11-07） + +| 仓库与工作线 | 状态 | 缺口 | +|---|---|---| +| mcpp 2026.11.5.1（#888） | 已发布，索引与 GLOBAL/CN 已验 | main 上的 `ci` `41960218833` 被取消后未重跑；修复分支 `ci/2026.11.5.1-macos-release-repair` 未删除 | +| mcpp 2026.11.7.1（#889） | `614227f1`，27 提交，src +1602/−185，tests +2009；`ci` 76 成功 1 失败 | Windows 交叉构建间歇失败（Q1）；范围过大（Q4） | +| mcpp-plugins 0.21.0（#52） | `79f7ba6d` CI 6/6 | 等核心公开后做最终集成；PR 正文过时 | +| openkal 0.24 组、os 0.3.0 | 已发布、已入索引 | musl#83、runtime#64、macOS 规则缓冲区、os 不认识扩展长度名字（Q9、Q1） | +| mcppls-clangd 0.0.20（#16） | 已发布 | 无；O4 证实其度量 | +| mcppls 0.0.20（#75） | 已发布；JetBrains 已补交（#78，run `42268066561`，HTTP 201，审核中） | GitCode 与 xim 索引仍是 0.0.19 | +| mcpp-ide 0.2.1（#4） | 已发布（含 JetBrains、Open VSX） | GitCode 与 xim 索引仍是 0.2.0；GUI 验证只在 Linux 上做过 | +| mcpp-index | — | `MCPP_VERSION` 为 2026.11.3.1，openkal 测试成员停在 0.23 组（Q10） | +| xim-pkgindex | okrun/okld/okprof 0.5.0 已入 | mcppls、mcpp-ide 条目落后 | + +## 3. 审查结论摘要 + +第一、二版的审查条目保留编号，正文以本节为准。 + +| 编号 | 结论 | 设计中的处置 | +|---|---|---| +| Q1 | #889 的 Windows 交叉失败由两个缺陷组成：A 类（扩展长度名字无法经 os 定位）是确定性的，已被 `8791c2ca` 绕开；B 类（记录发布瞬间读取出错被当成控制器已死）是 #889 引入的间歇性回归，失败率 10/40 | 4.1.1、4.2.3 | +| Q2 | “控制器已结束”的判定把瞬时读取错误等同于死亡（H1 已证实；H3、H4 已排除） | 4.1.1 | +| Q3 | `copy_atomic` 仍走同一个有缺陷的 os 调用，在 `settle`、备份恢复、缓存保留三处执行 | 4.1.1 | +| Q4 | #889 的范围远超“收尾” | D11 提交检查点，4.1.8 分系统审查 | +| Q5 | #839 已由 #841 修复但未关闭；#803 待处置 | 4.1.7 | +| Q6、Q7 | mcppls 0.0.20、IDE 0.2.1 分发不完整 | 4.5、4.6 | +| Q8 | 2026.11.5.1 发布后没有重跑 main `ci` | 4.1.9 | +| Q9 | openkal 0.24 组带着已知缺陷发布，发布门禁没有对照未关闭缺陷 | 4.2.3、A6 | +| Q10 | 索引与测试固定落后 | 4.7 | +| Q11 | 引擎冷 CPU 的通过来自产品本身（O4）；macOS 靠重跑通过的问题仍在 | 4.5 第 4 项 | +| Q12 | IDE 0.2.1 前向引用尚未发布的 2026.11.7.1 | 4.1.8、4.6 | +| Q13 | IDE #4 在 exact-head CI 被取消的情况下合入 | 4.8 | +| Q14 | #52、#75、#885 的公开记录过时 | 4.3、4.8 | +| Q15 | 证据在 /tmp（1711 项、41 GB，磁盘 100%） | D6，4.8 | +| Q16 | 已知限制缺少跟踪议题 | 4.1.3、4.8 | +| Q17 | 强杀持锁构建后，孤儿构建边与下一次构建竞争；#889 只修好了拆分调度 | 4.1.2、D8 | +| Q18 | 一个叶子修复引起六七级发布；两周 161 个家族版本中约 40 个只为跟随下层 | 4.2、D12 | +| Q19 | IDE 0.2.1 只认 mcppls 第 4 版载荷清单，遇到 0.0.20 的第 5 版载荷无法启动；载荷清单没有成文契约 | 4.6、D13 | +| Q20 | 映像引擎不带 zlib，读不了带 zlib 的 clangd 写出的索引分片，于是丢弃分片、重新索引并反复告警 | 4.5 第 4 项、D14 | +| Q21 | 各编辑器的运行入口只有文字：VS Code 是纯文字 CodeLens，Neovim 没有标识，CLion 的 gutter 图标没有验收 | 4.6.3、D16 | +| Q22 | VS Code 中测试没有输出：后端状态映射与 mcpp 契约不一致；通过的测试丢失输出；输出没有归属到测试项 | 4.6.4、D16 | + +## 4. 设计 + +### 4.1 mcpp 2026.11.7.1（继续使用 #889，按提交检查点组织） + +#889 继续作为唯一的核心 PR（D11）。每个检查点是一组相连的提交，满足三点： + +- 能单独构建，并通过它自己的本地验证； +- 在本设计记录中追加一段检查点记录：提交、验证、结论； +- 不改动其他检查点已经验证过的行为。 + +完整 CI、20 次重复验收与生态审查放在最后，对整个 PR 做一次。 + +| 检查点 | 内容 | 节 | +|---|---|---| +| C1 | 编译器事务记录协议 | 4.1.1 | +| C2 | 默认调度的构建代际 | 4.1.2 | +| C3 | Windows `build.mcpp` 的宿主工具链 | 4.1.3 | +| C4 | 平台版本组参与版本解析；默认平台改为版本系列 | 4.1.4 | +| C5 | 依赖与头文件收敛 | 4.1.5 | +| C6 | 默认平台、os 与启动器跟进 0.24.1 / os 0.3.1 | 4.1.6 | + +#### 4.1.1 C1 编译器事务记录协议 + +问题见 Q1–Q3，证据见附录 A.1。 + +1. **读取三分类。** 事务记录的每个读者（`controller`、witness、`.rc`、证书、指针）统一按三类处理： + - 名字不存在：返回“不存在”； + - 名字存在但打不开：在调用方既有的截止时间内有界退避重试； + - 内容损坏、或重试到期仍打不开：失败。 + + 不新增超时常量，也不放宽任何既有界限。 +2. **存活只看进程身份（D4）。** `controller_alive` 改为三值：活、死、尚不可知。记录暂时不可读时为“尚不可知”。 + 阶段一以已持有的 supervisor 身份继续判断；阶段二在既有启动宽限内继续等待。只有进程身份确认结束，并且再读 + `.rc` 仍然不存在时，才报控制器丢失。 +3. **单一命名空间（D2）。** 事务模块内的记录、备份与恢复，包括 `copy_atomic`，统一使用 `std::filesystem`。 + 这样既绕开 os 的缺陷，也与 os 0.3.1 的修复（4.2）互不依赖。 +4. **每个出口都留状态。** supervisor 的 `return 130` 补写 `.rc`。测量表明它不是失败原因，但违反了既有约定。 +5. **正式诊断。** 判定失败时输出一行：哪个条件成立、记录读取类别、两个身份的存活结果、日志与 `.rc` 的状态， + 不含路径与用户名。 +6. **回归与验收：** + - 在 Windows 扩展长度路径上实际执行 `settle` 的增量重建，以计数确认走到了 `copy_atomic`，补齐附录 A.4 + 没有证实的部分； + - 一个库层控制：打开与 rename 交错，旧实现失败、新实现通过； + - #890 的同一 workflow 重复 20 次全部通过，且没有出现判定诊断（D1）。修正无效时 20 次全过的概率约为 0.3%。 + +#### 4.1.2 C2 默认调度的构建代际（D8） + +默认调度的编译边也经 `__compiler-cap` 执行器运行。执行器登记带内核锁的参与者，并受同一个编译器许可约束； +下一次构建在上一代参与者全部结束前不进入构建目录，超出既有清理界限时明确拒绝。 + +不采用“把 Ninja 放进进程单元”：POSIX 上 Ninja 1.12.1 以 `POSIX_SPAWN_SETPGROUP` 把每条构建边放进各自的 +进程组，而 openkal-linux 的进程单元就是进程组。 + +验收： + +- 本机强杀控制在默认与拆分两种调度下各 75 轮，零失败，后续构建为空操作；Windows 原生 E2E 做同样的控制； +- `compile_commands.json` 与 `mcpp emit build-database` 的输出与改动前一致，仍是编译器本身的 argv，不带执行器 + 前缀。mcppls 与 mcpp-ide 读取这两份数据； +- O5：Linux 每条边约多 1 毫秒（附录 A.5）。Windows 以整次构建墙钟时间的变化报告。D8 的批准以此为前提。 + +#### 4.1.3 C3 Windows `build.mcpp` 的宿主工具链（Q16，D11） + +**现象。** 在默认使用 MSVC 的 Windows 宿主上，为裸交叉目标执行 `build.mcpp` 时，构建程序的宿主工具链解析把 +`msvc@system` 当作索引包，走进了安装路径。 + +**契约。** `@system` 表示使用本机已安装的工具链：解析器只做探测，探测不到就以“本机没有”拒绝，从不进入 +索引安装路径（`lifecycle` 中已有“mcpp never installs the machine's Visual Studio”的约定）。构建程序的宿主工具链 +与普通宿主构建走同一条解析，不另设分支。 + +**回归。** 把规范 ABI 布局 fixture 恢复为 #889 改动前的形式，即由构建程序提供 include 目录，在 Windows MSVC 宿主上 +为裸交叉目标执行。旧实现失败，新实现通过；32/64 位布局断言保持不变。 + +#### 4.1.4 C4 平台版本组参与版本解析，默认平台改为版本系列（D12 的核心部分） + +1. **版本组是约束，不再只是事后检查。** 目标图选定平台后，平台 `[platform.versions]` 中的每一项都以精确版本参与 + 该图的解析，与各请求方的约束取交集。 + - 交集为空时，用既有的 `platform-version-group` 拒绝，并同时写出平台与请求方； + - 只服务宿主的实例与来自目录或仓库的开发树，沿用现有的排除规则； + - 现有的 `step4b_check_platform_versions` 保留为断言：解析结果仍须满足版本组。 + + 这是 SPEC-013 的通用语义，不是 openkal 的特例。旧的家族包本来就精确固定，在新语义下结果不变。 +2. **默认平台是版本系列。** 目标表把 `x86_64-openkal`、`aarch64-openkal` 的默认平台从 + `mcpplibs.openkal-platform@0.24.0` 改为系列 `^0.24`： + - 解析时取系列中最新、且平台声明的 `mcpp >=` 下限不高于当前引擎的版本； + - 结果写入 `mcpp.lock`，`--locked` 与离线构建按锁文件复现； + - 平台规格的字符串形式 `name@<约束>` 接受 SemVer 约束，表格形式沿用依赖语法。 + + 此后平台发布补丁版本不再需要发布 mcpp。 +3. **生成的启动器以兼容范围依赖 os。** `distribution.cppm` 写出的清单由 `os = "0.3.0"` 改为 `os = "^0.3.1"`， + 具体版本由目标图与锁文件决定。这同时消除了“mcpp 用内置平台构建启动器，所以平台 CI 需要候选 mcpp”的自举 + 循环（`MCPP_SOURCE_REF`）。 +4. **规范与文档：** SPEC-013 的版本组与默认平台两节、`docs/35-platforms.md`、`docs/05-dependencies.md` 中英文同步。 +5. **验收：** + - 解析单测与 E2E 覆盖以下情形：索引中有更新的补丁但版本组指定旧版本，结果为版本组的版本；交集为空时拒绝并写出 + 双方；默认系列跳过下限过高的平台；锁文件复现；离线构建； + - 用候选索引，让 2026.11.5.1 与新引擎分别解析 0.24.1 组（4.2），结果一致； + - 级联演练见 4.2 第 4 项。 + +#### 4.1.5 C5 依赖与头文件收敛（D10） + +| 来源 | 现状 | 设计 | +|---|---|---| +| zlib、zstd、xz | `[dependencies.compat]`；`src/pack/compress_codecs.cpp` 与 `src/pack/distribution.cppm` 直接 `#include` | 依赖 `mcpplibs.mwrap.zlib`、`.zstd`、`.xz`（4.4）。这两个文件改为 `import`，宏常量取自 `.macros`，`LZMA_STREAM_INIT` 一类的初始化宏取自 `` | +| nlohmann/json | `modules/libs/src/json/json.hpp` 是复制进仓库的源码，由 `mcpp.libs.json` 包装，40 处 `import` | 依赖索引中的 `nlohmann.json`（3.12.0，原生模块包），删除复制的头文件与 `mcpp.libs.json`。`detail::json_sax_dom_callback_parser`、`unknown_size` 先核对索引包是否导出；不导出的改用公开 API，不重新复制 | +| C 标准库的宏 | 52 个文件的全局模块片段为宏引入 ``、``、``、``：`stderr` 203 处，`stdout` 79 处，`EOF` 20 处，`errno` 4 处实际使用，以及 `ENOENT`、`EACCES`、`ENOEXEC`、`EPIPE`、`BUFSIZ` | 只为函数而包含的直接删去，函数由 `import std` 提供。错误码改用 `std::errc`。`stderr`、`stdout`、`EOF`、`BUFSIZ`、`errno` 取自 `mcpplibs.mwrap.libc`（4.4.1） | +| 操作系统头文件 | 只在 `src/native/{posix,windows}/lock.cpp` | 保持不变，它们已经是按系统选择的最小区域 | + +另加仓库检查 `.github/tools/check_include_regions.sh`（与 `check_modules_wiring.sh` 同类）：`src/` 与 `modules/` 中除 +两个锁文件外不得出现 `#include`；测试目录不在检查范围内。 + +`nlohmann.json` 已由 mcpp-ide 以 GCC 16.1、由 mcppls 以 LLVM 23 实际构建并使用。 + +验收： + +- SPEC-014 §5 的容器往返测试与 pack E2E：gzip、raw DEFLATE、zstd、xz 的写出字节与改动前逐字节一致； +- json 的全部单测与 E2E 通过； +- 写到已关闭管道（EPIPE）与终端输出的现有测试通过； +- 四种工具链上的完整构建（风险见第 8 节）。 + +#### 4.1.6 C6 默认平台、os 与启动器跟进 + +4.2 的更正组公开后： + +- 默认系列解析到 openkal-platform 0.24.1； +- mcpp 自身与生成的启动器依赖 os `^0.3.1`。mcpp 有 25 处 `os::fs` 读、写、rename、stamp、identity、lock 调用，在 + Windows 上同样受扩展长度名字影响，所以 mcpp 自身也要跟进，不只是启动器。 + +#### 4.1.7 议题处置 + +- #839：以附录 A.3 的数据关闭，注明由 #841 修复；强杀残留由 C2 处理。 +- #803：外部贡献，与新的许可设计重叠。审查后给出合并方式，或说明后关闭。 +- #840、#843：过期探针，说明后关闭。 + +#### 4.1.8 分系统审查与跨仓库验证 + +按检查点 C1–C6，以及原有的启动配置与具名测试运行器、下载进度截止时间、中英文文档与规范，逐项记录发现、证据与 +结论。 + +跨仓库验证： + +- mcpp-plugins CI 以 `mcpp_source_ref` 指向候选； +- mcpp-ide 用候选核心跑 0.2.1 的调试启动场景，验证 `debuggee` 选择与旧引擎降级（Q12）； +- 发布 canary 覆盖 mcppls 与 xlings； +- 候选索引中全部 openkal 消费者在新解析语义下通过（mcpp-index 的跨命名空间候选测试）。 + +#### 4.1.9 发布 + +依次完成： + +1. 最终 CI → 生态审查 → 合入； +2. 取消 main CI → `release.yml` → 本地 `gtc` 补 GitCode → xim-pkgindex bump → 确认 CDN → GLOBAL/CN 安装； +3. 重跑 main `ci` 并记录 run，同时补上 2026.11.5.1 遗漏的那一次（Q8）； +4. 删除修复分支。 + +### 4.2 openkal 家族：0.24.1 更正组与级联架构（D12） + +#### 4.2.1 问题 + +附录 C 的统计：10-24 到 11-07 两周内，家族发布了 161 个版本。其中约 40 个只为跟随下层的新版本而发，自身源码未变： +runtime 约 23/27、musl 约一半、os 约 5/12。 + +根因有四条： + +1. 家族包之间写裸版本。裸版本等于一个已发布版本时按精确匹配处理，所以每一层都精确固定下层。 +2. 平台版本组只做事后检查，不参与选版本。 +3. mcpp 把默认平台编译进目标表，把启动器的 os 版本编译进生成的清单。 +4. 消费方在多处写死平台版本（mcppls-clangd 的 workflow 中有四处）。 + +结果是一个叶子修复要依次发布 image → musl → runtime → platform → os → mcpp，每一级都要 CI、GitCode、索引 PR +和约 5 分钟的 CDN 等待，最后还拖动 mcppls、mcppls-clangd、IDE 重新验证。 + +#### 4.2.2 架构：精确版本只出现在两处 + +| 编号 | 改动 | 所在 | +|---|---|---| +| A1 | 家族包之间改用兼容范围：musl 写 `openkal-linux = "^0.25.0"`，runtime 写 `openkal-musl = "^0.28.0"`，os 写 `openkal = "^0.24.0"`。补丁版本承诺可以直接替换：openkal 的 C ABI 由规范固定；C 库与 C++ 运行时的补丁不得删除或改变已导出的符号。家族现有 CI 只检查规范的数据布局，没有检查导出集合，所以 A1 同时新增导出集合比对：用 `llvm-nm` 比较新旧两版归档的导出符号，补丁版本只允许新增 | 各家族仓库的 0.24.1 PR | +| A2 | 平台版本组参与解析 | mcpp C4 | +| A3 | 默认平台为版本系列并写入锁文件；启动器以范围依赖 os | mcpp C4 | +| A4 | 消费方只在一处声明平台：mcppls、mcpp-ide 用系列加锁文件；需要以精确产物做资格的 mcppls-clangd 保留精确版本作为资格身份，但收敛到一处，按自己的节奏升级 | 各消费方 PR（4.5、4.6） | +| A5 | 一个版本组只开一个索引 PR：组内所有新版本与平台补丁一起提交，用跨命名空间候选测试一次验完，只等一次 CDN | mcpp-index | +| A6 | 家族发布 workflow（放在 openkal-platform，即版本组的所在）：输入版本组，核对每个成员的标签与 GitCode 镜像，生成 A5 的单一索引 PR，运行新旧两版 mcpp 的候选解析，并把各成员仓库未关闭的缺陷列入发布说明 | openkal-platform | + +精确版本此后只出现在两处：平台的版本组，以及各项目的锁文件。叶子修复的路径变为“叶子包 + 平台补丁”，两者以 +一个索引 PR 同时发布；musl、runtime、os、mcpp 与各消费方都不必跟着发布。只有当叶子以二进制形式进入别的产物时 +才需要重建，例如 okrun 静态链接系统实现，这时由 image 发补丁。 + +#### 4.2.3 0.24.1 更正组（本轮） + +| 包 | 修正 | 回归 | +|---|---|---| +| openkal-musl（musl#83） | 目录名缓存保留完整名字；放不下时记录溢出，改用目录自身身份解析，不再拼出截断的绝对路径 | 议题中 16 棵子树、每棵 10 个 72 字节分量的深树，`remove_all` 等相对操作 | +| openkal-llvm-runtime（runtime#64） | Windows 带卷前缀的路径判为绝对路径 | 议题中的最小程序，原生 MSVC 与映像运行时各跑一次 | +| openkal-macos | 隔离规则的缓冲区按授予数与路径上限计算容量；放不下时返回表示容量的错误码，而不是 `invalid` | 16 个最长路径授予的实际启动 | +| mcpplibs.os | `locate()` 及经它定位的所有 `os::fs` 操作接受扩展长度名字：`\\?\X:\…` 按 `X:/…` 匹配，`\\?\UNC\server\share\…` 按 `//server/share/…` 匹配 | Windows CI 以扩展形式名字覆盖读、写、rename、stamp、identity，并固定附录 A.1 的读取对照 | + +1. 每个家族仓库一个 PR，按提交检查点组织：修正 → A1 范围 → 发布准备。 +2. 这一组是最后一次逐级的精确级联，之后由 A6 一次发布：platform 0.24.1 的版本组名列全部新版本，os 0.3.1 随组发布。 +3. 合入后按家族约定把 main 强推到 `openkal-0.24`。 +4. **级联演练（D12 验收）。** 0.24.1 公开后，在候选索引上虚拟发布一个只改测试的 openkal-ld 补丁和对应的平台补丁， + 确认只有这两个包变化，新旧两版 mcpp 都解析到版本组，并记录所需的发布数与耗时。 +5. **过渡兼容。** + - 2026.11.5.1 没有 A2。对 0.24.1 组，A5 的原子发布保证“索引中最新的补丁”就是版本组的版本，旧引擎经事后检查 + 同样通过。 + - 0.24.0 及更早的组仍是精确固定，不受影响。 + - 旧引擎只有在一种情形下会被拒绝：显式固定较旧的平台，同时没有锁文件，又遇到之后发布的补丁。拒绝信息已经指出 + 应选哪个平台版本，`docs/35-platforms.md` 写明升级到 2026.11.7.1 即可解决。 + +os 0.3.1 是所有 os 消费者的修正。mcppls、mcpp-ide、mcpp-plugins 目前没有显式构造扩展长度名字，但这种名字也可能 +由系统 API 返回，所以它们各自的本轮 PR 都跟进 os `^0.3.1`。 + +### 4.3 mcpp-plugins 0.21.0（继续使用 #52） + +1. 立即更新 PR 正文：当前头 `79f7ba6d`、已验证范围、未验证组合。 +2. `src/plugins.cppm`、`src/tool.cppm` 中为宏引入的 C 标准库头文件改用 `mcpplibs.mwrap.libc` 与 `std::errc`（D10）。 +3. 核心 2026.11.7.1 公开后，把正式 CI 的固定版本改为它，并加入“子工程安装静态库并构建独立可执行程序”的 + 集成回归。 +4. 最终 CI → 审查 → 合入 → 发布 → 索引 → 实际消费者验证。 +5. 以实际消费者证据关闭 #887。SYCL 归档的目标身份混合：在 mcpp 建立 `upstream-bug` 议题，并向 SDK 上游报告； + 插件文档列出这一限制。 + +### 4.4 mwrap 包装（新仓库与 mcpp-index） + +| 仓库 | 包 | 版本 | 唯一实现 | 消费者 | +|---|---|---|---|---| +| `mcpplibs/libc-m` | `mcpplibs.mwrap.libc` | 17.0.0（4.4.1） | 目标的 C 库 | mcpp、mcppls、mcpp-plugins | +| `mcpplibs/zlib-m` | `mcpplibs.mwrap.zlib` | 1.3.2 | `compat.zlib` | mcpp | +| `mcpplibs/zstd-m` | `mcpplibs.mwrap.zstd` | 1.5.7 | `compat.zstd` | mcpp | +| `mcpplibs/xz-m` | `mcpplibs.mwrap.xz` | 5.8.3 | `compat.xz` | mcpp | +| `mcpplibs/xxhash-m` | `mcpplibs.mwrap.xxhash` | 0.8.3 | `compat.xxhash` | mcppls | +| `mcpplibs/libarchive-m` | `mcpplibs.mwrap.libarchive` | 3.8.7 | `compat.libarchive` | mcppls | + +每个仓库都按 mwrap 规范实现： + +- 标签系列 `mwrap-`； +- 审计全部公开头文件，以 `export using` 导出原拼写的实体； +- 对象常量映射到 `.macros`，其余宏留在 `/macros.hpp>`； +- 带默认模板、中英两份 README、许可、API 与行为测试、installed consumer 测试； +- 首发通过 mwrap 的发布审计。 + +包名沿用 compat 的身份（例如 `xz`，而不是 liblzma），使包装与唯一的实现一一对应。CI 覆盖实际构建消费者的全部 +工具链：mcpp 用的 GCC 16.1（Linux 与 x86_64-linux-musl 交叉）、LLVM 23.1.3（macOS、Windows）；mcppls 各宿主的 +工具链与 openkal 映像工具链。libarchive 的公开 API 最大，用仓库 `tools/` 中的生成与审计工具做完整清单，不能只导出 +mcppls 用到的部分。 + +mcpp-index 增加 `pkgs/mwrap/*.lua` 与 `tests/mwrap/*`，候选检查通过后发布。 + +#### 4.4.1 libc-m：C 标准库的模块形态 + +**是否标准、跨平台。** 需要映射的内容分三类： + +| 类别 | 内容 | 处理 | +|---|---|---| +| ISO C，各平台都有 | `stdin`、`stdout`、`stderr`、`EOF`、`BUFSIZ`、`errno`，以及 `EDOM`、`ERANGE`、`EILSEQ` | 进入 libc-m | +| POSIX，ISO C 没有 | `ENOENT`、`EACCES`、`ENOEXEC`、`EPIPE`。Windows CRT 也定义它们，但取值不同 | 用 C++ 的 `std::errc`，它可移植地给出这些条件 | +| 函数与类型 | `fflush`、`fputs`、`fwrite`、`fprintf`、`getenv` 等 | `import std` 已经提供，不映射 | + +所以 libc-m 的内容全部是 ISO C，与操作系统无关。这正是它不该放进 mcpplibs.os 的原因：os 负责“std 没有说的 +操作系统能力”。也不叫 `mcpplibs.os.libc`：libc 通常指 C 库的实现（如 musl），而这个包不实现任何东西。 + +**为什么 `std.compat` 不够。** 命名模块不能导出宏，`std` 与 `std.compat` 只导出函数和类型；`stdout`、`errno`、`EOF` +在每个标准库实现中都是宏（例如 UCRT 中 `stdout` 展开为 `__acrt_iob_func(1)`）。 + +**命名与版本。** + +- 仓库 `mcpplibs/libc-m`，包 `mcpplibs.mwrap.libc`，符合 mwrap 的 `-m` 与 `mcpplibs.mwrap.` + 约定。不用 `mlibc`：这个名字已被 managarm 项目的一个 C 库实现占用，而且同样会被误读为实现。 +- 上游是 ISO C 标准。C++23 以规范性引用纳入 ISO/IEC 9899:2018（C17），所以版本取 `17.0.0`，在 + `[package.metadata.mwrap]` 中记录这一映射。C++26 改为引用 C23 时，发布 `23.0.0`。 + +**接口。** + +- 根模块 `mcpplibs.mwrap.libc` 以 `export import std.compat;` 加上 `.macros`，导入一处即得到完整的 C 标准库。 +- `mcpplibs.mwrap.libc.macros` 在 `mcpplibs::mwrap::libc::macros` 中给出： + - 对象宏：`stdin()`、`stdout()`、`stderr()` 返回 `std::FILE*`；`errno()` 返回 `int&`；`EOF`、`BUFSIZ`、`EDOM` + 等为 `inline constexpr`； + - ISO C 其余头文件中可以映射的对象宏，按 mwrap 要求完整审计后一并映射。 +- 不能映射的宏留在 ``：`assert`、`va_*`、`offsetof`、`setjmp`。 + +**唯一实现。** 实现是目标的 C 库，由 mcpp 的目标基准选定（glibc、musl、UCRT、macOS libSystem、openkal-musl、 +mingw）。mwrap 规范目前只写了 compat 与内置源码两种实现来源，需要在 mcpp-index 的 `docs/mwrap.md` 中补一句： +“C 标准库的实现由目标的 C 库提供”。 + +**测试矩阵。** 覆盖上述每一种 C 库与 libstdc++、libc++、MSVC STL 的组合，检查映射值等于头文件中的宏值，并实际 +写入 `stdout()` / `stderr()`。 + +**意义。** + +- 每个模块化的 C++ 项目都会遇到同一个缺口：std 模块拿不到 C 的宏。目前 mcpp 有 52 个文件、mcppls 2 个、 + mcpp-plugins 2 个，各自在全局模块片段里包含 C 头文件。 +- 独立成包后，各 C 库之间的差异只在一处审计和测试，消费者的头文件包含收敛为零。 +- 它对生态以外的模块化项目同样可用。 + +### 4.5 mcppls 与 mcppls-clangd + +1. **0.0.20 分发补齐：** + - 本地 `gtc` 镜像到 `xlings-res/mcppls`，包括 `src-0.0.20.tar.gz`，逐个下载复核 SHA256； + - xim-pkgindex 的 `mcppls`、`mcppls-nvim` 跟进到 0.0.20。0.0.20 起只有一个通用载荷 + `mcppls-payload-0.0.20.tar.gz`（57 MB，各宿主共用），载荷清单为第 5 版：`server.path` 是 `bin/mcppls.okb`， + 运行器按宿主写在 `platforms..runner`（`libexec/okrun//okrun`）。配方的 xvm 别名按这一形态 + 启动；0.0.19 及更早版本保留原来的按平台载荷与第 4 版别名； + - GLOBAL/CN 洁净室中实际安装，并启动一次服务。 +2. **本轮 PR（合入 main，随下一个版本发布）：** + - libarchive、xxhash、C 标准库宏改用对应的 mwrap 包装与 `std::errc`； + - 平台与 os 改为系列加锁文件（A4），os `^0.3.1`； + - 完整 CI 通过后合入，随本轮的 mcppls 新版本发布（D14）。 +3. **mcppls-clangd：** 四处 workflow 中的 `openkal-platform@0.24.0` 收敛到一处，作为资格身份（A4），并更新 + `ci/test_release_gates.py` 的检查。 +4. **索引分片与 zlib（Q20，D14）：** + - **现象（用户实测）：** 0.0.20 的会话日志对每个索引分片报 + “Error while reading shard …: Compressed string table, but zlib is unavailable”。 + - **原因：** 映像引擎由 `okgen` 以 `-DLLVM_ENABLE_ZLIB=OFF` 构建，原生引擎则静态链接 zlib。启用 zlib 的 clangd + 写出的分片压缩了字符串表，映像引擎读不了，只能丢弃分片、重新索引。写出这种分片的可能是 0.0.19 及更早的原生 + 引擎，也可能是共用 `~/.cache/clangd/index` 或项目 `.cache/clangd` 的其他 clangd。语义结果仍然正确，但缓存 + 失效、CPU 与时间浪费；只要有带 zlib 的 clangd 写入同一个存储，警告就会反复出现。日志中的 + “Rebuilding module …: no BMI for this command” 是上下文中首次构建 BMI 的正常信息，不属于问题。 + - **修正：** + - 映像引擎以 compat.zlib 构建，启用 zlib，使原生与映像引擎读写同一种分片，这符合第 3 部分的原生/映像对等 + 原则； + - mcppls 让引擎把索引存储放在 mcppls 自有、按引擎身份区分的缓存目录中，不与其他 clangd 共用全局目录， + 旧分片随缓存回收清理。 + - **回归：** 原生引擎写出的分片由映像引擎读取，以及反向读取，都不再报错；同一工作区先后使用两种引擎时， + 不触发重新索引。 + - **发布：** 映像引擎的改动要发布新的引擎版本 0.0.21，按恢复后的 1.15 严格门槛与新登记的抽样规则做资格； + mcppls 随后以 0.0.21 的引擎发布新版本，两者都按 D5 完成。 +4. **性能门槛方法（#77）：** + - 把 O4 的结果写入 #77； + - 在引擎仓库登记严格门槛的抽样规则，合入后适用于下一个引擎版本： + - 每个宿主至少 4 对交替的冷会话，取比值中位数； + - 同一运行机内的离散上限以 O4 的 ±0.03 为基准； + - 规定无效样本的判据； + - 只在测量无效时允许重跑，重跑仍需满足完整样本数。 + +### 4.6 mcpp-ide（含 mcpp-vscode、mcpp-clion、mcpp-nvim） + +mcpp-vscode、mcpp-clion、mcpp-nvim 与后端都在 mcpp-ide 仓库中（`mide-plugins/`）。独立的 `mcpp-community/mcpp-vscode` +仓库自 1.0.0 起已迁入 mcpp-ide，本轮在其 README 写明，并在执行前经确认后归档。 + +本节的全部改动以 mcpp-ide 1.2.0 一次发布（4.6.1 的统一版本的首个版本），按 D5 完成分发。 + +#### 4.6.1 统一的版本语义（D15） + +**现状。** 后端、CLion、Neovim 为 0.2.1，VS Code 为 1.1.1。VS Code 扩展从独立仓库的 1.0.0 延续了市场身份， +所以与其他组件不同号。同一个版本号看不出组件之间是否兼容。 + +**规则。** 版本为 `X.Y.Z`： + +- `X.Y` 是**系列**，由后端与全部插件共享，代表协议与共同行为。系列相同的组件之间保证兼容； +- `Z` 留给各组件**自己的补丁**，各自递增，互不影响。 + +例如 VS Code 1.2.3 与 CLion 1.2.1 都与后端 1.2.0 兼容。 + +**首个系列取 1.2。** 两个市场都要求版本单调递增：VS Code 已经到 1.1.1，系列必须高于 1.1；CLion 与后端从 0.2.1 +升到 1.2.0 也是递增。本轮的改动（4.6.2–4.6.4，以及 A4）一起作为 1.2.0 发布，四个组件的 `Z` 都为 0。 + +**兼容判定。** 插件启动后端时，除现有的协议号检查外，还检查双方的系列相同；不同时给出明确提示，写出双方的版本 +与安装方法，不再按版本号大小推断。 + +**发布与标签：** + +| 情形 | 标签 | 发布内容 | 分发 | +|---|---|---|---| +| 系列发布（所有组件 `Z = 0`） | `v1.2.0` | 全部组件 | 两个市场、GitHub、GitCode、xim-pkgindex 的 `mcpp-ide` 与 `mcpp-ide-nvim` | +| 某个组件的补丁 | `<组件>-v1.2.Z`，组件为 `backend`、`vscode`、`clion`、`nvim` | 只有该组件 | 只更新该组件的渠道：`backend` 对应 xim 的 `mcpp-ide` 与 bundle-all，`nvim` 对应 `mcpp-ide-nvim` | + +CHANGELOG 按系列分节，节内再按组件列出补丁。现有的 release workflow 改为按标签前缀选择要发布的组件，并校验该 +组件清单中的版本与标签一致。 + +#### 4.6.2 与 mcppls 0.0.20 的载荷兼容（Q19，D13） + +IDE 通过 mcppls 的载荷找到语言服务：后端 `languageservice.cppm` 的 `payload_command`、CLion 的 +`Locate.runnablePayload`、Neovim 的会话代码，都读取载荷清单 `server.runner` 与 `server.path`。这是第 4 版的形态。 +0.0.20 的第 5 版去掉了 `server.runner`，运行器改为按宿主写在 `platforms..runner`。结果是，IDE 0.2.1 遇到 +0.0.20 的载荷时，会直接执行 `bin/mcppls.okb`，因而失败。IDE 的 CI 固定使用 mcppls 0.0.19，所以没有暴露。根因是 +载荷清单没有成文契约。 + +1. **mcppls 一侧：** 在 `docs/specs` 中发布载荷清单规范，附 JSON Schema，纳入 `validate.py`。规范覆盖第 4、5 两版， + 给出确定的启动规则： + - 第 5 版：在 `platforms` 中取本宿主一项，命令为 ` `； + - 第 4 版：命令为 `[server.runner] `； + - 本宿主不在 `platforms` 中：拒绝，并写出载荷服务的宿主。 + + 规范随 4.5 的本轮 PR 合入。 +2. **mcpp-ide 一侧：** 后端、CLion、Neovim 三处读取器按规范同时支持两版；单文件的原生 `mcppls`（PATH、xlings） + 照旧支持。VS Code 经后端取得命令。宿主名到 mcppls 平台键（`linux-x64`、`linux-arm64`、`darwin-arm64`、 + `win32-x64`）的映射只在后端一处。 +3. **测试：** + - 单测以真实发布中的 `payload.json`（0.0.19、0.0.20 各一份）作为夹具，另加越界路径、缺少本宿主等负向用例； + - VS Code、CLion、Neovim 各对 mcppls 0.0.19 与 0.0.20 跑一遍； + - `kRequired` 不提高。 + +#### 4.6.3 运行入口的图标（Q21，D16） + +**现状：** + +- VS Code 在 `main` 上方用 CodeLens 显示纯文字“Run”“Debug”； +- CLion 已有 `McppRunLineMarker`，会在 `main` 与测试用例旁的 gutter 显示 ▶，但只在后端返回该文件的入口时才出现， + GUI 检查没有断言它存在； +- Neovim 没有任何入口标识。 + +**设计：** 后端已经给出每个文件的入口（程序的 `main` 行、测试程序的 `main` 行、各用例行）。三个编辑器用各自原生的 +方式呈现同一份数据，提示中写明目标、配置与平台。 + +| 编辑器 | 程序的 `main` | 测试的 `main` 与用例 | +|---|---|---| +| VS Code | CodeLens 改为带图标的 `$(play) Run`、`$(debug-alt) Debug`；当前文件是某个目标的入口时，编辑器标题栏的运行按钮（`editor/title/run`）提供运行与调试 | Testing API 的 gutter 图标（测试项已带 `range`），显示上次结果：通过、失败、未运行 | +| CLion | 保留 `RunLineMarkerContributor` 的 ▶，补上入口取不到时的原因日志 | 同上，图标随上次结果变化（已有 `getTestStateIcon`） | +| Neovim | sign column 显示 `▶`，`` 映射与 `:Mcpp run` 调用 | sign 按上次结果显示 ✓、✗；失败行进入 quickfix | + +**验收：** 三个编辑器的 GUI 或冒烟检查分别断言：程序 `main` 行有运行图标；测试 `main` 与每个用例行有图标；运行后 +图标随结果变化。VS Code 断言 CodeLens 标题中含图标，以及编辑器标题栏按钮存在。 + +#### 4.6.4 测试输出与结果标识（Q22，D16） + +**现象（用户实测）：** VS Code 中 GoogleTest 用例没有输出；带 `main` 的普通测试程序也没有输出。 + +**原因（读源码确认）：** + +1. **状态映射有误。** mcpp 的测试流状态为 `pass`、`compile_fail`、`run_fail`、`not_run`、`built`，后端 + `views.cppm` 的 `stream_record` 却按 `fail`、`skip` 映射。结果是 `run_fail` 与 `compile_fail` 都落入 + `errored`，超时也永远识别不出来。 +2. **通过的测试丢失输出。** 后端把 `compile_output` 与 `run_output` 合成 `output` 转发，但 VS Code 只在失败时把它 + 放进 `TestMessage`。通过的测试，其输出没有交给任何测试项。 +3. **输出没有归属到测试项。** 两条运行路径都以 `run.appendOutput(text)` 写入，没有传测试项。VS Code 只在整次运行 + 的终端里显示，点开单个测试或用例看不到输出。用例模式下，GoogleTest 每个用例的输出也没有按用例切开。 + +**设计：** + +1. **后端：** + - 状态映射与 mcpp 的契约逐值一致：`pass` → passed、`run_fail` → failed（`timed_out` 为真时为 timedOut）、 + `compile_fail` → failed 并标注编译失败、`not_run` → skipped（带原因）、`built` → skipped（仅构建）。旧值按 + 未知处理，并记一次日志； + - 每个测试记录都携带完整输出，并附 `outputTruncated` 标志：单项输出上限 1 MiB，超出时保留头尾并注明截断； + - 用例模式下，按框架的边界切分输出。GoogleTest 用 `[ RUN ]` 与 `[ OK ]` / `[ FAILED ]`； + Catch2 用已有的 JSON 报告。每个用例的输出与失败位置（`file:line`）随该用例的结果一起发出。 +2. **VS Code：** 每条结果都调用 `run.appendOutput(output, location, item)` 归属到对应项，无论通过与否。失败时 + `TestMessage` 带上位置，在断言所在行内联显示；普通测试程序的失败，从输出中识别 `file:line:` 作为位置，识别不到 + 时挂在 `main` 行。 +3. **CLion：** SM 测试树为每个程序与用例发出 `testStdOut` 与 `testFailed`（带位置），通过的测试同样可以看到输出。 +4. **Neovim：** `:Mcpp test` 的结果写入一个结果缓冲区，按测试分节；失败位置进入 quickfix；sign 显示结果。 +5. **对 agent 与 CI 的契约不变：** `mcpp test --message-format json` 已为每个测试（包括通过的）给出 `run_output`， + 这一点由核心测试固定下来。用例级切分属于框架知识，只在 IDE 后端里做，不进入核心。mcpp 的 `mcpp-usage` 技能文档 + 补一段按用例运行与读取结果的做法（`--print-launch -- --gtest_filter=…`、`--gtest_output=json`），供 agent 使用。 + +**验收：** 夹具工程包含 GoogleTest 用例（通过、失败、输出很多）、Catch2 用例、带 `main` 的普通测试（通过、失败、 +超时、编译失败）。三个编辑器分别断言： + +- 每个测试与用例都能看到自己的输出； +- 失败显示在断言行； +- 状态与 mcpp 的契约一致； +- 截断标志生效。 + +#### 4.6.5 其余事项 + +1. **0.2.1 分发补齐：** 本地 `gtc` 镜像到 `xlings-res/mcpp-ide`；xim-pkgindex 的 `mcpp-ide`、`mcpp-ide-nvim` 跟进到 + 0.2.1；洁净室安装验证。1.2.0 发布后再按 4.6.1 跟进。 +2. **跨仓库：** 用 4.1.8 的候选核心实测 `debuggee` 选择；启动配置的形状若有变化，随 1.2.0 一起适配。 +3. **原生宿主：** 0.2.1 的 GUI 检查只在 Linux 上做过。在 Windows 与 macOS 运行机上，用同一个 packaged ZIP 运行 + 稳定版检查。运行机提供不了 GUI 的项目，在发布说明与文档中逐项写为限制。 +4. **A4：** 平台与 os 改为系列加锁文件，随 1.2.0 发布。 + +### 4.7 索引 + +- **mcpp-index：** + - 核心发布后，`xpkgindex-update.yml`、`openkal-compat.yml`、`tests/openkal/pins.toml` 的 `MCPP_VERSION` 改为 + 2026.11.7.1； + - `tests/examples/openkal` 与 `openkal-platform` 成员移到 0.24.1 组； + - 加入 4.4 的六个 mwrap 包与 `docs/mwrap.md` 的补充； + - 0.24.1 组以 A5 的单一 PR 发布； + - 更新 CHANGELOG。 +- **xim-pkgindex：** mcpp 2026.11.7.1 的 bot bump（重开或空提交触发 CI，合入后删除机器人分支），以及 4.5、4.6 的 + 条目。 + +### 4.8 收尾（原 T10） + +1. 第 4 部分设计以带日期的更正块写入最终实测数字与实际版本组，不改写原文。 +2. openkal 多角度审查（规范一致性、拒绝路径、安全、并发、跨平台），结合 4.2 的结果逐项给出结论。 +3. 生态全局审查；xlings 洁净室真实验证：中立 cwd，`XDG_*` 指向洁净室，不用 shim。 +4. 建立 Q16 的议题：SYCL 归档、`msvc@system`（在 C3 中关闭）、os 扩展长度名字（在 os 修复的 PR 中关闭），以及 + 映像分发（D9）。 +5. 证据归档（D6）：仍被引用的证据打包到 /tmp 以外的持久位置，附 SHA256 清单。之后清理 `/tmp/mcpp-885-*` 与 + 相关项目的过期构建缓存，并关闭本轮的探针 PR #890、mcppls-clangd#17。 +6. 更新 #52、#75 的正文；在 #885 发布最终台账，逐项列出“实现提交 / 局部验证 / 最终 CI / 公开发布 / 镜像与索引”。 +7. 流程：exact-head CI 被取消时不得合入（Q13）；家族发布经 A6，自动列出未关闭缺陷。 + +## 5. 决定 + +| 编号 | 决定 | 状态 | +|---|---|---| +| D1 | #889 须在带结构化诊断的 20 次重复验收下全部通过才能合入 | 已批准 | +| D2 | 编译器事务模块的记录、备份与恢复统一使用宿主命名空间 | 已批准 | +| D3 | 核心等待 openkal 0.24.1 更正组与 os 0.3.1，以更正后的默认平台发布 | 已批准 | +| D4 | 存活只由进程身份判定，记录读取错误按类别处理 | 已批准，O1 证实 | +| D5 | “已发布”指 GitHub、GitCode、索引、市场、洁净室安装全部完成 | 已批准 | +| D6 | 证据持久归档，并以 SHA256 清单引用 | 已批准 | +| D7 | 严格性能门槛用预先登记的抽样规则；更换计量口径时并列报告新旧结果 | 已批准，O4 证实 | +| D8 | 默认调度的编译边经 `__compiler-cap` 登记为代际参与者 | 有条件批准：以 O5 证明对构建性能影响很小为前提 | +| D9 | mcpp 本身以 ok 映像分发延后，分析见附录 B | 用户已决定 | +| D10 | 头文件收敛到最小范围：第三方库只出现在各自的 mwrap 包装中；C 标准库的宏取自 `mcpplibs.mwrap.libc`（版本 17.0.0），错误码用 `std::errc`；操作系统头文件留在按系统选择的源文件中；mcppls 的收敛随本轮的 mcppls 新版本发布 | 已批准 | +| D11 | #889 按提交检查点 C1–C6 组织，清单内的事项（包括 `msvc@system` 与 D12 的核心部分）全部进入；清单外新发现的问题建议题 | 已批准 | +| D12 | 级联架构 A1–A6 本轮实施：精确版本只出现在平台版本组与锁文件；0.24.1 是最后一次逐级的精确级联；家族发布经 A6 一次完成，并列出未关闭缺陷 | 已批准，含 4.2.3 第 5 项的过渡规则 | +| D13 | mcppls 的载荷清单成文为规范（第 4、5 版），客户端按规范读取并同时兼容两版与单文件原生 mcppls；随 mcpp-ide 1.2.0 发布 | 已批准 | +| D14 | 引擎映像启用 zlib；mcppls 为引擎使用自有、按引擎身份区分的索引存储；本轮发布引擎 0.0.21 与对应的 mcppls 版本 | 已批准 | +| D15 | mcpp-ide 版本为 `X.Y.Z`：`X.Y` 是后端与全部插件共享的系列，同系列即兼容；`Z` 为各组件自己的补丁；首个系列 1.2；系列发布标签 `v1.2.0`，组件补丁标签 `<组件>-v1.2.Z` | 已批准 | +| D16 | 运行入口与测试结果以同一份后端数据、各编辑器原生的图标呈现；测试输出逐项归属，状态与 mcpp 契约逐值一致；用例切分只在 IDE 后端 | 已批准 | + +## 6. 执行顺序 + +```mermaid +flowchart TD + C1[C1 记录协议] --> R[4.1.8 分系统审查与跨仓库验证] + C2[C2 默认调度代际] --> R + C3[C3 msvc@system] --> R + C4[C4 版本组约束与默认系列] --> R + L[4.4 libc / zlib / zstd / xz 包装入索引] --> C5[C5 依赖与头文件收敛] + C5 --> R + K[4.2.3 musl / runtime / macOS / os 修正 + A1 范围] --> A6[A6 一个索引 PR 发布 0.24.1 组] + A6 --> C6[C6 默认系列解析到 0.24.1，os ^0.3.1] + C6 --> R + R --> S[4.1.9 发布 2026.11.7.1 与 main ci] + S --> DR[4.2.3 级联演练] + S --> PL[4.3 插件 0.21.0 与 #887] + S --> IX[4.7 索引跟进] + S --> IDE2[4.6 debuggee 实测与 mcpp-ide 1.2.0 发布] + MX[4.4 xxhash / libarchive 包装] --> LS[4.5 mcppls PR] + A6 --> LS + A6 --> IDEP[4.6 版本语义、载荷兼容、运行图标、测试输出、A4] + LSP[4.6 mcppls 载荷规范] --> IDEP + IDEP --> IDE2 + D[4.5 / 4.6 分发补齐] --> F[4.8 收尾与关闭 #885] + W7[4.5 门槛方法] --> F + DR --> F + PL --> F + IX --> F + IDE2 --> F + LS --> EN[4.5 引擎 0.0.21 与 mcppls 新版本 D14] + EN --> F + IDEP --> F +``` + +立即可以并行开始的有：C1、C2、C3、C4、4.2.3 的修正与 A1、A6 的 workflow、4.4 的六个包装、4.5 与 4.6 的分发补齐、 +门槛方法、mcppls 载荷规范、映像引擎启用 zlib。 + +关键路径为：4.2.3 → A6 → C6 → 4.1.8 → 4.1.9 → 4.3。C4 与 A1 互不阻塞，因为 4.2.3 第 5 项的过渡规则让旧引擎也能 +解析 0.24.1 组。 + +## 7. 验收与关闭 #885 的条件 + +- 1 节的五项目标全部满足，且每一项都按 D5 完成。 +- C1 的 20 次重复、C2 的强杀控制、C3 的 fixture 回归、C4 的解析测试按所列标准通过；O5 已报告。 +- 全部字节一致性测试通过（C5）；`check_include_regions.sh` 通过。 +- 级联演练表明叶子补丁只需发布两个包，且新旧两版 mcpp 解析一致。 +- mcpp-ide 1.2.0： + - 对 mcppls 0.0.19 与 0.0.20 的兼容性测试在三个编辑器上全部通过； + - 运行图标与测试输出的检查全部通过（4.6.3、4.6.4）； + - release workflow 按标签前缀选择组件，在 dry-run 模式下验证系列发布与组件补丁两种情形。 +- 引擎 0.0.21 与对应的 mcppls 版本发布；跨引擎读取分片不再报错。 +- main 上的核心 `ci` 已在发布后的 HEAD 上通过并记录 run。 +- Q16 的议题均已建立或关闭；#887、#839、#803、#840、#843、#890、mcppls-clangd#17 均已处置。 +- 第 4 部分写入最终数字；openkal 审查与生态审查已记录；洁净室验证通过。 +- 证据已归档，临时文件已清理。 +- 性能后续只在 #77 中保留；映像分发只在附录 B 与其议题中保留。 + +## 8. 风险 + +| 风险 | 影响 | 控制 | +|---|---|---| +| C1 修正后，B 类以别的形式重现 | 核心发布推迟 | 正式诊断保留判定原因；验收按 20 次重复 | +| 解析语义改动（C4）影响索引中其他包 | 普遍的解析回归 | 只对选定平台的版本组成员生效；以全部 openkal 消费者与 mcpp-index 候选测试验证；旧家族包精确固定，结果不变 | +| 范围依赖降低可复现性 | 同一源码在不同时间解析出不同版本 | 发布与资格以锁文件为准；mcppls、mcppls-clangd、IDE 的 CI 用 `--locked` | +| 补丁版本违背兼容承诺 | 消费方静默拿到不兼容的补丁 | A1 新增的导出集合比对；A6 在发布前跑新旧两版 mcpp 的候选解析与消费者测试 | +| 旧引擎遇到之后的补丁时拒绝 | 使用 2026.11.5.1 并固定旧平台的用户需要升级 | 拒绝信息已写明解法，文档说明；只在没有锁文件时出现 | +| libarchive 的完整 API 审计工作量大 | mcppls PR 推迟 | 不在核心关键路径上；未完成时该 PR 不合入，#885 台账如实记录 | +| `nlohmann.json` 与 `mcpp.libs.json` 的导出不同 | 迁移受阻 | 先核对导出差异；缺失的 detail 符号改用公开 API | +| C5 的大量机械替换（约 300 处） | 引入行为差异 | 按文件分组提交；终端、管道与 EPIPE 的现有测试，以及四种工具链上的完整 E2E | +| mcpp 的模块图新增 5 个模块（libc、zlib、zstd、xz、nlohmann.json）后，GCC 16.1 崩溃（#863 等先例） | C5 推迟 | 包装入索引后先以探针在四种工具链上构建；触发编译器缺陷时建上游议题并在实现中规避 | +| 更正组级联中又发现新缺陷 | D3 让核心等待 | A6 列出；属于已验证失败的修入，其他建议题 | +| D8 在 Windows 上代价偏高 | 默认调度变慢 | O5 先测量；不可接受时改换机制并重新请批 | +| mcpp-ide 版本从 0.2.1 跳到 1.2.0 | 用户与索引误以为是大变动 | CHANGELOG、两个市场与 xim 条目写明统一系列的原因；兼容由系列检查保证 | +| 测试输出很大 | 测试视图变慢、内存上涨 | 单项上限 1 MiB，保留头尾并标注截断 | +| GoogleTest 输出边界被程序自身的打印打乱 | 用例输出切分错误 | 以 GoogleTest 的 JSON 报告校正用例结果，边界只用于切分输出；切不开的部分归属到程序项 | +| 磁盘 100% | 本机构建与测量失效 | 先按 D6 归档，再清理；重任务经 `/tmp/mcpp-heavy.lock` 串行 | + +## 9. 不在本轮的事项 + +- mcpp 以 ok 映像分发（D9）：延后理由与先决条件见附录 B，并建立议题保存。 +- mcppls #77 中的可选性能实验。 + +## 10. 多视角深度审查（第五版） + +### 10.1 生态 + +| 检查 | 结论与处置 | +|---|---| +| 各仓库的职责是否清楚 | 核心只承担通用契约：版本组参与解析、默认平台系列、测试流。框架知识（用例切分）在 IDE 后端；载荷格式归 mcppls 规范；C 标准库宏归 libc-m；操作系统能力归 os | +| 已发布产物之间是否互相兼容 | 发现并处置了两处已有的不兼容：IDE 0.2.1 与 mcppls 0.0.20（Q19）、映像与原生引擎的索引分片（Q20）。旧 mcppls 与旧引擎继续可用（D13、D14） | +| 版本号能否表达兼容 | mcpp-ide 改为“系列 + 组件补丁”（D15）；openkal 家族改为“范围 + 版本组 + 锁文件”（D12）。两者都把“是否兼容”交给可判定的规则，而不是比较版本号大小 | +| 分发是否完整 | D5 覆盖核心、插件、0.24.1 组、os、六个 mwrap 包、mcppls 0.0.20 补齐与新版本、引擎 0.0.21、mcpp-ide 1.2.0 | + +### 10.2 架构 + +| 检查 | 结论与处置 | +|---|---| +| 核心是否新增特例 | 没有。C3 把 `@system` 写成通用契约；C4 是 SPEC-013 的通用语义；测试流契约不变 | +| 依赖方向 | 去掉复制的 json.hpp；每个 mwrap 包装只有一个实现；IDE 读取 mcppls 载荷依据成文规范，不再按实现反推 | +| 头文件最小范围 | 第三方与 C 标准库头文件只在各自包装中；操作系统头文件只在两个锁文件中；由 `check_include_regions.sh` 固定 | +| 一份数据，多处呈现 | 运行入口与测试结果都来自后端的同一份数据，三个编辑器只负责呈现（D16），避免三套各自推断的逻辑 | +| 补丁兼容承诺 | A1 新增导出集合比对；mcpp-ide 的系列检查与协议号检查并存 | + +### 10.3 性能 + +| 检查 | 结论与处置 | +|---|---| +| 构建 | D8 每条边约 1 毫秒（Linux），mcpp 自举约多 0.06 秒；Windows 待 O5；C1 的重试只在名字存在但打不开时发生 | +| 编辑器 | D14 去掉跨引擎的重复索引；测试输出单项上限 1 MiB，超出时保留头尾，避免 VS Code 测试视图因为超大输出变慢；用例切分是流式的，不缓存整次输出 | +| 发布 | D12 把叶子修复的发布从六七级降到两级；A5 只等一次 CDN | + +### 10.4 交互 + +| 检查 | 结论与处置 | +|---|---| +| 入口是否一眼可见 | 三个编辑器都用原生图标：VS Code 的带图标 CodeLens、标题栏运行按钮与 Testing gutter；CLion 的 gutter ▶；Neovim 的 sign | +| 结果能否就地查看 | 每个测试与用例的输出都归属到自己的项；失败显示在断言行；Neovim 的失败进入 quickfix | +| 状态是否可信 | 状态与 mcpp 契约逐值一致，编译失败、超时、未运行、仅构建分别呈现，不再一律显示为“错误” | +| 版本不匹配时如何提示 | 插件与后端的系列不同时，给出明确提示，写出双方版本与安装方法 | + +### 10.5 用户体验 + +| 检查 | 结论与处置 | +|---|---| +| 升级是否无痛 | 旧版 mcppls 与 IDE 继续可用；分片告警的根因在 D14 修复；版本号从 0.2.1 到 1.2.0 的跳变在 CHANGELOG 与两个市场的说明中写明原因 | +| 告警是否有意义 | `Rebuilding module …` 是正常信息；分片告警属于缺陷，修复后不再出现；后端对未知状态值只记一次日志 | +| 跨平台一致 | Windows 与 macOS 的 GUI 检查用同一个 packaged ZIP 补做；运行机提供不了的项目逐项写为限制 | + +### 10.6 agent 使用体验 + +| 检查 | 结论与处置 | +|---|---| +| agent 能否拿到与 IDE 相同的信息 | 能。`mcpp test --message-format json` 为每个测试（包括通过的）给出状态与 `run_output`，这一点由核心测试固定；IDE 的状态映射与它逐值一致 | +| 用例级信息 | 用例切分不进入核心，`mcpp-usage` 技能文档补充按用例运行与读取的做法（`--print-launch -- --gtest_filter=…`、`--gtest_output=json`） | +| 诊断能否被程序读取 | C1 的判定诊断是单行结构化文本，不含路径；构建失败原因、平台拒绝都有既有的 NDJSON 事件 | +| 版本与兼容能否被程序判断 | 版本组、锁文件与 mcpp-ide 的系列都是可判定的规则，agent 不需要猜测版本号大小 | + +### 10.7 其他 + +| 视角 | 结论与处置 | +|---|---| +| 正确性 | 每个修正都有“旧实现失败、新实现通过”的控制，包括 Q21、Q22 的 GUI 与冒烟断言 | +| 证据 | Q22 的三个原因来自 `views.cppm` 与 `testing.ts` 的源码；Q21 来自三处插件的源码。在实现前，以夹具工程在三个编辑器中复现用户报告的现象，并把复现结果写入检查点记录 | +| 自举顺序 | D3 与 A1 互不等待，理由见 4.2.3 第 5 项 | +| 编译器 | mcpp 新增 5 个导入模块，先在四种工具链上试构建 | +| 安全与隐私 | 诊断不含路径；附录只出现运行机路径的形状 | +| 流程 | 延后项只有用户决定的 D9；不可逆操作（归档仓库、强推、市场发布）在执行前单独确认 | +| 范围与工作量 | 新增的 D15、D16 只在 mcpp-ide，不在核心关键路径上；mcpp-ide 1.2.0 依赖 mcppls 的载荷规范与候选核心的 `debuggee` 实测 | + +审查后对正文做了三处修正： + +- 4.6.4 补入单项输出上限与截断标志，防止超大输出拖慢测试视图； +- 4.6.4 补入 agent 侧的技能文档，使 agent 与 IDE 拿到同一份信息； +- 第 7 节把“组件补丁演练”改为 release workflow 的 dry-run 验证，不为验收真的发布一个补丁。 + +## 11. 实施计划与检查点记录 + +### 11.1 每个仓库一个 PR + +| 仓库 | PR | 版本 | 内容 | 依赖 | +|---|---|---|---|---| +| mcpp | #889（标题改为 2026.11.7.1） | 2026.11.7.1 | C1–C6、第 4 部分实测记录、本设计 | 0.24.1 组与 os 0.3.1（C6）；六个 mwrap 包中 mcpp 用到的四个（C5） | +| openkal-macos | `openkal-0.24.1` | 0.22.1 | 规则缓冲区容量与错误码；A1 | 无 | +| openkal-musl | `openkal-0.24.1` | 0.28.1 | musl#83；A1；导出集合比对 | 无（范围下限取已发布版本） | +| openkal-llvm-runtime | `openkal-0.24.1` | 0.19.1 | runtime#64；A1；导出集合比对 | 无 | +| openkal-image | `openkal-0.24.1` | 0.5.1 | 运行器以 macOS 0.22.1 重建；A1 | openkal-macos 0.22.1 已入索引 | +| openkal-platform | `openkal-0.24.1` | 0.24.1 | 版本组；A6 workflow | 组内各成员的标签 | +| mcpplibs/os | `openkal-0.24.1` | 0.3.1 | 扩展长度名字的定位；A1 | 无 | +| libc-m、zlib-m、zstd-m、xz-m、xxhash-m、libarchive-m | 各自首个 PR | 见 4.4 | mwrap 包装 | 无 | +| mcpp-index | 按依赖分批的 PR，每批一个 | — | mwrap 包；0.24.1 组；`MCPP_VERSION` 与测试成员 | 各成员的标签 | +| mcpp-plugins | #52 | 0.21.0 | libc 宏；最终集成 | mcpp 2026.11.7.1 公开 | +| mcppls-clangd | 一个 PR | 0.0.21 | 映像启用 zlib；平台版本收敛到一处；抽样规则 | 0.24.1 组 | +| mcpp-language-server | 一个 PR | 0.0.21 | 载荷规范；mwrap；A4；引擎索引存储 | 引擎 0.0.21；xxhash-m、libarchive-m | +| mcpp-ide | 一个 PR | 1.2.0 | D15、D13、D16、A4 | mcppls 载荷规范 | +| xim-pkgindex | 每个产品的发布各一个 PR | — | mcpp、mcppls、mcpp-ide、nvim 条目 | 各自的发布 | + +mcpp-index 与 xim-pkgindex 的 PR 数量由发布依赖决定：一个条目只能在它引用的标签存在之后提交。 + +### 11.2 依赖波次 + +1. **立即并行：** + - 0.24.1 组中不依赖他人的四项：macOS、musl、runtime、os； + - 六个 mwrap 包； + - mcpp 的 C1、C2、C3、C4； + - mcpp-ide 的 D15、D16； + - mcppls 的载荷规范； + - 引擎的 zlib 构建； + - mcppls 0.0.20 与 IDE 0.2.1 的分发补齐。 +2. **macOS 0.22.1 入索引后：** image 0.5.1 重建运行器。 +3. **组成员标签齐全后：** platform 0.24.1 的版本组；以一个 mcpp-index PR 发布全组。 +4. **全组与 mwrap 包公开后：** mcpp 的 C5、C6；mcppls 与引擎改用新组；mcpp-ide 的 A4。 +5. **完整 CI 与生态审查后：** 发布 mcpp 2026.11.7.1；随后 mcpp-plugins 0.21.0、引擎 0.0.21、mcppls 0.0.21、mcpp-ide 1.2.0； + 各自的索引条目；洁净室验证；处置议题。 + +等待 CI 期间推进同一波次或下一波次中不依赖该 CI 的任务。 + +### 11.3 检查点记录 + +各检查点完成时在此追加：提交、验证命令与结果、结论。 + +## 附录 A 测量（2026-11-07） + +### A.1 Windows 交叉构建（mcpp#890） + +源码为 #889 头 `614227f1`。探针只保留 Windows 本机构建与 Windows→x86_64-linux-musl 交叉构建两个 job，后者用 +同一个已构建的二进制重复执行。 + +- **基线（run `42253775511`）：** 5/20 失败。其中 4 次为“compiler controller ended without a final status”， + 1 次为“compiler witness: cannot read compiler transaction record”，发生在构建开始后 42–557 秒。成功的 15 次 + 各耗时 863–1774 秒。 +- **诊断（run `42253941066`，仅附加临时插桩）：** 5/20 失败，统计如下。 + +| 观察 | 数值 | +|---|---| +| 以扩展长度形式（`//?/D:/…`）拼写的记录 | 4228 条，OpenKal 读取全部失败（`invalid`），定位到 preopen `//`，相对名含冒号 | +| 以普通形式（`D:/…`）拼写的记录 | 122731 条，全部成功 | +| 失败判定 | 5 次，均在阶段一；判定时 supervisor 尚未创建日志 | +| 其中 `controller` 记录读取出错 | 3 次；三秒后记录可读，控制器与 supervisor 均存活 | +| 其中探测时记录已可读 | 2 次；一例三秒后 `.rc` 为 0，编译实际成功 | +| supervisor 出口 | 4241 次启动中，4228 次走正常出口；其余 13 次是失败构建中止时仍在运行的；没有其他出口，没有 terminate | + +每次完整构建发布 254 条扩展形式的 `.rc` 记录，所以 `8791c2ca` 之前用 OpenKal 发布最终状态时必然失败。插桩中对 +盘符 `D:` 的 identity 查询失败，是探针把盘符当作路径传入造成的，不构成发现。 + +### A.2 同一产物上的 v2 与 v3（mcppls-clangd#17，run `42254645814`） + +已发布的 0.0.20：Linux x64 native 与 okrun 下的 `clangd.okb`，两者摘要与 release 一致。在 3 台运行机上各做 4 对 +交替的冷会话；每次就绪读数取 v3，并在其前后各取一次未修改的 v2（`ae3956b^`）。 + +- 24 个会话中 v2 与 v3 全部一致（精度 0.01 秒）； +- image/native 比值中位数 1.082，最小 1.041，最大 1.103； +- 额外 CPU 中位数 0.74 秒； +- native 绝对 CPU 为 8.46–11.03 秒，随运行机变化； +- 全部会话成功。 + +### A.3 并发构建与强杀（本机 Linux，GCC 16.1，#839 的 24 模块工程） + +| 场景 | 2026.11.5.1 | #889 | +|---|---|---| +| 两个存活构建，错开 0.1–0.5 秒 | 10/10 成功，后者等待构建目录锁 | 10/10 成功 | +| 强杀持锁者，默认调度 | 等待者失败 3/75 | 等待者失败 4/75 | +| 强杀持锁者，拆分调度 | 等待者失败 12/15 | 0/15，后续构建均为空操作 | + +#889 的 75 轮中，60 轮使用 `614227f1` 的构建，15 轮使用同一分支此前构建的同版本二进制。失败形式为 +“failed to read compiled module”或依赖文件缺失；强杀后进程表中可以看到收养到 init 的 +`sh -c cp … .bak; g++ …`。用 SIGINT 的对照无效：后台进程继承了被忽略的 SIGINT，所以没有计入结果。 + +### A.4 Windows 增量重建（mcpp#890，run `42264255917`） + +交叉构建成功后触碰 `src/*.cppm` 并重建。3 次中 1 次交叉构建本身以 B 类失败；另外 2 次增量重建都成功，约 +9 分 45 秒。没有插桩确认是否调用了 `copy_atomic`，所以本项只说明未观察到失败。 + +### A.5 进程启动代价（本机 Linux） + +`614227f1` 的 mcpp 连续启动 100 次（`--version`）共 0.07 秒，约 0.7 毫秒/次。 + +经 `__compiler-cap --sem --cap 8 -- /bin/true` 执行 200 次，共 0.24–0.29 秒；直接执行 `/bin/true` 200 次 +为 0.07–0.08 秒。执行器每条边多出约 1 毫秒，其中包括取得许可锁。mcpp 自举约 464 条边，按 8 路并行约增加 +0.06 秒，相对 2 分 36 秒的构建不到 0.1%。Windows 上的数值待 O5 测量。 + +## 附录 B mcpp 以 ok 映像分发（延后，D9） + +**可行性。** 已有先例：mcpp-ide 0.2.0 起只发一个 `openkal-bundle-all` 归档（0.2.1 为 4 MB）。归档内有两种架构的 +映像、各宿主的启动器与四个宿主的 okrun，xim-pkgindex 的三张表引用的是同一个 URL。mcpp 本身已经建立在 os 与 +openkal 之上，Windows 与 macOS 上用 LLVM 23 构建。 + +**先决条件：** + +| 编号 | 条件 | 现状 | +|---|---|---| +| P1 | 宿主相关逻辑改为运行期判断 | 约 95 处 `if constexpr (os::is_windows / is_macos / is_linux)` 在映像中都为假；须逐处区分宿主行为与目标行为，前者改用 `os::host()`；`lock.cpp` 改用 `os::fs::lock` | +| P2 | 用映像工具链（LLVM 23、libc++，openkal 运行时）构建 | Linux 上 mcpp 用 GCC 构建；#666（Clang 构建的 mcpp 在 ELF 检查中崩溃）未关闭 | +| P3 | openkal 与 os 在 mcpp 负载下正确 | 依赖 4.2 的全部修正 | +| P4 | 每个进程的启动代价可接受 | mcpp 每条编译边都会再启动自身；原生约 0.7 毫秒，经 okrun 的代价未测 | +| P5 | 映像以不受限的宿主权限运行 | 授予至多 16 个，须确认 okrun 对宿主程序的不受限启动方式 | +| P6 | 内置 xlings | 原生归档带各宿主的 xlings（20 MB）；xim 渠道宜改为依赖 `xim:xlings@` | +| P7 | 终端与交互 | `mcpp run` 交出终端、进度状态行、原始模式、Windows 控制台代码页，须在映像中实测 | + +**恢复时的第一步：** 测量 O6（经 okrun 的单进程启动代价）与 O7（两种架构的 mcpp 映像在四个宿主上跑 E2E 子集）， +再写独立的设计记录。xim-pkgindex 一侧只列 bundle-all 一个资源，其他渠道继续使用原生归档。 + +## 附录 C 家族发布统计（2026-10-24 至 2026-11-07） + +| 仓库 | 版本数 | 只为跟随下层的版本（按发布说明） | +|---|---:|---| +| openkal（规范） | 9 | — | +| openkal-linux | 13 | — | +| openkal-macos | 13 | — | +| openkal-windows | 11 | — | +| openkal-emscripten | 7 | — | +| openkal-musl | 27 | 约一半，例如 0.27.2、0.27.4、0.27.7、0.28.0 写明“C 库源码未变” | +| openkal-llvm-runtime | 27 | 约 23 个，多数写明“runtime 源码未变” | +| openkal-image | 21 | 若干为跟随系统实现重建运行器 | +| openkal-platform | 21 | 全部是版本组数据，这是平台的职责 | +| mcpplibs.os | 12 | 约 5 个 | +| 合计 | 161 | 约 40 个纯跟随版本 | + +同期 mcpp 的四个版本中，2026.11.3.1 与 2026.11.5.1 都包含默认平台的更新。 diff --git a/.agents/docs/README.md b/.agents/docs/README.md index c9efb5146..83417d657 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -352 records. +353 records. ## By subject @@ -75,6 +75,7 @@ Records that declare one. Everything else is listed by date below. ### ecosystem +- [生态 #885 收尾设计：核心、插件、openkal、语言服务与编辑器的一轮闭合](2026-11-07-ecosystem-885-closeout-design.md) — active - [生态设计 part 2：openkal 的最小原子与二进制映像、其上的 `mcpplibs.os`，以及 part 1 的收尾与回归记录](2026-10-26-ecosystem-part2-os-library-over-openkal-design.md) — active - [控制契约、并发与性能、单一 C++ 运行时，以及 IDE 生态（生态级，#838、#839）](2026-10-26-control-contract-runtime-and-ide-ecosystem-design.md) — active @@ -176,6 +177,7 @@ Records that declare one. Everything else is listed by date below. ### 2026-11 +- [生态 #885 收尾设计：核心、插件、openkal、语言服务与编辑器的一轮闭合](2026-11-07-ecosystem-885-closeout-design.md) — active - [Launch configuration and named test runners](2026-11-06-launch-configuration-and-named-test-runners.md) — active - [Compiler attempt lifetime and output recovery](2026-11-06-compiler-attempt-recovery.md) — active - [BMI publication, compiler capacity and active download budgets](2026-11-06-bmi-publication-and-compiler-cap-closeout.md) — active From 2e6284fcd50dc870db4c326ccf60985da2a1f075 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 04:56:52 +0900 Subject: [PATCH 30/40] chore: bump version to 2026.11.7.1 Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .agents/docs/2026-11-06-compiler-attempt-recovery.md | 8 ++++---- ...6-11-06-launch-configuration-and-named-test-runners.md | 2 +- CHANGELOG.md | 2 +- docs/04-mcpp-toml.md | 2 +- docs/08-testing.md | 2 +- docs/24-openkal-cross.md | 2 +- docs/25-home-indices-and-store.md | 2 +- docs/30-build-mcpp.md | 2 +- docs/50-machine-output.md | 2 +- docs/specs/concurrency.md | 4 ++-- docs/zh/04-mcpp-toml.md | 2 +- docs/zh/08-testing.md | 2 +- docs/zh/24-openkal-cross.md | 2 +- docs/zh/25-home-indices-and-store.md | 2 +- docs/zh/30-build-mcpp.md | 2 +- docs/zh/50-machine-output.md | 2 +- mcpp.toml | 2 +- modules/versioning/src/version.cppm | 2 +- 18 files changed, 22 insertions(+), 22 deletions(-) diff --git a/.agents/docs/2026-11-06-compiler-attempt-recovery.md b/.agents/docs/2026-11-06-compiler-attempt-recovery.md index 91d0c8ddb..00fbfe8a3 100644 --- a/.agents/docs/2026-11-06-compiler-attempt-recovery.md +++ b/.agents/docs/2026-11-06-compiler-attempt-recovery.md @@ -212,7 +212,7 @@ No runtime generation becomes a dependency version or build-cache identity. | C++ runtime | 0.19.0 | [Runtime 66](https://github.com/mcpplibs/openkal-llvm-runtime/pull/66) | | Platform | 0.24.0 | Publication follows its public lower dependencies | | OS composition | 0.3.0 | [OS 17](https://github.com/mcpplibs/os/pull/17) | -| Core | 2026.11.6.1 | [mcpp 889](https://github.com/mcpp-community/mcpp/pull/889) | +| Core | 2026.11.7.1 | [mcpp 889](https://github.com/mcpp-community/mcpp/pull/889) | OS advances to 0.3.0 so an explicit range such as `^0.2.3` cannot silently absorb a different OpenKal minor group. Bare package versions name exact wire @@ -229,16 +229,16 @@ changes the state output from `empty` to `unknown`. Source-group controls establish implementation behaviour. The lower versions, including platform 0.24.0 and OS 0.3.0, are now publicly released and indexed; -their ordinary consumers have been checked. Core 2026.11.6.1 uses those pins +their ordinary consumers have been checked. Core 2026.11.7.1 uses those pins and still requires full exact-head CI, an independent ecosystem review, canonical tag archives, byte-identical CN mirrors, a published index entry and ordinary GLOBAL/CN consumers. Plugins PR 52 retains the published -2026.11.5.1 pin until core 2026.11.6.1 is published, then runs its actual +2026.11.5.1 pin until core 2026.11.7.1 is published, then runs its actual sidecar integration and final CI. ## 2026-11-06 final-head CI failure controls -The first 2026.11.6.1 CI run exposed four distinct boundaries. Clang/libc++ +The first 2026.11.7.1 CI run exposed four distinct boundaries. Clang/libc++ uses a wider file-clock tick representation than `std::to_string` accepts; the journal now rejects values outside its signed 64-bit schema and preserves in-range ticks. A direct exact-toolchain control compiled the failing expression diff --git a/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md b/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md index 74b143dc4..05818ebc0 100644 --- a/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md +++ b/.agents/docs/2026-11-06-launch-configuration-and-named-test-runners.md @@ -5,7 +5,7 @@ status: active # Launch configuration and named test runners -Status, 2026-11-06: implemented for candidate 2026.11.6.1. Focused local +Status, 2026-11-06: implemented for candidate 2026.11.7.1. Focused local controls pass; final public-dependency and cross-platform CI remain pending. A consumer choosing a debugger runner needs the configuration of the program diff --git a/CHANGELOG.md b/CHANGELOG.md index 7db928668..4338f214e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,7 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. -## [2026.11.6.1] - 2026-11-06 +## [2026.11.7.1] - 2026-11-07 ### Fixed diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index ef4c358c1..02ca671c0 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1049,7 +1049,7 @@ with two behaviours has to be. `bmi_schedule` decides when importers are unblocked. With GCC and `bmi_schedule = "on"`, ordinary compilation and detached code -generation share the same total `jobs` limit (2026.11.6.1+). Additional +generation share the same total `jobs` limit (2026.11.7.1+). Additional scheduling and join tasks do not increase the number of compiler processes. Cancelling the build ends its ordinary and detached compilers. An immediately following build waits for their cleanup before starting another compiler; diff --git a/docs/08-testing.md b/docs/08-testing.md index 45418b758..b8b9a7c41 100644 --- a/docs/08-testing.md +++ b/docs/08-testing.md @@ -170,7 +170,7 @@ or a QEMU exit code is what a bare-metal runner is chosen to produce. `--no-runner` exists for a host that can execute the binaries natively and should not pay for an emulator. -`--runner ` (2026.11.6.1+) selects a named runner instead of the default, +`--runner ` (2026.11.7.1+) selects a named runner instead of the default, using the same declarations and argument substitution as `mcpp run --runner`. It also applies to `--print-launch` and tests read from an archive. An absent runner name is refused; `--runner` and `--no-runner` cannot be combined. diff --git a/docs/24-openkal-cross.md b/docs/24-openkal-cross.md index 4210a02db..2caeebe36 100644 --- a/docs/24-openkal-cross.md +++ b/docs/24-openkal-cross.md @@ -411,7 +411,7 @@ Nothing else is declared (2026.10.27.1+): the target table selects `openkal-platform` for both rows, and the platform states the toolchain, the layers, the tools, the conventions below and the runner ([35](35-platforms.md)). -Both built-in openkal targets select platform 0.24.0 by default (2026.11.6.1+). +Both built-in openkal targets select platform 0.24.0 by default (2026.11.7.1+). An explicit `platform` configuration overrides that selection. This platform preserves the emulated TLS convention through the final ThinLTO link. diff --git a/docs/25-home-indices-and-store.md b/docs/25-home-indices-and-store.md index dd4814da8..c7347f63e 100644 --- a/docs/25-home-indices-and-store.md +++ b/docs/25-home-indices-and-store.md @@ -90,7 +90,7 @@ configuration, so it remains usable while the network is broken. An install tracks byte progress and source changes separately for each active download. Time spent waiting for a download slot does not consume that budget. Progress on another file does not renew a stalled file's budget. -From 2026.11.6.1, a stalled-download diagnostic names the file, elapsed interval +From 2026.11.7.1, a stalled-download diagnostic names the file, elapsed interval and source attempt. Provider preparation and process timeouts are reported separately. Mirror selection uses `mcpp self config --mirror auto`, `CN` or `GLOBAL`. diff --git a/docs/30-build-mcpp.md b/docs/30-build-mcpp.md index 2ba194da1..da98c93a7 100644 --- a/docs/30-build-mcpp.md +++ b/docs/30-build-mcpp.md @@ -1097,7 +1097,7 @@ none to rely on), and the only interpolations are a closed set: | `${mcpp.out_dir}` | the build output directory | | `${mcpp.bin_dir}` | where produced binaries land: `bin/`, or, for an action a workspace member's build program declares, that member's product directory (2026.9.29.4+) | | `${mcpp.compile_db}` | path to `compile_commands.json` (what clang-tidy's `-p` wants) | -| `${mcpp.contract}` *(2026.10.30.1+)* | path to the configuration's contract, `mcpp-contract.json`, beside its compile database: the compiler of each language with the words every unit is compiled with (target, sysroot, target-side headers, platform and profile flags), and the link driver, its words and the archiver. A foreign build system reads it to compile for the target as the engine does; a package's own include directories are not in it, nor the engine's C++ dialect and module words (`-std=`, `-fmodules`), which are its choice for its own units. Since 2026.11.6.1, link arguments describe the target configuration; the consuming package's link flags and libraries are excluded, since they can name outputs the foreign build has yet to produce | +| `${mcpp.contract}` *(2026.10.30.1+)* | path to the configuration's contract, `mcpp-contract.json`, beside its compile database: the compiler of each language with the words every unit is compiled with (target, sysroot, target-side headers, platform and profile flags), and the link driver, its words and the archiver. A foreign build system reads it to compile for the target as the engine does; a package's own include directories are not in it, nor the engine's C++ dialect and module words (`-std=`, `-fmodules`), which are its choice for its own units. Since 2026.11.7.1, link arguments describe the target configuration; the consuming package's link flags and libraries are excluded, since they can name outputs the foreign build has yet to produce | | `${mcpp.target_file:}` | the built file of target `` | | `${mcpp.stage_dir}` *(2026.9.11.1+)* | the tree `mcpp pack` staged, absolute. `artifact` role only, and only under `mcpp pack --pkg-format ` | | `${mcpp.self}` *(2026.9.13.1+)* | the engine's own executable, absolute. An action's command is an argv with no shell, so a build program has no portable way to copy a file; the engine is present wherever a build runs, and `${mcpp.self} stage --verify content --output ` copies one file, creates the destination's parent, and writes only when the bytes differ. That argument shape is a contract from 2026.9.13.1 on; a build program that names it declares that release as its floor | diff --git a/docs/50-machine-output.md b/docs/50-machine-output.md index a44c21cd3..204a69357 100644 --- a/docs/50-machine-output.md +++ b/docs/50-machine-output.md @@ -1002,7 +1002,7 @@ starting it. `data` is `program`, `argv`, `cwd`, `env` and `via`, as `launch` above. `--print-launch` takes no `--format`: on `mcpp run`, `--pkg-format` names a distributable. -`data.configuration` (2026.11.6.1+) contains the selected program's `triple`, +`data.configuration` (2026.11.7.1+) contains the selected program's `triple`, `profile` and resolved `platform`, using the same fields as the build record's configuration. For a target set it describes the leg selected for this launch. `configuration.runners` lists the available named runners, including project diff --git a/docs/specs/concurrency.md b/docs/specs/concurrency.md index 20dcabacf..bbf7cfcfb 100644 --- a/docs/specs/concurrency.md +++ b/docs/specs/concurrency.md @@ -7,7 +7,7 @@ | 状态 | 评审中 v1.1 | | 版本 | 1.1 | | 最后修改 | 2026-11-06 | -| 对应实现 | 基础保证: mcpp >= 2026.10.26.1; §5.1: mcpp >= 2026.11.6.1 | +| 对应实现 | 基础保证: mcpp >= 2026.10.26.1; §5.1: mcpp >= 2026.11.7.1 | | 相关设计文档 | `.agents/docs/2026-10-26-control-contract-runtime-and-ide-ecosystem-design.md` §3、§4、§5.4 | | 相关 issue | #839 | | 使用文档 | [50 — 机器可读输出](../zh/50-machine-output.md) §9、§11 | @@ -88,7 +88,7 @@ CI。在 2026.10.24.1 上,同一配置的两次冷构建相隔 0.1–0.5 s 启 ### 5.1 分离编译的执行单元与恢复 -以下保证适用于 GCC 的 `bmi_schedule = "on"`（2026.11.6.1+）: +以下保证适用于 GCC 的 `bmi_schedule = "on"`（2026.11.7.1+）: 1. 普通编译与分离代码生成**必须**共用实际的 `jobs` 总上限。等待任务与调度任务 **禁止**作为编译名额的替代。内核锁已释放而原编译单元仍被占用或状态未知时, diff --git a/docs/zh/04-mcpp-toml.md b/docs/zh/04-mcpp-toml.md index 7e0d1c3a4..00bc9dcc8 100644 --- a/docs/zh/04-mcpp-toml.md +++ b/docs/zh/04-mcpp-toml.md @@ -973,7 +973,7 @@ bmi_schedule = "off" # auto (default, = off) | on | off `bmi_schedule` 决定 importer 何时被解除阻塞。 使用 GCC 且 `bmi_schedule = "on"` 时，普通编译与分离的代码生成共用同一个 `jobs` 总上限 -（2026.11.6.1+）。额外的调度和等待任务不增加编译器进程数量。 +（2026.11.7.1+）。额外的调度和等待任务不增加编译器进程数量。 取消构建会结束本轮普通编译器和分离的编译器。紧接着的构建等待它们完成清理后 才启动新的编译器；未完成的输出会重新构建。已完成模块的 object 缺失时， 该次构建会重新生成它。输入和输出均不变的构建仍不进行编译。 diff --git a/docs/zh/08-testing.md b/docs/zh/08-testing.md index 6f9aca1c7..04dde53a5 100644 --- a/docs/zh/08-testing.md +++ b/docs/zh/08-testing.md @@ -151,7 +151,7 @@ mcpp test --target aarch64-macos --no-run # build the tests for the target an `--no-runner` 是为「本机能原生执行这些二进制、不该为模拟器付代价」的宿主准备的。 -`--runner `（2026.11.6.1+）选择具名运行器代替默认运行器，声明与参数替换 +`--runner `（2026.11.7.1+）选择具名运行器代替默认运行器，声明与参数替换 规则与 `mcpp run --runner` 相同。该选项同样适用于 `--print-launch` 与从归档读取的 测试。不存在的运行器名称会被拒绝；`--runner` 与 `--no-runner` 不能同时使用。 diff --git a/docs/zh/24-openkal-cross.md b/docs/zh/24-openkal-cross.md index 056c47987..3d6f91319 100644 --- a/docs/zh/24-openkal-cross.md +++ b/docs/zh/24-openkal-cross.md @@ -361,7 +361,7 @@ mcpp run --target x86_64-openkal # okrun bin/app.oke 无需其他声明（2026.10.27.1+）：目标表为两行选择 `openkal-platform`，由平台陈述工具链、各层、工具、下表的约定 与 runner（[35](35-platforms.md)）。 -两个内置 openkal 目标默认选择平台 0.24.0（2026.11.6.1+）。显式 `platform` 配置可覆盖默认选择； +两个内置 openkal 目标默认选择平台 0.24.0（2026.11.7.1+）。显式 `platform` 配置可覆盖默认选择； 该平台的最终 ThinLTO 链接保持 emulated TLS 约定。 OS 0.3 使用 OpenKal 0.24 依赖组。保留 OS 0.2.9 的工程显式选择匹配的 0.23 平台组； diff --git a/docs/zh/25-home-indices-and-store.md b/docs/zh/25-home-indices-and-store.md index 2079c863b..b759ea7ce 100644 --- a/docs/zh/25-home-indices-and-store.md +++ b/docs/zh/25-home-indices-and-store.md @@ -73,7 +73,7 @@ Windows 首次显式 `self init` 时，若 D 是可写的本地固定磁盘， 安装过程分别跟踪每个活动下载的字节进度与来源切换，等待下载槽位的时间不消耗该预算。 其他文件取得进度不会延长停滞文件的预算。 -从 2026.11.6.1 起，下载停滞诊断会标明文件、停滞时长与来源尝试次数；provider 准备阶段与进程超时单独报告。 +从 2026.11.7.1 起，下载停滞诊断会标明文件、停滞时长与来源尝试次数；provider 准备阶段与进程超时单独报告。 镜像选择使用 `mcpp self config --mirror auto`、`CN` 或 `GLOBAL`。 ### 构建产物的保留期（2026.10.31.3+） diff --git a/docs/zh/30-build-mcpp.md b/docs/zh/30-build-mcpp.md index 1f6dff233..3e666e48b 100644 --- a/docs/zh/30-build-mcpp.md +++ b/docs/zh/30-build-mcpp.md @@ -933,7 +933,7 @@ mcpp 会写出 `<暂存树>.stage-manifest` —— 一个兄弟文件，永不 | `${mcpp.out_dir}` | 构建输出目录 | | `${mcpp.bin_dir}` | 产出的二进制所在目录:`bin/`;工作区成员的构建程序声明的 action 中为该成员的产物目录(2026.9.29.4+) | | `${mcpp.compile_db}` | `compile_commands.json` 的路径（clang-tidy 的 `-p` 要的就是它） | -| `${mcpp.contract}` *(2026.10.30.1+)* | 配置的契约 `mcpp-contract.json` 的路径，与该配置的编译数据库同目录：每种语言的编译器与每个单元都带的词（目标、sysroot、目标侧的头文件、平台与 profile 的标志），以及链接驱动、它的词与归档工具。外部构建系统读它，以引擎的方式为目标编译；包自己的包含目录不在其中，引擎为自己的单元选择的 C++ 方言与模块词（`-std=`、`-fmodules`）也不在其中。自 2026.11.6.1 起，链接参数描述目标配置，不含消费包自己的链接标志和库；这些库可能正是外部构建尚待生成的产物 | +| `${mcpp.contract}` *(2026.10.30.1+)* | 配置的契约 `mcpp-contract.json` 的路径，与该配置的编译数据库同目录：每种语言的编译器与每个单元都带的词（目标、sysroot、目标侧的头文件、平台与 profile 的标志），以及链接驱动、它的词与归档工具。外部构建系统读它，以引擎的方式为目标编译；包自己的包含目录不在其中，引擎为自己的单元选择的 C++ 方言与模块词（`-std=`、`-fmodules`）也不在其中。自 2026.11.7.1 起，链接参数描述目标配置，不含消费包自己的链接标志和库；这些库可能正是外部构建尚待生成的产物 | | `${mcpp.target_file:}` | target `` 构建出的文件 | | `${mcpp.stage_dir}` *(2026.9.11.1+)* | `mcpp pack` 暂存出的那棵树，绝对路径。仅 `artifact` role 可用，且仅在 `mcpp pack --pkg-format ` 下可用 | | `${mcpp.self}` *(2026.9.13.1+)* | 引擎自己的可执行文件，绝对路径。action 的命令是没有 shell 的 argv，构建程序因此没有可移植的拷贝手段；引擎在构建运行的每台机器上都在，`${mcpp.self} stage --verify content --output ` 拷贝一个文件、创建目标的父目录、只在字节不同时写入。这个参数形状自 2026.9.13.1 起是契约；写下它的构建程序即以该版本为下限 | diff --git a/docs/zh/50-machine-output.md b/docs/zh/50-machine-output.md index 37fc7a997..f4a79b659 100644 --- a/docs/zh/50-machine-output.md +++ b/docs/zh/50-machine-output.md @@ -890,7 +890,7 @@ mcpp run [target] --print-launch `argv`、`cwd`、`env` 与 `via`，同上文的 `launch`。`--print-launch` 不接受 `--format`：在 `mcpp run` 上，`--pkg-format` 表示可分发物的格式。 -`data.configuration`（2026.11.6.1+）包含所选程序的 `triple`、`profile` 与解析后的 +`data.configuration`（2026.11.7.1+）包含所选程序的 `triple`、`profile` 与解析后的 `platform`，字段与构建记录的 configuration 相同。目标集合中，它描述本次启动所选的 目标。`configuration.runners` 列出可用的具名运行器，包括 platform 为 null 时 项目声明的运行器。旧引擎不提供 `configuration`；消费者检查该字段是否存在。 diff --git a/mcpp.toml b/mcpp.toml index c752c16ae..906945fd3 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.11.6.1" +version = "2026.11.7.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/modules/versioning/src/version.cppm b/modules/versioning/src/version.cppm index ae049ca57..4288a2395 100644 --- a/modules/versioning/src/version.cppm +++ b/modules/versioning/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.11.6.1"; +inline constexpr std::string_view MCPP_VERSION = "2026.11.7.1"; } // namespace mcpp From 4e01eb88ff6de7ba8b6fbcbdb2e22495304b3da5 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 05:27:12 +0900 Subject: [PATCH 31/40] fix(build): a record being published is waited out, and only an ended process is a lost controller On Windows the name a rename installs can be refused to a reader for a short interval. The waiter read that refusal as an unreadable controller record and declared a running compiler lost: 10 of 40 Windows-host cross builds failed this way (#890). Records and statuses are now opened with a bounded backoff (kRecordSettle, 250 ms) while the name exists; an absent name is absent at once and a refusal that outlasts the bound is an error. The controller's state is three-valued: only an identity that names an ended process is ended; an unreadable record is unknown and the waiter keeps waiting within its existing bounds. A loss is reported with the observation that decided it. copy_atomic publishes through the host namespace like every other record of the module, so a name in Windows' long spelling stays valid. A cancelled managed supervisor writes its status, and a registration failure reaches the supervisor log. Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- src/build/schedule/compiler_transaction.cppm | 75 ++++++++++++++----- src/build/schedule/detach_codegen.cppm | 28 +++++-- tests/e2e/helpers/record_open_refusal.c | 32 ++++++++ tests/unit/test_compiler_transaction.cpp | 78 ++++++++++++++++++++ 4 files changed, 187 insertions(+), 26 deletions(-) create mode 100644 tests/e2e/helpers/record_open_refusal.c diff --git a/src/build/schedule/compiler_transaction.cppm b/src/build/schedule/compiler_transaction.cppm index c9ecb4b66..3d3e634ff 100644 --- a/src/build/schedule/compiler_transaction.cppm +++ b/src/build/schedule/compiler_transaction.cppm @@ -40,18 +40,38 @@ std::string pack(std::string_view tag, const Fields& fields) { // Atomic visibility across process loss; storage durability is not promised. bool replace_record(const fs::path& path,std::string_view bytes,std::string* diagnostic=nullptr); bool publish(const fs::path& path,std::string_view tag,const Fields& fields); -Record read(const fs::path& path, std::string_view tag) { - std::ifstream input(path,std::ios::binary); - if (!input) { +// A RECORD THAT EXISTS AND CANNOT YET BE OPENED IS BEING PUBLISHED. On Windows +// the name a rename installs can be refused to a reader for a short interval +// (the rename's own handle, or a scanner holding the new file); a reader that +// took that refusal as the record's content declared a live compiler lost +// (mcpp-community/mcpp#889, measured 10 of 40 Windows cross builds). The open +// is retried with backoff for at most kRecordSettle; a name that is absent is +// absent at once, and a refusal that outlasts the bound is still an error. +inline constexpr auto kRecordSettle=std::chrono::milliseconds(250); +enum class Opened { yes, absent, refused }; +// Opens `path' for reading, waiting out a publication in progress. +Opened open_settled(std::ifstream& input,const fs::path& path) { + input.open(path,std::ios::binary); + if(input)return Opened::yes; + const auto limit=std::chrono::steady_clock::now()+kRecordSettle; + auto pause=std::chrono::milliseconds(1); + for(;;) { std::error_code ec; - if (!fs::exists(path,ec) && !ec) return std::optional{}; - if(ec)return std::unexpected("cannot read compiler transaction record"); - // An atomic publisher may install the record between the failed open - // and the existence probe. Retry once; a persistent refusal still - // fails rather than being mistaken for an absent record. + const bool present=fs::exists(path,ec); + if(!present&&!ec)return Opened::absent; + input.clear(); input.open(path,std::ios::binary); - if(!input)return std::unexpected("cannot read compiler transaction record"); + if(input)return Opened::yes; + if(std::chrono::steady_clock::now()>=limit)return Opened::refused; + std::this_thread::sleep_for(pause); + pause=std::min(pause*2,std::chrono::milliseconds(32)); } +} +Record read(const fs::path& path, std::string_view tag) { + std::ifstream input; + const auto opened=open_settled(input,path); + if(opened==Opened::absent)return std::optional{}; + if(opened==Opened::refused)return std::unexpected("cannot read compiler transaction record"); std::string bytes(65537,'\0'); input.read(bytes.data(),bytes.size());bytes.resize(static_cast(input.gcount())); if(input.bad())return std::unexpected("cannot read compiler transaction record"); @@ -118,7 +138,11 @@ bool copy_atomic(const fs::path& from,const fs::path& to) { fs::copy_file(from,temp,fs::copy_options::overwrite_existing,ec);if(ec)return false; const auto time=fs::last_write_time(from,ec);if(ec)return false; fs::last_write_time(temp,time,ec);if(ec)return false; - return bool(os::fs::rename(temp,to,true)); + // The host namespace, as for every record of this module (the copy was + // written through it); a name in Windows' long spelling stays valid. + fs::rename(temp,to,ec); + if(ec) {fs::remove(temp,ec);return false;} + return true; } struct Attempt { fs::path directory, bmi, object, slot, dep; @@ -299,17 +323,26 @@ bool release_reservation(const fs::path& reservation,const fs::path& attempt) { if((**current)[0]!=spelling(attempt))return true; // a later, separately locked slot std::error_code ec;fs::remove(reservation,ec);return !ec; } -bool controller_alive(const Attempt& attempt) { +// WHETHER A CONTROLLER IS RUNNING IS A QUESTION ABOUT A PROCESS, NOT ABOUT A +// FILE. Only an identity that names an ended process answers `ended'; a record +// that cannot be read yet answers `unknown', and the caller keeps waiting +// within its own bounds. A record that is absent is the existing startup case. +enum class Controller { running, ended, unknown }; +Controller controller_state(const Attempt& attempt) { auto controller=read(attempt.directory/"controller","mcpp-controller-v1"); - if(!controller)return false; - if(!*controller)return true; // the existing startup bound applies - if((**controller).size()!=3)return false; + if(!controller)return Controller::unknown; + if(!*controller)return Controller::running; // the existing startup bound applies + if((**controller).size()!=3)return Controller::ended; os::process::program who{};const auto& fields=**controller; const auto id=std::from_chars(fields[0].data(),fields[0].data()+fields[0].size(),who.id); const auto birth=std::from_chars(fields[1].data(),fields[1].data()+fields[1].size(),who.started_ns); - return id.ec==std::errc{}&&id.ptr==fields[0].data()+fields[0].size() - &&birth.ec==std::errc{}&&birth.ptr==fields[1].data()+fields[1].size() - &&who.id&&who.started_ns&&os::process::alive(who); + const bool valid=id.ec==std::errc{}&&id.ptr==fields[0].data()+fields[0].size() + &&birth.ec==std::errc{}&&birth.ptr==fields[1].data()+fields[1].size()&&who.id&&who.started_ns; + if(!valid)return Controller::ended; + return os::process::alive(who) ? Controller::running : Controller::ended; +} +std::string_view describe(Controller c) { + return c==Controller::running ? "running" : c==Controller::ended ? "ended" : "unknown"; } std::expected receive(os::ipc::connection& connection,std::chrono::milliseconds bound) { @@ -399,7 +432,9 @@ private: if(*retired) {it=held_.erase(it);continue;} auto attempt=Attempt::load(directory); if(!attempt) {fail(attempt.error());++it;continue;} - if(!controller_alive(*attempt)) { + const auto state=controller_state(*attempt); + if(state==Controller::unknown) {++it;continue;} // asked again at the next audit + if(state==Controller::ended) { // Retirement can be published between the first read and the // controller's exit. Observe it again before reporting loss. retired=read(directory/"retired","mcpp-retired-v1"); @@ -430,7 +465,9 @@ private: const auto actor=named((**unit)[3]); if(actor.parent_path()!=generation_/"actors") {send(connection,"REFUSED");continue;} auto lease=os::fs::lock::acquire(actor,os::fs::lock::kind::exclusive,false); - if(lease||lease.error().code!=os::errc::again||!controller_alive(*prepared)) {send(connection,"REFUSED");continue;} + // The actor's lock is held, so its controller is running; a record + // still settling does not refuse the registration. + if(lease||lease.error().code!=os::errc::again||controller_state(*prepared)==Controller::ended) {send(connection,"REFUSED");continue;} auto retained=os::process::unit_observer::open((**unit)[0]); if(!retained) {send(connection,"REFUSED");continue;} std::lock_guard guard(admission_); diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index e886c7b8e..9673a2a2c 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -299,8 +299,8 @@ void withdraw_outputs(const std::filesystem::path& slot, // the supervisor's status is lost, and the unit is taken as failed rather than // waited for until the hard limit. std::optional read_rc(const std::filesystem::path& slot) { - std::ifstream in(suffixed(slot, ".rc")); - if (!in) return std::nullopt; + std::ifstream in; + if (transactions::open_settled(in, suffixed(slot, ".rc")) != transactions::Opened::yes) return std::nullopt; int rc = 0; if (!(in >> rc)) { std::println(std::cerr, "mcpp: the status of the compiler for {} was not written " @@ -548,6 +548,15 @@ void record_supervisor_error(const std::filesystem::path& slot,std::string_view if(log)log << "mcpp: " << reason << '\n'; } +// What the waiter observed when it concluded that the controller is gone: one +// line, without a path, so a failure states which observation decided it. +void report_controller_loss(const std::filesystem::path& slot,transactions::Controller controller,bool supervising) { + std::error_code ec; + const bool log=std::filesystem::exists(suffixed(slot,".log"),ec); + std::println(std::cerr,"mcpp: compiler controller observation: controller {}, supervisor {}, status absent, log {}", + transactions::describe(controller),supervising ? "running" : "ended",log ? "present" : "absent"); +} + void replay_supervisor_log(const std::filesystem::path& slot) { std::ifstream log(suffixed(slot,".log")); if(log)std::cerr << log.rdbuf(); @@ -827,12 +836,14 @@ int compile_release_at_bmi(const CompileRequest& req) { } if(attempt && clock::now()>=nextControllerCheck) { nextControllerCheck=clock::now()+std::chrono::milliseconds(50); - if(!transactions::controller_alive(*attempt) - ||(supervisor&&!mcpplibs::os::process::alive(*supervisor))) { + const auto controller=transactions::controller_state(*attempt); + const bool supervising=!supervisor||mcpplibs::os::process::alive(*supervisor); + if(controller==transactions::Controller::ended||!supervising) { // The final status can be published between the first read and // this liveness observation. The next iteration validates it. if(read_rc(actual.slot))continue; replay_supervisor_log(actual.slot); + report_controller_loss(actual.slot,controller,supervising); std::println(std::cerr,"mcpp: compiler controller ended without a final status; retaining its outputs until unit recovery"); return 1; } @@ -885,9 +896,10 @@ int supervise(const std::filesystem::path& slot, auto unit=std::make_shared(std::move(*created)); InterruptCleanup cleanup{mcpp::run::add_cleanup([actor,unit] {unit->terminate();})}; if(auto registered=attempt.register_unit(*unit,actor->path,permit->reservation);!registered) { - std::println(std::cerr,"mcpp: {}",registered.error());(void)write_rc(slot,2);return 2; + record_supervisor_error(slot,registered.error());(void)write_rc(slot,2);return 2; } - if(!actor->owner.active())return 130; + // Every exit leaves a status, a cancelled one included. + if(!actor->owner.active()) {(void)write_rc(slot,130);return 130;} if(!transactions::publish(attempt.directory/"admitted","mcpp-admitted-v1",{})) { (void)write_rc(slot,2);return 2; } @@ -982,11 +994,13 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o if (!actor.owner.active()) return 130; if(attempt&&clock::now()>=nextControllerCheck) { nextControllerCheck=clock::now()+std::chrono::milliseconds(50); - if(!transactions::controller_alive(*attempt)) { + const auto controller=transactions::controller_state(*attempt); + if(controller==transactions::Controller::ended) { // A controller may publish its status and exit after the read // in the loop condition. Let the normal result path inspect it. if((rc=read_rc(slot)))break; replay_supervisor_log(slot); + report_controller_loss(slot,controller,true); std::println(std::cerr,"mcpp: compiler controller ended without a status; unit recovery will retain or invalidate its outputs"); return 1; } diff --git a/tests/e2e/helpers/record_open_refusal.c b/tests/e2e/helpers/record_open_refusal.c new file mode 100644 index 000000000..0cdbf7dea --- /dev/null +++ b/tests/e2e/helpers/record_open_refusal.c @@ -0,0 +1,32 @@ +#define _GNU_SOURCE +#include +#include +#include +#include +#include + +/* Refuses opening an existing record for reading, as Windows refuses a name a + * rename has just installed: MCPP_TEST_REFUSE_NAME names the file and + * MCPP_TEST_REFUSE_COUNT how many opens are refused (-1: every one). Each + * refusal appends a line to `-refused`. */ +static FILE *(*native_open)(const char *, const char *); +static long refused; + +FILE *fopen64(const char *path, const char *mode) { + if (!native_open) native_open = dlsym(RTLD_NEXT, "fopen64"); + const char *name = getenv("MCPP_TEST_REFUSE_NAME"); + const char *count = getenv("MCPP_TEST_REFUSE_COUNT"); + const long limit = count ? strtol(count, NULL, 10) : 0; + if (name && strcmp(path, name) == 0 && mode[0] == 'r' && (limit < 0 || refused < limit)) { + ++refused; + char marker[512]; + snprintf(marker, sizeof marker, "%s-refused", name); + FILE *m = native_open(marker, "ab"); + if (m) { fputs("refused\n", m); fclose(m); } + errno = EACCES; + return NULL; + } + return native_open(path, mode); +} + +FILE *fopen(const char *path, const char *mode) { return fopen64(path, mode); } diff --git a/tests/unit/test_compiler_transaction.cpp b/tests/unit/test_compiler_transaction.cpp index 0006aea98..222234015 100644 --- a/tests/unit/test_compiler_transaction.cpp +++ b/tests/unit/test_compiler_transaction.cpp @@ -288,6 +288,84 @@ TEST(CompilerTransaction, RecordAppearanceBetweenOpenAndExistsIsNotCorruption) { EXPECT_EQ(result.exit_code,0)< refused_driver(const fs::path& root,std::string_view filter, + std::vector> env) { + const auto library=root/"record-open-refusal.so"; + const auto source=fs::current_path()/"tests/e2e/helpers/record_open_refusal.c"; + auto compiled=mcpp::run::execute({"/usr/bin/cc","-shared","-fPIC","-Wall","-Wextra","-Werror", + source.string(),"-o",library.string(),"-ldl"},{.total=std::chrono::seconds(10)}); + if(compiled.exit_code!=0) {ADD_FAILURE()<interp.find("ld-linux")!=std::string::npos; +} +std::size_t refusals(const fs::path& marker) { + std::ifstream in(marker);std::size_t n=0;std::string line; + while(std::getline(in,line))++n; + return n; +} +} // namespace +TEST(CompilerTransaction, RecordRefusedWhilePublishedIsWaitedOut) { + if(!preloadable())GTEST_SKIP()<<"open interposition requires Linux and the dynamic glibc loader"; + Transaction t; + ASSERT_TRUE(tx::publish(t.root/"record","schema",{"published"})); + const auto result=refused_driver(t.root,"RecordRefusalDriver", + {{"MCPP_TEST_RECORD_REFUSAL_DRIVER","1"},{"MCPP_TEST_REFUSE_NAME","record"},{"MCPP_TEST_REFUSE_COUNT","4"}}); + ASSERT_TRUE(result); + EXPECT_EQ(result->exit_code,0)<output; + // Four refusals: the reader before 2026.11.7.1 retried once and failed. + EXPECT_EQ(refusals(t.root/"record-refused"),4u); +} +TEST(CompilerTransaction, PersistentRefusalIsABoundedError) { + if(!preloadable())GTEST_SKIP()<<"open interposition requires Linux and the dynamic glibc loader"; + Transaction t; + ASSERT_TRUE(tx::publish(t.root/"record","schema",{"published"})); + const auto result=refused_driver(t.root,"RecordRefusalDriver", + {{"MCPP_TEST_RECORD_REFUSAL_DRIVER","1"},{"MCPP_TEST_REFUSE_NAME","record"},{"MCPP_TEST_REFUSE_COUNT","-1"}}); + ASSERT_TRUE(result); + EXPECT_EQ(result->exit_code,0)<output; + EXPECT_GE(refusals(t.root/"record-refused"),2u); +} +TEST(CompilerTransaction, UnreadableControllerRecordIsNotAnEndedController) { + if(!preloadable())GTEST_SKIP()<<"open interposition requires Linux and the dynamic glibc loader"; + Transaction t; + auto a=t.prepare(); + ASSERT_TRUE(tx::publish(a.directory/"controller","mcpp-controller-v1",{"1","1",tx::spelling(t.generation/"actors/helper.lock")})); + const auto result=refused_driver(t.root,"ControllerRefusalDriver", + {{"MCPP_TEST_CONTROLLER_REFUSAL_DRIVER","1"},{"MCPP_TEST_ATTEMPT",a.directory.string()}, + {"MCPP_TEST_REFUSE_NAME",(a.directory/"controller").string()}, + {"MCPP_TEST_REFUSE_COUNT","-1"}}); + ASSERT_TRUE(result); + EXPECT_EQ(result->exit_code,0)<output; +} TEST(CompilerTransaction, FailedCloseCannotPublishOrAcknowledgeARecord) { if constexpr (!mcpplibs::os::is_linux)GTEST_SKIP()<<"Linux libc close failure control"; const auto runtime=mcpp::runtime::elf::inspect_elf_runtime(mcpp::invocation::self_exe_path()); From 30b900f4edf219fac31df2b978010f61a1dc67bf Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 05:42:27 +0900 Subject: [PATCH 32/40] fix(build): every compiler of a build belongs to its generation Killing the mcpp that holds a build directory left its compilers writing there while the next build entered: 2026.11.5.1 failed 3 of 75 such rounds under the default schedule and 12 of 15 under the split schedule; the split schedule's generations covered only its own compilers. Every strategy now caps its compilers, so every compile edge runs through the capped executor, registers in the build's generation, and the next build enters only after those compilers end. The compile database keeps the compiler's own argv. E2E 1016 kills the holding build twelve times under each schedule; it passes on this change and fails on 2026.11.5.1 (split, round 1). The executor costs about 1 ms per edge on Linux. Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- src/build/execute.cppm | 9 +++- src/build/ninja_backend.cppm | 9 ++-- src/build/schedule/policy.cppm | 16 +++++++ ...killed_build_does_not_race_the_next_one.sh | 45 +++++++++++++++++++ tests/unit/test_schedule_policy.cpp | 14 ++++++ 5 files changed, 88 insertions(+), 5 deletions(-) create mode 100755 tests/e2e/1016_a_killed_build_does_not_race_the_next_one.sh diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 410748143..8cc7e3bda 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -2177,7 +2177,14 @@ std::optional run_ninja_fast(const std::string& ninjaProgram, } std::optional compilerRun; - if (mcpp::build::read_schedule(ninjaPath) == "detach-codegen") { + // A graph whose compilers run through the capped executor replays inside a + // generation, as its first drive did. + const bool capped = [&] { + std::ifstream in(ninjaPath, std::ios::binary); + const std::string text{std::istreambuf_iterator(in), {}}; + return text.find(" __compiler-cap ") != std::string::npos; + }(); + if (capped) { auto opened = mcpp::build::schedule::detach::RunScope::begin(outputDir); if (!opened) { mcpp::ui::error(opened.error()); return 1; } compilerRun.emplace(std::move(*opened)); diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index adf995d76..99a49e9c8 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -1472,7 +1472,9 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements, // Ninja has extra slots for BMI publication and join edges. Ordinary // compilers must share the supervisors' semaphore rather than use that // larger slot count as their own concurrency bound. - const std::string compilerPrefix = splitBmi + // Every capped graph (2026.11.7.1): the executor also registers each + // compiler as a participant of the build's generation. + const std::string compilerPrefix = plan.scheduleCompilerCap > 0 ? std::format("$mcpp __compiler-cap --sem .mcpp-sched --cap {} --output $out --depfile $out.d -- ", plan.scheduleCompilerCap) : std::string{}; // The other split shape. Clang publishes no BMI early — it writes the BMI @@ -4542,13 +4544,12 @@ std::expected NinjaBackend::build(const BuildPlan& plan plan.outputDir.string(), ec.message()), plan.outputDir}); std::optional compilerRun; - if (!opts.dryRun && plan.scheduleTag == "detach-codegen") { + if (!opts.dryRun && plan.scheduleCompilerCap > 0) { auto opened = mcpp::build::schedule::detach::RunScope::begin(plan.outputDir); if (!opened) return std::unexpected(BuildError{opened.error(), plan.outputDir}); compilerRun.emplace(std::move(*opened)); - } - if (!opts.dryRun && plan.scheduleTag == "detach-codegen") reclaim_stale_schedule_tokens(plan.outputDir); + } auto ninja_path = plan.outputDir / "build.ninja"; // Written beside build.ninja and not into it: the fast path replays the diff --git a/src/build/schedule/policy.cppm b/src/build/schedule/policy.cppm index 8fe445cef..9cbc9f638 100644 --- a/src/build/schedule/policy.cppm +++ b/src/build/schedule/policy.cppm @@ -178,11 +178,27 @@ int resolve_jobs(const manifest::Manifest& m, // peaks near a gigabyte. Decision decide(const toolchain::Toolchain& tc, std::string_view requested, int hostJobs, int autoJobs = 0); +// The module-edge shape alone, before the generation's compiler cap is applied. +Decision decide_shape(const toolchain::Toolchain& tc, std::string_view requested, int hostJobs, + int autoJobs = 0); // --------------------------------------------------------------------------- Decision decide(const toolchain::Toolchain& tc, std::string_view requested, int hostJobs, int autoJobs) { + Decision d = decide_shape(tc, requested, hostJobs, autoJobs); + // EVERY COMPILER OF A BUILD IS A PARTICIPANT OF ITS GENERATION (2026.11.7.1). + // Under every strategy a compile edge runs through the capped executor, so a + // later build does not enter the build directory while a compiler of an + // earlier one -- whose mcpp was killed -- still writes there (design + // 2026-11-07 §4.1.2). The cap is the strategy's, else the job count, else + // what this machine would choose. + if (d.compilerCap <= 0) d.compilerCap = hostJobs > 0 ? hostJobs : autoJobs; + return d; +} + +Decision decide_shape(const toolchain::Toolchain& tc, std::string_view requested, int hostJobs, + int autoJobs) { Decision d; const int cap = hostJobs > 0 ? hostJobs : 0; diff --git a/tests/e2e/1016_a_killed_build_does_not_race_the_next_one.sh b/tests/e2e/1016_a_killed_build_does_not_race_the_next_one.sh new file mode 100755 index 000000000..4d200ae55 --- /dev/null +++ b/tests/e2e/1016_a_killed_build_does_not_race_the_next_one.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# requires: gcc +# A BUILD WHOSE mcpp IS KILLED DOES NOT RACE THE NEXT ONE (2026.11.7.1, design +# 2026-11-07 §4.1.2). Killing the mcpp that holds the build directory leaves +# its compilers running. The next build waited only for the directory lock, so +# it entered while those compilers still wrote BMIs and objects: 2026.11.5.1 +# failed 3 of 75 such rounds under the default schedule and 12 of 15 under the +# split schedule. Every compiler is now a participant of its build's +# generation, and the next build enters only after they end. Both schedules, +# twelve rounds each: the waiting build succeeds and the build after it does +# nothing. +set -e +MCPP="${MCPP:-mcpp}" +TMP=$(mktemp -d); trap 'rm -rf "$TMP"' EXIT +make_project() { # make_project + mkdir -p "$1/src" + printf '[package]\nname = "kg"\nversion = "0.1.0"\nstandard = "c++23"\n[toolchain]\ndefault = "gcc@16.1.0"\n' > "$1/mcpp.toml" + [ "$2" = split ] && printf '[build]\nbmi_schedule = "on"\n' >> "$1/mcpp.toml" + for i in $(seq 1 24); do + printf 'export module m%d;\nimport std;\nexport int f%d(){ std::vector v(%d); return (int)v.size(); }\n' "$i" "$i" "$i" > "$1/src/m$i.cppm" + done + { echo 'import std;'; for i in $(seq 1 24); do echo "import m$i;"; done + echo 'int main(){ std::println("{}", f1()+f24()); }'; } > "$1/src/main.cpp" +} +for schedule in default split; do + dir="$TMP/$schedule"; make_project "$dir" "$schedule" + ( cd "$dir" && "$MCPP" build > "$TMP/warm.log" 2>&1 ) || { cat "$TMP/warm.log"; exit 1; } + for round in $(seq 1 12); do + rm -rf "$dir/target" + ( cd "$dir" && exec "$MCPP" build > "$TMP/a.log" 2>&1 ) & a=$! + ( sleep 0.2; cd "$dir"; rc=0; "$MCPP" build > "$TMP/b.log" 2>&1 || rc=$?; echo "$rc" > "$TMP/b.rc" ) & b=$! + sleep 0.9; kill -9 "$a" 2>/dev/null || true; wait "$a" 2>/dev/null || true + wait "$b" || true + if [ "$(cat "$TMP/b.rc")" != 0 ]; then + echo "FAIL: $schedule round $round: the build after the killed one failed"; cat "$TMP/b.log"; exit 1 + fi + ( cd "$dir" && "$MCPP" build > "$TMP/c.log" 2>&1 ) || { cat "$TMP/c.log"; exit 1; } + if grep -q "Compiling" "$TMP/c.log"; then + echo "FAIL: $schedule round $round: the following build compiled again"; cat "$TMP/c.log"; exit 1 + fi + out=$(cd "$dir" && "$MCPP" run 2>/dev/null | tail -1) + [ "$out" = 25 ] || { echo "FAIL: $schedule round $round: the program printed '$out'"; exit 1; } + done + echo "OK: $schedule: 12 killed builds, each followed by a successful build and a no-op" +done diff --git a/tests/unit/test_schedule_policy.cpp b/tests/unit/test_schedule_policy.cpp index 92893bf3e..375d44d6d 100644 --- a/tests/unit/test_schedule_policy.cpp +++ b/tests/unit/test_schedule_policy.cpp @@ -15,6 +15,7 @@ import mcpp.manifest; using mcpp::build::schedule::Strategy; using mcpp::build::schedule::decide; +using mcpp::build::schedule::decide_shape; using mcpp::build::schedule::requested_switch; using mcpp::toolchain::CompilerId; using mcpp::toolchain::Toolchain; @@ -256,3 +257,16 @@ TEST(SchedulePolicy, EnvironmentBeatsManifestAndIsValidatedToo) { EXPECT_EQ(seen, "ON"); } } + +// EVERY STRATEGY CAPS ITS COMPILERS (2026.11.7.1): the capped executor is what +// registers a compiler in its build's generation, so a strategy without a split +// still has a cap -- the job count, else what this machine would choose. +TEST(SchedulePolicy, EveryStrategyCarriesACompilerCap) { + for (auto id : {CompilerId::GCC, CompilerId::Clang, CompilerId::MSVC, CompilerId::Unknown}) { + EXPECT_EQ(decide(with(id), "auto", 8).compilerCap, 8) << static_cast(id); + EXPECT_EQ(decide(with(id), "off", 0, 6).compilerCap, 6) << static_cast(id); + } + // The shape alone is unchanged: only the split strategies carry a cap there. + EXPECT_EQ(decide_shape(with(CompilerId::MSVC), "auto", 8).compilerCap, 0); +} + From dd535308566a3f3c4954ffdd763de70abcccbd2b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 05:42:27 +0900 Subject: [PATCH 33/40] fix(toolchain): msvc@system is located for a build program, never installed On a Windows machine whose default is msvc@system, a build program of a bare cross target resolved its host toolchain through the package index and looked for an msvc@system package. The spec now resolves the way a native build resolves it: the Visual Studio installation is located, and its absence is reported with the installation guidance. E2E 1017 builds openkal's 32-bit layout probe shape on Windows with msvc@system. Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- src/build/prepare/xlings.cpp | 23 +++++++ ...lds_the_build_program_of_a_cross_target.sh | 61 +++++++++++++++++++ 2 files changed, 84 insertions(+) create mode 100755 tests/e2e/1017_a_system_msvc_builds_the_build_program_of_a_cross_target.sh diff --git a/src/build/prepare/xlings.cpp b/src/build/prepare/xlings.cpp index 4445f3380..27e924c2a 100644 --- a/src/build/prepare/xlings.cpp +++ b/src/build/prepare/xlings.cpp @@ -18,6 +18,8 @@ import mcpp.manifest; import mcpp.source_kind; import mcpp.toolchain.hostflags; // the compile-token producer the package std module reuses import mcpp.toolchain.detect; +import mcpplibs.os.platform; +import mcpp.toolchain.msvc; import mcpp.toolchain.dialect; import mcpp.toolchain.fingerprint; import mcpp.toolchain.registry; @@ -235,6 +237,27 @@ step3_define_host_tc_closures_and_refresh_index(PrepareState& state) { return std::unexpected(std::format( "toolchain spec '{}' is invalid for the build.mcpp host resolve", hostSpecText)); } + // `@system` NAMES THIS MACHINE'S INSTALLATION, AND IS LOCATED, + // NEVER INSTALLED (2026.11.7.1). A Windows machine whose default is + // `msvc@system`, building for a bare cross target, handed that spec to + // the package resolver below, which looked for `msvc@system` in the + // index (design 2026-11-07 §4.1.3). The build program is located the + // way a native build locates the same spec. + if (mcpp::toolchain::origin_of(*spec) == mcpp::toolchain::Origin::SystemMsvc) { + if constexpr (!mcpplibs::os::is_windows) { + return std::unexpected(std::format( + "host toolchain for build.mcpp ('{}') is only available on Windows hosts", hostSpecText)); + } + auto inst = mcpp::toolchain::msvc::detect_installation(); + if (!inst) return std::unexpected(mcpp::toolchain::msvc::install_guidance()); + auto htc = mcpp::toolchain::detect( + inst->clPath, state.runtimePayload, state.runtimeBindingSnapshot.contractHash); + if (!htc) return std::unexpected(htc.error().message); + mcpp::ui::info("Resolved", std::format( + "host toolchain for build.mcpp: {} → {}", hostSpecText, htc->label())); + state.hostTcCache = std::pair{inst->clPath, *htc}; + return std::pair{state.hostTcCache->first, as_host(state.hostTcCache->second)}; + } // Deliberately NO target injection: the spec resolves for the host. auto pkg = mcpp::toolchain::to_xim_package(*spec); auto cfgH = state.get_cfg(true); diff --git a/tests/e2e/1017_a_system_msvc_builds_the_build_program_of_a_cross_target.sh b/tests/e2e/1017_a_system_msvc_builds_the_build_program_of_a_cross_target.sh new file mode 100755 index 000000000..a4de1e817 --- /dev/null +++ b/tests/e2e/1017_a_system_msvc_builds_the_build_program_of_a_cross_target.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# requires: windows msvc llvm +# `msvc@system` IS LOCATED FOR A BUILD PROGRAM, NEVER INSTALLED (2026.11.7.1). +# On a Windows machine whose default toolchain is `msvc@system`, a project with +# a build program built for a bare cross target resolved the build program's +# host toolchain through the package index and looked for an `msvc@system` +# package there (openkal's 32-bit layout probe, design 2026-11-07 §4.1.3). The +# build program is now compiled by the located Visual Studio, as a native build +# of the same machine is, and its directive reaches the cross build. +set -e + +CONF="${MCPP_HOME:-$HOME/.mcpp}/config.toml" +ORIG_DEFAULT="" +if [[ -f "$CONF" ]]; then + ORIG_DEFAULT=$(sed -n '/^\[toolchain\]/,/^\[/p' "$CONF" \ + | grep -E '^default[[:space:]]*=' | head -1 | cut -d'"' -f2 || true) +fi +TMP=$(mktemp -d) +restore() { + if [[ -n "$ORIG_DEFAULT" ]]; then + "$MCPP" toolchain default "$ORIG_DEFAULT" >/dev/null 2>&1 || true + fi + rm -rf "$TMP" +} +trap restore EXIT + +"$MCPP" toolchain default msvc >/dev/null 2>&1 \ + || { echo "FAIL: msvc@system not selectable"; exit 1; } + +mkdir -p "$TMP/probe/src" "$TMP/probe/inc" +cd "$TMP/probe" +cat > mcpp.toml <<'EOF2' +[package] +name = "layout" +version = "0.1.0" +EOF2 +cat > build.mcpp <<'EOF2' +#include +int main() { + std::puts("mcpp:include-dir=inc"); + std::puts("mcpp:rerun-if-changed=build.mcpp"); + return 0; +} +EOF2 +cat > inc/layout.h <<'EOF2' +struct pair_layout { unsigned size; unsigned long long stamp; }; +EOF2 +cat > src/main.cpp <<'EOF2' +#include "layout.h" +static_assert(__builtin_offsetof(pair_layout, stamp) == 8); +extern "C" void _start() { for (;;) {} } +EOF2 + +out=$("$MCPP" build --target riscv64-none-elf 2>&1) \ + || { echo "FAIL: the cross build with a build program failed:"; echo "$out"; exit 1; } +grep -q "host toolchain for build.mcpp: msvc@system" <<< "$out" \ + || { echo "FAIL: the build program was not compiled by the located msvc@system:"; echo "$out"; exit 1; } +if grep -qi "xim:msvc@system\|msvc@system.*not found in the" <<< "$out"; then + echo "FAIL: msvc@system was looked up as a package:"; echo "$out"; exit 1 +fi +echo "OK: msvc@system compiled the build program of a riscv64-none-elf build" From 135434f7712b714a9e55a723d5a49c46a37eeb9b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 06:36:02 +0900 Subject: [PATCH 34/40] [temp] probe 885: what the failing slot lock sees on macOS Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/workflows/probe-885-macos.yml | 44 ++++++++++++++++++++++++++ src/build/schedule/detach_codegen.cppm | 13 ++++++++ 2 files changed, 57 insertions(+) create mode 100644 .github/workflows/probe-885-macos.yml diff --git a/.github/workflows/probe-885-macos.yml b/.github/workflows/probe-885-macos.yml new file mode 100644 index 000000000..88d40f9ae --- /dev/null +++ b/.github/workflows/probe-885-macos.yml @@ -0,0 +1,44 @@ +name: probe-885-macos + +# [temp][do not merge] #889 C2 on macOS: `__compiler-cap` reports +# "locking /../slots/slot-0.lock: not found" for some edges. The +# instrumented acquire_permit prints what the failing lock saw. + +on: + pull_request: + +permissions: + contents: read + +jobs: + build-macos: + uses: ./.github/workflows/build.yml + with: + host: macos-arm64 + runs-on: macos-15 + + e2e: + needs: build-macos + runs-on: macos-15 + timeout-minutes: 40 + steps: + - uses: actions/checkout@v4 + with: + submodules: recursive + - uses: ./.github/actions/setup-macos-llvm + with: + image: macos-15 + - uses: ./.github/actions/use-built-mcpp + with: + host: macos-arm64 + - name: The failing tests, instrumented + run: | + export MCPP MCPP_BOOT MCPP_VENDORED_XLINGS + export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL + "$MCPP" self config + "$MCPP" toolchain default "llvm@${MCPP_LLVM_VER}" + for t in 128_feature_forwarding 76_compile_commands_generated 924_visible_imports_resolve_transitive_modules 35_workspace; do + echo "::group::$t" + bash "tests/e2e/$t.sh" 2>&1 | tail -60 || true + echo "::endgroup::" + done diff --git a/src/build/schedule/detach_codegen.cppm b/src/build/schedule/detach_codegen.cppm index 9673a2a2c..c721ed547 100644 --- a/src/build/schedule/detach_codegen.cppm +++ b/src/build/schedule/detach_codegen.cppm @@ -364,6 +364,19 @@ acquire_permit(const Actor& actor, const std::filesystem::path& requested, int c } if (permit.error().code != mcpplibs::os::errc::again) { std::println(std::cerr, "mcpp: {}", permit.error().message()); + { // [temp] probe 885 macOS: what the failing lock saw. + std::error_code pec; + const auto lockFile = dir / std::format("slot-{}.lock", i); + std::println(std::cerr, "probe885: dir='{}' exists={} lockexists={} cwd='{}' env='{}'", + dir.string(), std::filesystem::exists(dir, pec), std::filesystem::exists(lockFile, pec), + std::filesystem::current_path(pec).string(), + mcpp::invocation::get(kRunEnv).value_or("")); + for (const auto& name : mcpplibs::os::fs::preopens()) std::println(std::cerr, "probe885: preopen '{}'", name); + std::ofstream touch(dir / "probe-touch"); + std::println(std::cerr, "probe885: ofstream create in dir: {}", touch.good()); + auto again = mcpplibs::os::fs::lock::acquire(lockFile, mcpplibs::os::fs::lock::kind::exclusive, false); + std::println(std::cerr, "probe885: second acquire: {}", again ? std::string("ok") : again.error().message()); + } return std::unexpected(1); } } From 4af8445fe6879ac189d1bd370a241da35e6d3bac Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 06:51:10 +0900 Subject: [PATCH 35/40] [temp] probe 885: repeat the macOS tests and run a shard Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/workflows/probe-885-macos.yml | 28 +++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/.github/workflows/probe-885-macos.yml b/.github/workflows/probe-885-macos.yml index 88d40f9ae..76ab1143d 100644 --- a/.github/workflows/probe-885-macos.yml +++ b/.github/workflows/probe-885-macos.yml @@ -31,14 +31,30 @@ jobs: - uses: ./.github/actions/use-built-mcpp with: host: macos-arm64 - - name: The failing tests, instrumented + - name: The failing tests, instrumented, repeated run: | export MCPP MCPP_BOOT MCPP_VENDORED_XLINGS export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL - "$MCPP" self config + "$MCPP" self config > /dev/null "$MCPP" toolchain default "llvm@${MCPP_LLVM_VER}" - for t in 128_feature_forwarding 76_compile_commands_generated 924_visible_imports_resolve_transitive_modules 35_workspace; do - echo "::group::$t" - bash "tests/e2e/$t.sh" 2>&1 | tail -60 || true - echo "::endgroup::" + fails=0 + for rep in $(seq 1 15); do + for t in 128_feature_forwarding 76_compile_commands_generated 924_visible_imports_resolve_transitive_modules 35_workspace; do + if ! bash "tests/e2e/$t.sh" > "$RUNNER_TEMP/$t.$rep.log" 2>&1; then + fails=$((fails+1)); echo "::group::FAIL $t rep $rep"; grep -E "probe885|mcpp:|error" "$RUNNER_TEMP/$t.$rep.log" | head -40; echo "::endgroup::" + fi + done done + echo "failures: $fails" + - name: One e2e shard as CI runs it + if: always() + env: + E2E_SHARD: 1/3 + E2E_EXCLUDE: 721_*.sh + E2E_TIMINGS: tests/e2e/timings/macos.tsv + run: | + export MCPP MCPP_BOOT MCPP_VENDORED_XLINGS + export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL + XLINGS_HOME="$HOME/.mcpp/registry" "$MCPP_VENDORED_XLINGS" install mcpplibs:cmdline@0.0.1 mcpplibs:cmdline@0.0.2 -y > /dev/null + bash tests/e2e/run_all.sh > "$RUNNER_TEMP/shard.log" 2>&1 || true + grep -E "probe885|^FAIL|E2E Summary" "$RUNNER_TEMP/shard.log" | head -80 From eda7bf24a161ad8caa16b420ae6dd4758f59d39e Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 07:20:54 +0900 Subject: [PATCH 36/40] [temp] probe 885: the errno of a contended F_OFD_SETLK on macOS Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/probe/ofd.c | 38 +++++++++++++++++++++++++++ .github/workflows/probe-885-macos.yml | 9 +++++++ 2 files changed, 47 insertions(+) create mode 100644 .github/probe/ofd.c diff --git a/.github/probe/ofd.c b/.github/probe/ofd.c new file mode 100644 index 000000000..259da6c0a --- /dev/null +++ b/.github/probe/ofd.c @@ -0,0 +1,38 @@ +/* [temp] probe 885: the errno of a contended non-waiting F_OFD_SETLK on macOS. + Eight processes open (O_CREAT) and try-lock four files in turn, as + mcpp's compiler slots do, and count every errno they see. */ +#include +#include +#include +#include +#include +#include +#include +#define F_OFD_SETLK 90 +int main(int argc, char** argv) { + const char* dir = argc > 1 ? argv[1] : "."; + for (int p = 0; p < 8; ++p) { + if (fork() == 0) { + int counts[128] = {0}; + for (int round = 0; round < 4000; ++round) { + for (int i = 0; i < 4; ++i) { + char path[512]; + snprintf(path, sizeof path, "%s/slot-%d.lock", dir, i); + int fd = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0666); + if (fd < 0) { counts[errno < 128 ? errno : 127]++; continue; } + struct flock fl = {0}; + fl.l_type = F_WRLCK; fl.l_whence = SEEK_SET; + int r = fcntl(fd, F_OFD_SETLK, &fl); + if (r == 0) { usleep(50); fl.l_type = F_UNLCK; fcntl(fd, F_OFD_SETLK, &fl); counts[0]++; close(fd); break; } + counts[errno < 128 ? errno : 127]++; + close(fd); + } + } + for (int e = 0; e < 128; ++e) + if (counts[e]) printf("child %d: %s (%d): %d\n", p, e ? strerror(e) : "acquired", e, counts[e]); + _exit(0); + } + } + while (wait(NULL) > 0) {} + return 0; +} diff --git a/.github/workflows/probe-885-macos.yml b/.github/workflows/probe-885-macos.yml index 76ab1143d..142119ef8 100644 --- a/.github/workflows/probe-885-macos.yml +++ b/.github/workflows/probe-885-macos.yml @@ -11,6 +11,15 @@ permissions: contents: read jobs: + ofd-errno: + runs-on: macos-15 + steps: + - uses: actions/checkout@v4 + - run: | + clang -O1 -o ofd .github/probe/ofd.c + mkdir -p "$RUNNER_TEMP/slots" && ./ofd "$RUNNER_TEMP/slots" + ./ofd "$RUNNER_TEMP/slots" | awk -F': ' '{print $2}' | sort | uniq -c + build-macos: uses: ./.github/workflows/build.yml with: From c2f2d78689dc377c7aa70b0e414c9e1730e6c81c Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 07:26:22 +0900 Subject: [PATCH 37/40] [temp] probe 885: flush before _exit Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/probe/ofd.c | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/probe/ofd.c b/.github/probe/ofd.c index 259da6c0a..2f357088d 100644 --- a/.github/probe/ofd.c +++ b/.github/probe/ofd.c @@ -30,6 +30,7 @@ int main(int argc, char** argv) { } for (int e = 0; e < 128; ++e) if (counts[e]) printf("child %d: %s (%d): %d\n", p, e ? strerror(e) : "acquired", e, counts[e]); + fflush(stdout); _exit(0); } } From 877e83142dfd8e592659e02b07a7a7e7e379e2e8 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 07:29:11 +0900 Subject: [PATCH 38/40] [temp] probe 885: mcpplibs.os lock::acquire under contention on macOS Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/probe/oslock/mcpp.toml | 6 ++++++ .github/probe/oslock/src/main.cpp | 23 +++++++++++++++++++++++ .github/workflows/probe-885-macos.yml | 10 ++++++++++ 3 files changed, 39 insertions(+) create mode 100644 .github/probe/oslock/mcpp.toml create mode 100644 .github/probe/oslock/src/main.cpp diff --git a/.github/probe/oslock/mcpp.toml b/.github/probe/oslock/mcpp.toml new file mode 100644 index 000000000..02ce66fbf --- /dev/null +++ b/.github/probe/oslock/mcpp.toml @@ -0,0 +1,6 @@ +[package] +name = "oslock" +version = "0.1.0" + +[dependencies.mcpplibs] +os = "0.3.0" diff --git a/.github/probe/oslock/src/main.cpp b/.github/probe/oslock/src/main.cpp new file mode 100644 index 000000000..135e07ed5 --- /dev/null +++ b/.github/probe/oslock/src/main.cpp @@ -0,0 +1,23 @@ +// [temp] probe 885: mcpplibs.os lock::acquire under contention, as mcpp's +// compiler slots take it, counting each error it reports. +import std; +import mcpplibs.os.fs; + +int main(int argc, char** argv) { + const std::filesystem::path dir = argc > 1 ? argv[1] : "."; + std::map seen; + for (int round = 0; round < 3000; ++round) { + for (int i = 0; i < 4; ++i) { + auto lock = mcpplibs::os::fs::lock::acquire(dir / std::format("slot-{}.lock", i), + mcpplibs::os::fs::lock::kind::exclusive, false); + if (lock) { + ++seen["acquired"]; + std::this_thread::sleep_for(std::chrono::microseconds(50)); + break; + } + const auto message = lock.error().message(); + ++seen[message.substr(message.rfind(':') + 1)]; + } + } + for (auto& [what, n] : seen) std::println("{}: {}", what, n); +} diff --git a/.github/workflows/probe-885-macos.yml b/.github/workflows/probe-885-macos.yml index 142119ef8..75190422a 100644 --- a/.github/workflows/probe-885-macos.yml +++ b/.github/workflows/probe-885-macos.yml @@ -40,6 +40,16 @@ jobs: - uses: ./.github/actions/use-built-mcpp with: host: macos-arm64 + - name: mcpplibs.os lock::acquire under contention + run: | + cd .github/probe/oslock + "$MCPP" build --release > build.log 2>&1 || { tail -30 build.log; exit 1; } + bin=$(ls target/*/*/bin/oslock | head -1) + mkdir -p "$RUNNER_TEMP/oslots" + for p in 1 2 3 4 5 6 7 8; do "$bin" "$RUNNER_TEMP/oslots" > "$RUNNER_TEMP/os-$p.txt" 2>&1 & done + wait + cat "$RUNNER_TEMP"/os-*.txt | sort | awk -F': ' '{a[$1]+=$2} END {for (k in a) print k": "a[k]}' + - name: The failing tests, instrumented, repeated run: | export MCPP MCPP_BOOT MCPP_VENDORED_XLINGS From 8ef58f9d94e54cfc8468263f626915ec5901d2bd Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 07:50:59 +0900 Subject: [PATCH 39/40] [temp] probe 885: the first OFD lock attempt of a process Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/probe/ofd-first.c | 38 +++++++++++++++++++++++++++ .github/workflows/probe-885-macos.yml | 2 ++ 2 files changed, 40 insertions(+) create mode 100644 .github/probe/ofd-first.c diff --git a/.github/probe/ofd-first.c b/.github/probe/ofd-first.c new file mode 100644 index 000000000..7da073f40 --- /dev/null +++ b/.github/probe/ofd-first.c @@ -0,0 +1,38 @@ +/* [temp] probe 885: the FIRST F_OFD_SETLK of a process on a lock another + process holds. A holder keeps slot-0 locked; 2000 short-lived processes + each open it and try once; their errnos are counted. */ +#include +#include +#include +#include +#include +#include +#include +#define F_OFD_SETLK 90 +static int try_once(const char* path) { + int fd = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0666); + if (fd < 0) return 1000 + errno; + struct flock fl = {0}; + fl.l_type = F_WRLCK; fl.l_whence = SEEK_SET; + int r = fcntl(fd, F_OFD_SETLK, &fl); + int e = r == 0 ? 0 : errno; + close(fd); + return e; +} +int main(int argc, char** argv) { + char path[512]; + snprintf(path, sizeof path, "%s/slot-0.lock", argc > 1 ? argv[1] : "."); + int holder = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0666); + struct flock fl = {0}; + fl.l_type = F_WRLCK; fl.l_whence = SEEK_SET; + if (fcntl(holder, F_OFD_SETLK, &fl) != 0) { perror("holder"); return 1; } + int counts[2000] = {0}; + for (int i = 0; i < 2000; ++i) { + pid_t p = fork(); + if (p == 0) _exit(try_once(path) & 0xff); + int st = 0; waitpid(p, &st, 0); + counts[WEXITSTATUS(st)]++; + } + for (int e = 0; e < 256; ++e) if (counts[e]) printf("first attempt errno %d (%s): %d\n", e, strerror(e), counts[e]); + return 0; +} diff --git a/.github/workflows/probe-885-macos.yml b/.github/workflows/probe-885-macos.yml index 75190422a..fe14366a6 100644 --- a/.github/workflows/probe-885-macos.yml +++ b/.github/workflows/probe-885-macos.yml @@ -19,6 +19,8 @@ jobs: clang -O1 -o ofd .github/probe/ofd.c mkdir -p "$RUNNER_TEMP/slots" && ./ofd "$RUNNER_TEMP/slots" ./ofd "$RUNNER_TEMP/slots" | awk -F': ' '{print $2}' | sort | uniq -c + clang -O1 -o ofd-first .github/probe/ofd-first.c + mkdir -p "$RUNNER_TEMP/first" && ./ofd-first "$RUNNER_TEMP/first" build-macos: uses: ./.github/workflows/build.yml From d2a28862f65fd31c7bd7731837f192e94601f21f Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 7 Nov 2026 07:53:53 +0900 Subject: [PATCH 40/40] [temp] probe 885: OFD lock holders that end by exiting Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/probe/ofd-exit.c | 41 +++++++++++++++++++++++++++ .github/workflows/probe-885-macos.yml | 2 ++ 2 files changed, 43 insertions(+) create mode 100644 .github/probe/ofd-exit.c diff --git a/.github/probe/ofd-exit.c b/.github/probe/ofd-exit.c new file mode 100644 index 000000000..fde439038 --- /dev/null +++ b/.github/probe/ofd-exit.c @@ -0,0 +1,41 @@ +/* [temp] probe 885: contenders against OFD locks whose holders end by exiting + (no unlock, no close), as mcpp's compiler executors do. 4000 processes in + waves of 16 each take the first free of four slots, hold it 200 us and exit; + every errno other than EAGAIN is counted. */ +#include +#include +#include +#include +#include +#include +#include +#define F_OFD_SETLK 90 +static int child(const char* dir) { + for (int attempt = 0; attempt < 2000; ++attempt) { + for (int i = 0; i < 4; ++i) { + char path[512]; + snprintf(path, sizeof path, "%s/slot-%d.lock", dir, i); + int fd = open(path, O_RDWR | O_CREAT | O_CLOEXEC, 0666); + if (fd < 0) return 100 + (errno & 0x7f); + struct flock fl = {0}; + fl.l_type = F_WRLCK; fl.l_whence = SEEK_SET; + if (fcntl(fd, F_OFD_SETLK, &fl) == 0) { usleep(200); _exit(0); } + if (errno != EAGAIN) return errno & 0x7f; + close(fd); + } + usleep(20); + } + return 99; +} +int main(int argc, char** argv) { + const char* dir = argc > 1 ? argv[1] : "."; + int counts[256] = {0}; + for (int wave = 0; wave < 250; ++wave) { + pid_t pids[16]; + for (int k = 0; k < 16; ++k) if ((pids[k] = fork()) == 0) _exit(child(dir)); + for (int k = 0; k < 16; ++k) { int st = 0; waitpid(pids[k], &st, 0); counts[WEXITSTATUS(st)]++; } + } + for (int e = 0; e < 256; ++e) + if (counts[e]) printf("exit-holder contender: %d (%s): %d\n", e, e == 0 ? "acquired" : e >= 100 ? "open failed" : strerror(e), counts[e]); + return 0; +} diff --git a/.github/workflows/probe-885-macos.yml b/.github/workflows/probe-885-macos.yml index fe14366a6..3fcaa6f47 100644 --- a/.github/workflows/probe-885-macos.yml +++ b/.github/workflows/probe-885-macos.yml @@ -21,6 +21,8 @@ jobs: ./ofd "$RUNNER_TEMP/slots" | awk -F': ' '{print $2}' | sort | uniq -c clang -O1 -o ofd-first .github/probe/ofd-first.c mkdir -p "$RUNNER_TEMP/first" && ./ofd-first "$RUNNER_TEMP/first" + clang -O1 -o ofd-exit .github/probe/ofd-exit.c + mkdir -p "$RUNNER_TEMP/exit" && ./ofd-exit "$RUNNER_TEMP/exit" build-macos: uses: ./.github/workflows/build.yml