2026.9.28.1: interface update_packages reports its progress, and nothing but NDJSON reaches stdout

docs/spec/interface-ndjson-v1.md

@@ -1,15 +1,16 @@
 > 编写日期: 2026-05-17 | 版本: 2026.9.27.1
 
-# NDJSON 接口协议规范 v1.1
+# NDJSON 接口协议规范 v1.2
 
 ## 1. 概述
 
-`xlings interface` 提供面向程序的结构化 API，使外部客户端（IDE 插件、CI 脚本、AI agent 等）可通过标准 IO 与 xlings 交互。协议版本为 **1.1**，基于 NDJSON（Newline-Delimited JSON）。
+`xlings interface` 提供面向程序的结构化 API，使外部客户端（IDE 插件、CI 脚本、AI agent 等）可通过标准 IO 与 xlings 交互。协议版本为 **1.2**，基于 NDJSON（Newline-Delimited JSON）。
 
 | 协议版本 | xlings | 变化 |
 |----------|--------|------|
 | 1.0 | 0.4.36 起 | 初始版本 |
 | 1.1 | 2026.9.27.1 起 | 增补：`install_targets` 事件；`install_plan` 条目的第三个元素 `revision`（§6.3.1） |
+| 1.2 | 2026.9.28.1 起 | 增补：`update_packages` 的 `progress` 事件（`index_sync`、`index_rebuild`，§6.1.1）与 `download_progress` 数据事件（§6.3.2）；任何能力都不向 stdout 写 NDJSON 以外的内容（§5） |
 
 次版本号的变化只做增补，1.0 客户端无需修改即可读取 1.1 的输出。客户端应通过**探测能力**
 判断服务端是否提供某项功能（例如 `install_targets` 事件是否出现），而不是比较版本号。
@@ -42,7 +43,7 @@ xlings interface --version
 服务端输出一行后退出：
 
 ```json
-{"protocol_version":"1.1"}
+{"protocol_version":"1.2"}
 ```
 
 ### 3.2 查询可用能力
@@ -122,7 +123,13 @@ xlings interface install_packages --args '{"targets":["gcc@14"],"yes":true}'
 
 ## 5. stdout 事件格式
 
-每行为一个 JSON 对象，必含 `"kind"` 字段标识类型。
+每行为一个 JSON 对象，必含 `"kind"` 字段标识类型。stdout 上不会出现这一行 JSON
+之外的任何字节——包括某能力在执行期间于进程内运行的、xlings 自身不控制的代码
+（例如索引缓存失效后，通过库的 Lua 沙箱执行下载来的索引构建脚本）。这类代码
+即使直接写系统级 stdout，也会被截获：可识别为进度的输出转换成一条 `progress`
+事件，其余转发到 stderr，两种情况都不出现在 stdout 上——此前 `update_packages`／
+`install_packages` 触发的索引重建会把这类脚本的原始终端文本（含 `\r`、ANSI
+清行序列）直接混进 NDJSON 流，客户端解析到该行即失败。
 
 ## 6. 事件类型
 
@@ -140,6 +147,21 @@ xlings interface install_packages --args '{"targets":["gcc@14"],"yes":true}'
 | percent | number | 0-100 整数，-1 表示不确定 |
 | message | string | 人类可读描述 |
 
+#### 6.1.1 `update_packages` 的 progress 事件
+
+`update_packages` 在刷新索引（未指定 `target`，或指定 `target` 时刷新之前）的
+过程中，按下列 phase 报告进度；三者都可能一次都不出现——本地/离线源没有网络
+字节可报，缓存未失效时不会重建：
+
+| phase | 何时出现 | i/n 的含义 |
+|-------|---------|-----------|
+| `index_sync` | 每同步一个配置的索引仓库 | 该仓库在其所属分组（顶层 index_repos，或一批已发现的子索引）中的序号/该分组总数 |
+| `index_rebuild` | 索引缓存失效、重新解析 pkgs/ 时 | 已处理/待处理的包文件总数 |
+
+`percent` 为 `i/n`（分组总数为 0 时为 0）；`message` 同时给出仓库名或包文件名，
+便于在没有专门 UI 的客户端里也能打印出有意义的一行。索引工件下载的字节级进度
+走 `download_progress`（§6.3.2），不走这里。
+
 ### 6.2 log
 
 日志消息。
@@ -230,6 +252,28 @@ xlings interface install_packages --args '{"targets":["gcc@14"],"yes":true}'
 {"kind":"data","dataKind":"install_targets","payload":{"targets":[{"request":"xim:nosuchpkg","namespace":"","name":"","version":"","revision":0,"status":"failed","payload_dir":""}]}}
 ```
 
+#### 6.3.2 download_progress
+
+字节级下载进度，`install_packages` 与 `update_packages` 共用同一形状——后者
+用它报告索引工件（当索引仓库以工件方式获取、且字节数已知时；git/本地源没有
+这一层进度）的下载，客户端因此无需为两者各写一套渲染逻辑。
+
+```json
+{"kind":"data","dataKind":"download_progress","payload":{"files":[{"name":"xim","totalBytes":204800,"downloadedBytes":102400,"started":true,"finished":false,"success":false}],"nameWidth":3,"elapsedSec":1.2,"sizesReady":true,"prevLines":0}}
+```
+
+| 字段 | 类型 | 说明 |
+|------|------|------|
+| files | array | 本次渲染帧里的每一个下载任务 |
+| files[].name | string | 任务标签——`update_packages` 用索引仓库名，`install_packages` 用文件名 |
+| files[].totalBytes | number | 总字节数；`sizesReady` 为 false 时无意义 |
+| files[].downloadedBytes | number | 已下载字节数 |
+| files[].started / finished / success | boolean | 该任务的生命周期 |
+| nameWidth | number | 渲染对齐用的标签列宽；结构化客户端可忽略 |
+| elapsedSec | number | 自本次刷新/安装开始的已用秒数 |
+| sizesReady | boolean | 总字节数是否已知（服务器在响应头给出前不知道） |
+| prevLines | number | 上一帧渲染的终端行数；结构化客户端可忽略 |
+
 ### 6.4 prompt
 
 服务端向客户端请求用户输入。

mcpp.toml

@@ -29,7 +29,7 @@ members = [
 
 [package]
 name = "xlings"
-version = "2026.9.27.1"
+version = "2026.9.28.1"
 description = "Universal package management infrastructure tool with SubOS isolation"
 license = "Apache-2.0"
 repo = "https://github.com/openxlings/xlings"

modules/platform/src/platform.cpp

@@ -6,8 +6,10 @@ module;
 #include <sys/wait.h>
 #include <unistd.h>
 #include <fcntl.h>
+#include <cerrno>
 #else
 #include <io.h>
+#include <fcntl.h>
 #define NOMINMAX
 #include <windows.h>
 #endif
@@ -163,6 +165,126 @@ bool is_tui_mode() {
         return tui_mode_.load(std::memory_order_relaxed);
     }
 
+int dup_stdout_fd() {
+#if defined(_WIN32)
+        return ::_dup(1);
+#else
+        return ::dup(1);
+#endif
+    }
+
+bool write_fd(int fd, std::string_view data) {
+        if (fd < 0) return false;
+        const char* p = data.data();
+        std::size_t remaining = data.size();
+        while (remaining > 0) {
+#if defined(_WIN32)
+            unsigned int chunk = static_cast<unsigned int>(
+                std::min<std::size_t>(remaining, 1u << 20));
+            int n = ::_write(fd, p, chunk);
+            if (n <= 0) return false;
+#else
+            ssize_t n = ::write(fd, p, remaining);
+            if (n < 0) {
+                if (errno == EINTR) continue;
+                return false;
+            }
+            if (n == 0) return false;
+#endif
+            p += n;
+            remaining -= static_cast<std::size_t>(n);
+        }
+        return true;
+    }
+
+void close_fd(int fd) {
+        if (fd < 0) return;
+#if defined(_WIN32)
+        ::_close(fd);
+#else
+        ::close(fd);
+#endif
+    }
+
+StdoutCapture::StdoutCapture(std::function<void(std::string_view)> onLine) {
+        std::fflush(stdout);
+        savedFd_ = dup_stdout_fd();
+        if (savedFd_ < 0) return;  // best-effort: leave fd 1 untouched
+
+        int fds[2];
+#if defined(_WIN32)
+        if (::_pipe(fds, 1u << 16, _O_BINARY) != 0) {
+            close_fd(savedFd_);
+            savedFd_ = -1;
+            return;
+        }
+        if (::_dup2(fds[1], 1) != 0) {
+            ::_close(fds[0]);
+            ::_close(fds[1]);
+            close_fd(savedFd_);
+            savedFd_ = -1;
+            return;
+        }
+        ::_close(fds[1]);
+#else
+        if (::pipe(fds) != 0) {
+            close_fd(savedFd_);
+            savedFd_ = -1;
+            return;
+        }
+        if (::dup2(fds[1], 1) < 0) {
+            ::close(fds[0]);
+            ::close(fds[1]);
+            close_fd(savedFd_);
+            savedFd_ = -1;
+            return;
+        }
+        ::close(fds[1]);
+#endif
+        pipeRead_ = fds[0];
+
+        // EOF on pipeRead_ (a zero-length read) is the signal that fd 1 was
+        // handed back to the real terminal in the destructor -- that closes
+        // the pipe's last write-end reference, which is what unblocks this
+        // read() without any cooperative cancellation.
+        reader_ = std::thread([this, onLine = std::move(onLine)]() {
+            std::string buf;
+            char chunk[4096];
+            for (;;) {
+#if defined(_WIN32)
+                int n = ::_read(pipeRead_, chunk, sizeof(chunk));
+                if (n <= 0) break;
+#else
+                ssize_t n = ::read(pipeRead_, chunk, sizeof(chunk));
+                if (n < 0 && errno == EINTR) continue;
+                if (n <= 0) break;
+#endif
+                for (int i = 0; i < n; ++i) {
+                    char c = chunk[i];
+                    if (c == '\n' || c == '\r') {
+                        if (!buf.empty()) { onLine(buf); buf.clear(); }
+                    } else {
+                        buf.push_back(c);
+                    }
+                }
+            }
+            if (!buf.empty()) onLine(buf);
+        });
+    }
+
+StdoutCapture::~StdoutCapture() {
+        if (savedFd_ < 0) return;
+        std::fflush(stdout);
+#if defined(_WIN32)
+        ::_dup2(savedFd_, 1);
+#else
+        ::dup2(savedFd_, 1);
+#endif
+        close_fd(savedFd_);
+        if (reader_.joinable()) reader_.join();
+        if (pipeRead_ >= 0) close_fd(pipeRead_);
+    }
+
 int exec(const std::string& cmd) {
         std::string actual_cmd = cmd;
         if (tui_mode_.load(std::memory_order_relaxed)) {

modules/platform/src/platform.cppm

@@ -139,6 +139,53 @@ namespace platform {
 
     export bool is_tui_mode();
 
+    // ── Real stdout, decoupled from whatever fd 1 currently means ──────
+    //
+    // The interface protocol (xlings.interface) writes NDJSON to fd 1 and
+    // forbids anything else there. In-process code this program does not
+    // control -- a Lua build script run through a vendored package-index
+    // loader, for the one measured case -- can still put raw text straight
+    // onto the C library's `stdout`, which is fd 1 underneath regardless of
+    // who wrote it. `StdoutCapture` below redirects fd 1 away from the real
+    // terminal for as long as such code might run; a writer that must keep
+    // reaching the real terminal throughout (the NDJSON writer itself) has
+    // to hold ITS OWN duplicate of fd 1, taken with dup_stdout_fd() BEFORE
+    // any capture starts, so the redirection cannot swallow the very output
+    // it exists to protect.
+
+    // A duplicate of the process's current fd 1, usable with write_fd() /
+    // close_fd() regardless of what fd 1 is later redirected to. -1 on
+    // failure (descriptor limit, or an unsupported platform).
+    export int dup_stdout_fd();
+
+    // Write `data` in full to `fd`, retrying across short writes and EINTR.
+    // Returns false on a hard I/O error (fd closed, broken pipe, ...).
+    export bool write_fd(int fd, std::string_view data);
+
+    export void close_fd(int fd);
+
+    // RAII: while alive, redirects the real fd 1 into an internal pipe and
+    // delivers whatever is written there to `onLine`, one line at a time --
+    // split on '\n' AND on a bare '\r', since a self-refreshing terminal
+    // progress line never sends '\n' until it is done. Restores the
+    // original fd 1 on destruction and joins the reader thread.
+    //
+    // Mechanical only: it does not parse what it captures. Not reentrant --
+    // at most one instance should be alive per process, since a nested one's
+    // destructor would hand fd 1 back to the outer capture's pipe rather than
+    // to the real terminal.
+    export class StdoutCapture {
+    public:
+        explicit StdoutCapture(std::function<void(std::string_view)> onLine);
+        ~StdoutCapture();
+        StdoutCapture(const StdoutCapture&) = delete;
+        StdoutCapture& operator=(const StdoutCapture&) = delete;
+    private:
+        int savedFd_   { -1 };
+        int pipeRead_  { -1 };
+        std::thread reader_;
+    };
+
     export int exec(const std::string& cmd);
 
     export [[nodiscard]] std::string shell_quote(const std::string& arg);

src/core/config.cppm

@@ -10,7 +10,7 @@ import xlings.core.xvm.db;
 namespace xlings {
 
 export struct Info {
-    static constexpr std::string_view VERSION = "2026.9.27.1";
+    static constexpr std::string_view VERSION = "2026.9.28.1";
     static constexpr std::string_view REPO = "https://github.com/openxlings/xlings";
 };
 

src/core/xim/commands.cpp

@@ -2708,8 +2708,49 @@ int cmd_clear_xpkg(const std::string& what) {
 }
 
 int cmd_update(const std::string& target, bool yes, EventStream& stream) {
-    // Sync repos
-    if (!sync_all_repos(true)) {
+    // Sync repos. Reported per repository so an interface client can show
+    // progress for a step that otherwise runs silently until it either
+    // finishes or times out (#599's XLINGS_UPDATE_TIMEOUT budget) -- there
+    // used to be nothing on the wire for this at all, only the heartbeat.
+    auto onRepoSync = [&stream](std::string_view repoName, int index, int total) {
+        stream.emit(ProgressEvent{
+            .phase = "index_sync",
+            .percent = total > 0
+                ? static_cast<float>(index) / static_cast<float>(total) : 0.0f,
+            .message = std::format("syncing index repo {}/{}: {}",
+                                   index, total, repoName),
+        });
+    };
+    // Bytes-level progress for whichever repo is fetched as an artifact
+    // (a git clone or a local-source link never calls this). Shaped exactly
+    // like install_packages' own `download_progress` (see DownloadProgressRenderer
+    // above in this file) so a client renders an index download the same way
+    // it already renders a package download, rather than needing a second
+    // code path for one more `files[]` shape.
+    const auto downloadStart = std::chrono::steady_clock::now();
+    auto onIndexBytes = [&stream, downloadStart](std::string_view label,
+                                                 double total, double downloaded) {
+        const bool sizesReady = total > 0.0;
+        const bool finished = sizesReady && downloaded >= total;
+        nlohmann::json files = nlohmann::json::array();
+        files.push_back({
+            {"name", std::string(label)},
+            {"totalBytes", total},
+            {"downloadedBytes", downloaded},
+            {"started", true},
+            {"finished", finished},
+            {"success", finished}
+        });
+        nlohmann::json payload;
+        payload["files"] = std::move(files);
+        payload["nameWidth"] = static_cast<int>(label.size());
+        payload["elapsedSec"] = std::chrono::duration<double>(
+            std::chrono::steady_clock::now() - downloadStart).count();
+        payload["sizesReady"] = sizesReady;
+        payload["prevLines"] = 0;
+        stream.emit(DataEvent{"download_progress", payload.dump()});
+    };
+    if (!sync_all_repos(true, onRepoSync, onIndexBytes)) {
         log::error("failed to sync repositories");
         return 1;
     }