feat(api-server)!: Move WebUI metadata, compression, file-listing, and stream-file operations to the API server.

[junhaoliao]

Description

Moves metadata queries, compression submission, filesystem listing, and stream-file extraction out of the WebUI server and into typed Rust API-server endpoints. The WebUI now consumes the generated OpenAPI client for these operations, and the obsolete Fastify routes and database/S3 managers are removed.

This also fixes the issues found while reviewing the original metadata branch and merged PR #2484: CLP text-mode table names now match Python's no-dataset convention; CLP-S aggregate counts decode without silently becoming null; S3 stream files use pre-signed URLs; extraction uses shared job enums, configured collection/target-size values, and an overall timeout; filesystem listing uses the generated snake_case contract and is contained to the configured read-only input root; compression requests preserve the timestamp-key contract and the scheduler's zstd-msgpack framing; compressed files retain host-absolute original paths by stripping the shared /mnt/logs container prefix; timestamp-column names are de-duplicated and deterministically ordered; native search remains active until both result streams finish; and the WebUI proxy has no five-minute Undici timeout. Because all WebUI modes now depend on the migrated endpoints, package validation and Helm rendering reject WebUI deployments without an API server, including Presto deployments.

The WebUI cleanup also removes 19 settings that no longer have readers, its unused Fastify MySQL plugin, and the AWS, MySQL, and MessagePack dependencies orphaned by the migration.

A follow-up audit also aligns compression-path display with the Jobs table, preserves API-server error details in the UI, restores the legacy compression-request validation, removes a superseded timestamp-column query, reduces ingestion-details queries to one scan per table, supports every configured AWS authentication mode, makes metadata table-set selection explicit, checks file-listing containment before probing the filesystem, removes the now-unreachable nullable API-server settings path, and safely handles omitted nullable timestamp fields. Focused unit coverage was added for path resolution, compression-config encoding, query construction, and dataset parsing.

The API-server container now receives the read-only filesystem-input and AWS configuration mounts in both Compose and Helm. Constraining /os/ls and compression paths to logs_input.directory is hardening for a pre-existing unrestricted listing surface, not a regression fix.

Rust linting and formatting now use the known-good nightly-2026-08-21 snapshot instead of a rolling nightly. The 2026-08-22 compiler snapshot crashes internally while checking clp-rust-utils; the same crash occurs on unchanged main, so pinning makes the gate reproducible without changing production builds or runtime behavior.

This is a breaking change because the WebUI routes /api/compress, /api/compress-metadata, /api/os, /api/stream-files, and /api/archive-metadata are removed. Their replacements are served under /api/v1 by the API server.

The API server's own /api/v1/column_metadata/{dataset_name}/timestamp route also moves to /api/v1/metadata/column_metadata/{dataset_name}/timestamp, so any direct consumer of that endpoint needs to be updated.

Checklist

• The PR satisfies the contribution guidelines.
• This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
• Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Final committed source: b7cd0da67c1c5055838dff34d59e512f988b9011; repository: /home/junhao/.codex/worktrees/6905/clp. Unless noted otherwise, commands below exited 0.

Baseline

At clean merged main SHA fcfe3aee252fc8ac0ad5a0942ea029e65ac17f1d in /home/junhao/workspace/clp/.claude/worktrees/verify-main-fcfe3ae, task lint:check-rust, task lint:check-py, task lint:check-js, task lint:check-yaml, task tests:rust-all, task codegen:openapi && task codegen:webui-generate-api-client-schema, and a clean pnpm install --frozen-lockfile && pnpm run build all exited 0; Rust reported 62/62 tests and code generation produced no drift. The live baseline reproduced HTTP 500 responses from all CLP text metadata endpoints because they queried clp_default_archives/clp_default_files, while the database contained clp_archives/clp_files. On CLP-S datasets a,b, the pre-fix ingestion-details endpoint returned num_files: null and num_messages: null while the equivalent aggregate SQL returned numeric values, reproducing the DECIMAL decode problem.

Required automated gates

• task lint:fix-rust — exit 0; installed nightly-2026-08-21 (rustc 1.100.0-nightly (8925ea358 2026-08-20)), then Clippy and rustfmt completed without changing Rust source.
• task lint:check-rust — exit 0; cargo +nightly-2026-08-21 fmt --all -- --check and cargo +nightly-2026-08-21 clippy --all-targets --all-features -- -D warnings passed.
• GitHub Actions clp-rust-checks — passed in 15m28s on a clean runner, including lock validation, lint, and all Rust tests.
• task lint:check-py — exit 0; the task printed the same configured non-blocking mypy diagnostics as baseline.
• task lint:check-js — exit 0; 6/6 Turbo tasks passed.
• task lint:check-yaml — exit 0.
• task tests:rust-all — exit 0; 110/110 tests passed, including the API-server path/config/query tests and the AWS-authentication/config helper tests, and the task stopped its LocalStack container successfully.
• task codegen:openapi && task codegen:webui-generate-api-client-schema — exit 0; neither generated artifact changed.
• From components/webui, old root/package node_modules and dist directories were moved aside, then pnpm install --frozen-lockfile and pnpm run build ran with a client cache miss — exit 0 after the generated nullable request adapter was corrected. On the current tree, pnpm exec turbo build --force also exited 0 with 5/5 uncached builds, including the client, server, generated API client, common package, and Log Viewer.
• task — exit 0 at the final head; the native binaries, Rust services, WebUI, package image, and build/clp-package were rebuilt.
• pnpm --filter @webui/server test from components/webui — exit 0; 4/4 tests passed.
• pnpm install --frozen-lockfile from components/webui — exit 0 across all six workspaces after regenerating the lockfile with the generated Log Viewer workspace present.
• A direct Python validator smoke test set webui.query_engine to Presto and api_server to null; validate_webui_config raised the expected Presto-specific API-server requirement.
• git diff --check — exit 0.

Follow-up audit plan checks

Plans 005–024 were re-verified against the current branch, accepted, implemented, and split into one commit per plan. In addition to the branch-wide gates above:

• Client TypeScript checks and targeted ESLint checks passed for the compression-path display, API error formatting, and optional timestamp-field changes. Direct Node checks covered five display-path cases, eight error-message cases, and six nullable/absent timestamp cases, including epoch zero.
• Common and server TypeScript builds passed after making ApiServerUrl non-nullable; the server test suite remained 4/4. Helm rendered a string ApiServerUrl by default and rejected clpConfig.api_server=null with the expected invariant message.
• Compose configuration and Helm render assertions confirmed that database, AWS, and logs-input credentials/mounts were removed only from the WebUI while the API server retained the mounts it needs.
• The API-server focused test suite reached 38 passing tests before the final workspace run. It covers compression validation, one-scan ingestion queries, containment-before-existence behavior, path resolution, zstd-MessagePack config round trips, and dataset parsing. clp-rust-utils reached 31 passing focused tests, including profile and env_vars authentication and explicit metadata table-set naming.
• Re-running OpenAPI generation at the final head produced no tracked changes, and a clean frozen WebUI install followed by pnpm exec turbo build --force completed 5/5 uncached package builds.
• Polling-backoff tests cover configured initial delay, capped exponential growth, and extraction timeout. Migrated handlers log internal failures before returning redacted HTTP 500 responses, and focused contract tests cover unsupported content types, not-found extraction metadata, nullable fields, and generated schemas.
• Package setup tests verify CLP_HOME=/opt/clp, named-profile mounts for all S3 consumers, and environment credentials scoped to exactly the six services that access S3. Credential files were written with mode 0600, stale credentials were cleared, and missing variables failed setup.
• The API server constructs one stream-output S3 client and reuses it for search-result streaming and pre-signed extraction URLs.

Live deployments and issue-specific checks

The package image used for the deployment matrix was rebuilt from 818bad8122b537af498d6798b2bc66a51c886f5d; task package was run again successfully after merging current origin/main at dfabec6b1401552a1dc75153bb4a9c626db25d93, and the full task build passed after the CI toolchain pin at final head b7cd0da67c1c5055838dff34d59e512f988b9011.

• CLP text: ./sbin/start-clp.sh brought up healthy database, scheduler, worker, API-server, and WebUI containers, then exited 1 because Docker Compose v5.3.1 reports intentionally zero-replica services as missing during up --wait. A plain docker compose up --detach completed. A real filesystem compression job succeeded (1 file, 11 messages); /metadata/time_range, /metadata/space_savings, and /metadata/ingestion_details returned HTTP 200 through both ports 3001 and 4000 with populated values. Playwright showed the populated Ingest cards and expanded the configured input directory to simple.txt. ./sbin/stop-clp.sh exited 0.
• CLP-S / JSON: ten real compression jobs across datasets a and b succeeded, producing 50 matches in ten archives. With max_num_results=5, sorted=true streamed the exact newest timestamps 1311208074120, 1311208074120, 1311207834960, 1311207834960, and 1311207595800. mongosh confirmed all 50 documents and the timestamp-descending index {timestamp: -1, _id: 1}. A Playwright route delayed aggregation results by 8 seconds; after table results arrived the UI remained Running with Cancel available until the aggregation stream finished.
• File browsing and containment: the API-server container saw the read-only filesystem-input mount. /os/ls returned is_expandable and parent_path, Playwright expanded the configured directory and displayed its file, and /etc returned HTTP 400 because it is outside the configured root.
• Multi-dataset ingestion counts: /metadata/ingestion_details?dataset=a,b returned numeric 0,0, not null, so the explicit CAST(... AS SIGNED) values decode correctly and the review's DECIMAL hypothesis was refuted. This fixture also confirmed that the current CLP-S compressor leaves its per-dataset files tables empty; populating those tables is separate database/data-pipeline work and is intentionally excluded from this PR.
• S3 stream files: tools/scripts/localstack/start.py, create-bucket.py, and stop.py all exited 0; cleanup ran successfully. With stream_output.storage.type: s3, env_vars authentication, a custom endpoint, a nondefault key prefix, and target_uncompressed_size: 1024, compression and both S3-backed and Mongo-buffered searches succeeded. The six scoped services received mode-0600 credential files. Clicking a result caused /stream_files/extract to return HTTP 200 with a pre-signed S3 URL; after applying bucket CORS for the browser origin, the object GET returned HTTP 200 and the Log Viewer rendered 3 events with no console errors.
• Long proxy request: a Node stub withheld response headers for 305 seconds behind the real WebUI proxy. curl http://127.0.0.1:4100/api/v1/delay exited 0 with HTTP 200 and time_total=305.027806; the server logged responseTime=305025.92 ms, confirming Undici did not fail at 300 seconds. Both temporary processes were stopped.
• API-server requirement: package --setup-only succeeded for Presto with the API server enabled and rejected api_server: null with the expected message explaining that Presto also reads metadata through the API server. Helm enforces the same invariant, and the official Presto helper no longer disables the API server.

Helm rendering

From the current working tree, helm dependency build tools/deployment/package-helm, task lint:check-helm, and the default helm template tools/deployment/package-helm render exited 0. A complete S3 fixture with credentials also rendered successfully. helm template tools/deployment/package-helm --set clpConfig.api_server=null exited nonzero with the new WebUI/API-server invariant instead of rendering a broken WebUI. The rendered settings JSON parsed successfully with 7 public and 9 server keys. The rendered API-server Deployment had a read-only logs-input mount only for filesystem input and an AWS-config mount when configured. A kind deployment was not run.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}