Skip to content

docs: sphinx-build -j auto everywhere the HTML build runs - #4501

Merged
borisbat merged 1 commit into
masterfrom
bbatkin/sphinx-jobs
Nov 9, 2026
Merged

borisbat merged 1 commit into
masterfrom
bbatkin/sphinx-jobs

Conversation

@borisbat

@borisbat borisbat commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Behavior change: every Sphinx HTML build in the tree runs with -j auto - the Pages step, the two CMake doc targets, preflight's docs gate, and the local scripts and skill recipes that mirror them.

Why. The HTML build ran on one process everywhere. All five extensions the docs load (daslang, myst_parser, sphinx_sitemap, tutorial_video, icon_role) declare parallel_read_safe and parallel_write_safe, so nothing stood in the way - the flag was simply never passed.

What changes.

  • .github/workflows/pages.yml: the "Build Sphinx HTML" step passes -j auto.
  • doc/CMakeLists.txt: das_documentation and build_doc (the run_doc lane) pass -j auto.
  • utils/internal/preflight/main.das: the docs/sphinx-html gate passes -j auto.
  • doc/make.sh, doc/make.bat, doc/make_site.sh, site/README.md, modules/dasImgui/README.md, skills/internal/documentation_rst.md, skills/internal/preflight.md, skills/internal/make_pr.md step 4f (review round one): the same flag in every recipe a reader copies; preflight.md says where it does and does not parallelize.
  • Left alone: doc/make_pdf.ps1 and doc/make_pdf.bat (Windows-only, Sphinx driven from a worker thread).

Observable behavior.

  • Sphinx parallelizes only where it can fork: Linux and macOS spread the read and write phases over every core; on Windows the same flag runs one process and changes nothing, with no warning, so -W is unaffected.
  • Pages run 42526727151 on ubuntu-latest (4 vCPU): the Sphinx step took 3 min 55 s serial. Pages runs on a master push, so the parallel number comes from the first Pages run after this merges.
  • Local WSL timing (Ubuntu 24.04, 96 cores, Sphinx 7.2.6, the tree read over the /mnt/d mount, output on native disk, -W --keep-going, serial then -j auto back to back from an empty doctree): serial 585.2 s -> -j auto 62.2 s, both exit 0 with no warning, the same page set written.

Where to look. .github/workflows/pages.yml "Build Sphinx HTML", doc/CMakeLists.txt build_doc, utils/internal/preflight/main.das the docs/sphinx-html gate.

Validation, claims, ledger

Validation

  • utils/lint/main.das -- utils/internal/preflight/main.das: PASS, 0 issues.
  • dastest --test utils/internal/preflight/tests: 32 tests, 32 passed.
  • .github/workflows/REVIEW.das: clean. git diff --check: clean.
  • Sphinx 7.2.6 (the pin in doc/requirements.txt) in a WSL venv, -W --keep-going, the full doc/source with the generated pages and the staged recordings, serial and -j auto back to back: 585.2 s and 62.2 s, both rc=0, neither log carries a warning. The mount inflates the serial time (every source read crosses /mnt/d), so the ratio on a native disk and on the 4-vCPU runner is smaller; the first Pages run after the merge is the CI number.
  • Windows: sphinx.util.parallel.parallel_available is False (Sphinx gates it on os.name == 'posix' and a fork context); the builder checks that flag before the job count, so -j auto is inert there and prints nothing.

Claims - stated, not tested

  • The CI speed-up on the 4 vCPU runner is bounded by the four cores; the first Pages run after the merge is the measurement.

Not done

  • Newer Sphinx (9.x) still gates parallel builds on fork; the Windows preflight gate stays serial until Sphinx's parallel rewrite (their issue 6881) lands.

Copilot AI balanced review requested due to automatic review settings November 9, 2026 01:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

One documentation inconsistency remains: the PR’s claim that all Sphinx HTML build instructions use -j auto is contradicted by skills/internal/make_pr.md still showing a sphinx-build -b html ... command without it, so the guidance needs alignment before approval.

1 open issue
What changed in this PR

This pull request updates Sphinx HTML build invocations across CI, CMake doc targets, preflight gating, and developer-facing scripts/docs to pass -j auto, enabling parallel Sphinx builds on platforms where Sphinx can fork (and remaining effectively a no-op on Windows).

Changes:

  • Enable parallel Sphinx HTML builds (sphinx-build -j auto) in GitHub Pages workflow and CMake documentation targets.
  • Update the preflight docs/sphinx-html gate to use -j auto.
  • Align local helper scripts and documentation recipes with the parallel build flag (noting platform behavior).

Review notes (unresolved):

  • doc/make.sh: site uses -j auto, but the html target still runs serial; consider updating dirhtml as well for consistency. (moderate)
  • doc/make.bat: site uses -j auto, but the html target below still runs without it; consider updating dirhtml as well. (moderate)
  • skills/internal/preflight.md: references indicate skills/internal/make_pr.md still documents an HTML build command without -j auto; update the recipe or narrow the “everywhere” claim. (nit)
File Summary
utils/​internal/​preflight/​main.das Passes -j auto in the docs/sphinx-html preflight gate.
skills/​internal/​preflight.md Updates preflight docs to mirror the parallel Sphinx HTML command and platform behavior notes.
skills/​internal/​documentation_rst.md Updates documentation workflow recipe to include -j auto.
site/​README.md Updates site/docs build instructions to use -j auto.
modules/​dasImgui/​README.md Updates module doc build instructions to include -j auto.
doc/​make.sh Adds -j auto to the site build path; html target still appears serial.
doc/​make.bat Adds -j auto to the site build path; html target still appears serial.
doc/​make_site.sh Updates the site build helper script to invoke Sphinx with -j auto.
doc/​CMakeLists.txt Adds -j auto to Sphinx HTML invocations in documentation-related targets.
.github/​workflows/​pages.yml Updates the Pages “Build Sphinx HTML” step to run with -j auto.

This pull request requires changes.

Review effort: Balanced

Comment thread skills/internal/preflight.md Resolved
@borisbat
borisbat force-pushed the bbatkin/sphinx-jobs branch from 4ce4aac to e661437 Compare November 9, 2026 01:42
Copilot AI balanced review requested due to automatic review settings November 9, 2026 01:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟢 Approval recommended

The documented changes consistently propagate -j auto to the key CI, CMake, preflight, and developer-facing Sphinx HTML build paths, with only minor nits noted and no outstanding blockers reported in the review evidence.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

Two moderate issues remain where the default html targets in doc/make.sh and doc/make.bat still invoke Sphinx without -j auto, conflicting with the stated intent and leaving the primary HTML build path unoptimized.

2 open issues

Review effort: Balanced

Comment thread doc/make.bat Resolved
Comment thread doc/make.sh Resolved
Every Sphinx HTML build in the tree ran on one process: the pages.yml
step, the two CMake targets behind the doc lane, preflight's docs gate,
and the local scripts and skill recipes that mirror them. All five of
our extensions (daslang, myst_parser, sphinx_sitemap, tutorial_video,
icon_role) declare parallel_read_safe and parallel_write_safe, so the
flag was simply never passed.

Every call site now passes -j auto. Sphinx spreads the read and write
phases over every core on Linux and macOS. It parallelizes only where it
can fork, so on Windows the same flag runs one process and changes
nothing - the preflight gate on a Windows box stays as it was.

The two rinohtype PDF scripts are left alone: they are Windows-only and
drive Sphinx from a worker thread.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat
borisbat force-pushed the bbatkin/sphinx-jobs branch from 9643dce to 382769d Compare November 9, 2026 02:18
Copilot AI balanced review requested due to automatic review settings November 9, 2026 02:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟢 Approval recommended

The change is a straightforward, consistent update to Sphinx build invocations and related documentation to add the platform-safe -j auto flag across CI, CMake targets, and helper scripts, with no active issues identified as blocking approval.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@borisbat
borisbat merged commit 414a552 into master Nov 9, 2026
42 checks passed
@borisbat
borisbat deleted the bbatkin/sphinx-jobs branch November 9, 2026 03:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants