Add automated QA flow: packages/qa runner + qa-flow workflow - #8215
Draft
alfonso-noriega wants to merge 2 commits into
Draft
Add automated QA flow: packages/qa runner + qa-flow workflow#8215alfonso-noriega wants to merge 2 commits into
alfonso-noriega wants to merge 2 commits into
Conversation
Automates the QA doc steps that need no human interaction by driving the CLI directly: execa for non-interactive commands, node-pty for interactive ones (app dev, app config link, hydrogen init). Emits a summary that mirrors the QA doc structure, with manual/delegated items reported as skipped. Verified locally so far: Preparation (version), Hydrogen init + build (hydrogen dev blocked locally by Santa denying workerd; fine on CI runners). Assisted-By: devx/2410058d-4b68-49a7-ba23-7233a4497d19
- app dev block reclassified as manual (interactive session, team decision): no store/password needed by the runner at all - config link driven like the e2e helper: --config + --organization-id skip prompts; create-new-app + app name answered over the pty - exec runs with CI=1 (fail fast on unexpected prompts) and app init writes frozen-lockfile=false, matching GitHub Actions behavior - pty kill() now kills the process group (workerd/vite orphans kept ports) - hydrogen dev pinned to a free port; storefront probe tries IPv4 and IPv6 - qa-flow.yml: non-blocking PR/merge_group job (repo build) + dispatch (published version, default nightly), macOS first Assisted-By: devx/2410058d-4b68-49a7-ba23-7233a4497d19
github-actions
Bot
added
the
no-changelog
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| } | ||
|
|
||
| function escapeCell(text: string): string { | ||
| return text.replace(/\|/g, '\\|').replace(/\n/g, ' ') |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Automates the human-free steps of the CLI Pre-release QA flow doc as a CI job, so release QA gardeners only run the checks that genuinely need a human.
Two pieces:
packages/qa— a standalone runner (no Playwright, no test framework) that drives the CLI directly: non-interactive commands viaexeca, interactive ones (app config link,hydrogen init/dev) vianode-ptywith the same keypresses the doc describes (Enter through defaults, CTRL+C). Every checklist item of the doc is represented in order and reported — automated (auto), needing a human (manual, includes the wholeapp devblock by team decision), or owned by the themes team (delegated). Nothing is silently dropped..github/workflows/qa-flow.yml— non-blocking job on PRs / merge queue testing the CLI built from the branch, plusworkflow_dispatchtesting a published version (defaultnightly). Publishes a summary that mirrors the doc structure into the run summary, ending with the "Remaining manual checklist" for gardeners. macOS first; ubuntu/windows are follow-up matrix flips.What gets automated
Local run (macOS, repo build): 16 passed · 0 failed in ~2m40s.
shopify versionmatches expectedapp init(reactRouter), 5×generate extension(admin_action, theme_app_extension, discount+ts, flow_action, admin_block),function build,function run(doc's JSON via stdin),deploy --version v1,versions list,config link→ create new app (pty), seconddeploy --config stagingshopify app devblock (dev console, admin action, GraphiQL, theme extension, q/dev clean)hydrogen init(pty, default answers ⇒ Mock.shop),build,dev+ storefront HTTP probe + CTRL+CNotable traps the runner handles (all found empirically)
spawn-helper(posix_spawnp failed) — chmod'ed at startupINIT_CWD, and cli-kit'scwd()prefers it over the real cwd — scrubbed and re-pointed per invocationCI=1makes pnpm default tofrozen-lockfile, breakinggenerate extensioninstalls — the runner appendsfrozen-lockfile=falseto the app.npmrc(same fix aspackages/e2e)::1only — the probe tries both families;hydrogen devis pinned to a free--portNPM_CONFIG_NODE_VERSIONfrom exotic environments poisons pnpm engine checks — scrubbedCI setup needed before enabling
Two new repo secrets (Genghis e2e org, mirroring the
E2E_*pattern):QA_ORG_ID— org for the throwawayqa-ci-*appsQA_APP_AUTOMATION_TOKEN— exported asSHOPIFY_APP_AUTOMATION_TOKENfor headless authRunning locally
Follow-ups (not in this PR)
ubuntu-latest, thenwindows-latestin the matrix once macOS is stableqa-ci-*apps in the e2e org (runner names them predictably)