From a06130fc091c22dd438e1c960f643f07d342ab42 Mon Sep 17 00:00:00 2001 From: Duane Nykamp Date: Tue, 18 Aug 2026 17:35:47 -0500 Subject: [PATCH] Add the content.isDescription column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A problem set has no way to include an introduction, a section header, or shared setup text — every child document is a scored, numbered problem. The viewer already has the concept: `SingleDocSource.isDescription` marks a document as unscored and unnumbered. Nothing here can set it. This adds only the column, with no code reading it, so that the migration can be applied on its own. That matters more than usual here: `ADD COLUMN` on `content` cannot use instant DDL — its two FULLTEXT indexes force a full `ALGORITHM=COPY` rebuild, measured at 7 min 3 s on dev3 against a 5-minute deploy timeout. The deploy kills it mid-rebuild, and because MySQL DDL is not transactional the column commits while Prisma's bookkeeping does not, leaving `_prisma_migrations` unfinished. Every subsequent deploy then fails with `P3009` — including a rollback — until someone runs `prisma migrate resolve` by hand. This already happened on dev3 (2026-08-15 02:30 UTC). `MANUAL_MIGRATION_PLAN.md` is the runbook for applying it to prod by hand, with the pre-flight results already gathered. Tracked in #3027. Co-Authored-By: Claude Opus 5 --- MANUAL_MIGRATION_PLAN.md | 197 ++++++++++++++++++ .../migration.sql | 2 + apps/api/prisma/schema.prisma | 3 + 3 files changed, 202 insertions(+) create mode 100644 MANUAL_MIGRATION_PLAN.md create mode 100644 apps/api/prisma/migrations/20260814120000_problem_set_description_doc/migration.sql diff --git a/MANUAL_MIGRATION_PLAN.md b/MANUAL_MIGRATION_PLAN.md new file mode 100644 index 0000000000..8e7fd5304c --- /dev/null +++ b/MANUAL_MIGRATION_PLAN.md @@ -0,0 +1,197 @@ +# Manual migration plan — `20260814120000_problem_set_description_doc` + +Runbook for adding `content.isDescription` to **prod** by hand. + +```sql +ALTER TABLE `content` ADD COLUMN `isDescription` BOOLEAN NOT NULL DEFAULT false; +``` + +A one-line additive column, but on `content` it is a full table rebuild that +takes longer than the deploy's health-check timeout. Left to the deploy it fails +partway and leaves prod undeployable ([why](#why-this-cannot-just-be-deployed)). +So it is applied out-of-band, and the migration ships in this PR alone — ahead +of any code that reads the column. + +**Cost:** ~8–9 min of writes blocked (estimated from 7 min 3 s on dev3). Reads +keep working; the site is read-only during the window, not down. Budget 15–20 +minutes and decide the abort point before starting. + +--- + +## The sequence + +Run steps 1–4 against prod's MySQL, then merge and deploy this PR. + +### 1. Pre-flight + +Take a database backup and open a maintenance window. + +Confirm prod is at exactly 20 applied migrations with nothing broken — otherwise +the deploy in step 5 will also run whatever else is outstanding: + +```sql +SELECT COUNT(*) FROM _prisma_migrations +WHERE finished_at IS NOT NULL AND rolled_back_at IS NULL; -- expect 20 + +SELECT migration_name, started_at FROM _prisma_migrations +WHERE finished_at IS NULL OR rolled_back_at IS NOT NULL; -- expect empty +``` + +Both must match before continuing. If the count is not 20, stop and find out +what else landed. + +### 2. Apply the migration + +Exactly the SQL from +`apps/api/prisma/migrations/20260814120000_problem_set_description_doc/migration.sql`: + +```sql +ALTER TABLE `content` ADD COLUMN `isDescription` BOOLEAN NOT NULL DEFAULT false; +``` + +**Run it somewhere that survives a dropped connection** — `screen`, `tmux`, or +`nohup mysql -e "…" &`. An ECS exec session dying at minute 7 kills the `ALTER`. +That is recoverable (`ALGORITHM=COPY` discards the temp table and leaves the +original intact) but means starting over. + +To watch progress from a second session: + +```sql +SHOW PROCESSLIST; -- the ALTER shows State = "copy to tmp table" +``` + +### 3. Record it in Prisma's bookkeeping + +Prisma did not run the migration, so it has no row for it. Insert one, or the +next deploy tries the `ALTER` again: + +```sql +INSERT INTO _prisma_migrations + (id, checksum, migration_name, started_at, finished_at, applied_steps_count) +VALUES + (UUID(), + 'cffbbc8e54852ec48adcffca534a0c8eeb0c0d10ae13101f8b84274cd1985a2b', + '20260814120000_problem_set_description_doc', + NOW(3), NOW(3), 1); +``` + +`logs` and `rolled_back_at` are nullable and correctly left unset. + +The checksum is the SHA-256 of `migration.sql`, verified against the row Prisma +itself wrote on dev3. It must be exact — Prisma checks applied migrations against +it and a mismatch fails the next deploy with "migration has been modified". If +`migration.sql` is ever edited, this value changes. + +> `prisma migrate resolve --applied` would normally do this, but it requires the +> migration directory to exist locally, and the currently-running prod image +> predates it — that path fails with `P3017`. + +### 4. Verify + +```sql +SHOW COLUMNS FROM content LIKE 'isDescription'; +-- expect: tinyint(1), Null = NO, Default = 0 + +SELECT COUNT(*) FROM _prisma_migrations +WHERE finished_at IS NOT NULL AND rolled_back_at IS NULL; -- now 21 +``` + +Both must pass before step 5. This is the gate — nothing above has been executed +against prod before. + +### 5. Merge and deploy this PR + +Deploy normally. `prisma migrate deploy` finds 21 of 21 applied and goes straight +to `npm run start`; the task should reach steady state in about a minute, as +healthy rollouts do. Nothing in this PR reads the column, so there is no +user-visible change. + +If it instead sits unhealthy for 5 minutes, step 3 did not take. **Do not simply +retry** — a retry cannot fix a bookkeeping problem. Read the task's stdout first: + +```bash +aws logs tail prod --region us-east-2 --since 15m \ + --log-stream-name-prefix doenet/doenet/ --format short +``` + +### Backing out + +Reverting the schema means another full rebuild of the same length and another +write freeze: + +```sql +ALTER TABLE content DROP COLUMN isDescription; +DELETE FROM _prisma_migrations +WHERE migration_name = '20260814120000_problem_set_description_doc'; +``` + +Usually the better option is to leave the column and roll back only the +application image. The column is additive, and the old code neither reads nor +writes it. + +--- + +## Why this cannot just be deployed + +`content` carries two FULLTEXT indexes (`content_name_idx`, `content_source_idx`). +InnoDB cannot use `ALGORITHM=INSTANT` or `INPLACE` for `ADD COLUMN` on such a +table, so it falls all the way back to `ALGORITHM=COPY` — a full table rebuild. +MySQL says so itself: + +``` +mysql> ALTER TABLE content ADD COLUMN _x BOOLEAN NOT NULL DEFAULT false, ALGORITHM=INSTANT; +ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: InnoDB presently +supports one FULLTEXT index creation at a time. Try ALGORITHM=COPY/INPLACE. +``` + +Measured on dev3 (403 MB): **7 min 3 s**. The backend deploy allows **5 minutes** +(`timeout-minutes` defaults to `5` in `.github/actions/update-cluster-with-rollback`, +not overridden by `reusable-deploy-backend.yml`), and `entrypoint.sh` runs +`prisma migrate deploy` _before_ the server listens — so the task never passes a +health check while the rebuild runs, and the deploy kills it partway. + +MySQL has no transactional DDL: the column commits but Prisma never writes +`finished_at`. Every later deploy then fails with `P3009` — **including a +rollback to `main`** — until someone runs `prisma migrate resolve` by hand. This +happened on dev3 on 2026-08-15 and left it undeployable until cleared manually. + +Tracked in #3027. Until that is addressed, migrations touching `content` must be +applied out-of-band. + +| Table size | dev3 | prod | +| --------------- | --------- | ----------------------- | +| `content` data | 383 MB | 459 MB | +| `content` index | 20 MB | 24 MB | +| Rebuild time | 7 min 3 s | **~8–9 min (estimate)** | + +Scale from dev3 with care — prod's MySQL container may not perform identically, +and the FULLTEXT rebuild may not scale linearly. + +## Alternative to steps 2–3: let Prisma do its own bookkeeping + +Avoids the manual `ALTER` and `INSERT` entirely, at the cost of a manual image +push. The deploy pushes to a **mutable** image tag and only runs +`update-service --force-new-deployment` — it never registers a new task +definition — so a one-off task launched after the push picks up the new image: + +1. Build and push the image to the prod tag **without** updating the service + (the workflow does both in one job, so this means running the docker + build/push steps by hand). +2. `aws ecs run-task --cluster prod --task-definition ` — no load + balancer and no health-check clock, so the migration can take as long as it + needs, and Prisma writes its own row. +3. Watch the task's logs; stop it once the migration completes. +4. Verify as in step 4, then deploy normally. + +More faithful to what Prisma expects. + +## Provenance + +The rebuild timing, the `ALGORITHM=INSTANT` error, the checksum, and the +`_prisma_migrations` schema were all verified against **dev3**, and the deploy +timings against the workflows in `.github/`. **None of it has been executed +against prod.** + +Prod facts confirmed 2026-08-17: cluster `prod`, service `doenet-FARGATE`, +`HealthCheckGracePeriod` 600s (the deploy script still gives up at 300s), +`content` at 459 MB + 24 MB index, 20 migrations applied and none broken. diff --git a/apps/api/prisma/migrations/20260814120000_problem_set_description_doc/migration.sql b/apps/api/prisma/migrations/20260814120000_problem_set_description_doc/migration.sql new file mode 100644 index 0000000000..9ac49007e9 --- /dev/null +++ b/apps/api/prisma/migrations/20260814120000_problem_set_description_doc/migration.sql @@ -0,0 +1,2 @@ +-- AlterTable +ALTER TABLE `content` ADD COLUMN `isDescription` BOOLEAN NOT NULL DEFAULT false; diff --git a/apps/api/prisma/schema.prisma b/apps/api/prisma/schema.prisma index 68e89a203a..2a84dc676d 100644 --- a/apps/api/prisma/schema.prisma +++ b/apps/api/prisma/schema.prisma @@ -50,6 +50,9 @@ model content { accessibilityCheck AuditState @default(unchecked) // Specific to documents inside problem sets repeatInProblemSet Int @default(1) + // A description is rendered like any other document but is not one of the + // scored items: no problem number, no credit, no item state of its own. + isDescription Boolean @default(false) // // ====== Problem Sets ========= shuffle Boolean @default(false)