ChatGPT status enum does not include "started"

[frockett]

The expected types in the SDK do not match the actual types returned by the ChatGPT scraper:

{"timestamp":1780467628875,"level":"WARNING","logger":"utils.response","message":"unexpected API response shape for datasets/v3/progress","error":"✖ Invalid option: expected one of \"running\"|\"ready\"|\"failed\"|\"cancelled\"|\"error\"\n  → at status","data":"{\"status\":\"starting\",\"snapshot_id\":\"sd_mpxoge3f2aes4c6gd2\",\"dataset_id\":\"gd_m7aof0k82r803d5bjm\"}"}
[scrape-worker] Job 56b9bf14-0130-4cbc-8780-960514ed3007 failed: Error: ChatGPT scrape failed: Unexpected response from datasets/v3/progress: ✖ Invalid option: expected one of "running"|"ready"|"failed"|"cancelled"|"error"
  → at status
    at scrapeChatGPT (/home/crockett/repos/geo-tool/apps/app/src/lib/server/brightdata/chatgpt.ts:48:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async processJob (/home/crockett/repos/geo-tool/apps/app/src/lib/server/workers/scrape-worker.ts:62:13)
    at async Promise.all (index 0)
    at async handleScrapeJob (/home/crockett/repos/geo-tool/apps/app/src/lib/server/workers/scrape-worker.ts:21:2)
    at async resolveWithinSeconds (file:///home/crockett/repos/geo-tool/node_modules/.pnpm/pg-boss@12.18.2/node_modules/pg-boss/dist/tools.js:38:18)
    at async #processJobs (file:///home/crockett/repos/geo-tool/node_modules/.pnpm/pg-boss@12.18.2/node_modules/pg-boss/dist/manager.js:211:28)
    at async Worker.onFetch (file:///home/crockett/repos/geo-tool/node_modules/.pnpm/pg-boss@12.18.2/node_modules/pg-boss/dist/manager.js:304:21)
    at async Worker.run (file:///home/crockett/repos/geo-tool/node_modules/.pnpm/pg-boss@12.18.2/node_modules/pg-boss/dist/worker.js:56:21)

[jrhizor]

Yeah, not just the ChatGPT but all web scrapers are missing starting from the status enum (SnapshotStatusResponseSchema).

[karaposu]

Hi, thanks for the detailed report, and to @jrhizor for confirming the scope.

The fix: instead of just adding "starting" to the allowed list (which would crash again on the next unknown value), the progress endpoint's status is now validated by shape (any non-empty string) rather than against a fixed list. The poll loop only acts on the terminal states (ready/failed/error), so unknown lifecycle values like starting flow through and polling continues, closing the whole class, not just this one value. Malformed responses are still rejected.
We also fixed an adjacent bug in the same path: cancelled wasn't treated as terminal, so a cancelled snapshot polled until timeout instead of stopping. So this resolves both your crash and the broader issue @jrhizor raised, with regression tests added.

Please watch for the next release and let me know if anything's still off. Thanks again!