feat: Crawler API, CLI-credential auth, and platform fixes - #31
Merged
Conversation
Basename filenames before stripping reserved chars so "../" and absolute
paths can't escape cwd via saveResults / SnapshotAPI.download.
BREAKING CHANGE: filenames passed to saveResults and snapshot download
are now reduced to their final path segment. Callers relying on nested
subdirectory paths (e.g. "output/data.json") will now write to the
basename only ("data.json") in the working directory.
Adds the Bright Data Crawl API as a top-level service on bdclient,
mirroring brightdata.crawler from the Python SDK. Backed by the same
/datasets/v3/{scrape,trigger,progress,snapshot} endpoints already used
by the platform scrapers — dataset_id gd_m6gjtfmeh43we6cqc.
API surface:
- client.crawler.crawl(urls) — sync /scrape → CrawlResult
- client.crawler.trigger(urls) — async /trigger → ScrapeJob (CrawlJob alias)
- client.crawler.status(snapshotId) — GET /progress → status string
- client.crawler.download(snapshotId) — poll + fetch → CrawlResult
Design choices:
- CrawlResult is a new BaseResult subclass with pageCount + snapshotId,
matching the per-service Result pattern used by ScrapeResult,
SearchResult, and DiscoverResult.
- CrawlJob is a type alias for ScrapeJob — the snapshot-job wrapper is
already generic over SnapshotOperations, no fork needed.
- crawl() and download() never throw on HTTP/network errors, matching
the never-throws-on-orchestrated convention used by toResult().
Tests: 33 unit + 3 gated integration.
fix!: prevent path traversal in FilenameSchema
SnapshotStatusResponseSchema validated the /datasets/v3/progress status field against a fixed list (running|ready|failed|cancelled|error). The API owns this vocabulary and emits values the SDK had not enumerated (e.g. "starting"), so Zod validation threw and killed the polling loop — affecting every async platform scraper. Validate the status field by shape (a non-empty string) instead of membership. pollUntilReady already acts only on terminal states, so unknown lifecycle values flow through and polling continues; malformed responses are still rejected. Also fix an adjacent bug in the same path: "cancelled" was not treated as terminal, so a cancelled snapshot polled until timeout instead of stopping. Adds regression tests for unknown/empty/non-string status and for the cancelled terminal case.
Bun's bundled undici omits the optional `dns` interceptor, so composing the dispatcher chain called `dns()` and threw "dns is not a function", crashing the Transport constructor at client init on Bun. Detect each interceptor by capability (typeof === 'function') and compose only those the runtime provides. On Node this is unchanged (dns + retry present); on Bun `dns` is skipped (undici falls back to the platform resolver, requests still work), and it self-retires if Bun later ships dns.
…resolution
Shipped declarations used extensionless relative imports (e.g. `export { bdclient } from './client'`), which node16/nodenext resolution rejects (TS2834), silently degrading the SDK's public types to `any` for consumers on those modes. They resolved only under moduleResolution:bundler — the mode the declaration build itself used.
Author explicit `.js` on every relative specifier in src/ (551 across 184 files) so tsc emits nodenext-resolvable .d.ts under the existing build; `.js` still resolves to `.ts` under bundler and rollup, so the JS bundle and typecheck are unchanged.
Add scripts/smoke-dist.mjs (wired into `build`): it loads all four entry points in both ESM and CJS from the built dist/ and constructs the client, so a broken emitted specifier fails the build instead of shipping. scripts/add-js-extensions.mjs is the AST codemod that applied the change.
Verified in a nodenext consumer project (skipLibCheck:false): no TS2834, and real types resolve on both the import and require paths.
…user-agent Adds a third token source so `new bdclient()` works with zero config on a machine where the user has run `brightdata login`. Resolution precedence: apiKey param → env (BRIGHTDATA_API_TOKEN / BRIGHTDATA_API_KEY) → CLI credentials store → actionable AuthenticationError. The resolved source (param / env / cli_credentials) is appended to the User-Agent as `brightdata-sdk-js/<version> (auth=<source>)` so SDK onboarding is measurable. src/utils/cli-credentials.ts is a read-only reader whose per-platform path mirrors the CLI exactly (bd-cli get_config_dir): Windows homedir()/AppData/Roaming (not %APPDATA%), macOS ~/Library/Application Support, Linux ~/.config (no XDG). Only credentials.json is read; config.json is never touched, and the token is never logged or put in the user-agent. Tests: per-platform paths, precedence, empty-env fallthrough, malformed/missing/empty credentials, no-credentials error, and the composed user-agent — isolated from the real machine's env + credential store.
…ring
SnapshotStatusResponseSchema validates status as a non-empty string (not an
enum) since the API can add lifecycle values without an SDK release. The
derived public type followed suit and widened to plain string, losing
editor autocomplete/exhaustiveness hints for the values we do know about.
Type it as KnownSnapshotStatus | (string & {}) instead: same runtime
behavior, but IDEs still surface running/ready/failed/cancelled/error while
still accepting any other value the API may return.
It already did its job (all relative specifiers under src/ carry explicit .js extensions now) and isn't wired into build/test/CI, so it was dead weight in the tree. Keeping it around risked someone re-running it against a codebase it was never re-validated for. If we want to keep enforcing explicit extensions on new files going forward, that's better done with an ESLint rule (e.g. import/extensions) than an ad-hoc script someone has to remember to run.
CLI-credential auth resolution + fixes (nodenext types, Bun compat, snapshot status)
# Conflicts: # src/client.ts # src/core/transport.ts
artemo-brd
changed the title
artemo-brd
changed the title
…tors Real Bun (verified on 1.3.11 and 1.3.14) ships an Agent that is a bare, largely-inert stub: no compose(), no close(), no dispatch anywhere on the instance or its prototype chain — and Bun's request()/stream() ignore the dispatcher option entirely, routing through Bun's own native HTTP client regardless of what's passed. The prior fix (#24) only guarded the interceptors passed *into* compose(); it still called .compose(...) and .close() unconditionally, both of which throw under this shape, so bdclient still crashed under real Bun — just later, and with a different error than originally reported. - Transport constructor: only call rawAgent.compose(...) when it exists; otherwise use the raw Agent directly. - Transport.close(): only call agent.close() when it exists. - Rewrote tests/transport-bun.test.ts's undici mock to match Bun's actual shape (no compose/close/dispatch) instead of an invented one that assumed compose existed; added construct+close lifecycle assertions. - README: corrected the Bun caveat — it's not just 'no DNS cache', none of Transport's tuning (pooling/timeouts/retry/DNS-cache) takes effect under Bun, though requests still succeed via Bun's own HTTP client.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features & Services:
client.crawler): NewCrawlerServicefor full-page crawling (crawl()sync,trigger()/download()async via snapshots).CrawlJobis exported as an alias ofScrapeJobso consumers porting from the Python SDK keep the same vocabulary. Reuses the existing snapshot-poll machinery instead of duplicating it.new bdclient()with zero config): The API token is now resolved by precedence —apiKeyparam →BRIGHTDATA_API_TOKEN/BRIGHTDATA_API_KEYenv → credentials stored bybrightdata login(Bright Data CLI) → actionableAuthenticationError. The resolved source (param/env/cli_credentials) is appended to theUser-Agent(brightdata-sdk-js/<version> (auth=<source>)) for onboarding visibility. Credential file paths mirror the CLI exactly per platform (Windows/macOS/Linux); onlycredentials.jsonis ever read, the token is never logged.Fixes:
fix!Path traversal inFilenameSchema(CWE-22) — filenames passed tosaveResults/SnapshotAPI.downloadare now reduced to their basename (path.basename) before sanitizing reserved characters, so../sequences and absolute paths can no longer escape the working directory.BREAKING CHANGE: callers relying on nested subdirectory paths (e.g.
"output/data.json") will now write to the basename only ("data.json") in the working directory..jsextensions to every relative import/export specifier undersrc/so the shipped.d.tsresolve correctly for consumers onmoduleResolution: "node16"/"nodenext"(previously TS2834 / silentany). Added a build-time smoke test (scripts/smoke-dist.mjs, wired intonpm run build) that loads all 4 published entry points in both ESM and CJS from the actualdist/output.Transportnow capability-detects whichundiciinterceptors are available instead of assuming Node's full set — Bun's bundledundiciomitsdns, which previously crashed the constructor with "dns is not a function".SnapshotStatusResponseSchema.statusnow accepts any non-empty string instead of a closed enum, so new lifecycle values from the API (e.g.starting,collecting) no longer throw mid-poll.pollUntilReadynow also treatscancelledas a terminal status.main, carried through the merge):discover()ergonomics,dataFormat/env aliases, and a sharedbeforeExitlistener instead of one perTransportinstance (fixesMaxListenersExceededWarningwhen an app creates many clients).Type safety / cleanup:
SnapshotStatusis now an open union (KnownSnapshotStatus | (string & {})) instead of a barestring, keeping editor autocomplete for the known lifecycle values while still accepting values the API may add later.add-js-extensions.mjscodemod now that it has done its job (not wired into build/test/CI).Testing & validation:
tests/cli-credentials.test.ts,tests/client-auth.test.ts,tests/transport-bun.test.ts,tests/files.test.ts(path-traversal, CWE-22), plus additions totests/polling.test.tsandtests/response-schemas.test.ts.sanity(lint + typecheck),build+ built-package smoke (all 4 entries load in ESM and CJS), and the full suite (380/392 tests, 12 skipped integration) are green ondevafter merging in the latestmain.