Catalog: Data Product browse (virtual bucket) — dx demo - #5086
Draft
nl0 wants to merge 112 commits into
Draft
Conversation
Completes the previous commit: the integration edits (Layout shell, Bucket.tsx pre slot, Bookmarks index) that pair with the BucketNav/Sidebar drawer removals.
…ding, nested indent
…n, sign out as links)
…divider/highlight
…nt, B: bookmarks/buckets)
…sidebar everywhere
…reorder, paddings
Metadata parity with the in-bucket package list, computed client-side over the fixed member set (no search model): - packagesMeta.ts derives user-meta paths by flattening the loaded members' userMeta into JSON-pointer leaves (plain non-empty objects recurse; scalars, arrays, empty objects are leaves), inferring Number/Boolean/Text from uniform leaf kinds, sorted by coverage. - The leading few paths (strict majority coverage, capped at 3) surface as default table columns (shared CellValue UserMetaValue cells); a compact Columns menu toggles the rest. Members without meta in hand render the usual NoValue cells. - The filter field now understands whitespace-separated terms: `key:value` matches meta (key resolves by full path or last segment; value is a case-insensitive substring, empty value = presence check), everything else AND-matches names. All in memory. - Search/Table re-exports ColumnUserMetaCreate for hand-built column lists (value companion to the already-exported Column types). Read-only stays absolute; all links remain DP-local via PackageLinkBuilder; matchLocations stubs stay all-false. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Athena/ES console pair moves out of the bucket band
(containers/Bucket/Queries -> containers/Queries) and mounts at a
root-level /queries screen with Athena and ElasticSearch subsection
tabs; bare /queries lands on Athena. Gated by requireAuth (the console
always required an authenticated actor).
New named routes: queries (/queries), queriesAthena (/queries/athena,
with { bucket, table } search opts for tabulator deep links),
queriesAthenaWorkgroup (/queries/athena/:workgroup),
queriesAthenaExecution (/queries/athena/:workgroup/:queryExecutionId),
queriesEs (/queries/es). Athena deep-link semantics are preserved.
Bucket becomes a parameter, not a home:
- ES: the index scope is an input on the console (defaults to all
readable indexes; picking a bucket also loads that bucket's saved
queries config and seeds the placeholder index)
- Athena tabulator autofill: the bucket rides in the ?bucket= search
param (set by bucket-page deep links and the legacy redirect)
- create-package from results defaults its destination to the bucket
the manifest entries physically live in
The bucket Queries/ElasticSearch tabs are removed from the bucket
sub-nav (per-bucket ui.nav.queries gating dies with them); old
/b/:bucket/queries... URLs redirect to the global equivalents, keeping
workgroup/executionId/?table= (plus the bucket, as ?bucket=) intact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On the search page (which provides its SearchUIModel above the Layout) the ContentBar binds to the model: the input follows the URL-held searchString and edits are pushed back debounced (500ms, as the in-body field did); suggestions become the model-aware in-search set. The screen hides its in-body field via the existing hideSearch mechanism, and the bar autofocuses when bound so submitting a search from any other page lands on /search with the query preserved and the caret still in the (same-placed) input. Off /search the bar keeps its seed-and-navigate behavior unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ive-state highlighting, rename Qurator
Search links to urls.search({}) after Volumes; Queries now points at the new
global /queries route instead of a hardcoded bucket. Volumes/Search/Queries/Admin
render selected via useRouteMatch against their paths, reusing the ListItem
`selected` affordance already used by the workspace switcher row. Qurator label
becomes "Ask Qurator".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ult count, home filter wrap - Sidebar: highlight the Volumes entry on /data-products/* (data products are volumes), matching the existing /b/* handling. - Search ResultsCount: name the count after the result type — "N packages" for QuiltPackage, "N objects" for S3Object (was "N results" outside the package table view), matching the DP Packages tab. - Landing/Buckets: let the filter row wrap the All|Buckets|Data products and view toggles below the filter input at narrow widths (<=~800px) instead of overflowing and clipping the toggle label. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
At workspace scope with no workgroup context, useWorkgroup resolves to an Error, which surfaced as a red "Workgroup not found" banner. Render the workgroup selector with a neutral "Select a workgroup to run queries" prompt instead. A genuine failure listing workgroups still shows a hard error; query semantics are unchanged (the model still yields the Error). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Listings whose items carry no per-item size (e.g. Data Product object members) otherwise show a misleading "0 B" aggregate in the footer. Add a backward-compatible `hideSize` prop that omits both the size column and the footer size cell; default false preserves existing behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Tabs: size the three fixed tabs to content (drop the 160px sm floor) and collapse a disabled scroll chevron to 0 width, so the narrow-width strip no longer clips Overview into a sliver. - Packages: default sort to 'Most recent first' (in-bucket parity). - Packages: home-page empty-state idiom — an h4 that echoes the filter query instead of a small muted body line. - Packages: 'Sort by:' label with a visually distinct value (match Search/Sort via SelectDropdown's value class). - Packages: aria-labels on the view-toggle icon buttons. - Packages: anchor the metadata Columns menu to its button's right edge so it opens within the card instead of spilling past it. - Objects: hide the size affordance (members carry no size; a registry size increment is deferred). - Overview: render Created as relative time with a full-date tooltip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- BucketGrid: floor every card variant (and the Add tile) to one shared min-height so /?view=card grid rows are uniform, not ragged. - Queries: wrap the active tab panel in a white M.Paper card so the header card no longer reads as an orphaned float. - Sidebar: give the active nav item a stronger treatment (0.24 fill + medium-weight label) to distinguish it from the always-selected workspace chip. - Landing/Buckets: hide the standalone Add Bucket button whenever the in-list add row shows, so only one add affordance appears. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FIX 1: dataProductObjects inserted the logical key into the URL raw, so a key containing #, %, or ? broke the URL. Encode per segment via s3paths.encode (mirroring dataProductPackage / bucketFile) and decode symmetrically in the ObjectsTab consumer. Extend routes.spec.ts with a 'weird #1/file %20.csv' case. FIX 3b: move effectiveRevision + toPackageItem (and the PackageItem/ MemberRevision types) out of DataProduct.tsx into a pure ./packageItems module (the packagesMeta sibling idiom), unchanged, and unit-test the pin-matching / unknown-stats fallback semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FIX 2: the shared package-listing leaves (Search List/Hit Package, Table/CellValue, Table/Entries) take an OPTIONAL links?: PackageLinkBuilder and silently fall back to /b/<bucket>/ bucket routes when omitted — a DP call site that forgets the prop leaks physical links with no signal. Add a regression test under containers/DataProduct rendering the real card + table leaves with the DP link builder (asserting every href is /data-products/-local) and, as the negative, without it (asserting the /b/ fallback), and annotate each `links ?? bucketLinks` fallback site with the DP-invariant dependency. FIX 3a: extract the legacy bucket-queries redirect components (BucketQueriesRedirect + Athena root/workgroup/execution) from App.jsx into ./queryRedirects unchanged (App.jsx behavior identical), and unit-test the redirect targets: /b/:bucket/queries/athena?table=drugs -> /queries/athena?bucket=:bucket&table=drugs, plus the es and workgroup/execution variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bucket tab bar: rename the file-tree tab label 'Bucket' -> 'Files' (routes/values unchanged) and reorder to Overview · Files · Packages · Workflows, matching the sidebar Nav (which already used this label and order). DataProduct tab bar: rename 'Objects' -> 'Files' (route/value stay 'objects'). Both tabbed containers now share Overview · Files · Packages as their common prefix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s" flag, default off Admin > Settings gets a "Data Products" switch beside the beta-features toggle, persisted the same way (catalog/settings.json in the service bucket via CatalogSettings.useWriteSettings, read via the suspending CatalogSettings.use hook, so consumers always render with the final value — no flash of DP content). With the flag off the app carries zero data-products surface: - App.jsx mounts the four /data-products routes only when on; off, the paths fall through to the catch-all 404 - the landing volume list pauses the DataProducts GraphQL query (urql pause — never fired), passes no DP rows/cards down, and hides the type toggle entirely (with DPs off, All == Buckets), ignoring any ?type= riding in the URL - the sidebar Volumes highlight no longer matches /data-products paths when off (they 404) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vitest config Asserts the landing renders no DP rows / no type toggle and keeps the DataProducts query paused when the flag is off, and renders rows, the toggle, and an unpaused query when on (settings hook mocked). The website/* alias mirrors tsconfig baseUrl resolution so vitest can resolve (and mock) website/... imports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…orkgroup not found"
During cold-load credential warmup the AWS credentials identity flaps, and
each flap yields a new athena client -> useWorkgroups' [athena] effect re-fires
the entire ~90-workgroup getWorkGroup fan-out (observed 7 passes / ~930 calls).
The resulting throttling made the one accessible workgroup's probe fail on some
passes; because fetchWorkgroup returned null on ANY error, a transient throttle
was indistinguishable from a by-design AccessDenied, so the list resolved empty
and useWorkgroup reported Error('Workgroup not found').
Two layers:
A. Data correctness (fetchWorkgroup): distinguish transient failures
(Throttling/TooManyRequests/5xx/network) from definitive
AccessDenied/InvalidRequest. Retry only transient failures with bounded
exponential backoff + jitter (3 attempts, 200ms base); by-design denials
stay exactly one call each. Bound the per-page probe fan-out with a p-limit
pool of 10 to reduce throttling at the source. An empty list is now a
genuine signal.
B. Amplification (useWorkgroups): dedupe the fan-out with an in-flight ref so a
re-fired effect (from athena-client identity flapping) joins the existing
pass instead of launching a new one. AWS.Athena.use() suspends until
credentials are usable, so the first captured client is already valid.
Tests: retry-transient-then-success (fake timers), AccessDenied-not-retried
call-count assertion, concurrency-pool bound. Existing genuinely-empty ->
error test stays valid.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…) me useIsAdmin destructured `me.isAdmin` directly; `me` is null when signed out. Harmless while this component only rendered on the always-authed PRODUCT landing, but it's about to also mount on the anonymously reachable OPEN landing, where it would throw on first paint. Guard with optional chaining and add a spec case covering the null-me path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One landing presentation for both modes: OpenLanding now renders the same combined buckets + (flag-gated) data-products list, type filter, text filter, and list/card toggle as the PRODUCT landing, instead of its own plain BucketGrid + local search field. Hero/search stays; only the bucket-grid section is swapped. Drops the dark background wrapper the old, heading-less grid sat in — the shared Buckets component renders its own heading/filter row directly on the page background (as it already does on the PRODUCT landing), so keeping that wrapper would have put low-contrast text on a dark panel. Anonymous mode (OPEN allows signed-out visitors, unlike PRODUCT): useRelevantBuckets/useBuckets already degrade to [] when `alwaysRequiresAuth && !authenticated`; the data-products query is paused unless the admin flag is on and folds to [] on any error either way; the previous commit closes the one gap (isAdmin crashing on a null `me`). Verified live by pointing a local dev config at mode=OPEN: signed-in rendering matches the PRODUCT landing pixel-for- pixel (same rows, toggle, footer) with no console errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scaleX(0.94)->1 expand (and opacity collapse back) read as an unwanted enter animation on the package table view; every view is effectively full-width now, so the transition is removed outright (not gated). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operator ruling (7/06) narrows the Data Product Packages tab from the search-parity metadata surface to a lean fixed-column list. - Drop the Columns show/hide menu (MetaColumnsMenu), the derived user-metadata columns, and the visibleMeta state/toggle. Table columns are now a fixed system-meta set (name, modified, size, file count, comment) plus the workflow column only when a member carries one. - Replace the key:value metadata filter grammar with a plain case-insensitive substring match over the full prefix/suffix package name (virtualName); the placeholder becomes "Filter packages". Count, sort, view toggle, pagination, and the empty/no-match state are kept. - Delete the now-unused packagesMeta.ts / packagesMeta.spec.ts. - Align the card/table toggle and the Sort dropdown to the same 36px height, matching the in-bucket package-list toolbar idiom (Search/Layout/Results.tsx): padding-5 bordered toggle buttons paired with a medium outlined Sort button; centers aligned, spacing(1) gap. packageItems.ts is unchanged apart from a stale comment: hit.meta and tableHit.meta stay populated because the shared leaf hit types require them, and they are harmless with no meta columns. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tab displayed the authored virtual name; the demo DP's members were seeded with single-token virtual names (hurricanes, cord19), so no namespace showed. Display the physical package name (member.name — always prefix/suffix) for the row label, sort, filter, fallback rows, and drill-in breadcrumb; the virtual name still rides the URL segment for routing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* origin/master: (39 commits) catalog: design-context scaffolding (PRODUCT.md + DESIGN.md) (#5124) build(deps): bump actions/setup-python from 6 to 7 (#5123) build(deps): bump actions/setup-node from 6 to 7 (#5122) docs: cover 1.71.0 user-facing changes (#5119) Close preview lambda's streaming response via context manager (#5121) docs: update for 1.70 release (#4981) build(deps): bump mistune from 3.2.1 to 3.3.0 in /lambdas/preview (#5118) build(deps): bump soupsieve from 2.8 to 2.8.4 in /api/python (#5116) build(deps): bump soupsieve from 2.8 to 2.8.4 in /gendocs (#5115) build(deps): bump mistune from 3.2.1 to 3.3.0 in /gendocs (#5117) build(deps): bump soupsieve from 2.6 to 2.8.4 in /lambdas/preview (#5114) catalog: treat any registry 401 as auth-loss (redirect to sign-in) (#5113) build(deps): bump requests from 2.32.5 to 2.33.0 in /gendocs (#5112) build(deps): bump pygments from 2.19.2 to 2.20.0 in /lambdas/pkgevents (#5111) build(deps): bump pygments from 2.19.2 to 2.20.0 in /lambdas/indexer (#5103) build(deps): bump requests from 2.32.5 to 2.33.0 in /api/python (#5104) build(deps): bump pygments from 2.19.2 to 2.20.0 in /lambdas/shared (#5099) build(deps): bump pygments from 2.19.2 to 2.20.0 in /lambdas/pkgpush (#5100) build(deps): bump urllib3 from 2.5.0 to 2.7.0 in /api/python (#5097) build(deps): bump tornado from 6.5.5 to 6.5.7 in /gendocs (#5098) ...
Member
Author
|
Split: the shell/IA re-home now lives in #5129 (branch |
… tree unchanged — this branch already contains the shell work) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * dx-shell-rehome: catalog: workspace-home shell re-home (sidebar, search bar, global queries, files tabs, unified home list)
* dx-shell-rehome: catalog: drop dead website marketing code and the unused websiteTheme
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — the catalog side of the data-exchange DP browse, built on top of Max's workspace-sidebar branch (
catalog-sidebar) for the dx dev-stack demo. Pairs with registryquiltdata/enterprise#1097(DP v0 + own-content) and the spec inquiltdata/quilt-specsdata-exchange(changes/data-exchange).A Data Product is browsed as a self-contained virtual bucket — the physical buckets its members live in are abstracted away (provenance only; no navigation to
/b/<bucket>/…). All browsing stays under/data-products/:id/….What's here
README.mdmember file rendered below (a README is a member, not a content field).shared/graphqlschema re-synced with the registry (DataProduct+title/description+ own-content ops);containers/DataProduct/*; DP routes inconstants/routes.ts+App.jsx.Verification — deployed to the dx dev stack (
dx.dev.quilttest.com) and driven end-to-end (Overview README, packages-list, in-DP package/file drill-in). typecheck/lint/build/routes-test green. No outbound/b/links from the DP browse (grep-verified).Base note — targets
catalog-sidebarfor now (Max's sidebar work is the substrate); re-target tomasterwhen that lands. Draft: demo/POC on the dev stack, not for merge yet.Update 2026-07-03 (pm) — packages-list parity + unified home:
08f099ef): the package card + table row/cells incontainers/Searchgain injectablePackageLinkBuilder+displayNameprops (bucket-route defaults — search screens behavior-identical). Resolves the earlier fork honestly: the DP Packages tab now shares the presentational leaves instead of replicating chrome.202cb307): the in-bucket experience over the fixed member set — real Modified/Size/File count/Comment columns from memberPackagehydration, sortable headers, card↔table toggle, filter, count. Read-only (no create-package). All links DP-local.359c5744+2c78b41d): Data products merged into the single dense volume list (row variant: type chip + member counts), type filter (All | Volumes | Data products) riding?type=, text filter matches DP rows.b4b1ef27): adversarial review found two/b/leaks via shared components reachable from DP surfaces —JsonDisplays3-string links (nownoS3Linksmode on DP cards) and the Markdown preview loader's relative-link rewrite (now an injectableprocessLink; DP READMEs resolve links DP-locally).58b88d31); graphcache keys for DP embedded types — console warning-free.🤖 Generated with Claude Code