diff --git a/.dev/tech-debt.md b/.dev/tech-debt.md index 4b5a30e13..1ee84f858 100644 --- a/.dev/tech-debt.md +++ b/.dev/tech-debt.md @@ -34,6 +34,10 @@ context: tsup@6.7.0 is ~2 years old. Upgrading to 8.5.1 is blocked by the npm ho standalone: yes context: `apps/search-server` depends on `modules/graphql-router` via `"file:../../modules/graphql-router"`, resolved through that package's `dist/` (its `package.json`'s `main`), never live source. Running `search-server`'s or `integration-tests/server`'s tests without first running `npm run build -w modules/graphql-router` silently tests against whatever `dist/` was last built, no warning that it's stale. Concretely hit during the multicatalogue partial-availability work (2026-07-24): the `{ cause: err }` fix in `fetchMapping.ts`/`router.ts` passed every unit test (which import from source via internal path aliases, never crossing the package boundary) but silently produced `unknown_error` instead of `index_not_found` when exercised through a real integration test, because `dist/` was 8 days stale at that point. Only caught because a real end-to-end integration test against live Elasticsearch was written and run (see `integration-tests/server/test/partialAvailability.test.ts`); a unit test alone could not have caught this, by construction. fix: add a `pretest` step to `apps/search-server` and `integration-tests/server` that rebuilds their local `file:` dependencies first, or wire `turbo:test`'s dependency graph to do this automatically (Turbo already tracks the monorepo's build graph); at minimum, document prominently in `AGENTS.md`'s "Running tests" section that changes to `modules/*` require an explicit rebuild before testing any consumer app, the current guidance to "always run from the monorepo root" doesn't by itself guarantee a fresh build. +### No OpenSearch service in `docker-compose.yml` + +standalone: yes +context: `docker-compose.yml` defines only `elasticsearch`, `kibana`, `server` and `ui`, so there is no way to bring up a local OpenSearch cluster. The Makefile carried a `start-os` target that did `up -d opensearch` against this file; since no such service exists it could never have worked, and it was removed 2026-07-30 while correcting the sibling `start-server` target (which referenced `arranger-server`, the *container name* of `server`, rather than the service key). Arranger supports OpenSearch 1.x+ and `SEARCH_ENGINE=opensearch` is a documented `apps/search-server` env var, so local OpenSearch cannot currently be exercised the way Elasticsearch can. Fix: add an `opensearch` service (plus optional OpenSearch Dashboards) to `docker-compose.yml` and restore a `start-os` target. Note the Makefile's existing `COMPOSE_PROJECT_NAME=arranger_es` / `arranger_os` split implies the two engines are meant to be alternatives sharing port 9200, so they should not be started together under the same project name. ## apps/mcp-server @@ -333,6 +337,15 @@ The preferred pattern is **(B)**. Mixing the two makes it harder to find tests, **Fix:** Add a docs page or section covering both endpoints: path (configurable via `PING_PATH`/`READY_PATH`), response shape, HTTP status semantics (`/ready` returns `503` only when `unhealthy`), and the liveness-vs-readiness distinction with the reasoning for why liveness stays catalogue-blind. Cross-link from `GET /introspection` in `05-introspection.md`, since its top-level `status` there is the same computation `/ready` uses. **Standalone:** yes; documentation addition only, no code changes +### README, package.json engines and the Dockerfiles disagree on the Node version + +**Files:** `README.md:21`; `package.json` (`engines.node`); `docker/Dockerfile.local:13,27,83`; `docker/Dockerfile.jenkins:13,36` +**Severity:** medium (a contributor following the README may install a version the tooling does not actually want, and the published `engines` constraint is what consumers of the packages resolve against) +**Kind:** prerequisite drift across three sources of truth +**Issue:** Three different Node versions are stated for the same project. `README.md` lists "Node.js (v22 or higher)" under Development Environment, `package.json` declares `engines.node: ">=20.0.0"`, and every stage in both Dockerfiles builds `FROM node:24-alpine`. There is no `.nvmrc`, `.node-version`, or `volta` block to break the tie, and no CI workflow in this repo to infer the tested version from. Found while fixing link hygiene in the README, so the docs half was in scope but resolving the disagreement is not a documentation question: which value is correct depends on what the tooling actually requires and what the published packages intend to support. +**Fix:** Decide the authoritative version first, then make the three agree. Likely shape: pin the intended development version in a `.nvmrc` (or `volta`) so there is one machine-readable source, set `engines.node` to the lowest version actually supported by consumers (which may legitimately stay below the development version), align the Dockerfiles, and have the README cite the pinned value rather than restating a number. Note the README claim was deliberately left untouched pending this decision. +**Standalone:** no; needs a decision on the supported and intended Node versions before any file changes + --- ## modules/sqon diff --git a/CHANGELOG.md b/CHANGELOG.md index 9feba9e80..cadeffc7d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,7 @@ This file covers high-level release notes for the Arranger project as a whole. W - **Docker image `arranger-server` renamed to `arranger-search-server`**: Update `docker-compose.yml`, Helm values, and any deployment manifests. - **`MAX_RESULTS_WINDOW` is now enforced**: Previously present in the env schema but not applied; now caps query results at `10000` by default. Deployments that return more than 10,000 documents must set this explicitly (via env var or per-catalogue `table.json`). -See [docs/migration/v3.1.md](docs/migration/v3.1.md) for upgrade instructions. +See [docs/reference/08-Migration/v3.1.md](docs/reference/08-Migration/v3.1.md) for upgrade instructions. --- diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index 69e5f2393..54b9ef6e1 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -43,11 +43,13 @@ integration-tests/ Start a local search engine and seed test data: ```bash -make start # starts Elasticsearch via docker-compose -make seed-es # seeds test documents +make start-es # starts Elasticsearch alone, for a host-run dev server +make seed-es # seeds test documents into file_centric_1.0 ``` -The local stack runs without authentication. If you need to test against a secured cluster (OpenSearch or Elasticsearch with the security plugin enabled), see the [search engine permissions reference](docs/setup.md#search-engine-permissions) in the setup documentation for the minimum permissions required per feature. +`make start` brings up Elasticsearch, Kibana and a containerized Arranger server together; prefer `make start-es` when you are going to run the server yourself with `npm run dev:server`. + +The local cluster **does** run with authentication: `docker-compose.yml` sets `xpack.security.enabled: "true"`, and the Makefile passes the credentials it defines (`ES_USER=elastic`, `ES_PASS=unsafePassword123`) through to both the cluster and the containerized server. Use those same values in `apps/search-server/.env` when running the server on the host. For the minimum permissions each feature needs on a cluster you do not control, see the [search engine permissions reference](docs/setup.md#search-engine-permissions) in the setup documentation. Start the development server (watches `sqon`, `types`, `graphql-router`, and `search-server`): diff --git a/Makefile b/Makefile index 5e31a53e9..c8ba3a779 100644 --- a/Makefile +++ b/Makefile @@ -166,15 +166,9 @@ start-es: @echo $(GREEN)$(INFO_HEADER) Succesfully started this service! $(GREEN) @echo $(MAGENTA) "You may have to populate it before using it with the Server. (Use 'make seed-es' for mock data)" $(END) -start-os: - @echo $(YELLOW)$(INFO_HEADER) "Starting the following service: OpenSearch" $(END) - @COMPOSE_PROJECT_NAME=arranger_os $(DC_UP_CMD) opensearch - @echo $(GREEN)$(INFO_HEADER) Succesfully started this service! $(GREEN) - @echo $(MAGENTA) "You may have to populate it before using it with the Server. (Use 'make seed-es' for mock data)" $(END) - start-server: @echo $(YELLOW)$(INFO_HEADER) "Starting the following service: Arranger Server" $(END) - @COMPOSE_PROJECT_NAME=arranger_server $(DC_UP_CMD) arranger-server + @COMPOSE_PROJECT_NAME=arranger_server $(DC_UP_CMD) server @echo $(GREEN)$(INFO_HEADER) Succesfully started this service! $(GREEN) test: diff --git a/README.md b/README.md index 6a1bd8035..0a52e1da3 100644 --- a/README.md +++ b/README.md @@ -10,11 +10,11 @@ Arranger is a versatile, model-agnostic data discovery API for OpenSearch and El ## Documentation -Technical resources for those working with or contributing to the project are available from our official documentation site, the following content can also be read and updated within the `/docs` folder of this repository. +Technical resources for those working with or contributing to the project live in the `/docs` folder of this repository, and are also published, fully rendered, on our [official documentation site](https://docs.overture.bio/develop/Arranger/overview). -- **[Arranger Overview](https://docs.overture.bio/docs/core-software/Arranger/overview)** -- [**Setting up the Development Enviornment**](https://docs.overture.bio/docs/core-software/Arranger/setup) -- [**Common Usage Docs**](https://docs.overture.bio/docs/core-software/Arranger/setup) +- **[Arranger Overview](./docs/overview.md)** +- [**Setting up the Development Environment**](./docs/setup.md) +- [**Reference Docs**](./docs/reference/reference.mdx) ## Development Environment @@ -27,7 +27,7 @@ Technical resources for those working with or contributing to the project are av ## Support & Contributions - For support, feature requests, and bug reports, please see our [Support Guide](https://docs.overture.bio/community/support). -- For detailed information on how to contribute to this project, please see our [Contributing Guide](https://docs.overture.bio/docs/contribution). +- For detailed information on how to contribute to this project, please see our [Contributing Guide](./CONTRIBUTING.md). ## Related Software @@ -45,8 +45,6 @@ The Overture platform includes the following components: | [Lyric](https://github.com/overture-stack/lyric) | A model-agnostic, tabular data submission system | | [Lectern](https://github.com/overture-stack/lectern) | Schema Manager, designed to validate, store, and manage collections of data dictionaries. | -If you'd like to get started using our platform [check out our quickstart guides](https://docs.overture.bio/guides/getting-started) - ## Funding Acknowledgement Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research. diff --git a/apps/mcp-server/README.md b/apps/mcp-server/README.md index aaa665260..8bfbb8a26 100644 --- a/apps/mcp-server/README.md +++ b/apps/mcp-server/README.md @@ -4,6 +4,17 @@ This app is an MCP server that learns how to talk to Arranger by consuming Arran The current scaffold implements the Streamable HTTP MCP transport using **v1.x** of the official [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk/tree/v1.x). +## Tools + +The server registers four tools that cover the full query lifecycle: + +| Tool | Purpose | +| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `list_catalogues` | Returns the catalogues the connected Arranger exposes. | +| `get_sqon_schema` | Returns a compact SQON quick reference (grammar, operators, worked examples) plus the full machine-readable SQON JSON Schema. | +| `get_catalogue_fields` | Returns field introspection for one catalogue: each field's type, display name, unit, description, and valid operators. | +| `execute_query` | Builds, confirms, and executes a SQON-filtered query against a catalogue and returns the matching records. | + ## Folder Structure ```text diff --git a/docs/assets/charts-bar.png b/docs/assets/charts-bar.png new file mode 100644 index 000000000..f07385910 Binary files /dev/null and b/docs/assets/charts-bar.png differ diff --git a/docs/assets/charts-dashboard.png b/docs/assets/charts-dashboard.png new file mode 100644 index 000000000..46d659057 Binary files /dev/null and b/docs/assets/charts-dashboard.png differ diff --git a/docs/assets/charts-sunburst.png b/docs/assets/charts-sunburst.png new file mode 100644 index 000000000..6dd743d98 Binary files /dev/null and b/docs/assets/charts-sunburst.png differ diff --git a/docs/charts.md b/docs/charts.md new file mode 100644 index 000000000..07352a4d5 --- /dev/null +++ b/docs/charts.md @@ -0,0 +1,244 @@ +--- +sidebar_position: 4 +--- + +# Arranger Charts + +Arranger Charts (`@overture-stack/arranger-charts`) is a React chart library for visualizing the aggregation data an Arranger server returns. Charts read the catalogue and SQON state of the search interface they sit in, so every chart re-queries when a user changes a filter. + +Charts under one `ChartsProvider` are fetched together: each chart registers the field it needs on mount, and the provider builds a **single GraphQL query** covering all of them. + +The library is published from the [`modules/charts`](https://github.com/overture-stack/arranger/tree/main/modules/charts) package in the Arranger repository. Charts are rendered with [Nivo](https://nivo.rocks/). + +![A data portal dashboard built with Arranger Charts](./assets/charts-dashboard.png 'Bar and sunburst charts alongside Arranger Components facets in a data portal') + +## Installation + +```bash +npm i @overture-stack/arranger-charts @overture-stack/arranger-components +``` + +Arranger Charts requires an `ArrangerDataProvider` from [`@overture-stack/arranger-components`](https://github.com/overture-stack/arranger/tree/main/modules/components) as a parent component: that provider supplies the API fetcher, the current SQON, the document type, and the extended mapping the charts validate against. It expects React 18, Arranger Components 3, and `@emotion/react`. + +:::caution One catalogue, document type `file` + +Arranger Charts does not yet support the multiple catalogues introduced in Arranger 3.1. It works against a single index, whose `documentType` must be `file`. Progress is tracked in [arranger#1084](https://github.com/overture-stack/arranger/issues/1084). + +::: + +## Quick start + +Wrap your charts in the three providers: `ArrangerDataProvider`, `ChartsProvider`, and `ChartsThemeProvider`. Each chart fills its parent container, so give the container a height. + +```jsx +import { ArrangerDataProvider } from '@overture-stack/arranger-components'; +import { BarChart, ChartsProvider, ChartsThemeProvider } from '@overture-stack/arranger-charts'; + +function App() { + return ( + + + +
+ console.log(data) }} + /> +
+ + + + ); +} +``` + +:::caution `BarChart` needs a `theme` + +`theme.axisBottom` is read directly when the chart renders, so a `BarChart` without a `theme` object throws a `TypeError` as soon as data arrives. The axis legends also default to the literal placeholders `Axis-Bottom-Legend` and `Axis-Left-Legend`, so set both legends (or set them to empty strings) unless you want that text on screen. + +::: + +--- + +## Providers + +### ChartsProvider + +Manages chart registration, query building, and data fetching for every chart below it. + +**Props:** + +- `debugMode` (boolean, default `false`): verbose logging to the browser console +- `loadingDelay` (number, default `50`): milliseconds to hold the loading state, which stops a fast response from flashing the loader +- `disableIncludeMissing` (boolean, default `false`): drop the `__missing__` bucket by querying with `include_missing: false`, so records with no value for the field are excluded + +### ChartsThemeProvider + +Provides the colour palette and the fallback components used by all charts below it. You can nest multiple `ChartsThemeProvider`s under a single `ChartsProvider` to theme groups of charts differently. + +**Props:** + +- `colors` (string[]): hex colours assigned to buckets in order, wrapping around if there are more buckets than colours. Defaults to a 12-colour [d3 categorical palette](https://observablehq.com/@d3/color-schemes). +- `components`: replacements for the three fallback states, which otherwise render as the plain text `Loading...`, `Error`, and `No Data Available` + - `Loader`: shown while the query is in flight + - `ErrorData`: shown when the query fails or the field fails validation + - `EmptyData`: shown when the field has no buckets + +```jsx + + {/* Charts */} + +``` + +--- + +## Charts + +### BarChart + +A horizontal bar chart of one field's buckets. + +![Arranger Charts bar chart](./assets/charts-bar.png 'A BarChart in a consumer application; the card and title are not part of the library') + +**Props:** + +- `fieldName` (string, required): GraphQL field name to visualize. Nested fields use `__` for each level (`primary_diagnosis__age_at_diagnosis`). +- `maxBars` (number, required): how many bars to display. Throws if omitted or `0`. +- `theme` (required): chart configuration, merged over the library's defaults and passed to Nivo's `ResponsiveBar`, so any `ResponsiveBar` prop can be set here + - `axisBottom.legend`, `axisLeft.legend`: axis labels + - `axisBottom.customTickValueSize` (number): place x-axis ticks at multiples of this value instead of Nivo's four automatic ticks + - `sortByKey` (string[]): display the bars in this key order. **Any bucket whose key is not in the list is dropped**, so account for every value the field can take, including `__missing__`. +- `ranges` (Range[]): required for numeric fields, rejected for categorical ones. See [Field types](#field-types). +- `handlers.onClick`: called with the clicked bar; `data.label`, `data.value`, and `data.key` carry the bucket +- `disableTopBarsCount` (boolean, default `false`): hide the `Top N of M` badge that appears when the field has more buckets than `maxBars` + +```jsx + { + console.log('Clicked', data.label, data.value); + }, + }} +/> +``` + +Without `sortByKey`, the chart takes the `maxBars` largest buckets and draws them ascending from the axis, so the largest bar is at the top. Bar labels are truncated to seven characters; the full value is in the tooltip. + +### SunburstChart + +Two concentric rings showing specific values grouped into broader categories. The inner ring holds the categories your `mapper` returns, the outer ring the field's own values, and the legend lists the categories. A category and its values share a hue, with the inner ring drawn at half opacity. + +![Arranger Charts sunburst chart](./assets/charts-sunburst.png 'A SunburstChart in a consumer application; the card and title are not part of the library') + +**Props:** + +- `fieldName` (string, required): GraphQL field name to visualize +- `mapper` (function, required): maps one of the field's values to the category it belongs to. Throws if omitted. A value the mapper returns nothing for is left out of the chart, so the mapper doubles as a filter; if it maps nothing at all, the chart renders its empty state. +- `maxSegments` (number, required): how many **categories** (inner-ring segments) to display. Every value belonging to a displayed category is drawn, so the number of outer segments is not capped directly. Throws if omitted or `0`. +- `handlers.onClick`: called with the clicked segment plus an `ids` array — the values under a category when the inner ring is clicked, or the single value when the outer ring is clicked, which is what you would feed into a SQON filter + +```jsx + { + // Map specific diagnosis codes to broader categories + if (diagnosisCode.startsWith('C78')) return 'Metastatic'; + if (diagnosisCode.startsWith('C50')) return 'Breast Cancer'; + return diagnosisCode; // falling through to the raw value keeps it in the chart + }} + handlers={{ + onClick: (data) => { + console.log('Selected category:', data.ids); + }, + }} +/> +``` + +Categories are ordered by total, largest first. `SunburstChart` accepts a `theme` prop, but nothing currently reads it: the chart's only theming is the palette from `ChartsThemeProvider`. + +### NetworkNodesChart + +A bar chart of the nodes in a [federated search](./federated-search.md) network, one bar per node showing its hit count. It takes no `fieldName`: it reads the `network.nodes` part of the response rather than a field aggregation, and asks the provider to include the network query on mount. + +**Props:** + +- `theme` (required): as for `BarChart`, plus `sortAlphabetically` (boolean, default `true`). Set it to `false` to sort by hit count descending instead. +- `maxBars` (number, default unlimited): how many nodes to display +- `handlers.onClick`: called with the clicked bar; the label is the node name and the value its hit count +- `disableTopBarsCount` (boolean, default `false`): hide the `Top N of M` badge + +```jsx + +``` + +--- + +## Field types + +Charts resolve a field's type from Arranger's [extended mapping](./reference/00-index-mappings.md) and query it accordingly: + +| Aggregation type | Index field types | `ranges` | +| --------------------- | ---------------------------------------------------------------------------------------------------- | -------- | +| `Aggregations` | `keyword`, `text`, `string`, `boolean`, `object`, `id` | not used | +| `NumericAggregations` | `integer`, `long`, `double`, `float`, `half_float`, `scaled_float`, `unsigned_long`, `bytes`, `date` | required | + +A numeric field without `ranges` fails validation and renders the error state, with the reason logged to the console. Note that `date` counts as numeric here, so date fields need `ranges` too. A field that isn't in the extended mapping at all is treated as categorical, which is what a misspelled `fieldName` looks like: no validation error, just an empty or failed query. + +Ranges are `{ key, from, to }`, with `from` inclusive and `to` exclusive: + +```jsx + 65', from: 66 }, + ]} +/> +``` + +--- + +## Behaviour worth knowing + +- **One chart per field, per provider.** Registration is keyed by field name, so if two charts under the same `ChartsProvider` request the same field, the second registration is ignored and both render the first one's data. Two charts on one field with different `ranges` is therefore not supported: put them under separate providers. +- **Colours are stable.** Each bucket key keeps its colour as data changes, and the assignment is cached in `sessionStorage` under `arranger-charts-`, so it survives a remount within the browser session. +- **Records with no value** appear as a bucket labelled `No Data` (`__missing__` in `sortByKey` and in the raw data). Set `disableIncludeMissing` on the provider to leave them out. +- **Counts of 0 still draw a bar.** A zero-count bucket renders a minimum-width bar whose tooltip reads `Too few` rather than `0`. +- **Tooltips are styleable.** The built-in tooltip carries `tooltip-container`, `tooltip-label` (with a `data-label` attribute), and `tooltip-data` class hooks; the `Top N of M` badge carries `top-chart-bar-items-count`. Passing a custom tooltip component is not supported yet. + +## Debugging + +Set `debugMode` on `ChartsProvider` to log the data pipeline to the browser console: which fields registered and deregistered, and the aggregation type each field resolved to. It is the first thing to turn on when a chart renders blank, since a field name that doesn't match the extended mapping is reported there as a missing mapping. Validation failures are logged whether or not `debugMode` is set. diff --git a/docs/federated-search.md b/docs/federated-search.md new file mode 100644 index 000000000..d12fd2b10 --- /dev/null +++ b/docs/federated-search.md @@ -0,0 +1,242 @@ +--- +sidebar_position: 6 +--- + +# Arranger Federated search + +Arranger can answer a single query using data held on several Arranger servers at once. Each server keeps its own index and never ships documents anywhere; only aggregate counts cross the wire. The querying server merges those counts into one response. + +The feature has three names in the codebase and in older documentation: **federated search**, **network search**, and **network aggregation**. They all mean this feature. Configuration and GraphQL field names use `network`. + +:::info Aggregates only + +Federated search returns **counts, not documents**. You get per-field bucket counts and a total hit count per node. There is no federated `hits` list: a user cannot page through records held on a remote node through the federated query. Point them at that node's own portal for record-level access. + +::: + +--- + +## When to use it + +Use federated search when several organisations each run their own Arranger over their own data, and you want one portal that reports totals across all of them without any organisation handing over its records. + +A typical arrangement: each site runs a normal Arranger server, and one of them (or a separate server) additionally carries a `network` configuration listing the others. That server exposes the federated query. Sites remain independent, each controlling its own index, access rules, and uptime. + +--- + +## How it works + +**At startup**, the querying server contacts every remote node listed in its `network` configuration and asks which aggregation fields that node has, using a GraphQL `__type` introspection query against `Aggregations`. It builds the federated schema from the **union** of the fields discovered across all nodes, plus the local node's own fields. + +**Per query**, the server: + +1. Sends each remote node a query for only the requested fields that node actually has, plus that node's total hits. +2. Queries the local node, if one is configured, through its in-process resolvers rather than over HTTP. +3. Merges the returned buckets field by field, summing `doc_count` for buckets that share a `key`. +4. Returns the merged aggregations alongside a per-node status list. + +All node requests run concurrently, and one node failing does not fail the query. See [Node status and failures](#node-status-and-failures). + +### Field discovery is a union, not an intersection + +A field present on only one node still appears in the federated schema. When a node does not have a requested field, that node contributes a single sentinel bucket instead: + +```json +{ "key": "___aggregation_not_available___", "doc_count": 4210 } +``` + +`doc_count` is that node's total hits for the query. This keeps the arithmetic honest: the merged buckets plus the sentinel bucket account for every matching document across the network, so an interface can show "4,210 records at Node B, not broken down by this field" rather than silently under-reporting. + +:::warning Nodes must agree on field names + +Merging is by **field name and aggregation type**. Two nodes only combine on a field if both call it the same thing. `donor__gender` on one node and `donor_sex` on the other produce two separate fields, each carrying a sentinel bucket from the node that lacks it. Agreeing on index field names across the network is a prerequisite, not something Arranger can reconcile for you. + +::: + +--- + +## Configuration + +Configuration differs between the two ways Arranger is run. Pick the one matching your deployment. + +### With `search-server` (configuration files) + +Add a `network.json` to the catalogue's configuration directory, alongside `base.json` and the other files described in [Catalogue configuration](./reference/01-arranger-configs.md). + +```json +{ + "network": { + "localNode": { + "displayName": "Toronto", + "nodeId": "toronto" + }, + "remoteNodes": [ + { + "displayName": "Montreal", + "documentType": "file", + "graphqlUrl": "https://montreal.example.org/graphql", + "nodeId": "montreal" + }, + { + "displayName": "Vancouver", + "documentType": "file", + "graphqlUrl": "https://vancouver.example.org/graphql", + "nodeId": "vancouver", + "requests": { + "headers": ["Authorization", "X-Api-Key"] + } + } + ], + "remoteRequests": { + "headers": ["Authorization"] + } + } +} +``` + +| Field | Required | Description | +| -------------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `localNode.displayName` | yes, to include local data | Label for this server's own catalogue in the results. Omit the whole `localNode` block to federate over remote nodes only. | +| `localNode.nodeId` | recommended | Stable identifier, used by `nodesFilter`. | +| `remoteNodes[].displayName` | yes | Label for this node in the results. Also used to match responses back to nodes, so keep it unique across the network. | +| `remoteNodes[].documentType` | yes | The remote catalogue's `documentType`, meaning its root GraphQL field (for example `file`). Arranger appends `Aggregations` to this value during field discovery, so give the bare document type, not an aggregations type name. | +| `remoteNodes[].graphqlUrl` | yes | The remote Arranger's GraphQL endpoint. | +| `remoteNodes[].nodeId` | recommended | Stable identifier, used by `nodesFilter`. | +| `remoteNodes[].requests.headers` | no | Header names to forward to this node. **Replaces** `remoteRequests.headers` for this node rather than adding to it. | +| `remoteRequests.headers` | no | Header names to copy from the incoming request onto every outgoing remote request. | + +**Header passthrough is how authorization reaches remote nodes.** Listing `Authorization` copies the caller's token onto each remote request, letting every node apply its own access rules to the caller's identity. Only list headers the remote nodes should genuinely receive: each name listed is forwarded verbatim to every node it applies to. + +### With `graphql-router` (library) + +When embedding [`@overture-stack/arranger-graphql-router`](https://github.com/overture-stack/arranger/tree/main/modules/graphql-router) directly, the `network` block takes `localNode` and `remoteNodes` as above, but **not** `remoteRequests` or `remoteNodes[].requests`. Those two are `search-server` configuration-file conveniences: the server normalizes them into a `customizeRemoteRequest` function before handing the configuration to the library. At the library level, supply that function yourself. + +```ts +const router = await arrangerRouter({ + configs: { + documentType: "file", + esHost: "http://localhost:9200", + esIndex: "file_centric", + network: { + customizeRemoteRequest: ({ context, remoteNode }) => ({ + headers: { + Authorization: context.request.headers.get("Authorization") ?? "", + }, + }), + localNode: { displayName: "Toronto", nodeId: "toronto" }, + remoteNodes: [ + { + displayName: "Montreal", + documentType: "file", + graphqlUrl: "https://montreal.example.org/graphql", + nodeId: "montreal", + }, + ], + }, + }, +}); +``` + +`customizeRemoteRequest` runs once per node per query and receives that node's configuration, so it can vary credentials by destination. + +### Remote nodes must allow GraphQL introspection + +Field discovery uses a GraphQL `__type` query. A remote node running with [`disableGraphQLIntrospection`](./reference/07-feature-flags.md) set to `true` fails discovery and is reported as an errored node for the lifetime of the querying server's process. + +This matters because `disableGraphQLIntrospection` defaults to `true` when `NODE_ENV=production`. **Any node serving as a remote target in a federated deployment must explicitly set it to `false`.** Replacing this dependency with the REST [Introspection API](./reference/05-introspection.md) is tracked as tech debt. + +--- + +## Querying + +`network` is a root field on the catalogue's GraphQL schema, sitting beside the document type field. + +```graphql +query FederatedFacets($filters: JSON) { + network(filters: $filters) { + nodes { + nodeId + name + hits + status + errors + } + aggregations { + donor__gender { + bucket_count + buckets { + key + doc_count + } + } + } + } +} +``` + +### Arguments + +| Argument | Type | Description | +| -------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------- | +| `aggregations_filter_themselves` | `Boolean` | Passed through to each node's aggregation query. | +| `filters` | `JSON` | A [SQON](./reference/04-sqon-in-detail.md) applied on every node. A malformed value is rejected with an error. | +| `include_missing` | `Boolean` | Passed through to each node's aggregation query. | +| `nodesFilter` | `[String]` | Restrict the query to these `nodeId` values. Absent or empty means all nodes. | + +`nodesFilter` matches on `nodeId` only. A node configured without a `nodeId` is silently excluded whenever `nodesFilter` is non-empty, so give every node a `nodeId` if you intend to use it. + +### Response + +**`aggregations`** holds the merged result per field. `buckets` is the union of keys across nodes, with `doc_count` summed per key. `bucket_count` is recomputed from the merged bucket list, so it reflects distinct keys across the network rather than any single node's count. + +**`nodes`** reports every configured node, including those that failed. + +| Field | Description | +| -------- | ----------------------------------------------------------------------- | +| `errors` | Error message when `status` is `ERROR`; an empty string otherwise. | +| `hits` | Total documents matching `filters` on that node; `0` for a failed node. | +| `name` | The configured `displayName`. | +| `nodeId` | The configured `nodeId`, if any. | +| `status` | `OK` or `ERROR`. | + +Nodes are sorted by `name`, so local and remote nodes interleave rather than grouping by kind. + +--- + +## Node status and failures + +Federated search degrades rather than failing. There are three distinct failure points. + +**Startup discovery failure.** A node was unreachable, or returned an unusable schema, when the server booted. The server logs the reason, starts anyway, and reports that node in `nodes` with `status: "ERROR"` and `hits: 0`. There is no runtime retry: the node stays errored until the querying server restarts. + +**Query-time failure.** A node was reachable at startup but failed or timed out on this query. It appears with `status: "ERROR"`, `hits: 0`, and the error message. Every other node's data is still returned. + +**No fields anywhere.** If no node exposes any supported aggregation field, the federated schema cannot be built. The server logs an error and starts **without** the `network` field on the schema, so federated queries fail schema validation. Check that remote nodes allow introspection and that their `documentType` values are correct. + +Because a failed node reports `hits: 0` rather than an absence, an interface that sums `hits` across `nodes` silently under-counts during an outage. Read `status` before presenting network totals as complete. + +--- + +## Limitations + +- **Categorical fields only.** Only the `Aggregations` type federates. Numeric and date fields, which Arranger exposes as `NumericAggregations`, are excluded from the federated schema entirely: they do not appear under `network.aggregations` even when every node has them. +- **No document-level results.** Counts only. See the note at the top of this page. +- **One local node.** A server contributes at most one of its own catalogues to the network. Federating multiple local catalogues from a multicatalogue server is not yet supported. +- **Version parity is assumed.** Nodes are expected to run compatible Arranger versions. There is no capability negotiation and no version check at startup. +- **No live health monitoring.** Node status reflects the current query and the startup discovery result. Nothing polls nodes in between. + +--- + +## In the interface + +[Arranger Charts](./charts.md) includes `NetworkNodesChart`, a bar chart of hit counts per node that reads `network.nodes` from the federated response. Charts using the network query accept a `networkNodesFilter` array, passed through as the `nodesFilter` argument, letting a portal offer per-node toggles. + +--- + +## Related pages + +- [Catalogue configuration](./reference/01-arranger-configs.md): the other configuration files in a catalogue directory +- [Feature flags](./reference/07-feature-flags.md): including `disableGraphQLIntrospection` +- [Introspection API](./reference/05-introspection.md): the REST alternative to GraphQL introspection +- [SQON in detail](./reference/04-sqon-in-detail.md): the filter format accepted by `filters` +- [Arranger Charts](./charts.md): `NetworkNodesChart` and network-aware chart queries diff --git a/docs/usage/06-ai-and-automation.md b/docs/mcp-server.md similarity index 86% rename from docs/usage/06-ai-and-automation.md rename to docs/mcp-server.md index f5d6b79ea..ff6968562 100644 --- a/docs/usage/06-ai-and-automation.md +++ b/docs/mcp-server.md @@ -1,14 +1,16 @@ -# AI and automation - -Arranger exposes its catalogue data and query tools to AI models, scripts, and pipelines through two surfaces: a REST introspection API and a dedicated MCP server. This page introduces both and points to where to go next. - +--- +sidebar_position: 5 --- -## MCP server +# Arranger MCP server + +The Arranger MCP server exposes a running Arranger instance's catalogue data and query tools to AI models, scripts, and pipelines. It is one of two surfaces for that purpose; the other is the [Introspection API](./reference/05-introspection.md), a set of read-only REST endpoints that any client can call directly. The `arranger-mcp-server` package implements the [Model Context Protocol](https://modelcontextprotocol.io/) over Streamable HTTP. Connect any MCP-compatible AI client to it and the client can discover available catalogues, retrieve field metadata and the SQON schema, and construct search queries: without needing Arranger-specific integration code on the model side. -### Quick start +--- + +## Quick start ```bash # from the monorepo root @@ -24,7 +26,7 @@ The server starts on `http://localhost:3100/mcp` by default. Two environment var All other variables (host, port, path, log level, request timeout) have sensible defaults. Copy `apps/mcp-server/.env.schema` to `apps/mcp-server/.env` to start from a working local baseline. -### What the server exposes +## What the server exposes **Instructions** (sent once, in the `initialize` response): @@ -47,7 +49,7 @@ The server returns a short set of usage instructions that most clients fold into - `query_arranger`: accepts the user's goal as an input, and returns three messages containing the "system prompt" (workflow instructions), a SQON cheat sheet, and the user's goal -### Connecting a client +## Connecting a client Any MCP-compatible client that supports Streamable HTTP can connect. Point it at the MCP server URL (`http://127.0.0.1:3100/mcp` with default config) and use transport type `streamable-http`. @@ -63,7 +65,7 @@ For **LM Studio** and other model hosts, follow the client's documentation to ad ## SQON generation -When constructing SQONs from a script, pipeline, or model, use the [introspection API](./05-introspection.md) to derive field names, types, and valid operators at runtime rather than hard-coding them. This keeps the client current when a catalogue mapping changes. +When constructing SQONs from a script, pipeline, or model, use the [introspection API](./reference/05-introspection.md) to derive field names, types, and valid operators at runtime rather than hard-coding them. This keeps the client current when a catalogue mapping changes. Safe defaults for programmatic SQON construction: @@ -75,7 +77,7 @@ Safe defaults for programmatic SQON construction: - Do not invent `pivot` values; derive them from the live catalogue mapping or omit them - Use `not-in` for value exclusion, not `not { in: [...] }`: combining the two is a double negative -For a detailed walkthrough of the SQON format and how to compose queries, see [Building SQON queries](./03-building-sqon-queries.md). +For a detailed walkthrough of the SQON format and how to compose queries, see [Building SQON queries](./reference/03-building-sqon-queries.md). --- diff --git a/docs/overview.md b/docs/overview.md index e55622ca1..423ab8692 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -6,13 +6,13 @@ sidebar_position: 1 Arranger is a versatile, model-agnostic data discovery API for OpenSearch and Elasticsearch, designed to simplify building search interfaces for complex datasets. A React component library is available for generating interactive search UIs. - :::info Supported search engines +:::info Supported search engines - Arranger supports **OpenSearch 1.x or higher** and **Elasticsearch 7.x** (minimum 7.0, licensed/default distribution only; ES OSS and ES 8.x are not supported; the bundled client is `@elastic/elasticsearch` v7). +Arranger supports **OpenSearch 1.x or higher** and **Elasticsearch 7.x** (minimum 7.0, licensed/default distribution only; ES OSS and ES 8.x are not supported; the bundled client is `@elastic/elasticsearch` v7). - OpenSearch maintains API compatibility with Elasticsearch 7.x, so query syntax and conventions documented here apply to both engines. +OpenSearch maintains API compatibility with Elasticsearch 7.x, so query syntax and conventions documented here apply to both engines. - ::: +::: ## Key Features @@ -23,6 +23,7 @@ Arranger is a versatile, model-agnostic data discovery API for OpenSearch and El - SQON integration for human-readable and machine-processable search queries - **Model-Agnostic:** Works with any properly structured OpenSearch or Elasticsearch index. - **Integration-Ready:** The search API integrates with any web front end; a React component library is included for building search UIs. +- **Federated Search:** One query can report aggregate counts across several independently operated Arranger servers. Each site keeps its own index and records; only counts cross the wire. See [Federated search](./federated-search.md). ## System Architecture @@ -31,7 +32,7 @@ Arranger integrates with your OpenSearch or Elasticsearch cluster to generate a - **Arranger Server:** The back-end search API service that: - Generates a GraphQL API from Elasticsearch mappings - Acts as middleware between the UI and Elasticsearch - - Simplifies querying and filtering using Serializable Query Object Notation ([SQON](./usage/04-sqon-in-detail.md)) + - Simplifies querying and filtering using Serializable Query Object Notation ([SQON](./reference/04-sqon-in-detail.md)) - Provides an intermediary layer to avoid direct interaction with complex Elasticsearch queries - **Arranger Components:** A library of React components for building interactive search UIs, communicating with Arranger Server to fetch and display data. @@ -55,23 +56,26 @@ The Arranger Components image above highlights three key features: The Arranger repository can be accessed from our Overture-Stack GitHub page [located here](https://github.com/overture-stack/arranger). - ``` - arranger/ - ├── apps/ - │ ├── mcp-server/ - │ └── search-server/ - ├── docker/ - ├── integration-tests/ - │ └── server/ - ├── modules/ - │ ├── admin-ui/ - │ ├── charts/ - │ ├── components/ - │ ├── graphql-router/ - │ ├── sqon/ - │ └── types/ - └── scripts/ - ``` +``` +arranger/ +├── apps/ +│ ├── mcp-server/ +│ └── search-server/ +├── docker/ +├── integration-tests/ +│ ├── admin/ +│ ├── import/ +│ ├── mcp-server/ +│ └── server/ +├── modules/ +│ ├── admin-ui/ +│ ├── charts/ +│ ├── components/ +│ ├── graphql-router/ +│ ├── sqon/ +│ └── types/ +└── scripts/ +``` - **`apps/`**: Runnable server applications: - **`search-server/`**: The Arranger search server: a GraphQL service that interfaces with OpenSearch/Elasticsearch and hosts the configuration API. @@ -100,9 +104,9 @@ You need a running server connected to your search engine, with at least one cat 1. [Concepts](./concepts.md): the domain model: catalogues, facets, buckets, SQONs 2. [Setup](./setup.md): prerequisites, environment variables, search engine permissions -3. [Index mappings](./usage/00-index-mappings.md): what your ES/OS index mapping drives in Arranger -4. [Catalogue configuration](./usage/01-arranger-configs.md): the four JSON files that define each catalogue -5. [Feature flags](./usage/08-feature-flags.md): security hardening flags to review before going to production +3. [Index mappings](./reference/00-index-mappings.md): what your ES/OS index mapping drives in Arranger +4. [Catalogue configuration](./reference/01-arranger-configs.md): the four JSON files that define each catalogue +5. [Feature flags](./reference/07-feature-flags.md): security hardening flags to review before going to production --- @@ -111,9 +115,10 @@ You need a running server connected to your search engine, with at least one cat You're implementing a data portal using Arranger Components or writing UI code that queries Arranger. 1. [Concepts](./concepts.md): understand catalogues, facets, and SQONs before writing code -2. [Catalogue configuration](./usage/01-arranger-configs.md): configure which fields are visible and facetable -3. [Query processing](./usage/02-query-processing.md): how a user action becomes an Elasticsearch query -4. [Building SQON queries](./usage/03-building-sqon-queries.md): the `SqonBuilder` API and `addFilterClause` +2. [Catalogue configuration](./reference/01-arranger-configs.md): configure which fields are visible and facetable +3. [Query processing](./reference/02-query-processing.md): how a user action becomes an Elasticsearch query +4. [Building SQON queries](./reference/03-building-sqon-queries.md): the `SqonBuilder` API and `addFilterClause` +5. [Arranger Charts](./charts.md): React charts that visualize a catalogue's aggregation data --- @@ -121,10 +126,21 @@ You're implementing a data portal using Arranger Components or writing UI code t You're building an API client, pipeline, or script that sends queries to Arranger. -1. [Query processing](./usage/02-query-processing.md): the SQON to GraphQL to ES pipeline -2. [Building SQON queries](./usage/03-building-sqon-queries.md): constructing valid SQONs in TypeScript -3. [SQONs in detail](./usage/04-sqon-in-detail.md): operator reference, aliases, pivot, edge cases -4. [Introspection API](./usage/05-introspection.md): discover available fields and operators at runtime +1. [Query processing](./reference/02-query-processing.md): the SQON to GraphQL to ES pipeline +2. [Building SQON queries](./reference/03-building-sqon-queries.md): constructing valid SQONs in TypeScript +3. [SQONs in detail](./reference/04-sqon-in-detail.md): operator reference, aliases, pivot, edge cases +4. [Introspection API](./reference/05-introspection.md): discover available fields and operators at runtime + +--- + +**Federating across several Arranger servers** + +Multiple organisations each run their own Arranger, and you want one portal reporting totals across all of them. + +1. [Federated search](./federated-search.md): how federation works, configuration, the query shape, and its limitations +2. [Catalogue configuration](./reference/01-arranger-configs.md): the per-catalogue files each participating node needs +3. [Feature flags](./reference/07-feature-flags.md): `disableGraphQLIntrospection` must be `false` on every remote node +4. [Arranger Charts](./charts.md): `NetworkNodesChart` for showing per-node hit counts --- @@ -132,12 +148,12 @@ You're building an API client, pipeline, or script that sends queries to Arrange You're connecting an AI model, MCP client, or automated pipeline to Arranger. -1. [AI and automation](./usage/06-ai-and-automation.md): MCP server setup, available tools, SQON generation rules -2. [Introspection API](./usage/05-introspection.md): the live source of truth for field metadata +1. [Arranger MCP server](./mcp-server.md): MCP server setup, available tools, SQON generation rules +2. [Introspection API](./reference/05-introspection.md): the live source of truth for field metadata --- **Upgrading from 3.0.x or consolidating instances** -- [Migrating to 3.1](./migration/v3.1.md): breaking changes (env var renames, image rename, multicatalogue layout) -- [Consolidating multiple instances](./migration/v3.1.md#consolidating-multiple-single-catalogue-instances): step-by-step guide to the multicatalogue directory layout +- [Migrating to 3.1](./reference/08-Migration/v3.1.md): breaking changes (env var renames, image rename, multicatalogue layout) +- [Consolidating multiple instances](./reference/08-Migration/v3.1.md#consolidating-multiple-single-catalogue-instances): step-by-step guide to the multicatalogue directory layout diff --git a/docs/usage/00-index-mappings.md b/docs/reference/00-index-mappings.md similarity index 96% rename from docs/usage/00-index-mappings.md rename to docs/reference/00-index-mappings.md index d20332f28..1607ad62a 100644 --- a/docs/usage/00-index-mappings.md +++ b/docs/reference/00-index-mappings.md @@ -29,4 +29,4 @@ An OpenSearch (or Elasticsearch) index mapping defines the fields in your docume ## Further reading -For a comprehensive guide on creating and managing index mappings in the Overture platform, covering file-centric vs analysis-centric indexing, index templates, aliases, analyzers, and the Maestro indexing pipeline, see the [index mappings guide](https://docs.overture.bio/guides/administration-guides/index-mappings) in the Overture platform documentation. +For a comprehensive guide on creating and managing index mappings in the Overture platform, covering file-centric vs analysis-centric indexing, index templates, aliases, analyzers, and the Maestro indexing pipeline, see the [index mappings guide](https://docs.overture.bio/use/administration/index-mappings) in the Overture platform documentation. diff --git a/docs/usage/01-arranger-configs.md b/docs/reference/01-arranger-configs.md similarity index 94% rename from docs/usage/01-arranger-configs.md rename to docs/reference/01-arranger-configs.md index 64ec7328b..04c51706b 100644 --- a/docs/usage/01-arranger-configs.md +++ b/docs/reference/01-arranger-configs.md @@ -2,11 +2,13 @@ Each catalogue in Arranger is controlled by four JSON configuration files. Together they define which index to connect to, how fields are labelled for display, which columns appear in the data table, and which fields are exposed as facet panels. +A fifth file, `network.json`, is optional and only needed to federate queries across several Arranger servers. It is documented separately in [Federated search](../federated-search.md#with-search-server-configuration-files). + Templates for all four files are [in the Arranger repository](https://github.com/overture-stack/arranger/tree/main/apps/search-server/configTemplates). The full JSON schema describing every available option is at [`configTemplates/configs.json.schema`](https://github.com/overture-stack/arranger/blob/main/apps/search-server/configTemplates/configs.json.schema). ## File locations -Configuration files must be placed in the `configs/` directory under the server's working directory, or in the path specified by the `CONFIG_PATH` environment variable. +Configuration files must be placed in the `configs/` directory under the server's working directory, or in the path specified by the `CONFIGS_PATH` environment variable. In a multicatalogue setup, each catalogue gets its own subdirectory named after the catalogue ID: @@ -145,6 +147,6 @@ Only fields with `keyword` or `boolean` types in the index mapping are suitable :::tip Portal customization guide -For a step-by-step walkthrough of configuring a complete data portal, including mock data setup and Arranger component integration, see the [platform guide on customizing the data portal](https://docs.overture.bio/guides/administration-guides/customizing-the-data-portal). +For a step-by-step walkthrough of configuring a complete data portal, including mock data setup and Arranger component integration, see the [platform guide on customizing the data portal](https://docs.overture.bio/use/administration/customizing-the-data-portal). ::: diff --git a/docs/usage/02-query-processing.md b/docs/reference/02-query-processing.md similarity index 94% rename from docs/usage/02-query-processing.md rename to docs/reference/02-query-processing.md index 568cd0234..e32178faa 100644 --- a/docs/usage/02-query-processing.md +++ b/docs/reference/02-query-processing.md @@ -7,7 +7,7 @@ When a user applies filters in a search interface, the request flows through fou 3. **Arranger Server** translates the GraphQL query into an Elasticsearch query 4. **Elasticsearch** executes the query and returns results, which flow back through Arranger to the client -This pipeline separates client-side applications from backend Elasticsearch servers. +This pipeline separates client-side applications from backend Elasticsearch servers. For the concrete request and response shape — the endpoint, `hits`, and `aggregations` — see the [GraphQL API](./graphql-api.md) reference. :::info **What are SQONs?** SQON (Serializable Query Object Notation) is Overture's filter language for communicating queries between system components. The examples below show SQONs in action, followed by detailed explanations of [what SQON is](#sqon-at-a-glance), [why it exists](#why-sqon-exists). For a deeper reference on how to build them using operators, aliases, pivots, and covering common edge cases, see [SQON in detail](./04-sqon-in-detail.md). @@ -113,7 +113,7 @@ The client sends the SQON to Arranger as part of a GraphQL request: - **Sorting** (line 26): Order of results (empty in this example) :::tip -This example sets `first: 20` explicitly. If `first` is omitted entirely, it defaults to 10, along with a number of other non-obvious defaults for aggregations and downloads; see [Defaults and Limits](./07-defaults-and-limits.md) for the full list. +This example sets `first: 20` explicitly. If `first` is omitted entirely, it defaults to 10, along with a number of other non-obvious defaults for aggregations and downloads; see [Defaults and Limits](./06-defaults-and-limits.md) for the full list. ::: @@ -224,5 +224,5 @@ If you want the full SQON reference, including operators, aliases, pivot behavio ::: :::info **Need Help?** -If you encounter any issues or have questions, please don't hesitate to reach out through our relevant [**community support channels**](https://docs.overture.bio/community/support). +If you encounter any issues or have questions, please don't hesitate to reach out through our [**support page**](https://docs.overture.bio/community/support) or our [**discussion forum**](https://github.com/overture-stack/docs/discussions?discussions_q=). ::: diff --git a/docs/usage/03-building-sqon-queries.md b/docs/reference/03-building-sqon-queries.md similarity index 100% rename from docs/usage/03-building-sqon-queries.md rename to docs/reference/03-building-sqon-queries.md diff --git a/docs/usage/04-sqon-in-detail.md b/docs/reference/04-sqon-in-detail.md similarity index 100% rename from docs/usage/04-sqon-in-detail.md rename to docs/reference/04-sqon-in-detail.md diff --git a/docs/usage/05-introspection.md b/docs/reference/05-introspection.md similarity index 91% rename from docs/usage/05-introspection.md rename to docs/reference/05-introspection.md index 6be4bc23d..fe38b4921 100644 --- a/docs/usage/05-introspection.md +++ b/docs/reference/05-introspection.md @@ -134,4 +134,4 @@ By default, GraphQL introspection is disabled when `NODE_ENV=production` and ena Disabling GraphQL introspection is recommended in production to avoid exposing schema structure to clients through `__schema`/`__type` queries (OWASP A02). -**Caveat - network aggregation:** When network search federation is active, Arranger queries each remote node's GraphQL endpoint using `__type` to discover its aggregation field types at startup. If a remote node has GraphQL introspection disabled, that node's schema discovery fails and it is silently excluded from federation. Until this dependency is replaced with a REST-based discovery call, do not set `disableGraphQLIntrospection: true` on any node that serves as a remote target in a network aggregation deployment. +**Caveat - network aggregation:** When [federated search](../federated-search.md) is active, Arranger queries each remote node's GraphQL endpoint using `__type` to discover its aggregation field types at startup. If a remote node has GraphQL introspection disabled, that node's schema discovery fails and it is reported as an errored node with zero hits for the lifetime of the querying server's process. Until this dependency is replaced with a REST-based discovery call, do not set `disableGraphQLIntrospection: true` on any node that serves as a remote target in a federated deployment. diff --git a/docs/usage/07-defaults-and-limits.md b/docs/reference/06-defaults-and-limits.md similarity index 86% rename from docs/usage/07-defaults-and-limits.md rename to docs/reference/06-defaults-and-limits.md index d6127db2b..99a8afabc 100644 --- a/docs/usage/07-defaults-and-limits.md +++ b/docs/reference/06-defaults-and-limits.md @@ -29,6 +29,9 @@ Omitting an argument does not mean "use everything" or "no limit applies." It me | Behaviour | Default | Notes | | ---------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Sort order | Your `sort`, with an `_id: asc` tie-breaker appended | Not overridable. The tie-breaker guarantees deterministic ordering across paginated export batches; without it, ties in your sort field could cause rows to be skipped or repeated across batches. | +| Max rows (`DOWNLOAD_MAX_ROWS`) | `100` | Maximum number of rows a single export returns. Global default applied to all catalogues; set the env var to raise it. | +| Custom row caps (`ALLOW_CUSTOM_DOWNLOAD_MAX_ROWS`) | `false` | When `false`, requests cannot override the row cap above. Set to `true` to allow per-request row limits. | +| Stream buffer (`DOWNLOAD_STREAM_BUFFER_SIZE`) | `2000` | Number of rows buffered per batch while streaming an export. | ## Query validation limits @@ -36,10 +39,10 @@ Omitting an argument does not mean "use everything" or "no limit applies." It me | ------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------- | | `GRAPHQL_MAX_ALIASES` | 15 aliased fields per query | Set the env var, or `maxAliases` in a catalogue's `base.json` (per-catalogue config wins). | | `GRAPHQL_MAX_DEPTH` | 7 levels of selection nesting | Set the env var, or `maxDepth` in a catalogue's `base.json` (per-catalogue config wins). | -| `MAX_RESULTS_WINDOW` | 10000 hits per query | Set the env var, or `maxResultsWindow` in a catalogue's `table.json`. See [Migrating to 3.1](../migration/v3.1.md#max_results_window-is-now-enforced) for details. | +| `MAX_RESULTS_WINDOW` | 10000 hits per query | Set the env var, or `maxResultsWindow` in a catalogue's `table.json`. See [Migrating to 3.1](./08-Migration/v3.1.md#max_results_window-is-now-enforced) for details. | Both `GRAPHQL_MAX_ALIASES` and `GRAPHQL_MAX_DEPTH` apply their defaults whether or not you've set the corresponding environment variable: there is no "unset means unlimited" state for either. :::info -If you're generating queries programmatically, including through the [MCP server](./06-ai-and-automation.md), assume every default above applies unless you set the value yourself. Only `trackTotalHits` is visible via schema introspection; the rest require reading this page. +If you're generating queries programmatically, including through the [MCP server](../mcp-server.md), assume every default above applies unless you set the value yourself. Only `trackTotalHits` is visible via schema introspection; the rest require reading this page. ::: diff --git a/docs/usage/08-feature-flags.md b/docs/reference/07-feature-flags.md similarity index 93% rename from docs/usage/08-feature-flags.md rename to docs/reference/07-feature-flags.md index 66400fdcf..27e60523c 100644 --- a/docs/usage/08-feature-flags.md +++ b/docs/reference/07-feature-flags.md @@ -7,7 +7,7 @@ Arranger ships a set of boolean feature flags that turn optional behaviour on or `apps/search-server/.env.schema` is the canonical list of every env var Arranger reads, with its default value. This page explains what each feature flag actually does and, where relevant, why the default is what it is. -For numeric query-validation limits (`GRAPHQL_MAX_ALIASES`, `GRAPHQL_MAX_DEPTH`, `MAX_RESULTS_WINDOW`) and other invisible query defaults, see [Defaults and Limits](./07-defaults-and-limits.md) instead; they're a related but separate category from the on/off flags on this page. +For numeric query-validation limits (`GRAPHQL_MAX_ALIASES`, `GRAPHQL_MAX_DEPTH`, `MAX_RESULTS_WINDOW`) and other invisible query defaults, see [Defaults and Limits](./06-defaults-and-limits.md) instead; they're a related but separate category from the on/off flags on this page. --- @@ -17,7 +17,7 @@ These two flags close specific, identified attack surface. Both default to the p | Flag | Env var | Default | What it does | Recommendation | | --- | --- | --- | --- | --- | -| `disableGraphQLIntrospection` | `DISABLE_GRAPHQL_INTROSPECTION` | `false` (`true` when `NODE_ENV=production`) | Disables GraphQL's built-in `__schema`/`__type` introspection system, which otherwise exposes full schema structure (type names, field names, arguments) to any client. | Recommended in production (OWASP A02: Security Misconfiguration). See the [Introspection API](./05-introspection.md#graphql-introspection) page for full detail, including a caveat for federated (network aggregation) deployments: a node serving as a remote target must keep this disabled, since the aggregating node discovers its schema via `__type` at startup. | +| `disableGraphQLIntrospection` | `DISABLE_GRAPHQL_INTROSPECTION` | `false` (`true` when `NODE_ENV=production`) | Disables GraphQL's built-in `__schema`/`__type` introspection system, which otherwise exposes full schema structure (type names, field names, arguments) to any client. | Recommended in production (OWASP A02: Security Misconfiguration). See the [Introspection API](./05-introspection.md#graphql-introspection) page for full detail. **Caveat for [federated search](../federated-search.md):** a node serving as a remote target must keep this flag `false`, since the querying node discovers its schema via `__type` at startup. | | `enableGraphQLBatching` | `ENABLE_GRAPHQL_BATCHING` | `false` | Enables array-based GraphQL query batching (sending multiple operations in a single HTTP request, each executed in parallel). | Disabled by default and expected to stay that way: Arranger has no legitimate internal use for HTTP-level batching, and unrestricted batching can be used to bypass request-level rate limiting and amplify the cost of a single request. Only enable if a specific consumer genuinely relies on batched requests. | Field-name suggestions in GraphQL error messages (`"Did you mean ...?"`, which can leak schema structure even with introspection disabled) are stripped unconditionally and have no flag; there's nothing to configure. diff --git a/docs/migration/v3.1.md b/docs/reference/08-Migration/v3.1.md similarity index 99% rename from docs/migration/v3.1.md rename to docs/reference/08-Migration/v3.1.md index b22214fb5..70c47c3bf 100644 --- a/docs/migration/v3.1.md +++ b/docs/reference/08-Migration/v3.1.md @@ -118,7 +118,7 @@ const checkArrangerCatalogues = async () => { Your gateway or BFF layer will need a `GET /introspection` route that forwards to Arranger's introspection endpoint. Once this is in place, the `INDEX_NAME` env vars (which held ES index names that were never queried directly by the UI) can be removed from the frontend config. -See [Introspection API](../usage/05-introspection.md) for the full response schema. +See [Introspection API](../05-introspection.md) for the full response schema. ### Lock down CORS in production diff --git a/docs/reference/graphql-api.md b/docs/reference/graphql-api.md new file mode 100644 index 000000000..a08bda98f --- /dev/null +++ b/docs/reference/graphql-api.md @@ -0,0 +1,123 @@ +--- +sidebar_position: 2.5 +--- + +# GraphQL API + +Arranger exposes a single **GraphQL endpoint** per server — the primary programmatic interface for searching a catalogue. A client combines a [SQON](./03-building-sqon-queries.md) filter with field selections, pagination, and sorting; Arranger translates the whole request into an Elasticsearch query and returns the results (see [Query Processing](./02-query-processing.md) for the end-to-end flow). + +The schema is **generated per catalogue** from the catalogue's index mapping and [configuration](../concepts.md#catalogues-and-configuration) — there is no hand-written schema, so field names and types vary by catalogue. Discover them at runtime with the [Introspection API](./05-introspection.md). + +## Endpoint + +``` +POST /graphql +``` + +A running server serves GraphQL at `/graphql` (a local development server, for example, at `http://localhost:5050/graphql`). Requests are standard GraphQL over HTTP: a JSON body with a `query` string and an optional `variables` object. The paths for a given server — including the per-catalogue GraphQL paths used in multi-catalogue mode — are listed by `GET /introspection` (see the [Introspection API](./05-introspection.md)). + +## Schema shape + +For each catalogue, the root query exposes a field named after the catalogue's **document type** (for example `file` or `participant`, set in the catalogue configuration). That type carries: + +| Field | Purpose | +|---|---| +| `hits` | The matching records, as a paginated connection. | +| `aggregations` | Per-field [facet](../concepts.md#facets-buckets-and-aggregations) buckets — each a value and its document count — over the filtered result set. | +| `configs` | The catalogue's table, facet, and display configuration. | +| `mapping` | The raw Elasticsearch mapping, as JSON. | + +`hits` and `aggregations` both accept a `filters` argument that takes a SQON. + +## Querying records: `hits` + +```graphql +query SearchFiles($sqon: JSON, $first: Int, $offset: Int, $sort: [Sort]) { + file { + hits(filters: $sqon, first: $first, offset: $offset, sort: $sort) { + total + edges { + node { + id + # the remaining fields come from the catalogue's index mapping — + # discover them with the Introspection API + } + } + } + } +} +``` + +with variables: + +```json +{ + "sqon": { + "op": "and", + "content": [ + { "op": "in", "content": { "fieldName": "data.primary_site", "value": ["Brain"] } } + ] + }, + "first": 20, + "offset": 0, + "sort": [{ "fieldName": "data.primary_site", "order": "asc" }] +} +``` + +**`hits` arguments:** + +| Argument | Type | Purpose | +|---|---|---| +| `filters` | `JSON` (SQON) | The filter to apply; omit to match all records. | +| `first` | `Int` | Page size (defaults to 10 — see [Defaults and Limits](./06-defaults-and-limits.md)). | +| `offset` | `Int` | Number of records to skip. | +| `sort` | `[Sort]` | Ordering; each `Sort` is `{ fieldName, order, mode, missing }`. | +| `searchAfter` | `JSON` | Cursor for deep pagination, taken from a prior page's `edges.searchAfter`. | +| `trackTotalHits` | `Boolean` | Whether `total` counts all matches (default `true`). | + +**`hits` result:** a connection with `total` (the full match count, not just the current page) and `edges`, each holding a `node`. Every `node` has `id` and `score`; its remaining fields are those in the catalogue's index mapping. + +## Aggregations + +`aggregations` returns, for each requested field, the distinct values (**buckets**) and their document counts in the filtered result set — the data behind a facet panel: + +```graphql +query Facets($sqon: JSON) { + file { + aggregations(filters: $sqon) { + data__primary_site { + bucket_count + buckets { + key + doc_count + } + } + } + } +} +``` + +Each field returns a `bucket_count` (the number of distinct values) and `buckets`, where each bucket's `key` is a value and `doc_count` its document count. See [Concepts → Facets, buckets, and aggregations](../concepts.md#facets-buckets-and-aggregations). + +**`aggregations` arguments:** `filters` (a SQON), `include_missing` (also count documents missing the field), and `aggregations_filter_themselves` (whether a field's own facet selection constrains its buckets — set `false` for multi-select facet UIs). + +:::note Field names: dots become double underscores +GraphQL field names cannot contain dots, so a mapping field such as `data.primary_site` is exposed in the schema as `data__primary_site`. SQON `fieldName` values keep the dotted form (`data.primary_site`); GraphQL **selections** and aggregation names use the `__` form. The [Introspection API](./05-introspection.md) returns the dotted mapping names. +::: + +## Discovering the schema + +Because the schema is generated per catalogue, use the **[Introspection API](./05-introspection.md)** to discover what you can query without writing a GraphQL query first: + +- `GET /introspection/:catalogueId` — every queryable field, its type, and the SQON operators it accepts. +- `GET /introspection/sqon` — the SQON JSON Schema shared across catalogues. + +GraphQL's built-in introspection (`__schema` / `__type`) also works, but it is gated by the `disableGraphQLIntrospection` flag — disabled when `NODE_ENV=production` by default (see [Introspection API → GraphQL introspection](./05-introspection.md#graphql-introspection)). Prefer the REST introspection endpoints for tooling, since they are always available. + +## Filtering with SQON + +The `filters` argument on both `hits` and `aggregations` takes a **SQON**, Arranger's JSON filter language. Build it separately and pass it as a variable, as shown above. See [Building SQON queries](./03-building-sqon-queries.md) and [SQON in detail](./04-sqon-in-detail.md). + +:::info **Need Help?** +If you encounter any issues or have questions, please don't hesitate to reach out through our [**support page**](https://docs.overture.bio/community/support) or our [**discussion forum**](https://github.com/overture-stack/docs/discussions?discussions_q=). +::: diff --git a/docs/reference/reference.mdx b/docs/reference/reference.mdx new file mode 100644 index 000000000..18e040a57 --- /dev/null +++ b/docs/reference/reference.mdx @@ -0,0 +1,7 @@ +# Reference + +These pages cover Arranger's configuration, query model, integration options, and version migrations. If you're not sure where to start, the [overview](../overview.md#where-to-go-from-here) has pathway guidance by role. + +

+ + diff --git a/docs/setup.md b/docs/setup.md index ba6302f9e..6c6e07ca8 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -13,107 +13,120 @@ Before you begin, ensure you have the following installed on your system: ## Developer Setup -This guide will walk you through setting up a complete development environment, including Arranger and its complementary services. +The Arranger repository ships everything needed for local development: a `docker-compose.yml` defining a search engine, `make` targets to drive it, and a script that seeds test documents. No other repository is required. ### Setting up supporting services -We'll use the Overture quickstart service, a flexible Docker Compose setup, to spin up Arranger's complementary services. +1. Clone Arranger and navigate to its directory: -1. Clone the quickstart repository and navigate to its directory: + ```bash + git clone https://github.com/overture-stack/arranger.git + cd arranger + ``` + +2. Start Elasticsearch: ```bash - git clone -b quickstart https://github.com/overture-stack/prelude.git - cd prelude + make start-es ``` -2. Run the appropriate start command for your operating system: +3. Seed it with test documents: - | Operating System | Command | - | ---------------- | ------------------------ | - | Unix/macOS | `make arrangerDev` | - | Windows | `./make.bat arrangerDev` | + ```bash + make seed-es + ```
**Click here for a detailed breakdown** - This command will set up all complementary services for Arranger development as follows: + `make start-es` brings up the `elasticsearch` service from the repository's `docker-compose.yml`, and `make seed-es` loads the mock documents under `docker/elasticsearch/documents` into the `file_centric_1.0` index. + + | Service | Port | Description | Purpose in Arranger Development | + | ------------- | -------------- | --------------------------------------- | ---------------------------------------------------------------- | + | Elasticsearch | `9200`, `9300` | Distributed search and analytics engine | Provides fast and scalable search capabilities over indexed data | + + The cluster runs **with authentication enabled** (`xpack.security.enabled: "true"`). The Makefile defines the credentials it uses and passes them through to Docker Compose: - ![arrangerDev](./assets/arrangerDev.svg 'Arranger Dev Environment') + | Variable | Value | + | --------- | --------------------- | + | `ES_USER` | `elastic` | + | `ES_PASS` | `unsafePassword123` | + | `ES_HOST` | `http://localhost:9200` | - | Service | Port | Description | Purpose in Arranger Development | - | ------------- | ------ | ----------------------------------------------- | ---------------------------------------------------------------- | - | Conductor | `9204` | Orchestrates deployments and environment setups | Manages the overall development environment | - | Elasticsearch | `9200` | Distributed search and analytics engine | Provides fast and scalable search capabilities over indexed data | - | Stage | `3000` | Web Portal Scaffolding | Houses Arranger's search UI components | + Override them by exporting different values before running `make`, or with an `.env.testing` file at the repository root, which the Makefile includes when present. + + Two further targets bring up more of the stack, and are useful when you want a containerized server rather than one running on your host: + + | Command | Services started | + | ------------------- | ------------------------------------------------------------- | + | `make start-es` | Elasticsearch only | + | `make start-server` | The Arranger server only (`5050`) | + | `make start` | Elasticsearch, Kibana (`5601`), the server (`5050`), and a Stage UI (`3000`) | :::note Supported search engines - Arranger supports **OpenSearch 1.x or higher** and **Elasticsearch 7.x** (minimum 7.0, licensed/default distribution only; ES OSS and ES 8.x are not supported; the bundled client is `@elastic/elasticsearch` v7). OpenSearch maintains API compatibility with ES 7.x, so query syntax and conventions apply to both engines. + Arranger supports **OpenSearch 1.x or higher** and **Elasticsearch 7.x** (minimum 7.0, licensed/default distribution only; ES OSS and ES 8.x are not supported; the bundled client is `@elastic/elasticsearch` v7). OpenSearch maintains API compatibility with ES 7.x, so query syntax and conventions apply to both engines. Note that `docker-compose.yml` defines an Elasticsearch service only, so a local OpenSearch cluster has to be supplied separately. ::: - - Ensure all ports are free on your system before starting the environment. - - You may need to adjust the ports in the `docker-compose.yml` file if you have conflicts with existing services. - - For more information, see our [quickstart documentation linked here](https://docs.overture.bio/docs/other-software/quickstart). + - Ensure these ports are free on your system before starting the environment. + - You may need to adjust the ports in `docker-compose.yml` if you have conflicts with existing services. + - `make ps` shows what is running; `make clean` tears the stack down and removes its volumes.
### Running the Arranger-Server -1. Clone Arranger and navigate to its directory: - - ```bash - git clone https://github.com/overture-stack/arranger.git - cd arranger - ``` - -2. Rename the `.env.arrangerDev` file to `.env`: +1. Copy the search server's environment schema into place: ```bash - mv .env.arrangerDev .env + cp apps/search-server/.env.schema apps/search-server/.env ``` :::info - This `.env` file is preconfigured for the Arranger dev environment quickstart: + The server loads its `.env` from its own workspace directory, so the file must be at `apps/search-server/.env` rather than the repository root. A minimal configuration matching the Elasticsearch instance started above looks like this: ```env # ============================== # Arranger Environment Variables # ============================== - # Arranger Variables + # Server + SERVER_PORT=5050 ENABLE_LOGS=false - # Elasticsearch/Opensearch Variables - ES_HOST=http://elasticsearch:9200 + # Search engine connection + ES_HOST=http://localhost:9200 ES_USER=elastic - ES_PASS=myelasticpassword - SEARCH_ENGINE=elasticsearch + ES_PASS=unsafePassword123 - # Stage Variables - REACT_APP_BASE_URL=http://localhost:3000 + # Catalogue configuration + CONFIGS_PATH=../../docker/server ```
**Click here for a detailed explanation of Arranger's environment variables** - **Arranger Variables** + **Server** + - `SERVER_PORT`: The port the search server listens on - `ENABLE_LOGS`: Determines whether logging is enabled - **Elasticsearch Variables** - - `ES_HOST`: The URL of your Elasticsearch instance - - `ES_USER` and `ES_PASS`: The credentials for accessing Elasticsearch + **Search engine connection** + - `ES_HOST`: The URL of your Elasticsearch or OpenSearch instance. Use `localhost` when the server runs on your host and the cluster runs in Docker; the container hostname `elasticsearch` only resolves from inside the Compose network. + - `ES_USER` and `ES_PASS`: The credentials for accessing the cluster, matching the values the Makefile passes to Docker Compose + - `SEARCH_ENGINE`: Either `elasticsearch` or `opensearch`. Leave it unset to auto-detect from the cluster. + + **Catalogue configuration** + - `CONFIGS_PATH`: Directory holding the per-catalogue JSON config files, resolved relative to the server's workspace directory. The repository's example catalogue lives at `docker/server`, hence `../../docker/server`. Its `base.json` sets `index` to `file_centric_1.0` and `documentType` to `file`, matching the data `make seed-es` loads. + - `ES_INDEX` and `DOCUMENT_TYPE` are required, but are normally set per catalogue in `base.json` as above. Per-catalogue file values always take precedence over these environment defaults. - **Stage Variables** - - `REACT_APP_BASE_URL`: The base URL for your front-end application (Stage) - - `REACT_APP_ARRANGER_ADMIN_ROOT`: The URL for the Arranger GraphQL endpoint + The schema file lists the remaining variables, including feature flags, GraphQL security limits, and download settings.
::: -3. Install the required npm packages: +2. Install the required npm packages: ```bash npm install @@ -126,12 +139,18 @@ We'll use the Overture quickstart service, a flexible Docker Compose setup, to s ::: -4. Run the Arranger server: +3. Run the Arranger server: ```bash - npm run server + npm run dev:server ``` + :::tip + + `npm run dev:server` runs the server in watch mode, rebuilding `sqon`, `types`, and `graphql-router` as you change them. To run the built server instead, use `npm run server`. + + ::: + Once the server starts, you can access Arranger-Server at `http://localhost:5050/graphql`. ### Running the Arranger Components @@ -161,6 +180,6 @@ If you encounter any issues during setup: :::info Need Help? - If you encounter any issues or have questions about our API, please don't hesitate to reach out through our relevant [**community support channels**](https://docs.overture.bio/community/support). + If you encounter any issues or have questions about our API, please don't hesitate to reach out through our [**support page**](https://docs.overture.bio/community/support) or our [**discussion forum**](https://github.com/overture-stack/docs/discussions?discussions_q=). ::: diff --git a/docs/usage/usage.mdx b/docs/usage/usage.mdx deleted file mode 100644 index 886fe5cd3..000000000 --- a/docs/usage/usage.mdx +++ /dev/null @@ -1,7 +0,0 @@ -# Usage - -These pages cover Arranger's configuration, query model, and integration options. If you're not sure where to start, the [overview](../overview.md#where-to-go-from-here) has pathway guidance by role. - -

- - diff --git a/modules/charts/maintainer-docs/maintainer-docs.md b/modules/charts/maintainer-docs/maintainer-docs.md index 7d17fd00a..5eb36d7cc 100644 --- a/modules/charts/maintainer-docs/maintainer-docs.md +++ b/modules/charts/maintainer-docs/maintainer-docs.md @@ -62,7 +62,7 @@ Applications using the library must be wrapped in the required provider hierarch lookupMap[key]} + mapper={(key) => lookupMap[key]} handlers={{ onClick: handleSunburstClick }} /> diff --git a/modules/graphql-router/README.md b/modules/graphql-router/README.md index 77cfd1362..694071f19 100644 --- a/modules/graphql-router/README.md +++ b/modules/graphql-router/README.md @@ -113,30 +113,30 @@ A catalogue can federate aggregation queries across multiple remote Arranger nod ```ts const router = await arrangerRouter({ configs: { + documentType: 'file', esHost: 'http://localhost:9200', esIndex: 'file_centric', - documentType: 'File', network: { + // Runs once per node per query. Use it to forward auth to remote nodes. + customizeRemoteRequest: ({ context, remoteNode }) => ({ + headers: { + Authorization: context.request.headers.get('Authorization') ?? '', + }, + }), localNode: { displayName: 'Local', nodeId: 'local', }, - remoteRequests: { - headers: ['Authorization'], // forwarded to all remote nodes by default - }, remoteNodes: [ { displayName: 'Node A', - documentType: 'FileAggs', + documentType: 'file', // the remote's root field; `Aggregations` is appended internally graphqlUrl: 'http://node-a:5050/graphql', nodeId: 'node-a', - requests: { - headers: ['Authorization'], // per-node override; merged with remoteRequests.headers - }, }, { displayName: 'Node B', - documentType: 'FileAggs', + documentType: 'file', graphqlUrl: 'http://node-b:5050/graphql', nodeId: 'node-b', }, @@ -146,24 +146,46 @@ const router = await arrangerRouter({ }); ``` -When using `apps/search-server`, this config lives in `network.json` inside the catalogue's config directory. A template is at [`apps/search-server/configTemplates/network.json`](../../apps/search-server/configTemplates/network.json). - #### Network config fields -| Field | Description | -| -------------------------------- | ---------------------------------------------------------------------------------------------- | -| `localNode.displayName` | Human-readable label for this node's results in aggregation responses. | -| `localNode.nodeId` | Stable identifier for this node, used when filtering results by node. | -| `remoteRequests.headers` | Header names to forward from the incoming request to **all** remote nodes. | -| `remoteNodes[].graphqlUrl` | GraphQL endpoint URL of the remote Arranger instance. | -| `remoteNodes[].documentType` | Aggregation type name on the remote node. | -| `remoteNodes[].displayName` | Human-readable label for this remote node's results. | -| `remoteNodes[].nodeId` | Stable identifier for this node, used when filtering results by node. | -| `remoteNodes[].requests.headers` | Header names to forward to this specific node. Takes precedence over `remoteRequests.headers`. | +| Field | Description | +| ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `customizeRemoteRequest` | Callback invoked once per node per query, receiving `{ context, remoteNode }` and returning request properties (currently `headers`) to add to that node's outgoing request. | +| `localNode.displayName` | Human-readable label for this node's results in aggregation responses. Omit the whole `localNode` block to federate over remote nodes only. | +| `localNode.nodeId` | Stable identifier for this node, used by the `nodesFilter` query argument. | +| `remoteNodes[].displayName` | Human-readable label for this remote node's results. Also used to match responses back to nodes, so keep it unique across the network. | +| `remoteNodes[].documentType` | The remote catalogue's `documentType`, meaning its root GraphQL field (e.g. `file`). `Aggregations` is appended to this value during field discovery, so give the bare document type, not `fileAggregations`. | +| `remoteNodes[].graphqlUrl` | GraphQL endpoint URL of the remote Arranger instance. | +| `remoteNodes[].nodeId` | Stable identifier for this node, used by the `nodesFilter` query argument. | + +#### With `apps/search-server` + +When running `apps/search-server`, this config lives in `network.json` inside the catalogue's config directory. A template is at [`apps/search-server/configTemplates/network.json`](../../apps/search-server/configTemplates/network.json). + +A JSON file cannot express a callback, so `search-server` accepts two extra declarative properties **that this library does not**, and normalizes them into a `customizeRemoteRequest` function before calling `arrangerRouter`: + +| Field (`search-server` only) | Description | +| --------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| `remoteRequests.headers` | Header names to copy from the incoming request onto the outgoing request to **every** remote node. | +| `remoteNodes[].requests.headers` | Header names to forward to this specific node. **Replaces** `remoteRequests.headers` for that node, rather than merging with it. | + +Passing either of these to `arrangerRouter` directly has no effect: at this layer, supply `customizeRemoteRequest` instead. + +#### Field merging + +Nodes do **not** need identical field sets. The federated schema is the **union** of the supported aggregation fields found across all nodes, deduplicated by field name and type. A field present on only one node still appears in the schema, and each node is only queried for the fields it actually has. + +When a node lacks a requested field, it contributes one sentinel bucket carrying its total hits for that query, so counts still add up: + +```json +{ "key": "___aggregation_not_available___", "doc_count": 4210 } +``` + +Merging is keyed on field name and aggregation type, so nodes only combine on a field when both name it identically. Only the `Aggregations` type federates; `NumericAggregations` fields are excluded from the federated schema entirely. -All nodes must serve overlapping index field names. Fields with the same name and GraphQL type are merged across nodes; fields unique to one node are excluded from federation. +**Introspection requirement:** At startup, each remote node's aggregation field types are discovered via a `__type` GraphQL introspection query. A remote node with `disableGraphQLIntrospection: true` fails schema discovery and is reported as an errored node with zero hits for the lifetime of this server's process. Since the flag defaults to `true` when `NODE_ENV=production`, any node serving as a remote target must explicitly set it to `false`. A fix that replaces this with a REST `/introspection/fields` call is tracked in tech-debt and planned for the yoga migration. -**Introspection requirement:** At startup, each remote node's aggregation field types are discovered via a `__type` GraphQL introspection query. Remote nodes that have `disableGraphQLIntrospection: true` will fail schema discovery and be silently excluded from federation. Do not enable `disableGraphQLIntrospection` on any node that serves as a remote target in a network aggregation deployment. A fix that replaces this with a REST `/introspection/fields` call is tracked in tech-debt and planned for the yoga migration. +For the query shape, per-node status reporting, failure behaviour, and full limitations, see the [Federated search](https://github.com/overture-stack/arranger/blob/main/docs/federated-search.md) documentation. ---