Overture docs update

.dev/tech-debt.md

@@ -34,6 +34,10 @@ context: tsup@6.7.0 is ~2 years old. Upgrading to 8.5.1 is blocked by the npm ho
 standalone: yes
 context: `apps/search-server` depends on `modules/graphql-router` via `"file:../../modules/graphql-router"`, resolved through that package's `dist/` (its `package.json`'s `main`), never live source. Running `search-server`'s or `integration-tests/server`'s tests without first running `npm run build -w modules/graphql-router` silently tests against whatever `dist/` was last built, no warning that it's stale. Concretely hit during the multicatalogue partial-availability work (2026-07-24): the `{ cause: err }` fix in `fetchMapping.ts`/`router.ts` passed every unit test (which import from source via internal path aliases, never crossing the package boundary) but silently produced `unknown_error` instead of `index_not_found` when exercised through a real integration test, because `dist/` was 8 days stale at that point. Only caught because a real end-to-end integration test against live Elasticsearch was written and run (see `integration-tests/server/test/partialAvailability.test.ts`); a unit test alone could not have caught this, by construction.
 fix: add a `pretest` step to `apps/search-server` and `integration-tests/server` that rebuilds their local `file:` dependencies first, or wire `turbo:test`'s dependency graph to do this automatically (Turbo already tracks the monorepo's build graph); at minimum, document prominently in `AGENTS.md`'s "Running tests" section that changes to `modules/*` require an explicit rebuild before testing any consumer app, the current guidance to "always run from the monorepo root" doesn't by itself guarantee a fresh build.
+### No OpenSearch service in `docker-compose.yml`
+
+standalone: yes
+context: `docker-compose.yml` defines only `elasticsearch`, `kibana`, `server` and `ui`, so there is no way to bring up a local OpenSearch cluster. The Makefile carried a `start-os` target that did `up -d opensearch` against this file; since no such service exists it could never have worked, and it was removed 2026-07-30 while correcting the sibling `start-server` target (which referenced `arranger-server`, the *container name* of `server`, rather than the service key). Arranger supports OpenSearch 1.x+ and `SEARCH_ENGINE=opensearch` is a documented `apps/search-server` env var, so local OpenSearch cannot currently be exercised the way Elasticsearch can. Fix: add an `opensearch` service (plus optional OpenSearch Dashboards) to `docker-compose.yml` and restore a `start-os` target. Note the Makefile's existing `COMPOSE_PROJECT_NAME=arranger_es` / `arranger_os` split implies the two engines are meant to be alternatives sharing port 9200, so they should not be started together under the same project name.
 
 ## apps/mcp-server
 
@@ -333,6 +337,15 @@ The preferred pattern is **(B)**. Mixing the two makes it harder to find tests,
 **Fix:** Add a docs page or section covering both endpoints: path (configurable via `PING_PATH`/`READY_PATH`), response shape, HTTP status semantics (`/ready` returns `503` only when `unhealthy`), and the liveness-vs-readiness distinction with the reasoning for why liveness stays catalogue-blind. Cross-link from `GET /introspection` in `05-introspection.md`, since its top-level `status` there is the same computation `/ready` uses.
 **Standalone:** yes; documentation addition only, no code changes
 
+### README, package.json engines and the Dockerfiles disagree on the Node version
+
+**Files:** `README.md:21`; `package.json` (`engines.node`); `docker/Dockerfile.local:13,27,83`; `docker/Dockerfile.jenkins:13,36`
+**Severity:** medium (a contributor following the README may install a version the tooling does not actually want, and the published `engines` constraint is what consumers of the packages resolve against)
+**Kind:** prerequisite drift across three sources of truth
+**Issue:** Three different Node versions are stated for the same project. `README.md` lists "Node.js (v22 or higher)" under Development Environment, `package.json` declares `engines.node: ">=20.0.0"`, and every stage in both Dockerfiles builds `FROM node:24-alpine`. There is no `.nvmrc`, `.node-version`, or `volta` block to break the tie, and no CI workflow in this repo to infer the tested version from. Found while fixing link hygiene in the README, so the docs half was in scope but resolving the disagreement is not a documentation question: which value is correct depends on what the tooling actually requires and what the published packages intend to support.
+**Fix:** Decide the authoritative version first, then make the three agree. Likely shape: pin the intended development version in a `.nvmrc` (or `volta`) so there is one machine-readable source, set `engines.node` to the lowest version actually supported by consumers (which may legitimately stay below the development version), align the Dockerfiles, and have the README cite the pinned value rather than restating a number. Note the README claim was deliberately left untouched pending this decision.
+**Standalone:** no; needs a decision on the supported and intended Node versions before any file changes
+
 ---
 
 ## modules/sqon

CHANGELOG.md

@@ -15,7 +15,7 @@ This file covers high-level release notes for the Arranger project as a whole. W
 - **Docker image `arranger-server` renamed to `arranger-search-server`**: Update `docker-compose.yml`, Helm values, and any deployment manifests.
 - **`MAX_RESULTS_WINDOW` is now enforced**: Previously present in the env schema but not applied; now caps query results at `10000` by default. Deployments that return more than 10,000 documents must set this explicitly (via env var or per-catalogue `table.json`).
 
-See [docs/migration/v3.1.md](docs/migration/v3.1.md) for upgrade instructions.
+See [docs/reference/08-Migration/v3.1.md](docs/reference/08-Migration/v3.1.md) for upgrade instructions.
 
 ---
 

DEVELOPMENT.md

@@ -43,11 +43,13 @@ integration-tests/
 Start a local search engine and seed test data:
 
 ```bash
-make start          # starts Elasticsearch via docker-compose
-make seed-es        # seeds test documents
+make start-es       # starts Elasticsearch alone, for a host-run dev server
+make seed-es        # seeds test documents into file_centric_1.0
 ```
 
-The local stack runs without authentication. If you need to test against a secured cluster (OpenSearch or Elasticsearch with the security plugin enabled), see the [search engine permissions reference](docs/setup.md#search-engine-permissions) in the setup documentation for the minimum permissions required per feature.
+`make start` brings up Elasticsearch, Kibana and a containerized Arranger server together; prefer `make start-es` when you are going to run the server yourself with `npm run dev:server`.
+
+The local cluster **does** run with authentication: `docker-compose.yml` sets `xpack.security.enabled: "true"`, and the Makefile passes the credentials it defines (`ES_USER=elastic`, `ES_PASS=unsafePassword123`) through to both the cluster and the containerized server. Use those same values in `apps/search-server/.env` when running the server on the host. For the minimum permissions each feature needs on a cluster you do not control, see the [search engine permissions reference](docs/setup.md#search-engine-permissions) in the setup documentation.
 
 Start the development server (watches `sqon`, `types`, `graphql-router`, and `search-server`):
 

Makefile

@@ -166,15 +166,9 @@ start-es:
 	@echo $(GREEN)$(INFO_HEADER) Succesfully started this service! $(GREEN)
 	@echo $(MAGENTA) "You may have to populate it before using it with the Server. (Use 'make seed-es' for mock data)" $(END)
 
-start-os:
-	@echo $(YELLOW)$(INFO_HEADER) "Starting the following service: OpenSearch" $(END)
-	@COMPOSE_PROJECT_NAME=arranger_os $(DC_UP_CMD) opensearch
-	@echo $(GREEN)$(INFO_HEADER) Succesfully started this service! $(GREEN)
-	@echo $(MAGENTA) "You may have to populate it before using it with the Server. (Use 'make seed-es' for mock data)" $(END)
-
 start-server:
 	@echo $(YELLOW)$(INFO_HEADER) "Starting the following service: Arranger Server" $(END)
-	@COMPOSE_PROJECT_NAME=arranger_server $(DC_UP_CMD) arranger-server
+	@COMPOSE_PROJECT_NAME=arranger_server $(DC_UP_CMD) server
 	@echo $(GREEN)$(INFO_HEADER) Succesfully started this service! $(GREEN)
 
 test:

README.md

@@ -10,11 +10,11 @@ Arranger is a versatile, model-agnostic data discovery API for OpenSearch and El
 
 ## Documentation
 
-Technical resources for those working with or contributing to the project are available from our official documentation site, the following content can also be read and updated within the `/docs` folder of this repository.
+Technical resources for those working with or contributing to the project live in the `/docs` folder of this repository, and are also published, fully rendered, on our [official documentation site](https://docs.overture.bio/develop/Arranger/overview).
 
-- **[Arranger Overview](https://docs.overture.bio/docs/core-software/Arranger/overview)**
-- [**Setting up the Development Enviornment**](https://docs.overture.bio/docs/core-software/Arranger/setup)
-- [**Common Usage Docs**](https://docs.overture.bio/docs/core-software/Arranger/setup)
+- **[Arranger Overview](./docs/overview.md)**
+- [**Setting up the Development Environment**](./docs/setup.md)
+- [**Reference Docs**](./docs/reference/reference.mdx)
 
 ## Development Environment
 
@@ -27,7 +27,7 @@ Technical resources for those working with or contributing to the project are av
 ## Support & Contributions
 
 - For support, feature requests, and bug reports, please see our [Support Guide](https://docs.overture.bio/community/support).
-- For detailed information on how to contribute to this project, please see our [Contributing Guide](https://docs.overture.bio/docs/contribution).
+- For detailed information on how to contribute to this project, please see our [Contributing Guide](./CONTRIBUTING.md).
 
 ## Related Software
 
@@ -45,8 +45,6 @@ The Overture platform includes the following components:
 | [Lyric](https://github.com/overture-stack/lyric)        | A model-agnostic, tabular data submission system                                          |
 | [Lectern](https://github.com/overture-stack/lectern)    | Schema Manager, designed to validate, store, and manage collections of data dictionaries. |
 
-If you'd like to get started using our platform [check out our quickstart guides](https://docs.overture.bio/guides/getting-started)
-
 ## Funding Acknowledgement
 
 Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research.

apps/mcp-server/README.md

@@ -4,6 +4,17 @@ This app is an MCP server that learns how to talk to Arranger by consuming Arran
 
 The current scaffold implements the Streamable HTTP MCP transport using **v1.x** of the official [MCP TypeScript SDK](https://github.com/modelcontextprotocol/typescript-sdk/tree/v1.x).
 
+## Tools
+
+The server registers four tools that cover the full query lifecycle:
+
+| Tool                   | Purpose                                                                                                                        |
+| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| `list_catalogues`      | Returns the catalogues the connected Arranger exposes.                                                                        |
+| `get_sqon_schema`      | Returns a compact SQON quick reference (grammar, operators, worked examples) plus the full machine-readable SQON JSON Schema. |
+| `get_catalogue_fields` | Returns field introspection for one catalogue: each field's type, display name, unit, description, and valid operators.        |
+| `execute_query`        | Builds, confirms, and executes a SQON-filtered query against a catalogue and returns the matching records.                    |
+
 ## Folder Structure
 
 ```text