Overture docs update by MitchellShiell · Pull Request #215 · overture-stack/lyric

MitchellShiell · 2026-07-30T13:42:55Z

Summary

Documentation for the Overture docs site. The placeholder docs/02-lyric.md is replaced by two real pages, and the README's developer-docs links are repointed at the current site.

No Source code changes

Changes

docs/01-overview.md (new). Feature overview
docs/02-Setup.md (new). Task-oriented development-environment setup: prerequisites (pnpm, Node 20+, Docker), dependent services, configuration, and verification.
docs/02-lyric.md (deleted). Superseded by the two pages above.
README.md. The two Developer Documentation links moved off docs.overture.bio/docs/under-development/lyric/ onto docs.overture.bio/build/core-software/Lyric/overview.

Restructure the Lyric docs into the standard Overture core-software layout (Overview + Setup), mirroring the other components. Replaces the single 02-lyric.md overview, fills in the repository-structure detail, and points the README's developer-docs link at Lyric's new core-software home on docs.overture.bio.

The two new pages replace docs/02-lyric.md, which main had since updated for the Kafka document topic and category aliases (#208, #213). Port those changes onto the replacement pages so nothing regresses, and document the KAFKA_* and VALIDATOR_CONFIG variables main added to the README.

The docs site has since renamed /build/core-software/* to /build/*, then /build/* to /develop/*, so the previous target only resolved through two redirect hops.

The site renamed /docs/core-software/* to /build/* to /develop/*, moved /docs/contribution to /develop/contributing, /guides/getting-started to /deploy, and /guides/administration-guides to /use. Each old path only resolved through a redirect hop. Verified every new target against a local site build.

MitchellShiell · 2026-08-04T19:19:02Z

Related to the following main Docs site PR: Overture docs V2 update docs#48
Deploy preview found here: https://deploy-preview-48--overturedocs.netlify.app/develop/Lyric/overview
- Note, docs links will return 404 until deployed

joneubank · 2026-08-04T19:10:19Z

+- **Comprehensive data management:** Offers complete CRUD operations (Create, Read, Update, Delete) through a RESTful API documented in Swagger.
+- **Detailed change history:** Maintains a complete audit trail of all data modifications, tracking changes from committed submissions and updates ensuring data governance and accountability.
+- **SQON query endpoint:** Provides an endpoint for SQON (Structured Query Object Notation) based queries allowing complex search operations through combinations of simple field operations (`in`, `<=`, `>=`) and logic (`and`, `or`, `not`). This allows complex queries to be expressed in a simple JSON format.
+- **Multi-dictionary support:** Handles multiple Lectern dictionaries simultaneously, allowing organizations to manage different data categories with distinct schemas while maintaining data integrity and relationships. A category can optionally be given an alias, a human-readable identifier usable anywhere a numeric category ID is accepted.


alias is superfluous information for this feature summary. i would remove the second sentence.

joneubank · 2026-08-04T19:13:43Z

+- **SQON query endpoint:** Provides an endpoint for SQON (Structured Query Object Notation) based queries allowing complex search operations through combinations of simple field operations (`in`, `<=`, `>=`) and logic (`and`, `or`, `not`). This allows complex queries to be expressed in a simple JSON format.
+- **Multi-dictionary support:** Handles multiple Lectern dictionaries simultaneously, allowing organizations to manage different data categories with distinct schemas while maintaining data integrity and relationships. A category can optionally be given an alias, a human-readable identifier usable anywhere a numeric category ID is accepted.
+
+## System Architecture


I expected this to be a breakdown of Lyric's architecture, but this is actually how lyric fits into the overture system.

This is fine, but I would reorder the points here

Lyric functionality and required dependencies

lyric web api

lyric data storage (postgres)

Overture system integrations

maestro reads from lyric for search indexing, uses lyric api

song reads from lyric for external data validation, lyric provides configurable endpoints to facilitate this integration

Optional Integration - Kafka

publishes messages after data submission events to notifying other systems that new data is available

Also, you have a "why elasticsearch" section but lyric doesnt interact with elasticsearch at all. why kafka may be worthwhile (standardized message queue across overture, high throughput and performance)

removed elasticsearch, reordered the info and added lectern and postgres as required dependencies, dropped score

joneubank · 2026-08-04T19:19:01Z

+
+## Repository Structure
+
+The repository is a PNPM monorepo, organized with deployable applications under `apps/` and shared libraries under `packages/`:


pnpm monorepo link: https://pnpm.io/workspaces

updated accordingly

joneubank · 2026-08-04T19:21:08Z

+```bash
+# From the repository root
+docker-compose up -d
+```


docker compose for modern docker installations (docker-compose no longer distributed as a separate command, instead just a command inside the docker cli)

The primary difference is that docker-compose (with a hyphen) is the deprecated legacy version written in Python, while docker compose (with a space) is the modern, actively supported version integrated directly into the Docker CLI using Go.

Also, you have them reference the compose file before cloning the repo in step 2. This is out of order.

joneubank · 2026-08-04T19:22:39Z

+   A minimal development configuration looks as follows:
+
+   ```env
+   # Server
+   PORT=3030
+   LOG_LEVEL=info
+
+   # PostgreSQL (matches the docker-compose defaults)
+   DB_HOST=localhost
+   DB_PORT=5432
+   DB_NAME=lyric
+   DB_USER=postgres
+   DB_PASSWORD=secret
+
+   # Lectern schema service
+   LECTERN_URL=http://localhost:3000
+
+   # Local ID generation
+   ID_USELOCAL=true
+   ID_CUSTOM_ALPHABET=0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
+   ID_CUSTOM_SIZE=21
+
+   # Behaviour flags
+   AUDIT_ENABLED=true
+   CORS_ENABLED=false
+   ALLOWED_ORIGINS=
+   PLURALIZE_SCHEMAS_ENABLED=true
+   ```


Would be best to rely on the content of the .env.schema to have functioning defaults. maintaining them in two places will 100% lead to them drifting and these instructions failing. I would remove this example and rely on the example .env file and the main readme.

Env var official docs: https://github.com/overture-stack/lyric/blob/main/README.md#environment-variables

done, that said .env.schema didn't have functioning defaults, updated it to have values matching the bundled compose file.

joneubank · 2026-08-04T19:23:07Z

+   **Server**
+
+   - `PORT`: Server port (default: 3030)
+   - `LOG_LEVEL`: Log verbosity (default: info)
+
+   **PostgreSQL**
+
+   - `DB_HOST`: Database hostname
+   - `DB_PORT`: Database port
+   - `DB_NAME`: Database name
+   - `DB_USER`: Database user
+   - `DB_PASSWORD`: Database password
+
+   **Schema Service**
+
+   - `LECTERN_URL`: URL of the Lectern service supplying dictionary schemas
+
+   **ID Generation**
+
+   - `ID_USELOCAL`: Generate record IDs locally (default: true)
+   - `ID_CUSTOM_ALPHABET`: Custom alphabet for local ID generation
+   - `ID_CUSTOM_SIZE`: Length of locally generated IDs (default: 21)
+
+   **Behaviour**
+
+   - `AUDIT_ENABLED`: Log all modifications to submitted data (default: true)
+   - `CORS_ENABLED`: Enable CORS (default: false)
+   - `ALLOWED_ORIGINS`: Comma-separated list of permitted CORS origins
+   - `PLURALIZE_SCHEMAS_ENABLED`: Automatically pluralize schema names for compound documents (default: true)
+
+   **Kafka (optional)**
+
+   - `KAFKA_BROKERS`: Comma-separated broker addresses (for example `localhost:9092`). When set, Lyric publishes each committed record to a Kafka topic for Maestro to index. Omit to run without Kafka publishing
+   - `KAFKA_TOPIC`: Topic that committed records are published to, created on startup if it does not exist. Required when `KAFKA_BROKERS` is set
+   - `KAFKA_CLIENT_ID`: Kafka client identifier, unique per environment to distinguish producers in broker logs (default: `lyric`)
+
+   **Record Validation (optional)**
+
+   - `VALIDATOR_CONFIG`: JSON array enabling the endpoint that checks whether a record exists with a given field value. Each entry needs `categoryId`, `entityName`, and `fieldName`, where `categoryId` accepts either a category's numeric ID or its alias — for example `[{"categoryId": "1", "entityName": "sample", "fieldName": "sample_id"}]`


All of this should have a table to reference, its nicely formatted here, but again this will lead to drift.

Env var official docs: https://github.com/overture-stack/lyric/blob/main/README.md#environment-variables

Deleted the whole block. The README table is now the only place these are documented, linked from the setup step as the single source of truth.

joneubank · 2026-08-04T19:25:05Z

+
+### Submission Testing
+
+1. Navigate to the Swagger UI.


Probably include the link: [Swagger UI](http://localhost:3030/api-docs)

joneubank · 2026-08-04T19:25:29Z

+### Submission Testing
+
+1. Navigate to the Swagger UI.
+2. Register a Lectern dictionary against a category using the dictionary-registration endpoints.


Which endpoint is that? Can you link here to a sample dictionary to use?

updated to use curl commands for each step, updated a section preceeding this to state that all commands shown can be achieved via the swagger ui.

joneubank · 2026-08-04T19:25:46Z

+
+1. Navigate to the Swagger UI.
+2. Register a Lectern dictionary against a category using the dictionary-registration endpoints.
+3. Submit a tabular data file and verify that it is validated against the registered schema.


Again, which endpoint and sample data.

same as above

joneubank · 2026-08-04T19:26:22Z

+
+**Troubleshooting:**
+
+- Ensure PostgreSQL and Lectern are running and reachable at the configured hosts and ports.


hows? consider adding more details like docker ps to ensure container status.

expanded with docker ps and troubleshooting tips

MitchellShiell mentioned this pull request Jul 30, 2026

Overture docs V2 update overture-stack/docs#48

Open

5 tasks

MitchellShiell added 6 commits July 30, 2026 10:27

usage > reference

327e5de

docs update

b269879

docs: repoint README developer-docs links at /develop/Lyric/overview

ef2650b

The docs site has since renamed /build/core-software/* to /build/*, then /build/* to /develop/*, so the previous target only resolved through two redirect hops.

MitchellShiell force-pushed the overtureDocsUpdate branch from 17a2069 to a2d6776 Compare July 30, 2026 17:16

MitchellShiell requested a review from joneubank July 30, 2026 17:43

docs: use full docs.overture.bio URLs for cross-repo links

4243539

joneubank requested changes Aug 4, 2026

View reviewed changes

docs: address review feedback, add working .env.schema defaults

6bc7f0b

MitchellShiell requested a review from joneubank August 5, 2026 17:43

MitchellShiell added 2 commits August 5, 2026 14:03

removed quickstart references from readme

0ba9d37

contributing link updated to in repo ref

c8db6b2

joneubank approved these changes Aug 5, 2026

View reviewed changes


		## Repository Structure

		The repository is a PNPM monorepo, organized with deployable applications under `apps/` and shared libraries under `packages/`:


		Troubleshooting:

		- Ensure PostgreSQL and Lectern are running and reachable at the configured hosts and ports.

Uh oh!

Conversation

MitchellShiell commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

MitchellShiell commented Aug 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MitchellShiell Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MitchellShiell commented Jul 30, 2026 •

edited

Loading

MitchellShiell Aug 5, 2026 •

edited

Loading