Skip to content

fix(cli): support JSON-LD ingestion - #2506

Open
branarakic wants to merge 8 commits into
testnet-canaryfrom
feat/issue-15-jsonld-ingest
Open

branarakic wants to merge 8 commits into
testnet-canaryfrom
feat/issue-15-jsonld-ingest

Conversation

@branarakic

@branarakic branarakic commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

The CLI advertises .jsonld input but rejects documents that use @context. Convert JSON-LD through jsonld.js and the shared private N3 quad parser, preserving named graphs, blank nodes, lists, language tags and datatypes. JSON and legacy JSON-LD quad arrays share one typed decoder. The neutral parser returns source provenance alongside quads; its existing array facade remains available.

File input supplies its file URL as the document base. Safe conversion rejects expansion that would discard statements. Contexts must be inline: an offline loader rejects external @context and @import references without making requests. The CLI README documents the input and context policy.

Use the canonical RDF literal formatter so quotes, backslashes and control characters survive ingestion without producing invalid RDF.

Default-finalizing ka create rejects JSON-LD named-graph documents before connecting to the daemon, with actionable --no-finalize guidance. WM-only imports preserve their named graphs; existing quad-array inputs retain their prior contract. The Knowledge Asset command owns this policy and uses parser provenance to distinguish new JSON-LD document conversion from legacy arrays; no lifecycle flags pass through shared parsing helpers.

Working-memory writes accept validated RDF blank-node terms so JSON-LD lists and nested objects survive ingestion and the existing canonical finalization step. Malformed or injectable blank-node labels fail at the HTTP boundary. Inline --triples uses the same legacy quad-array decoder as JSON files.

Validation: all 60 parser and actual CLI smoke tests pass on this revision, covering syntax provenance, named/default graphs, lists/nested objects, literal escaping, file bases, legacy arrays and offline contexts. CLI/dependency builds, public types/package-boundary checks, repository lint, the 1,801-file test inventory, SPARQL and merge-tree checks pass. Earlier revisions additionally exercised blank-node writes and canonical finalization through a local daemon backed by the test chain.

The parser refactor keeps the array compatibility adapter one-way: parseRdf calls parseRdfInput, whose RDF and converted JSON-LD branches both call private parseN3Quads directly. No branch calls back through the compatibility adapter.

Closes #15.

@branarakic
branarakic requested a review from Jurij89 as a code owner September 7, 2026 20:21
Comment thread packages/cli/src/rdf-parser.ts Outdated
Comment thread packages/cli/test/rdf-parser.test.ts
Comment thread packages/cli/src/rdf-parser.ts Outdated
Comment thread packages/cli/src/rdf-parser.ts Outdated
Comment thread packages/cli/src/rdf-parser.ts Outdated
Comment thread packages/cli/test/knowledge-asset-cli-smoke.test.ts Outdated
Comment thread packages/cli/src/commands/knowledge-asset.ts
Comment thread packages/cli/src/cli-helpers.ts Outdated
Comment thread packages/cli/src/commands/knowledge-asset.ts
Comment thread packages/cli/src/daemon/http-utils.ts
Comment thread packages/cli/src/rdf-parser.ts Outdated
@branarakic
branarakic force-pushed the feat/issue-15-jsonld-ingest branch from 60eab8e to 8d0617b Compare September 14, 2026 21:37
@branarakic

Copy link
Copy Markdown
Contributor Author

Rebased the existing JSON-LD ingestion fix onto current testnet-canary and force-updated the PR branch with an explicit lease.

New head: 8d0617be5

Validation on the rebased tree:

  • pnpm install --frozen-lockfile
  • workspace dependency closure builds for RDF utils, core, storage, chain, publisher, agent, MCP, node-ui, EPCIS, OKF, and CLI
  • pnpm --filter @origintrail-official/dkg exec vitest run test/rdf-parser.test.ts — 40/40 passed
  • pnpm --filter @origintrail-official/dkg exec vitest run test/jsonld-knowledge-asset-lifecycle.test.ts test/knowledge-asset-cli-smoke.test.ts test/writable-quad-terms.test.ts — 74/74 passed
  • pnpm --filter @origintrail-official/dkg build — passed
  • pnpm lint and git diff --check — passed

The dependency and lockfile follow-up is limited to the CLI's JSON-LD and shared RDF utility dependencies.

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 20th, 2026 9:00 AM.

@branarakic
branarakic force-pushed the feat/issue-15-jsonld-ingest branch from 8d0617b to 50ecdb0 Compare September 14, 2026 22:08

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 20th, 2026 9:00 AM.

@branarakic
branarakic force-pushed the feat/issue-15-jsonld-ingest branch from 50ecdb0 to 81c4e8f Compare September 14, 2026 23:15
}

describe('JSON-LD CLI writes through the real daemon', () => {
it.each(['list', 'nested'] as const)('writes and finalizes linked %s RDF without losing blank-node edges', async (kind) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Blank-node collisions across separate JSON-LD writes are never exercised by a test

What's wrong
The change newly allows JSON-LD anonymous nodes (_:bN) through validation and storage, and jsonld.js assigns those labels per document, not per write. Because empty-graph quads all land in one per-KA WM graph, two separate ka write -f invocations on the same KA with different JSON-LD files silently merge their anonymous nodes. The only end-to-end test writes a single document per KA, so a conflated result (wrong graph structure at finalize/skolemization) still passes green and the suite gives the impression that JSON-LD blank-node writes are fully validated.

Example
File A: {"@id":"urn:docA","https://example.org/child":{"https://example.org/name":"A"}} → <urn:docA> <.../child> _:b0 and _:b0 <.../name> "A". File B: {"@id":"urn:docB","https://example.org/child":{"https://example.org/name":"B"}} → also _:b0. After two separate ka write calls both nodes are _:b0 in the single per-KA graph, so urn:docA and urn:docB point at one node carrying both "A" and "B". Expected: the two anonymous nodes stay distinct. The existing test only ever runs ka write once per KA, so this conflation is not detected.

Suggested direction
Add a live-daemon regression that creates one KA, runs two separate ka write -f (or ka create) invocations with different JSON-LD files that each contain an anonymous nested node, then reads /wm/quads and (after finalize) asserts the two documents' anonymous nodes are distinct — e.g. each document's child carries only its own literal. If the intended design is to disambiguate on ingest, add the relabeling in rdf-parser.ts/cli-helpers.ts and keep the current single-write test.

For Agents
Look at packages/cli/test/jsonld-knowledge-asset-lifecycle.test.ts and extend it: after the single-write cases, add a case that writes two different JSON-LD documents (each with an anonymous nested node) to the same KA via two separate CLI ka write -f invocations, then assert wm/quads contains two distinct blank-node subjects/objects and that finalization preserves both edges. If the fix is to relabel blank nodes at ingest, implement it in rdf-parser.ts/cli-helpers.ts (where quads are produced) and keep the existing single-write assertions unchanged.

}

/** Array-only compatibility facade for callers that do not need source provenance. */
export async function parseRdf(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Suggestion: parseRdf is now a pass-through facade whose only callers are tests.

Why it matters
The PR renamed the real parser to parseRdfInput and reintroduced parseRdf as an array-only wrapper, but every production call site goes through loadRdfFromInput -> parseRdfInput; only test files import parseRdf. An exported entry point kept alive solely by tests makes the module look like it has two APIs and invites the two signatures to drift.

Suggestion
Remove the test-only parseRdf wrapper so the module exposes a single parse entry point.

@branarakic-agent branarakic-agent added this to the 10.0.21 milestone Sep 25, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants