Skip to content

Fix dataset export fidelity and atomic upload failures - #234

Open
Gautam Sharma (GautamSharma99) wants to merge 1 commit into
langchain-ai:mainfrom
GautamSharma99:fix/dataset-roundtrip-226
Open

Gautam Sharma (GautamSharma99) wants to merge 1 commit into
langchain-ai:mainfrom
GautamSharma99:fix/dataset-roundtrip-226

Conversation

@GautamSharma99

Copy link
Copy Markdown
Contributor

Summary

Closes #226

Dataset export/upload now has a versioned, loss-aware round-trip format and validates the complete import before mutating LangSmith.

What changed

  • Export uses version 1 of a JSON envelope and preserves dataset name/description plus example IDs, creation timestamps, inputs, outputs, metadata, native split membership, and attachment URLs.
  • Upload validates every array member before creating the destination dataset. Non-object members include their array index in the error, and malformed inputs, IDs, timestamps, split values, and unsupported attachment restoration fail early.
  • Upload uses the generated SDK bulk-example endpoint so metadata, IDs, timestamps, and splits are sent as native API fields instead of being dropped or encoded in metadata.
  • If bulk creation fails after the dataset is created, upload deletes that dataset. If cleanup also fails, the returned error includes both failures so the operator has an actionable dataset ID.
  • Legacy array/object input remains accepted, but now requires an explicit inputs object rather than silently treating arbitrary fields as inputs.
  • The README documents the versioned schema and attachment behavior.

Tests

  • Reject mixed arrays with an indexed validation error.
  • Verify metadata and multi-split values survive parsing.
  • Reject attachment-bearing imports before remote mutation.
  • go test ./...
  • go test -race ./internal/cmd
  • go vet ./...
  • make build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dataset export/upload loses metadata and leaves partial datasets on failure

1 participant