Skip to content

docs(data-quality): add check_output_custom section (#212)#1658

Open
philipljh wants to merge 2 commits into
apache:mainfrom
philipljh:212-docs-check-output-custom
Open

docs(data-quality): add check_output_custom section (#212)#1658
philipljh wants to merge 2 commits into
apache:mainfrom
philipljh:212-docs-check-output-custom

Conversation

@philipljh

Copy link
Copy Markdown

Closes #212 - adds documentation for @check_output_custom

It walks through the full lifecycle of a custom validator:

  • When you need one: positions @check_output_custom relative to @check_output's default validators (data type, range, allowed values, ...), with a cross-reference to the check_output API reference.
  • Writing the validator: a complete example class, AllPositiveValidator, that subclasses hamilton.data_quality.base.DataValidator and checks a pandas Series for negative values. Each required method is then explained individually: applies_to, description, name, validate returning a ValidationResult (with passes, message, and the free-form diagnostics dict), plus the importance constructor argument.
  • Applying it: decorating a function with a validator instance, including that multiple validators go into a single decorator call and that stacking @check_output_custom decorators is not supported.
  • Runtime semantics: importance="warn" logs through the standard logging module and continues; importance="fail" raises DataValidationError, with all validators evaluated first so every failure is reported at once.
  • What happens to the DAG: the {name}_raw / per-validator / final-node split, the hamilton.data_quality.contains_dq_results and hamilton.data_quality.source_node tags, and a driver example showing that validator nodes can simply be requested as outputs to inspect their ValidationResult.

Changes

Adds a "Custom validators" section to the "Data quality" how-to page (docs/how-tos/run-data-quality-checks.rst) covering:

  • when to write a custom validator vs. using @check_output's built-in validators
  • a complete, runnable DataValidator subclass example
  • applying one or more validator instances with @check_output_custom (and that decorators can't be stacked)
  • runtime behavior of importance="warn" (logs, continues) vs "fail" (raises DataValidationError)
  • how validation results appear as tagged nodes in the DAG and how to request them as outputs

How I tested this

  • Ran every code snippet in the section against a local editable install; outputs match the docs verbatim.
  • Built the docs locally with sphinx-build -b dirhtml -W; no warnings from the changed page.
  • Screenshot of the rendered section attached.
image

Notes

  • Docs-only change

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested (n/a - docs only)
  • New functions are documented (n/a - docs only)
  • Placeholder code is flagged / future TODOs are captured in comments (n/a)
  • Project documentation has been updated if adding/changing functionality.

@philipljh

Copy link
Copy Markdown
Author

Hello, made my first oss pr, open to any feedback and guidance. Cheers~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Documentation for check_output_custom

1 participant