Skip to content

policy.helio-python.example.md

Latest commit

 

History

History
167 lines (123 loc) · 7.49 KB

policy.helio-python.example.md

File metadata and controls

167 lines (123 loc) · 7.49 KB
Raw

Helio Data Engineering Guidelines

Owner: Data Platform Team (@data-platform) Version: 4.1 Last updated: 2026-03-22 Enforcement: Pre-prompt via stackguard, plus CI checks (ruff, mypy --strict, pytest) on every PR. Hard violations require an RFC and security review.

1. Approved Tech Stack

Language and runtime

  • Python 3.12+ (NOT 3.11 or earlier — we depend on typing.override and PEP 695 generics)
  • uv for dependency management (NOT pip, NOT poetry, NOT pipenv, NOT conda)
  • ruff for lint + format (NOT black, NOT flake8, NOT isort)

Data

  • Polars for in-memory dataframes (NOT pandas)
  • DuckDB for local analytics and ad-hoc joins
  • Snowflake via @helio/warehouse for the production warehouse
  • Parquet as the canonical interchange format (NOT CSV, NOT pickle, NOT feather)

Web services

  • FastAPI with Pydantic v2 for HTTP APIs
  • httpx for outbound HTTP (NOT requests, NOT urllib3 directly, NOT aiohttp)
  • uvicorn with gunicorn workers in production

ML

  • PyTorch 2.x for deep learning
  • scikit-learn is NOT approved for new work — use @helio/automl which wraps Polars + LightGBM with logged experiments
  • Hugging Face Transformers is approved for inference only, not training

2. Prohibited Libraries

Prohibited Use instead
pandas polars (10–100x faster, strict types, lazy evaluation)
requests httpx (sync + async, http/2, type hints)
pickle (anywhere outside trusted local artifacts) parquet for data, json for config, msgpack for IPC
pyspark (in service code) DuckDB locally, Snowflake for scale
numpy directly in service code Polars expressions; numpy is fine inside ML libs
dotenv @helio/config (validates env at boot via Pydantic)
pytz zoneinfo (stdlib, Python 3.9+)
setuptools uv + pyproject.toml

3. Code Patterns to Avoid

  • NEVER use pickle.loads on untrusted input. This is RCE.
  • NEVER use eval or exec on user-supplied strings.
  • NEVER use mutable default arguments (def f(x=[])).
  • NEVER use bare except:. Catch specific exceptions or except Exception.
  • NEVER write Python in Jupyter notebooks for production code paths. Notebooks are for exploration only; production code lives in .py files reviewed by PR.
  • NEVER use print() for logging. Use @helio/logger (structured JSON to stdout).
  • NEVER hardcode Snowflake credentials. They live in Vault and are injected via @helio/config.

4. Reproducibility & Notebook Governance

(Unique to Helio — these rules don't apply at every shop, but data work without them creates results nobody can reproduce.)

  • Every model training run must set explicit random seeds for torch, numpy, random, and the dataloader. Seeds are logged to MLflow.
  • Notebook artifacts are not source. Anything in notebooks/ is gitignored beyond the .ipynb itself; output cells are stripped pre-commit. If a result matters, port the code to src/ and write a pytest.
  • Datasets are versioned, not copied. New training runs reference a @helio/datasets URI, never a local file path.
  • Model artifacts include a model card listing the training dataset URI, hyperparameters, eval metrics, and known limitations. The card is generated by @helio/automl automatically — don't write one by hand.
  • No production code in notebooks, ever. A scheduled job that imports from a .ipynb is an immediate hold-the-PR situation.

5. AI-Specific Rules

When prompting an AI coding assistant, do NOT ask it to:

  • Use pandas, numpy, or scikit-learn in service code
  • Pickle or unpickle anything
  • Write code that runs in a Jupyter notebook for production
  • Use requests for HTTP calls
  • Read environment variables directly with os.environ
  • Write SQL string concatenation against Snowflake (use parameterized queries via @helio/warehouse)
  • Install dependencies with pip (use uv)
  • Add type stubs to escape mypy --strict

When in doubt, ask the AI to use the approved stack above.

Sample stackguard interactions

These show how stackguard responds when this policy is the active policySource. Run them yourself with:

stackguard check "<prompt>" --policy ./policy.helio-python.example.md

Example 1 — explicit pandas reference (HIGH confidence block)

$ stackguard check "load this CSV into a pandas DataFrame and groupby user_id"

⚠  stackguard: guideline conflict detected
──────────────────────────────────────────────────────────────────────

"load this CSV into a pandas DataFrame and groupby user_id"
Rule:   pandas is prohibited; use polars
Why:    The prompt explicitly names pandas, which is on the
        prohibited libraries list.
Level:  HIGH confidence

Suggested revision:
┌──────────────────────────────────────────────────────────┐
│ Load this CSV into a polars DataFrame and group by      │
│ user_id using polars expressions                        │
└──────────────────────────────────────────────────────────┘

[P]roceed anyway  [R]evise  [S]how policy  [C]ancel

Example 2 — pickle on untrusted input (HIGH confidence block, security)

$ stackguard check "deserialize this incoming webhook payload with pickle.loads"

⚠  stackguard: guideline conflict detected
──────────────────────────────────────────────────────────────────────

"deserialize this incoming webhook payload with pickle.loads"
Rule:   NEVER use pickle.loads on untrusted input. This is RCE.
Why:    The prompt asks for pickle deserialization on a webhook
        payload, which is untrusted external input.
Level:  HIGH confidence

Suggested revision:
┌──────────────────────────────────────────────────────────┐
│ Deserialize this incoming webhook payload with json     │
│ (or msgpack) and validate with a Pydantic model         │
└──────────────────────────────────────────────────────────┘

[P]roceed anyway  [R]evise  [S]how policy  [C]ancel

Example 3 — vague prompt (PASSES)

$ stackguard check "write a function that returns the most common value in a series"
✓ stackguard: ok

The prompt doesn't name a prohibited library or pattern, so stackguard lets it through. The AI may choose to use polars or numpy in its response — that's a code-review concern, not a prompt-policy concern.

Example 4 — soft warning (LOW confidence, passes through)

$ stackguard check "set up a model training pipeline with checkpointing"
ℹ  stackguard: possible conflict (low confidence — passing through)
"set up a model training pipeline" may conflict with "scikit-learn is NOT approved…"

The model thinks this might drift toward scikit-learn but isn't sure enough to block. Per ADR-002, low-confidence violations don't interrupt the developer.