Skip to content

make_origs drops columns missing from cleaning_sd once any add_orig fires #914

Description

@paddymul

Problem

Both PandasAutocleaning.make_origs (buckaroo/dataflow/autocleaning.py:184) and PolarsAutocleaning.make_origs (buckaroo/polars_buckaroo.py:47) rebuild the cleaned frame solely from cleaning_sd entries. When no column sets add_orig (changed == 0), cleaned_df is returned unchanged. Once any column sets add_orig, the output is constructed only from columns that appear in cleaning_sd with an orig_col_name resolvable in cleaned_df — any column present in cleaned_df but absent from cleaning_sd is silently dropped.

Impact

A cleaning op or command that synthesizes a new column (one with no cleaning_sd entry) loses that column whenever another column triggers add_orig. No error is raised; the column just disappears. The built-in configs don't synthesize columns today, so this is latent, but it constrains what op generators and commands can safely do.

Suggested fix

Iterate cleaned_df.columns as the source of truth and consult cleaning_sd only for the add_orig flag, so columns without an sd entry pass through untouched. Should be folded into the autocleaning op-generation rebuild (#879).

Context

Identified during review of #912; the polars make_origs rewrite there deliberately mirrors the pandas behavior, so both implementations share the hazard.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions