Skip to content

MASTR: load any dated open-mastr dump (date-agnostic paths + glob fn)#300

Closed
MaykThewessen wants to merge 1 commit into
PyPSA:masterfrom
MaykThewessen:feature/mastr-date-agnostic-loading
Closed

MASTR: load any dated open-mastr dump (date-agnostic paths + glob fn)#300
MaykThewessen wants to merge 1 commit into
PyPSA:masterfrom
MaykThewessen:feature/mastr-date-agnostic-loading

Conversation

@MaykThewessen

Copy link
Copy Markdown

What

Makes the MASTR() loader work with any dated open_mastr dump instead of only the hardcoded bnetza_open_mastr_2025-02-09 release. Fixes parts 2 and 3 of #299.

Why

Today the loader cannot consume a newer MaStR dump even if you point the config at one:

  1. The storage-units CSV is read from a hardcoded path that embeds the release date, so a newer dump (different bnetza_open_mastr_<date>/ inner folder) raises KeyError.
  2. ThermischeNutzleistung exists in the Zenodo CSV dump but not in the open_mastr bulk export, so a non-Zenodo-shaped dump breaks the CHP-detection assign.

See #299 for the full description.

Changes

  • data.py — match bnetza_mastr_storage_units_raw.csv by suffix rather than the hardcoded dated folder prefix.
  • data.py — CHP detection falls back to KwkMastrNummer alone when ThermischeNutzleistung is absent.
  • utils.py (get_raw_file) — a glob fn selects the newest matching local file (ISO dates sort chronologically); falls back to downloading the URL's basename when nothing local matches. Non-glob fn is unchanged.
  • config.yaml — default MASTR.fn to the glob bnetza_open_mastr_*.zip, keeping the frozen Zenodo url as the download seed.

Compatibility

Behavior-preserving on the current Zenodo dump: a fresh checkout has no local match, so get_raw_file falls back to downloading bnetza_open_mastr_2025-02-09.zip, and the suffix match reads the same inner CSV as before. Pin a specific dump by setting fn to a literal filename.

Not included here (deliberately, to keep this focused): the optional helper that rebuilds a current dump from the open_mastr bulk export. Happy to add it in a follow-up if maintainers want in-repo refresh tooling.

Test plan

  • import powerplantmatching / module parse
  • Maintainer CI (Duke/Java-gated tests)

The loader hardcoded the 2025-02-09 dump's internal folder, so pointing
the source at any newer dated dump raised KeyError on the storage-units
read. Make the loader date-agnostic:

- data.py: match bnetza_mastr_storage_units_raw.csv by suffix instead of
  the hardcoded bnetza_open_mastr_2025-02-09/ folder prefix.
- data.py: fall back to KwkMastrNummer alone for CHP detection when
  ThermischeNutzleistung is absent (present in the Zenodo CSV dump,
  missing from the open-mastr bulk export).
- utils.py get_raw_file: support a glob fn, selecting the newest matching
  local dump and falling back to the URL basename when none exists.
- config.yaml: default MASTR.fn to the glob bnetza_open_mastr_*.zip, with
  the frozen Zenodo url kept as the download seed.

Behaviour-preserving on the current Zenodo dump. Addresses PyPSA#299.
@MaykThewessen

Copy link
Copy Markdown
Author

Closing as a duplicate of #298, which makes the identical date-agnostic changes (storage_units suffix-match, ThermischeNutzleistung guard, glob fn in get_raw_file, config glob) and additionally ships the build helper script. Apologies for the noise: #298 is the one to review.

@MaykThewessen MaykThewessen deleted the feature/mastr-date-agnostic-loading branch June 18, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant