MASTR: load any dated open-mastr dump (date-agnostic paths + glob fn)#300
Closed
MaykThewessen wants to merge 1 commit into
Closed
MASTR: load any dated open-mastr dump (date-agnostic paths + glob fn)#300MaykThewessen wants to merge 1 commit into
MaykThewessen wants to merge 1 commit into
Conversation
The loader hardcoded the 2025-02-09 dump's internal folder, so pointing the source at any newer dated dump raised KeyError on the storage-units read. Make the loader date-agnostic: - data.py: match bnetza_mastr_storage_units_raw.csv by suffix instead of the hardcoded bnetza_open_mastr_2025-02-09/ folder prefix. - data.py: fall back to KwkMastrNummer alone for CHP detection when ThermischeNutzleistung is absent (present in the Zenodo CSV dump, missing from the open-mastr bulk export). - utils.py get_raw_file: support a glob fn, selecting the newest matching local dump and falling back to the URL basename when none exists. - config.yaml: default MASTR.fn to the glob bnetza_open_mastr_*.zip, with the frozen Zenodo url kept as the download seed. Behaviour-preserving on the current Zenodo dump. Addresses PyPSA#299.
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes the
MASTR()loader work with any datedopen_mastrdump instead of only the hardcodedbnetza_open_mastr_2025-02-09release. Fixes parts 2 and 3 of #299.Why
Today the loader cannot consume a newer MaStR dump even if you point the config at one:
bnetza_open_mastr_<date>/inner folder) raisesKeyError.ThermischeNutzleistungexists in the Zenodo CSV dump but not in theopen_mastrbulk export, so a non-Zenodo-shaped dump breaks the CHP-detection assign.See #299 for the full description.
Changes
data.py— matchbnetza_mastr_storage_units_raw.csvby suffix rather than the hardcoded dated folder prefix.data.py— CHP detection falls back toKwkMastrNummeralone whenThermischeNutzleistungis absent.utils.py(get_raw_file) — a globfnselects the newest matching local file (ISO dates sort chronologically); falls back to downloading the URL's basename when nothing local matches. Non-globfnis unchanged.config.yaml— defaultMASTR.fnto the globbnetza_open_mastr_*.zip, keeping the frozen Zenodourlas the download seed.Compatibility
Behavior-preserving on the current Zenodo dump: a fresh checkout has no local match, so
get_raw_filefalls back to downloadingbnetza_open_mastr_2025-02-09.zip, and the suffix match reads the same inner CSV as before. Pin a specific dump by settingfnto a literal filename.Not included here (deliberately, to keep this focused): the optional helper that rebuilds a current dump from the
open_mastrbulk export. Happy to add it in a follow-up if maintainers want in-repo refresh tooling.Test plan
import powerplantmatching/ module parse