MASTR: support refreshing from the open-mastr bulk export#298
Open
MaykThewessen wants to merge 3 commits into
Open
MASTR: support refreshing from the open-mastr bulk export#298MaykThewessen wants to merge 3 commits into
MaykThewessen wants to merge 3 commits into
Conversation
The MASTR source assumes the Zenodo CSV dump, whose last published version is frozen at 2025-02-09. This adds support for building the source from a current open-mastr bulk export instead: - get_raw_file: a glob fn selects the newest matching local file (ISO-dated names sort chronologically). The newest local dump wins even on update=True, so a forced refresh never regresses to the frozen URL. - MASTR(): match the storage_units CSV by suffix instead of a hardcoded bnetza_open_mastr_2025-02-09/ folder, so any dated dump loads. - MASTR(): guard ThermischeNutzleistung, which is present in the Zenodo CSV but absent from the bulk export (CHP detection falls back to KwkMastrNummer alone). - config: MASTR fn is now a glob (bnetza_open_mastr_*.zip), Zenodo URL kept as the fallback seed. - scripts/build_mastr_zip_from_open_mastr.py: build the ppm-format zip from an open-mastr SQLite DB.
for more information, see https://pre-commit.ci
This was referenced Jun 18, 2026
The new helper script lacked the REUSE-required SPDX header, failing the reuse pre-commit.ci check on PyPSA#298. Add the MIT header used across the package.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lets the
MASTRsource be refreshed from a current open-mastr bulk export instead of only the Zenodo CSV dump (whose last published version is frozen at 2025-02-09, record 14783581is_last:true, while the registry is months newer).Changes
get_raw_file: a globfnselects the newest matching local file (ISO-dated names sort chronologically). The newest local dump wins even whenupdate=True- the URL is only a seed and can never be newer than a dated local build, so a forced refresh no longer silently regresses to stale data.MASTR(): match thestorage_unitsCSV by suffix instead of the hardcodedbnetza_open_mastr_2025-02-09/folder, so any dated dump loads.MASTR(): guardThermischeNutzleistung, which is present in the Zenodo CSV but absent from the bulk export; CHP detection falls back toKwkMastrNummeralone (previously raisedKeyError).config.yaml:MASTR.fnis now a glob (bnetza_open_mastr_*.zip); the Zenodo URL is kept as the fallback seed.scripts/build_mastr_zip_from_open_mastr.py: helper to build the ppm-format zip from an open-mastr SQLite DB (Mastr().download()), so users can produce a current dump.Why
Zenodo's open-mastr dump is no longer updated frequently, so the German fleet ages. The open-mastr bulk export is current to the day; these changes make ppm able to consume it with no manual path edits, and without a forced refresh quietly falling back to the frozen 2025 file.
Notes / backwards compatibility
fn(every other source).update=Falseandupdate=Trueresolve to the newest local dump; fullpowerplants()build runs end-to-end on a fresh 2026 export.Related
There is a companion issue (#295) about the open-mastr bulk download silently dropping the combustion table on a multi-code
WeitereBrennstoffevalue - relevant when building the dump this PR consumes.