Use temporary local Zarr for regionprops to prevent file errors - #189
Merged
Conversation
to avoid too many open file errors
Updated ZarrNii file loading to simplify the code.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves robustness and scalability of the Snakemake workflow for segmentation region property computation by increasing resources for memory-intensive steps, adding an additional wildcard constraint for group-level analysis, and adjusting regionprops processing to use a temporary local extracted Zarr store when inputs are zip-based.
Changes:
- Increased memory/runtime resources for
compute_filtered_regionpropsand increased memory formerge_into_segstats_tsvto reduce OOM/timeouts on large datasets. - Updated
compute_filtered_regionprops.pyto extract zip-based OME-Zarr inputs into a temporary directory before computing region properties. - Added an
acqwildcard constraint for group-level workflows to improve pattern matching.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| spimquant/workflow/Snakefile | Adds acq to wildcard_constraints for group-level analyses. |
| spimquant/workflow/scripts/compute_filtered_regionprops.py | Changes regionprops computation to operate on a temporary extracted local store rather than reading directly from a zip-based store. |
| spimquant/workflow/rules/segstats.smk | Increases memory allocation for TSV merging to reduce OOM risk. |
| spimquant/workflow/rules/regionprops.smk | Increases memory/runtime allocation for regionprops computation to better handle large datasets. |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several important improvements to the workflow's resource allocation and file handling for region property computation and segmentation statistics merging. The main changes focus on increasing resource limits for memory-intensive tasks and ensuring robust extraction and handling of zipped OME-Zarr files.
Resource allocation improvements:
compute_filtered_regionpropsrule inregionprops.smkto better handle large datasets (mem_mbfrom 256000 to 500000,runtimefrom 180 to 360).merge_into_segstats_tsvrule insegstats.smk(mem_mbfrom 1500 to 16000) to prevent out-of-memory errors during merging.File handling and robustness:
compute_filtered_regionprops.pyto extract the input OME-Zarr zip archive to a secure temporary directory before analysis, improving compatibility and preventing issues with direct zip file access.Workflow configuration:
acqwildcard constraint to the workflow'swildcard_constraintsfor more robust pattern matching in group-level analyses.