Skip to content

Use temporary local Zarr for regionprops to prevent file errors - #189

Merged
akhanf merged 5 commits into
mainfrom
regionprops-resources
Jul 30, 2026
Merged

Use temporary local Zarr for regionprops to prevent file errors#189
akhanf merged 5 commits into
mainfrom
regionprops-resources

Conversation

@akhanf

@akhanf akhanf commented Jul 30, 2026

Copy link
Copy Markdown
Member

This pull request introduces several important improvements to the workflow's resource allocation and file handling for region property computation and segmentation statistics merging. The main changes focus on increasing resource limits for memory-intensive tasks and ensuring robust extraction and handling of zipped OME-Zarr files.

Resource allocation improvements:

  • Increased memory and runtime limits for the compute_filtered_regionprops rule in regionprops.smk to better handle large datasets (mem_mb from 256000 to 500000, runtime from 180 to 360).
  • Increased memory allocation for the merge_into_segstats_tsv rule in segstats.smk (mem_mb from 1500 to 16000) to prevent out-of-memory errors during merging.

File handling and robustness:

  • Updated compute_filtered_regionprops.py to extract the input OME-Zarr zip archive to a secure temporary directory before analysis, improving compatibility and preventing issues with direct zip file access.

Workflow configuration:

  • Added an acq wildcard constraint to the workflow's wildcard_constraints for more robust pattern matching in group-level analyses.

akhanf and others added 2 commits July 30, 2026 17:00
to avoid too many open file errors
Updated ZarrNii file loading to simplify the code.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves robustness and scalability of the Snakemake workflow for segmentation region property computation by increasing resources for memory-intensive steps, adding an additional wildcard constraint for group-level analysis, and adjusting regionprops processing to use a temporary local extracted Zarr store when inputs are zip-based.

Changes:

  • Increased memory/runtime resources for compute_filtered_regionprops and increased memory for merge_into_segstats_tsv to reduce OOM/timeouts on large datasets.
  • Updated compute_filtered_regionprops.py to extract zip-based OME-Zarr inputs into a temporary directory before computing region properties.
  • Added an acq wildcard constraint for group-level workflows to improve pattern matching.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
spimquant/workflow/Snakefile Adds acq to wildcard_constraints for group-level analyses.
spimquant/workflow/scripts/compute_filtered_regionprops.py Changes regionprops computation to operate on a temporary extracted local store rather than reading directly from a zip-based store.
spimquant/workflow/rules/segstats.smk Increases memory allocation for TSV merging to reduce OOM risk.
spimquant/workflow/rules/regionprops.smk Increases memory/runtime allocation for regionprops computation to better handle large datasets.

Comment thread spimquant/workflow/scripts/compute_filtered_regionprops.py Outdated
Comment thread spimquant/workflow/scripts/compute_filtered_regionprops.py Outdated
Comment thread spimquant/workflow/scripts/compute_filtered_regionprops.py Outdated
akhanf and others added 2 commits July 30, 2026 15:14
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@akhanf
akhanf merged commit dd84853 into main Jul 30, 2026
2 checks passed
@akhanf
akhanf deleted the regionprops-resources branch July 30, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants