New snippet for jpeg/zip: 4tu-60cdce27-liposome-yolo-boxes-confocal

snippets/jpeg-zip-glob/README.md

@@ -0,0 +1,53 @@
+# jpeg-zip-glob
+
+Reads a set of JPEG images that are members of a zip, decoded with `imagecodecs`, and presents it as one
+array: one image per z plane, a colour JPEG's samples as the c axis. The zip may be a plain URL or a member of
+another zip, which is the case this snippet was written for: a dataset published as one zip whose images sit in
+zips inside it.
+
+- **URLs.** Each URL names one zip that holds images: a plain `<zip url>`, or `<outer zip url>::<inner zip
+  member>` for a zip nested in another. An outer member glob (`<outer zip url>::*/obj.zip`) is expanded with
+  `io.expand` and stands for every inner zip it matches. Several URLs are concatenated in binding order.
+- **`glob`** (required) picks the images inside each zip: an `fnmatch` pattern over member names (`*` also
+  crosses `/`), matches sorted by name within a zip. A zip with no match raises. The pattern is a param and
+  not a third `::` part of the URL because the harness itself resolves binding URLs (`harness fetch` and
+  `harness stage` call `io.expand` and `io.fetch` on each one) and understands one `::` level; a URL of the
+  form `<zip>::<inner zip>::<glob>` raises here with that explanation. Anchor the pattern at the image folder
+  (`val/*.jpg`): zips made on macOS carry `__MACOSX/val/._name.jpg` sidecars that a bare `*.jpg` matches, and
+  those are not JPEGs (the snippet raises `... is not a JPEG` when it reads one).
+- **Files.** Baseline or progressive JPEG, grey `(Y, X)` or three-sample `(Y, X, 3)` as `imagecodecs.jpeg_decode`
+  returns it (YCbCr is converted to RGB by libjpeg). All images of one array must have the same shape and
+  dtype; one that differs from the first raises at `read`. Four-sample (CMYK) JPEGs are refused.
+- **Axes.** `z` when there is more than one image, `c` when more than one sample is exposed, then `yx`:
+  `zcyx`, `zyx`, `cyx` or `yx`. The z axis is a stack of independent images in name order, not depth.
+- **`samples`** (optional) names the samples of a colour JPEG to expose, as distinct letters of `rgb` in c
+  order: `rg` gives c 0 = red, c 1 = green and drops blue; one letter gives a single plane and no c axis.
+  Without it all three samples are c. Use it when the record says channels were saved into some of the colour
+  planes and the rest is empty: the harness draws crops from every c plane, an empty one included. A grey JPEG
+  takes no `samples`.
+- **Access.** `plane`: an image is decoded whole whatever the region. The last decoded image is kept, so the c
+  planes of one image cost one decode. A zip that `io.open` can read by range (a plain URL, a stored member)
+  is read that way: its directory at open, each image's bytes when `read` asks. A deflated inner zip, or a
+  server ignoring Range, raises `NeedsFetch` and the whole zip is staged with `io.fetch` at open.
+  `open_array` reads every bound zip's directory and decodes the first image (shape, dtype, sample count).
+- **dtype.** What the decoder returns: `uint8` (`uint16` for a 12-bit JPEG). **Voxel size:** `None`; a JPEG's
+  JFIF density is a print size.
+
+## Known limits
+
+- JPEG is lossy: values carry compression noise, and with chroma subsampling a strong red or green signal
+  leaks a little into the other samples (the example's "empty" blue plane reaches 64 of 255 in places).
+- A deflated inner zip is fetched whole at open even when one image is wanted (67 MB for the example's
+  largest), since its directory is at its end.
+- Image shapes are checked against the first image at `read`, not at open.
+- No labels: this snippet reads images only. The example dataset's YOLO box tables (`.txt` beside each jpg) are
+  not read by any snippet; the binding format has no box representation yet.
+
+## Example
+
+`examples/4tu-60cdce27-liposome-yolo-boxes-confocal.yaml`: one 972 MB zip at data.4tu.nl holding four deflated
+inner zips (`obj.zip` and `val.zip` for a 1-class and a 2-class YOLO set) with 560, 140, 32 and 8 confocal
+images of liposomes, 512x512 RGB `uint8` JPEGs with the two channels in red and green. Bound as four raws,
+`samples: rg`, each `zcyx` `(N, 2, 512, 512)`. Which of red and green is which stain is not stated in the
+record. The example binding's `snippets_commit` is the commit before this snippet's own (a binding can't name
+the commit that adds it); a binding written by the worker names a commit that contains the snippet.

snippets/jpeg-zip-glob/examples/4tu-60cdce27-liposome-yolo-boxes-confocal.yaml

@@ -0,0 +1,25 @@
+record: 4tu-60cdce27-liposome-yolo-boxes-confocal
+record_commit: 3f75709f1d2778fe86fd9c436b2b5ff948a72c7b
+snippets_commit: c6f6a6d90f446630e6157532eb084e5c9c9650d6
+raw:
+  - snippet: jpeg-zip-glob
+    urls: ["https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data - YOLO training and testing/Fig3 - FigS4/MinD 1 class/20240108_MinD_1class_training data and results/obj.zip"]
+    params: {glob: "obj/*.jpg", samples: rg}
+  - snippet: jpeg-zip-glob
+    urls: ["https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data - YOLO training and testing/Fig3 - FigS4/MinD 1 class/20240108_MinD_1class_training data and results/val.zip"]
+    params: {glob: "val/*.jpg", samples: rg}
+  - snippet: jpeg-zip-glob
+    urls: ["https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data - YOLO training and testing/Fig3 - FigS4/MinCDE 2 classes/20241107_MinCDE_2classes_training data and results/obj.zip"]
+    params: {glob: "obj/*.jpg", samples: rg}
+  - snippet: jpeg-zip-glob
+    urls: ["https://data.4tu.nl/file/60cdce27-5bd5-44ec-9d8f-53f2ef6b9d58/884eb9fc-0111-48b3-93b7-68368a764753::data - YOLO training and testing/Fig3 - FigS4/MinCDE 2 classes/20241107_MinCDE_2classes_training data and results/val.zip"]
+    params: {glob: "val/*.jpg", samples: rg}
+labels: []
+reasons:
+  - "jpeg-zip-glob: the images are JPEGs inside obj.zip / val.zip, which are themselves deflated members of the record's one zip; no other snippet decodes JPEG or looks inside a nested zip"
+  - "each url is the inner zip as a member of the 4TU zip (`<zip>::<...>/obj.zip`), not the draft's `<zip>::<inner zip>::<glob>`: the harness's own fetch and stage call io.expand and io.fetch on every binding url, which resolve one `::` level, so the url names a file the harness can fetch and cache (the inner zip) and the pattern for the images inside it is params.glob"
+  - "four raws, one per inner zip, so the whole set is bound (560 + 140 + 32 + 8 = 740 images, the record's n_items) rather than the 8-image sample, and the 1-class / 2-class sets and their obj (training) / val splits stay separate volumes; each is a z stack of independent 512x512 fields in member-name order, not a 3D volume (record dimensionality 2D)"
+  - "glob obj/*.jpg and val/*.jpg, anchored at the folder: each inner zip also holds macOS sidecars __MACOSX/<folder>/._<name>.jpg (AppleDouble, not JPEG) that a bare *.jpg would match, and the .txt box files"
+  - "samples rg: the record says the two confocal channels look stored as R and G with B near 0; decoding all 740 images confirmed it (per-set B mean 0.3-1.5 of 255, B max 4 in the 2-class set and 56-64 in the 1-class set, against R and G reaching 255), so B is JPEG chroma residue and is dropped; c 0 = red sample, c 1 = green sample. Which of them is eGFP-MinC and which DOPE-Cy5 (or the 1-class set's stains) is not stated in the record or the files"
+  - "no labels bound: the annotations are YOLO box tables (record encoding boxes-table, one .txt per jpg), which have no segmentation_type or label_key kind in the binding format; a human has to decide how boxes are represented before a snippet can read them"
+  - "voxel size: the snippet reports None (a JPEG carries no pixel size); the record's 250 nm x/y comes from the paper"

snippets/jpeg-zip-glob/snippet.py

@@ -0,0 +1,115 @@
+"""JPEG images chosen by glob from the members of a zip, one image per z plane. The zip is whatever `io` can
+open or fetch: a plain URL, or a member of another zip (a zip nested in the dataset's zip).
+
+Each url names one zip (`<zip url>` or `<outer zip url>::<inner zip member>`); `params.glob` picks the images
+inside it. A colour JPEG's samples become the c axis, all of them or the ones `params.samples` names.
+`open_array` reads each zip's directory and decodes the first image; `read` decodes the planes it needs.
+"""
+from fnmatch import fnmatchcase
+from zipfile import ZipFile
+
+import imagecodecs
+import numpy as np
+
+from mia_harness.api import IO, Array, NeedsFetch, Source
+
+SPEC_VERSION = 1
+
+SAMPLES = "rgb"     # the letters of params.samples, in the decoded image's sample order
+
+
+class _Images:
+    def __init__(self, files: list[tuple[ZipFile, str, str]], samples: str | None):
+        self._files = files                             # (open zip, its url, member name), one per z plane
+        self._last: tuple[int, np.ndarray] | None = None    # the last decoded image: c planes of one file share it
+        first = self._image(0)
+        self._page_shape, self._file_dtype = first.shape, first.dtype   # (Y, X) or (Y, X, 3); every file must match
+        if first.ndim == 2:
+            if samples is not None:
+                raise ValueError(f"{self._name(0)} is a grey JPEG with no sample axis; drop params.samples")
+            self._sel: list[int] | None = None
+        else:
+            self._sel = [SAMPLES.index(ch) for ch in samples or SAMPLES]
+        n, nc = len(files), len(self._sel or [0])
+        self.axes = ("z" if n > 1 else "") + ("c" if nc > 1 else "") + "yx"
+        self.shape = ((n,) if n > 1 else ()) + ((nc,) if nc > 1 else ()) + first.shape[:2]
+        self.dtype = np.dtype(first.dtype)
+        self.voxel_size_nm = None       # a JPEG's JFIF density is a print size, not a pixel size
+        self.access = "plane"           # one file per plane: any y/x region costs decoding that whole image
+
+    def read(self, region: dict[str, int | slice]) -> np.ndarray:
+        if "z" not in self.axes:
+            return self._cut(self._image(0), region)
+        z = region.get("z", slice(None))
+        if not isinstance(z, slice):
+            return self._cut(self._image(range(len(self._files))[z]), region)   # IndexError like numpy
+        planes = [self._cut(self._image(k), region) for k in range(len(self._files))[z]]
+        if planes:
+            return np.stack(planes)
+        blank = self._cut(np.empty(self._page_shape, self._file_dtype), region)   # an empty z range, as numpy
+        return np.empty((0, *blank.shape), blank.dtype)
+
+    def _cut(self, img: np.ndarray, region: dict[str, int | slice]) -> np.ndarray:
+        """The c, y, x part of `region` out of one decoded image (Y, X) or (Y, X, 3), as ([c,] y, x)."""
+        y, x = region.get("y", slice(None)), region.get("x", slice(None))
+        if self._sel is None:
+            return img[y, x]
+        if "c" not in self.axes:
+            return img[:, :, self._sel[0]][y, x]
+        c = region.get("c", slice(None))
+        if isinstance(c, slice):
+            return np.moveaxis(img[:, :, self._sel[c]], 2, 0)[:, y, x]
+        return img[:, :, self._sel[c]][y, x]
+
+    def _name(self, k: int) -> str:
+        return f"{self._files[k][1]}::{self._files[k][2]}"
+
+    def _image(self, k: int) -> np.ndarray:
+        """The decoded image of file k, (Y, X) or (Y, X, 3), checked against file 0."""
+        if self._last is not None and self._last[0] == k:
+            return self._last[1]
+        data = self._files[k][0].read(self._files[k][2])    # zipfile inflates the member and checks its CRC
+        if data[:2] != b"\xff\xd8":
+            raise ValueError(f"{self._name(k)} is not a JPEG (starts {data[:4]!r})")
+        a = np.asarray(imagecodecs.jpeg_decode(data))
+        if a.ndim not in (2, 3) or (a.ndim == 3 and a.shape[2] != 3):
+            raise ValueError(f"{self._name(k)}: decoded as {a.dtype} {a.shape}; this snippet reads grey (Y, X) "
+                             "and three-sample (Y, X, 3) JPEGs only")
+        if k > 0 and (a.shape != self._page_shape or a.dtype != self._file_dtype):
+            raise ValueError(f"{self._name(k)}: {a.dtype} {a.shape} differs from the first file's "
+                             f"{self._file_dtype} {self._page_shape}; all files of one array must agree")
+        self._last = (k, a)
+        return a
+
+
+def _open_zip(url: str, io: IO) -> ZipFile:
+    """The zip at `url`, read by range when it can be (a plain URL, a stored member of an outer zip), else staged
+    whole (a deflated member of an outer zip, a server without Range)."""
+    try:
+        return ZipFile(io.open(url))
+    except NeedsFetch:
+        return ZipFile(io.fetch(url))
+
+
+def open_array(src: Source, io: IO) -> Array:
+    glob, samples = src.params.get("glob"), src.params.get("samples")
+    if not isinstance(glob, str) or not glob:
+        raise ValueError(f"params.glob must be a member name pattern such as 'val/*.jpg', got {glob!r}")
+    if samples is not None and (not isinstance(samples, str) or not samples or len(set(samples)) != len(samples)
+                                or not set(samples) <= set(SAMPLES)):
+        raise ValueError(f"params.samples must be distinct letters of {SAMPLES!r} such as 'rg', got {samples!r}")
+    for url in src.urls:
+        if url.count("::", max(url.find("/", url.find("://") + 3), 0)) > 1:
+            raise ValueError(f"{url}: a url names the zip that holds the images (at most one '::'); the pattern "
+                             "for the images inside it goes in params.glob")
+    urls = [u for url in src.urls for u in io.expand(url)]      # an outer `<zip>::<glob>` over several inner zips
+    if len(set(urls)) != len(urls):
+        raise ValueError("the urls (after glob expansion) name the same zip twice")
+    files: list[tuple[ZipFile, str, str]] = []
+    for url in urls:
+        z = _open_zip(url, io)
+        names = sorted(n for n in z.namelist() if not n.endswith("/") and fnmatchcase(n, glob))
+        if not names:
+            raise ValueError(f"no member of {url} matches {glob!r}")
+        files += [(z, url, n) for n in names]
+    return _Images(files, samples)

snippets/jpeg-zip-glob/snippet.yaml

@@ -0,0 +1,11 @@
+name: jpeg-zip-glob
+spec_version: 1
+summary: JPEG images chosen by glob from the members of a zip that is a plain URL or itself a member of another zip (a nested zip), stacked along z; a colour JPEG's samples (RGB) are the c axis, all of them or the ones `samples` names.
+formats: [jpeg]
+containers: [zip, http]
+params:
+  glob: {type: str, required: true, doc: "fnmatch pattern over the member names of each zip the urls name (`*` also crosses `/`), e.g. 'val/*.jpg'; every match is one z plane, sorted by name. Anchor it at the folder: a bare '*.jpg' also matches macOS sidecars such as '__MACOSX/val/._a.jpg', which are not JPEGs and raise at read"}
+  samples: {type: str, required: false, doc: "which samples of a colour JPEG to expose, as distinct letters of 'rgb' in the order they become the c axis, e.g. 'rg' when two channels were saved as red and green and blue is empty; one letter gives a single plane with no c axis. Default: all three as c. Not allowed for a grey JPEG; reason required in binding"}
+requires: [numpy, imagecodecs]
+tested_on: [4tu-60cdce27-liposome-yolo-boxes-confocal]
+supersedes: null