New snippet for png/s3: aws-chammi-75-multichannel-pretraining

snippets/png-channels/README.md

@@ -0,0 +1,55 @@
+# png-channels
+
+Reads greyscale PNG files with `imagecodecs` (libpng), from plain HTTP(S) URLs or zip members, where a
+multi-channel image is stored as **one grey PNG per channel**. One URL is a `yx` image; several URLs are the
+channels of one field of view and are stacked as `cyx`. It has no params.
+
+- **URLs.** One array is one field of view.
+  - One URL: `yx`.
+  - Several URLs: `cyx`, `c` = position in the binding's `urls`. List the channel files in channel order
+    (`..._channel-0.png`, `..._channel-1.png`, …) and name the channels, when known, in the binding's
+    `reasons`; a PNG does not say which stain it holds.
+  - Globs are taken: a `<zip url>::<glob>` or `<dir url>/::<glob>` URL is expanded with `io.expand` to the
+    matching files, sorted by name, in place (so `field_ch*.png` is channels 0..9 in order; with 10 or more
+    channels, name sorting puts `ch10` before `ch2`, so list them explicitly). A plain S3 bucket has no
+    directory index to glob over: give explicit URLs.
+  - The same URL twice is refused.
+- **Files.** Each must be an 8- or 16-bit greyscale PNG (IHDR colour type 0), interlaced or not. Refused at
+  open with the colour type and bit depth in the message: RGB, palette and alpha PNGs (these are not one
+  channel per file), and 1/2/4-bit grey (packed samples; not read rather than expanded to 8 bits). All
+  channels of a field must have the first file's height, width and bit depth; one that differs raises at
+  `read`, naming the URL. A grey PNG whose `tRNS` chunk makes the decoder add an alpha sample also raises at
+  `read`.
+- **Access.** Always `file`: a PNG is one deflate stream with row filters, so there are no rows, strips or
+  tiles to read by range. `open_array` reads only the first file's 33-byte header (signature + `IHDR`); over
+  HTTP that is the file's first 1 MiB block, cached by `io`, which for a small PNG is the file. `read`
+  downloads (through `io.open`, so cached blocks are reused) and decodes each channel the region asks for,
+  whole, then slices; the channel decoded last is kept, so several patches of one channel decode once. A
+  deflated zip member, or a server without Range / Content-Length, raises `NeedsFetch` and is staged whole
+  with `io.fetch`.
+- **dtype.** `uint8` or `uint16` (native byte order), the stored sample values unchanged.
+- **Voxel size.** `None`. A PNG's optional `pHYs` chunk is a print resolution, not a pixel size; take the
+  voxel size from the record.
+
+## Known limits
+
+- **One field per array, named URL by URL.** A dataset of millions of per-channel PNGs is not enumerated by
+  this snippet: a binding lists the fields it wants, each as one `raw` entry. Covering such a dataset needs
+  the file list from elsewhere (an S3 `?list-type=2&prefix=` listing, or the dataset's own metadata table).
+- **Only the first file is checked at open.** A missing or mismatched later channel is found when that channel
+  is read (HTTP 404 from `io`, or the shape/dtype error above).
+- The harness draws a crop from one channel of a `cyx` array; which channel is the sampler's choice.
+- No colour, no palette, no alpha, no bit depths below 8, no voxel size, no channel names.
+- Every read of a new channel decodes the whole file, however small the region.
+
+## Example
+
+`examples/aws-chammi-75-multichannel-pretraining.yaml`: CHAMMI-75, about 2.8 million single-channel 8-bit grey
+PNGs pooled from 75 studies in the public S3 bucket `chammi-data`, one PNG per channel of a field, with shapes
+and channel counts that differ by study. The binding reads two fields, the ones the record inspected: an
+idr0001 field with its two channel files (`cyx` 2×1040×1376 uint8) and an hpa0018 field with its four (`cyx`
+4×512×512 uint8). The channel files of each field were found by listing the field's key prefix on
+`chammi-data.s3.amazonaws.com`; the bucket has no directory index, so they are explicit URLs. No labels: the
+set is for self-supervised pre-training. The example binding's `snippets_commit` is the commit before this
+snippet's own (a binding can't name the commit that adds it); a binding written by the worker names a commit
+that contains the snippet.

snippets/png-channels/examples/aws-chammi-75-multichannel-pretraining.yaml

@@ -0,0 +1,26 @@
+record: aws-chammi-75-multichannel-pretraining
+record_commit: 3f75709f1d2778fe86fd9c436b2b5ff948a72c7b
+snippets_commit: c6f6a6d90f446630e6157532eb084e5c9c9650d6
+raw:
+  - snippet: png-channels
+    urls:
+      - "https://chammi-data.s3.amazonaws.com/CHAMMI-75/CHAMMI-75_train/idr0001/idr0001-plate_100A-converted/Meas_01%282013-01-14_11-13-43%29_001001001_series-0_z-9_t-0_channel-0.png"
+      - "https://chammi-data.s3.amazonaws.com/CHAMMI-75/CHAMMI-75_train/idr0001/idr0001-plate_100A-converted/Meas_01%282013-01-14_11-13-43%29_001001001_series-0_z-9_t-0_channel-1.png"
+    params: {}
+  - snippet: png-channels
+    urls:
+      - "https://chammi-data.s3.amazonaws.com/CHAMMI-75/CHAMMI-75_train/hpa0018/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0-ch0.png"
+      - "https://chammi-data.s3.amazonaws.com/CHAMMI-75/CHAMMI-75_train/hpa0018/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0-ch1.png"
+      - "https://chammi-data.s3.amazonaws.com/CHAMMI-75/CHAMMI-75_train/hpa0018/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0-ch2.png"
+      - "https://chammi-data.s3.amazonaws.com/CHAMMI-75/CHAMMI-75_train/hpa0018/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0-ch3.png"
+    params: {}
+labels: []
+reasons:
+  - "png-channels instead of the draft's tiff-zip-member: every raw is a single-channel 8-bit grey PNG (record technical.arrays: format png, axes yx, uint8, deflate), which tifffile refuses ('not a TIFF file: header=b'\\x89PNG''); png-channels decodes PNG and has no params"
+  - "raw 0 is the record's sampled idr0001 field with both of its channel files as cyx 2x1040x1376 instead of the draft's channel-0 alone: the dataset stores one PNG per channel of a field (record path_pattern <field>_channel-<c>.png), and a listing of the field's key prefix on chammi-data.s3.amazonaws.com (?list-type=2&prefix=...001001001_) returns exactly channel-0 and channel-1"
+  - "raw 1 is the record's probed hpa0018 field (00070df0-..., <uuid>-ch<c>.png) with its four channel files ch0..ch3 as cyx 4x512x512, so the binding covers both file layouts and both shapes the record saw (shape_varies: true); the same prefix listing returns exactly ch0..ch3"
+  - "all urls on chammi-data.s3.amazonaws.com, not the s3.us-west-2 host of the record's hpa0018 url: the record notes say the regional host is not on the allowlist and that the global host serves the same keys"
+  - "explicit urls, no glob: the bucket has no HTTP directory index (harness list on the prefix returns 404), so the channels of a field are listed one url each, in channel-number order"
+  - "two fields only: the other ~2.8M PNGs are not named in the record (listing_complete: false) and each field is its own array, since shapes and channel counts differ by study"
+  - "no voxel size and no channel names: neither is in a PNG, and the record sets neither (they differ by source study)"
+  - "no labels bound: record annotations.present is false (self-supervised pre-training set)"

snippets/png-channels/snippet.py

@@ -0,0 +1,86 @@
+"""Grey PNG files at plain URLs (or zip members), one file per channel of one field of view, as `cyx`.
+
+A PNG is one deflate stream, so a file is always read and decoded whole (`access: file`). One url is a `yx`
+image; several urls are the channels of one field, in binding order, stacked as `cyx`. `open_array` reads the
+first file's 33-byte header only (signature + IHDR: width, height, bit depth, colour type); `read` decodes the
+channels it is asked for and checks each against that header.
+"""
+import struct
+
+import imagecodecs
+import numpy as np
+
+from mia_harness.api import IO, Array, NeedsFetch, Source
+
+SPEC_VERSION = 1
+
+SIGNATURE = b"\x89PNG\r\n\x1a\n"
+HEADER = 33                 # signature (8) + IHDR chunk: length (4), type (4), 13 bytes of data, CRC (4)
+GREY = 0                    # IHDR colour type: one sample per pixel, no palette, no alpha
+DTYPES = {8: np.dtype("uint8"), 16: np.dtype("uint16")}     # IHDR bit depth -> dtype, stored values unchanged
+
+
+class _Png:
+    def __init__(self, io: IO, urls: list[str]):
+        self._io, self._urls = io, urls
+        self._plane_shape, self.dtype = _header(_data(io, urls[0], HEADER), urls[0])
+        self.shape: tuple[int, ...] = (len(urls), *self._plane_shape) if len(urls) > 1 else self._plane_shape
+        self.axes = "cyx" if len(urls) > 1 else "yx"
+        self.voxel_size_nm = None       # pHYs, when present, is a print resolution, not a pixel size
+        self.access = "file"            # one deflate stream per file: no rows or tiles to read by range
+        self._last: tuple[int, np.ndarray] | None = None    # the channel decoded last; patches of one file reuse it
+
+    def read(self, region: dict[str, int | slice]) -> np.ndarray:
+        y, x = region.get("y", slice(None)), region.get("x", slice(None))
+        if "c" not in self.axes:
+            return self._plane(0)[y, x]
+        sel = region.get("c", slice(None))
+        if isinstance(sel, slice):
+            picked = [self._plane(k)[y, x] for k in range(self.shape[0])[sel]]
+            return np.stack(picked) if picked else np.empty((0, *self._plane_shape), self.dtype)[:, y, x]
+        return self._plane(range(self.shape[0])[sel])[y, x]     # IndexError when out of range, like numpy
+
+    def _plane(self, k: int) -> np.ndarray:
+        if self._last is not None and self._last[0] == k:
+            return self._last[1]
+        url = self._urls[k]
+        data = _data(self._io, url)
+        shape, dtype = _header(data, url)
+        if shape != self._plane_shape or dtype != self.dtype:
+            raise ValueError(f"{url}: {dtype} {shape} differs from the first file's {self.dtype} "
+                             f"{self._plane_shape}; all channels of one field must agree")
+        plane = np.asarray(imagecodecs.png_decode(data))
+        if plane.shape != shape or plane.dtype != dtype:        # e.g. a tRNS chunk decoded as an alpha sample
+            raise ValueError(f"{url}: decoded as {plane.dtype} {plane.shape}, its header says {dtype} {shape}")
+        self._last = (k, plane)
+        return plane
+
+
+def _data(io: IO, url: str, n: int = -1) -> bytes:
+    """The first `n` bytes of the file, or all of it: by range when the server allows, else staged whole."""
+    try:
+        with io.open(url) as fh:
+            return fh.read(n)
+    except NeedsFetch:      # a deflated zip member, or a server without Range / Content-Length
+        data = io.fetch(url).read_bytes()
+        return data if n < 0 else data[:n]
+
+
+def _header(data: bytes, url: str) -> tuple[tuple[int, int], np.dtype]:
+    """(height, width) and dtype from the signature and IHDR chunk; ValueError for anything but 8/16-bit grey."""
+    if len(data) < HEADER or data[:8] != SIGNATURE or data[12:16] != b"IHDR":
+        raise ValueError(f"{url}: not a PNG file: header={data[:8]!r}")
+    width, height, depth, colour = struct.unpack(">IIBB", data[16:26])
+    if colour != GREY or depth not in DTYPES:
+        raise ValueError(f"{url}: PNG colour type {colour}, bit depth {depth}; this snippet reads greyscale "
+                         "(colour type 0) PNGs of 8 or 16 bits, one channel per file")
+    return (height, width), DTYPES[depth]
+
+
+def open_array(src: Source, io: IO) -> Array:
+    urls = [u for url in src.urls for u in io.expand(url)]
+    if not urls:
+        raise ValueError("png-channels needs at least one url")
+    if len(set(urls)) != len(urls):
+        raise ValueError("the urls name the same file twice")
+    return _Png(io, urls)

snippets/png-channels/snippet.yaml

@@ -0,0 +1,9 @@
+name: png-channels
+spec_version: 1
+summary: Grey 8- or 16-bit PNG files at plain URLs or as zip members, one file per channel of one field of view; one url is yx, several urls (or a glob) are the channels in binding order as cyx. Each file is read and decoded whole.
+formats: [png]
+containers: [http, zip]
+params: {}
+requires: [numpy, imagecodecs]
+tested_on: [aws-chammi-75-multichannel-pretraining]
+supersedes: null