New snippet for tiff/http: empiar-11399-dinoflagellate-fib-sem

snippets/tiff-series-http/README.md

@@ -0,0 +1,48 @@
+# tiff-series-http
+
+Reads a z-stack stored as one single-page grey TIFF per slice at plain HTTP(S) URLs and presents it as one
+`zyx` array (`yx` for a single file). What it adds over the other TIFF snippets is the **numbered series**:
+the binding gives the first file and the number of the last one, not one URL per slice, so a stack of
+thousands of files stays a binding of one line.
+
+- **URLs and `params.last`.**
+  - With `last`: exactly one URL, the first file. The counter is the last run of digits in the URL's file
+    name; it is counted up by 1 to `last`, keeping the zero padding when the first number has a leading zero
+    (`amst_crop_0101.tif`, `last: 2037` → `amst_crop_0101.tif` … `amst_crop_2037.tif`). z index = number −
+    first number. Every number in between must exist; take `last` and the absence of gaps from a complete
+    file listing and say so in the binding's `reasons`. A zip member URL (`::`) is refused.
+  - Without `last`: the URLs are the planes, in binding order.
+- **Files.** Each must be a single-series, single-page `YX` TIFF with the first file's shape and dtype; one
+  that differs raises at `read`, naming the URL. RGB tiles belong to `tiff-zip-glob-rgb`, multi-page stacks to
+  `tiff-zip-member`.
+- **Access.** `open_array` reads only the first file's header. `read` opens the files of the planes it is
+  asked for. When the first file's pixels are stored uncompressed in one contiguous run, `access` is `chunk`
+  and a y range is read as that range of rows (a 256-row patch of a 2718×2718 uint8 slice is 0.7 MB, not the
+  7.4 MB file); otherwise `access` is `plane` and the page is decoded whole. The choice is re-made per file,
+  so a compressed file inside a raw series still reads correctly. A server that ignores Range raises
+  `NeedsFetch` and the file is staged whole with `io.fetch`.
+- **dtype.** The file's, in native byte order. **Voxel size:** `None`; plain TIFF resolution tags describe
+  print size, not pixels. Take it from the record.
+
+## Known limits
+
+- **Only the first file is in the binding's `urls`.** The harness derives what it knows about sources from
+  `urls`: a crop's `crop.json` carries validators for the files `io` touched among them (the first file, read
+  at open), and `harness stage` prefetches and records in `derived_from` only that file; the other slices
+  are then read by range through `io` (still counted and host-checked, and their validators are in
+  `io.validators`), and an offline re-stage from the cache cannot reach them. Staging this kind of dataset
+  with full provenance needs the harness to expand the series, as it does a `<zip>::<glob>` URL; until then
+  list every URL (no `last`) when that matters.
+- A missing number inside the series is found only when its plane is read (HTTP 404 from `io`).
+- The series step is 1 and the counter is the last number of the file name; other numbering schemes need
+  explicit URLs.
+- No pyramid levels, no tiles, no colour, no voxel size.
+
+## Example
+
+`examples/empiar-11399-dinoflagellate-fib-sem.yaml`: EMPIAR-11399, a FIB-SEM volume of a dinoflagellate cell,
+1,937 uncompressed 2718×2718 uint8 slices (`amst_crop_0101.tif` … `amst_crop_2037.tif`, 14.3 GB) read as
+`zyx` 1937×2718×2718. No labels are bound: the deposited segmentation is surface meshes, and the voxel masks
+of the MoBIE project are N5 at half resolution on another host. The example binding's `snippets_commit` is
+the commit before this snippet's own (a binding can't name the commit that adds it); a binding written by the
+worker names a commit that contains the snippet.

snippets/tiff-series-http/examples/empiar-11399-dinoflagellate-fib-sem.yaml

@@ -0,0 +1,12 @@
+record: empiar-11399-dinoflagellate-fib-sem
+record_commit: bd965a656de8f35269c064a77912455b3d4058e7
+snippets_commit: 5ca9cacbe772e9af2c63ce829ccfb49088c3d66f
+raw:
+  - snippet: tiff-series-http
+    urls: ["https://ftp.ebi.ac.uk/empiar/world_availability/11399/data/FIBSEM/amst_crop_0101.tif"]
+    params: {last: 2037}
+labels: []
+reasons:
+  - "tiff-series-http with one url and last 2037: the raw is 1,937 single-page TIFFs FIBSEM/amst_crop_0101.tif ... amst_crop_2037.tif, one z slice each (record path_pattern FIBSEM/amst_crop_[0-9][0-9][0-9][0-9].tif, alignment_notes 'z = file index', listing_complete: true); 2037 - 101 + 1 = 1,937 files, so the numbering has no gaps and the url counts up to the whole stack as zyx 1937x2718x2718 uint8 (the shape of the MoBIE raw N5 in the record notes) without listing 1,937 urls, which made the match agent's draft too large"
+  - "no voxel size from the files: the TIFFs carry no calibration (record notes); 8 nm isotropic is the record's imaging.voxel_size_nm"
+  - "no labels: the EMPIAR labels are surface meshes (EMDB-SFF mesh_list in the .hff, STL files in individual_segments.tar.gz), not voxel arrays; the 14 binary organelle masks are N5 stores at 16 nm on s3.embl.de, a format this snippet does not read, on a host the record says is not allowlisted, and at half the raw's sampling (969x1359x1359), so they cannot pair shape-for-shape with this raw"

snippets/tiff-series-http/snippet.py

@@ -0,0 +1,129 @@
+"""A numbered series of single-page TIFF files over HTTP, one file per z plane, stacked as zyx.
+
+The binding names the first file only; `params.last` is the number in the last file's name. The counter is the
+last run of digits in the URL's file name, counted up one file at a time with the same zero padding, so a
+1,937-slice stack is one URL and one integer instead of 1,937 URLs. Without `last`, the urls are the planes,
+in binding order. `open_array` reads the first file's header only; `read` opens the files it needs, and reads
+an uncompressed strip-contiguous page by row range instead of whole.
+"""
+import re
+from typing import BinaryIO
+
+import numpy as np
+import tifffile
+
+from mia_harness.api import IO, Array, NeedsFetch, Source
+
+SPEC_VERSION = 1
+
+COUNTER = re.compile(r"\d+(?=\D*$)")     # the last digit run of the file name
+
+
+class _Series:
+    def __init__(self, io: IO, urls: list[str]):
+        self._io, self._urls = io, urls
+        with self._tiff(0) as tif:
+            page = self._page(tif, 0)
+            self._page_shape, self._file_dtype = tuple(page.shape), np.dtype(page.dtype)
+            contiguous = self._data_offset(page) is not None
+        ny, nx = self._page_shape
+        self.shape: tuple[int, ...] = (len(urls), ny, nx) if len(urls) > 1 else (ny, nx)
+        self.axes = "zyx" if len(urls) > 1 else "yx"
+        self.dtype = self._file_dtype
+        self.voxel_size_nm = None       # plain TIFF resolution tags describe print size, not pixels
+        self.access = "chunk" if contiguous else "plane"    # rows by range, or the whole decoded page
+
+    def read(self, region: dict[str, int | slice]) -> np.ndarray:
+        y, x = region.get("y", slice(None)), region.get("x", slice(None))
+        if "z" not in self.axes:
+            return self._plane(0, y, x)
+        sel = region.get("z", slice(None))
+        if isinstance(sel, slice):
+            return np.stack([self._plane(k, y, x) for k in range(self.shape[0])[sel]])
+        return self._plane(range(self.shape[0])[sel], y, x)     # IndexError when out of range, like numpy
+
+    def _tiff(self, k: int) -> tifffile.TiffFile:
+        url = self._urls[k]
+        try:
+            fh: BinaryIO = self._io.open(url)
+            return tifffile.TiffFile(fh)
+        except NeedsFetch:      # a server without Range: stage the whole file
+            return tifffile.TiffFile(self._io.fetch(url))
+
+    def _page(self, tif: tifffile.TiffFile, k: int) -> tifffile.TiffPage:
+        s = tif.series[0]
+        if len(tif.series) != 1 or len(tif.pages) != 1 or s.axes != "YX":
+            raise ValueError(f"{self._urls[k]}: tifffile axes {s.axes!r} {tuple(s.shape)}, {len(tif.pages)} "
+                             "page(s); this snippet reads one single-page grey YX TIFF per z plane")
+        page = tif.pages[0]
+        if k > 0 and (tuple(page.shape) != self._page_shape or np.dtype(page.dtype) != self._file_dtype):
+            raise ValueError(f"{self._urls[k]}: {page.dtype} {tuple(page.shape)} differs from the first file's "
+                             f"{self._file_dtype} {self._page_shape}; all files of one series must agree")
+        return page
+
+    @staticmethod
+    def _data_offset(page: tifffile.TiffPage) -> int | None:
+        """File offset of the pixel block when it is stored raw in one run (rows can be read by range)."""
+        if page.compression != 1 or page.predictor != 1 or page.fillorder != 1 or page.is_tiled:
+            return None
+        offsets, counts = page.dataoffsets, page.databytecounts      # strips, in row order
+        size = int(np.prod(page.shape)) * np.dtype(page.dtype).itemsize
+        if not offsets or len(offsets) != len(counts) or sum(counts) != size:
+            return None
+        if any(o + c != nxt for o, c, nxt in zip(offsets, counts, offsets[1:])):
+            return None
+        return int(offsets[0])
+
+    def _plane(self, k: int, y: int | slice, x: int | slice) -> np.ndarray:
+        with self._tiff(k) as tif:
+            page = self._page(tif, k)
+            offset = self._data_offset(page)
+            if offset is None:
+                return np.asarray(page.asarray())[y, x]
+            ny, nx = self._page_shape
+            rows = range(ny)[y] if isinstance(y, slice) else range(range(ny)[y], range(ny)[y] + 1)
+            if len(rows) == 0:
+                return np.empty((ny, nx), self.dtype)[y, x]
+            lo, hi = min(rows[0], rows[-1]), max(rows[0], rows[-1]) + 1
+            stored = np.dtype(page.dtype).newbyteorder(tif.byteorder)
+            fh = tif.filehandle
+            fh.seek(offset + lo * nx * stored.itemsize)
+            buf = fh.read((hi - lo) * nx * stored.itemsize)
+        if len(buf) != (hi - lo) * nx * stored.itemsize:
+            raise ValueError(f"{self._urls[k]}: file ends inside the pixel data")
+        block = np.frombuffer(buf, stored).reshape(hi - lo, nx).astype(self.dtype, copy=False)
+        if not isinstance(y, slice):
+            return block[0, x]
+        picked = block if rows.step == 1 else block[np.arange(rows.start, rows.stop, rows.step) - lo]
+        return picked[:, x]
+
+
+def _numbered(url: str, last: int) -> list[str]:
+    """`url` and its successors up to counter `last`: the last digit run of the file name, same zero padding."""
+    name_at = url.rfind("/") + 1
+    m = COUNTER.search(url, name_at)
+    if "::" in url[url.find("://") + 3:] or m is None:
+        raise ValueError(f"params.last needs a plain file url whose name holds a number, got {url!r}")
+    first, width = int(m.group()), len(m.group())
+    if last < first:
+        raise ValueError(f"params.last {last} is below the first file's number {first}")
+    if len(str(last)) > width and m.group().startswith("0"):
+        raise ValueError(f"params.last {last} does not fit the zero-padded width {width} of {url!r}")
+    pad = width if m.group().startswith("0") else 0
+    return [f"{url[:m.start()]}{n:0{pad}d}{url[m.end():]}" for n in range(first, last + 1)]
+
+
+def open_array(src: Source, io: IO) -> Array:
+    last = src.params.get("last")
+    urls = list(src.urls)
+    if not urls:
+        raise ValueError("tiff-series-http needs at least one url")
+    if last is not None:
+        if isinstance(last, bool) or not isinstance(last, int):
+            raise ValueError(f"params.last must be an integer, got {last!r}")
+        if len(urls) != 1:
+            raise ValueError(f"params.last counts up from one url (the first file), got {len(urls)} urls")
+        urls = _numbered(urls[0], last)
+    if len(set(urls)) != len(urls):
+        raise ValueError("the urls name the same file twice")
+    return _Series(io, urls)

snippets/tiff-series-http/snippet.yaml

@@ -0,0 +1,10 @@
+name: tiff-series-http
+spec_version: 1
+summary: A numbered series of single-page grey TIFF files at plain URLs, one file per z plane, stacked as zyx; the binding names the first file and the last file's number instead of every URL.
+formats: [tiff]
+containers: [http]
+params:
+  last: {type: int, required: false, doc: "the number in the last file's name. With it the binding gives exactly one url, the first file; the counter is the last run of digits in that url's file name and is counted up by 1 to `last` with the same zero padding (amst_crop_0101.tif, last 2037 -> amst_crop_0101.tif ... amst_crop_2037.tif, z = number - first). Every number in between must exist as a file. Without it the urls are the planes, in binding order; reason required in binding"}
+requires: [numpy, tifffile, imagecodecs]
+tested_on: [empiar-11399-dinoflagellate-fib-sem]
+supersedes: null