[mia-autodev]

Closes #60: For empiar-13156-hela-stard3-fib-sem-organelles: dataset empiar-13156-hela-stard3-fib-sem-organelles has no snippet that reads it.

Implemented by snippets: Claude Code running in a sandboxed job on the cluster. The agent's own
summary is the first comment.

• Merge: approve this PR, or comment /approve <head sha> (needed when running under a personal token; a
  trusted worker posts it when the output of this head is approved).
  snippets merges once an approval matches the current head and checks are not failing.
• Request changes: submit a review with Request changes, or comment /revise <what to change>.
  Up to 3 revision rounds.
• Stop: close this PR.

[mia-autodev]

snippets: round 1 pushed 12b2e8a83326 to snippets/issue-60 · $3.04

Committed on snippets/issue-60 (not pushed): a new snippet snippets/tiff-series-glob/ that reads the issue's sub-folder TIFF series from a short binding. It passes harness smoke and harness crop --n 4 on the real EMPIAR-13156 data.

What changed

Only the new directory was added: snippet.py, snippet.yaml, README.md and examples/empiar-13156-hela-stard3-fib-sem-organelles.yaml.

• URLs: each is a plain file or a <dir>/::<glob> over one folder's index, so the 600-slice volume in six sub-folders is six URLs.
• Order: files within a glob are sorted by the numbers in their names, and the URLs are concatenated in binding order as zyx. Open fails if the file numbers do not increase along the whole list, so a folder out of place cannot shuffle z silently.
• count param (optional): the number of planes the URLs must stand for; open fails if the listings give another number.
• Reading: the per-file logic is copied from tiff-series-http — single-page grey TIFFs, read by row range when uncompressed.
• Harness provenance: the harness can expand every URL itself, so fetch/stage prefetch the whole series.

The example binding is the draft, corrected: the raw is now all 600 planes of HeLaSTARD3_FA_YA (count: 600), paired with the two binary ROI label stacks through tiff-zip-member.

Testing

• Smoke on real data (MIA_HARNESS_ALLOWED_HOSTS=ftp.ebi.ac.uk, about 7 minutes):

  {"ok": true,
   "shapes": {"raw": [[600, 1092, 2516]], "labels": [[600, 1092, 2516], [600, 1092, 2516]]},
   "warnings": [], "errors": [], "bytes_fetched": 46156408}

• Z order across folders: planes 100 and 599 read through the snippet equal 101-200/image101.tif and 501-600/image600.tif downloaded directly.
• harness crop --n 4: 4 crops made, no errors, 52 MB fetched. The one crop inside EMPIAR's stated ROI has non-empty endosome and mitochondria masks; the three outside it are empty, as expected.
• harness verify: failed, with one error per crop: "crop.json has no licence". The Record block in the issue has no license field, so this comes from my record.yaml, not the snippet.
• Local tests: 5 ad hoc tests through the real loader over the repo's range server pass. They are not committed, because a snippet PR may only add the snippet directory.
• Repo checks: ci/check_pr.py --base main passes and the harness suite passes (225 tests).

Things to know

• record.yaml was trimmed. I used only the record's technical.arrays roles and dtypes, which is all smoke reads, rather than the full Record block.
• The server is slow to connect. No transport errors occurred in my runs, but there are no retries, so the timeouts the match agent saw could recur in the worker's test.
• Still open for a human (unchanged from the match report):
  • HeLaWT's label page count (993 or 994) is not established, so that volume is not bound.
  • The whole-volume masks would collide on label_key with the ROI masks, so only the ROI masks are bound.
  • ER_ROI.tif, Endosomes.tif and Mitochondria.tif have unknown encoding and are left out.
  • Voxel size differs per volume and is not bound.

[mia-autodev]

Review judge: approve

The PR adds a new snippet, tiff-series-glob. It reads a numbered single-page TIFF series spread over several HTTP sub-folders, one glob URL per folder joined in order as zyx, with checks on number order and file count. Its example binding reads the 600-slice HeLaSTARD3_FA_YA raw volume and pairs it with the two binary ROI label stacks, which is what the issue asks for. The code follows the SPEC's interface, import and layout rules, but nothing in the evidence shows a real-data test of the binding.

item	severity	note
snippets/tiff-series-glob/snippet.py	ok	Does what the issue asks: one <dir>/::<glob> URL per sub-folder, files put in numeric order and joined in binding order as zyx, so the 600 FA_YA slices across six folders fit in a six-URL binding.
snippets/tiff-series-glob/snippet.py	ok	Imports only re, typing, numpy, tifffile and mia_harness.api, and all I/O goes through io.open, io.fetch and io.expand. It falls back to io.fetch on NeedsFetch and reads no voxel data at open beyond the first file's header, so it follows SPEC §2.
snippets/tiff-series-glob/snippet.py	ok	The order check (file numbers must increase over the whole list), the duplicate check and the optional count check catch a misordered or missing folder that would otherwise shift z against the 600-page label stacks.
snippets/tiff-series-glob/snippet.py	suspect	_page reads tif.series[0] before checking len(tif.series), so a TIFF with no series raises IndexError instead of the intended ValueError. Minor, and it does not affect the named dataset.
snippets/tiff-series-glob/snippet.py	ok	The row-range read for uncompressed contiguous strips handles int, slice and negative-step y, and re-checks each file, falling back to a whole-page decode when needed.
snippets/tiff-series-glob/snippet.yaml	ok	name matches the directory, the only declared param is count, requires is [numpy, tifffile, imagecodecs], and tested_on matches the one example binding.
snippets/tiff-series-glob/examples/empiar-13156-hela-stard3-fib-sem-organelles.yaml	ok	Binds the FA_YA raw as six folder globs with count 600 and pairs it with the two binary ROI label stacks the record marks {0,1}. It leaves out the masks whose encoding is unknown and the whole-volume masks whose label_key would collide; the reasons cover each decision.
snippets/tiff-series-glob/examples/empiar-13156-hela-stard3-fib-sem-organelles.yaml	suspect	snippets_commit names the base commit, which does not contain tiff-series-glob. The README explains this, but a reviewer should confirm the repo accepts it for new-snippet examples.
diff	ok	Only adds files under the new snippets/tiff-series-glob/ and touches no protected path (SPEC §5).
evidence/ci.json	suspect	CI was not awaited (shadow) and the evidence holds no smoke result for this binding, so neither the real-data read nor tiff-zip-member on the 600-page LZW labels, which failed with transport errors in the issue, is confirmed.

confidence 0.8 · model claude-opus-5-5 · prompt f5f9c39cc98e · policy v1 · $0.29

Shadow mode: this verdict is recorded and compared with the human label; it decides nothing.

[mia-autodev]

/revise the crops at 12b2e8a83326972ba51c16bfcf1cfe00dd1f209a failed; make them pass:

Crops at 12b2e8a83326972ba51c16bfcf1cfe00dd1f209a: failed, 0 of 0 verified (at least 1 needed); the crop job fetched 1176036 bytes.
crop job 155382211 ended EXIT (exit 1): ReadTimeout: timed out

• verify checked no crops

Binding (sha256 9db357d5102db6615db190d33cbed344ed4c064773f4b89ba7de31e596f92124):

record: empiar-13156-hela-stard3-fib-sem-organelles
record_commit: 3f75709f1d2778fe86fd9c436b2b5ff948a72c7b
snippets_commit: 12b2e8a83326972ba51c16bfcf1cfe00dd1f209a
raw:
- snippet: tiff-series-glob
  urls:
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/1-100/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/101-200/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/201-300/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/301-400/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/401-500/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/501-600/::image*.tif
  params:
    count: 600
labels:
- snippet: tiff-zip-member
  urls:
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/Endosomes_ROI.tif
  params: {}
  pairs_with: 0
  label_key: unknown-endosome_lysosome-semantic
  segmentation_type: binary
- snippet: tiff-zip-member
  urls:
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/Mitochondria_ROI.tif
  params: {}
  pairs_with: 0
  label_key: unknown-mitochondria-semantic
  segmentation_type: binary
reasons:
- 'tiff-series-glob with six glob urls, one per sub-folder in series order: the HeLaSTARD3_FA_YA
  raw is 600 single-page uint8 TIFFs image001.tif ... image600.tif, one z slice each
  (record alignment_notes ''z = slice number in the file name''), kept 100 to a folder
  in 1-100/ ... 501-600/ (harness list: all 600 listed by name, no gaps); a directory
  glob does not descend into sub-folders and tiff-series-http''s counter cannot change
  folder, so each folder is one `<dir>/::image*.tif` url and the snippet concatenates
  them as zyx 600x1092x2516'
- 'count 600: the record''s slice count for this volume (600 pages in each label stack,
  600 files in the listing); the snippet refuses to open if the six indexes give another
  number, so the raw cannot silently pair with the 600-page label stacks at a wrong
  z'
- 'only the HeLaSTARD3_FA_YA volume is bound: it is the one the record sampled; HeLaSTARD3_WT
  (810 slices, image_NNN.tif in 4 folders) and HeLaWT (993 slices, ROI_CICGFA_NNN.tif
  in 5 folders, number 154 absent) read the same way, but HeLaWT''s label page count
  (993 or 994) is not established, an open point of the match report'
- 'tiff-zip-member for the labels: each is one multi-page uint8 LZW TIFF at a plain
  URL (record technical.arrays); axes left to the file, no override'
- 'label keys: provenance unknown because annotations.source is mixed; class from
  the record''s classes (endosome/lysosome, mitochondria); kind semantic and segmentation_type
  binary because the record gives encoding binary, values {0,1}, for Endosomes_ROI.tif
  and Mitochondria_ROI.tif. Only the curated-ROI masks are bound: the whole-volume
  masks (X.tif) would take the same label_key for the same raw, and ER_ROI.tif, Endosomes.tif
  and Mitochondria.tif have encoding unknown in the record'
- 'no voxel size bound: record imaging.voxel_size_nm is null (differs per volume)
  and the TIFFs carry no calibration (record notes: ImageJ default of 1 px = 1 micron,
  ignored)'

[mia-autodev]

Round 2 started: agent job 155382219.

[mia-autodev]

snippets: round 2 pushed eec06753ae35 to snippets/issue-60 · $3.50

The crops now get made: a local harness crop on the real data wrote 4 of 4 crops with no errors, where the job at 12b2e8a died before making any. The fix is committed as eec0675 on snippets/issue-60, not pushed.

Why the crop job failed

ftp.ebi.ac.uk took 12–35 s to answer every request when I probed it. The harness's HTTP client gives up after 60 s and never retries. The failed job's 1,176,036 bytes match the six directory indexes plus the first raw header, so it most likely died on the next request, opening a label stack. The labels were bound to tiff-zip-member, whose open reads all 600 IFDs of each 22 MB stack in about 23 separate requests.

What changed (only snippets/tiff-series-glob/)

• Labels go through this snippet too. When the URLs stand for one multi-page file, its pages are the z planes. A stack up to 64 MiB is read whole at open in 8 MiB range requests, so each label costs three GETs.
• Retries. Each step that makes requests is tried up to 4 times on transport errors (timeouts, dropped or cut connections) and HTTP 5xx/429. Everything else, including BudgetExceeded, HostNotAllowed and 404, is raised at once. Because snippets may not import httpx, the errors are recognised by class name.
• Binding. Every snippet: in the example is now tiff-series-glob, with a reason for the label choice.
• params.count now means z planes: files of a series, or pages of a stack.
• snippet.yaml and the README describe the stack layout, the retries and their limits.

Testing

• Smoke on the real data (8m15s):

  ok: true
  shapes: raw [[600, 1092, 2516]], labels [[600, 1092, 2516], [600, 1092, 2516]]
  warnings: [], errors: []
  bytes_fetched: 46156408
  

• Crop on the real data (--n 4, 9m50s): made: 4, errors: [], bytes_fetched: 52320404. I did not run the worker's verify step on these crops.
• Repo tests: 246 passed (harness/tests and ci, run with the pinned environment's pytest rather than pixi run test), and ci/check_pr.py --base c6f6a6d exits 0.
• Snippet test, not committed (CI allows only the four snippet files): a local range server with both layouts, where every request fails once with a timeout and the reads still come back correct.

Caveats

• The record file I used for smoke was a trimmed copy of the issue's Record block (title, imaging, data and the technical.arrays entries smoke compares against), not the full block.
• Retries only help with intermittent failures: four tries per request will not survive the server being down for minutes.
• The job is slow by nature on this server, roughly 8 minutes to open the three arrays and about 40 s per crop candidate. If the crop job has a short wall-time limit, that would be the next thing to hit.
• The four open points in the match report (HeLaWT page count, the two-masks-per-organelle key collision, the unsampled encodings, per-volume voxel size) are unchanged; only the HeLaSTARD3_FA_YA volume is bound.

[mia-autodev]

Review judge: approve

The PR adds a new snippet, tiff-series-glob. It reads a numbered single-page TIFF series spread over several HTTP sub-folders, one directory glob per folder, as one zyx stack, and it also reads a lone multi-page TIFF as a stack. Requests that fail in transit are retried, for the slow EMPIAR server. The example binding pairs the full 600-slice HeLaSTARD3_FA_YA raw volume with its two binary ROI label stacks. This is what the issue asks for, within the spec's layout and import rules; the remaining concerns are partial coverage of the dataset and no CI evidence.

item	severity	note
diff	ok	Adds only files under one new directory, snippets/tiff-series-glob/. No protected path (harness/, ci/, SPEC.md, WITHDRAWN.yaml, .github/) is touched.
snippets/tiff-series-glob/snippet.py	ok	Does what the issue asks: reads a numbered TIFF series spread over sub-folders from a short binding. Each folder is one <dir>/::<glob> URL resolved by io.expand, files are sorted by number, the URLs are joined in binding order, the result is zyx, and an order check rejects folders listed out of order.
snippets/tiff-series-glob/snippet.py	ok	Imports only re, time, typing, numpy, tifffile and mia_harness.api. All I/O goes through io.open, io.fetch and io.expand. open_array has the exact signature SPEC §2 requires, and NeedsFetch falls back to io.fetch.
snippets/tiff-series-glob/snippet.py	suspect	Also reads lone multi-page label stacks and retries transport errors. The issue did not ask for this, but it targets the smoke timeouts the issue reports. Retried attempts still count against the job's byte cap, as the README says.
snippets/tiff-series-glob/snippet.py	suspect	Retries recognise errors by class name ('TransportError', 'HTTPStatusError'). If io wraps client errors in other types, the retries silently never fire. That would not cause a wrong read.
snippets/tiff-series-glob/snippet.yaml	ok	Metadata is complete. The count param is declared, requires matches what sibling TIFF snippets use, and tested_on matches the example binding's record id.
snippets/tiff-series-glob/examples/empiar-13156-hela-stard3-fib-sem-organelles.yaml	ok	Binding is at the path the issue asks for. It binds all 600 FA_YA raw slices (six folder globs, count 600) to the two 600-page binary ROI label stacks from the record, with distinct label_keys and a reason for each decision.
snippets/tiff-series-glob/examples/empiar-13156-hela-stard3-fib-sem-organelles.yaml	suspect	Only the FA_YA volume is bound. The issue's open points (HeLaWT's label page count, colliding label_keys for X.tif and X_ROI.tif, labels of unknown encoding) are left out and the reasons say why. That is reasonable, but coverage is partial.
ci.json	suspect	CI was not awaited (shadow), and the evidence has no smoke or crop result for the binding, so whether it reads the real data is not confirmed here.

confidence 0.75 · model claude-opus-5-5 · prompt f5f9c39cc98e · policy v1 · $0.33

Shadow mode: this verdict is recorded and compared with the human label; it decides nothing.

[mia-autodev]

Crops at eec06753ae35ba6c59290f559bb60e0e2924afaa: passed, 16 of 16 verified (at least 8 needed); the crop job fetched 76419928 bytes.

16 crops

raw	overlay	caption
		i235-e1794582923198-b0-000 · z=499 c=None · (508, 1111)
		i235-e1794582923198-b0-001 · z=34 c=None · (518, 846)
		i235-e1794582923198-b0-002 · z=150 c=None · (21, 60)
		i235-e1794582923198-b0-003 · z=520 c=None · (466, 1770)
		i235-e1794582923198-b0-004 · z=407 c=None · (693, 1059)
		i235-e1794582923198-b0-005 · z=22 c=None · (175, 586)
		i235-e1794582923198-b0-006 · z=230 c=None · (327, 1030)
		i235-e1794582923198-b0-007 · z=176 c=None · (826, 1328)
		i235-e1794582923198-b0-008 · z=586 c=None · (110, 635)
		i235-e1794582923198-b0-009 · z=220 c=None · (150, 344)
		i235-e1794582923198-b0-010 · z=260 c=None · (666, 548)
		i235-e1794582923198-b0-011 · z=408 c=None · (122, 465)
		i235-e1794582923198-b0-012 · z=12 c=None · (487, 118)
		i235-e1794582923198-b0-013 · z=232 c=None · (222, 2056)
		i235-e1794582923198-b0-014 · z=441 c=None · (113, 1694)
		i235-e1794582923198-b0-015 · z=183 c=None · (197, 1289)

Binding (sha256 51ed17b609fe1b89879705c0cd57064ba9685aeb3151c65e929c0a38af8f096a):

record: empiar-13156-hela-stard3-fib-sem-organelles
record_commit: 3f75709f1d2778fe86fd9c436b2b5ff948a72c7b
snippets_commit: eec06753ae35ba6c59290f559bb60e0e2924afaa
raw:
- snippet: tiff-series-glob
  urls:
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/1-100/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/101-200/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/201-300/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/301-400/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/401-500/::image*.tif
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/501-600/::image*.tif
  params:
    count: 600
labels:
- snippet: tiff-series-glob
  urls:
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/Endosomes_ROI.tif
  params: {}
  pairs_with: 0
  label_key: unknown-endosome_lysosome-semantic
  segmentation_type: binary
- snippet: tiff-series-glob
  urls:
  - https://ftp.ebi.ac.uk/empiar/world_availability/13156/data/HeLaSTARD3_FA_YA_Raw/Mitochondria_ROI.tif
  params: {}
  pairs_with: 0
  label_key: unknown-mitochondria-semantic
  segmentation_type: binary
reasons:
- 'tiff-series-glob with six glob urls, one per sub-folder in series order: the HeLaSTARD3_FA_YA
  raw is 600 single-page uint8 TIFFs image001.tif ... image600.tif, one z slice each
  (record alignment_notes ''z = slice number in the file name''), kept 100 to a folder
  in 1-100/ ... 501-600/ (harness list: all 600 listed by name, no gaps); a directory
  glob does not descend into sub-folders and tiff-series-http''s counter cannot change
  folder, so each folder is one `<dir>/::image*.tif` url and the snippet concatenates
  them as zyx 600x1092x2516'
- 'count 600: the record''s slice count for this volume (600 pages in each label stack,
  600 files in the listing); the snippet refuses to open if the six indexes give another
  number, so the raw cannot silently pair with the 600-page label stacks at a wrong
  z'
- 'only the HeLaSTARD3_FA_YA volume is bound: it is the one the record sampled; HeLaSTARD3_WT
  (810 slices, image_NNN.tif in 4 folders) and HeLaWT (993 slices, ROI_CICGFA_NNN.tif
  in 5 folders, number 154 absent) read the same way, but HeLaWT''s label page count
  (993 or 994) is not established, an open point of the match report'
- 'tiff-series-glob for the labels too, one url each and no params: each is one plain
  multi-page uint8 LZW TIFF at a plain URL (record technical.arrays), which the snippet
  reads as zyx with one page per z plane (record notes: the label page axis is z and
  the page count equals the raw slice count). tiff-zip-member reads the same files,
  but its open walks the 600 IFDs spread through each 22 MB file in about 23 separate
  1 MiB requests with no retry, and ftp.ebi.ac.uk takes 12-35 s to answer each request
  and drops some (the match report''s three smoke runs and the first crop job ended
  in ReadTimeout / RemoteProtocolError); this snippet reads such a file whole in three
  8 MiB requests and retries a request that fails in transit'
- 'label keys: provenance unknown because annotations.source is mixed; class from
  the record''s classes (endosome/lysosome, mitochondria); kind semantic and segmentation_type
  binary because the record gives encoding binary, values {0,1}, for Endosomes_ROI.tif
  and Mitochondria_ROI.tif. Only the curated-ROI masks are bound: the whole-volume
  masks (X.tif) would take the same label_key for the same raw, and ER_ROI.tif, Endosomes.tif
  and Mitochondria.tif have encoding unknown in the record'
- 'no voxel size bound: record imaging.voxel_size_nm is null (differs per volume)
  and the TIFFs carry no calibration (record notes: ImageJ default of 1 px = 1 micron,
  ignored)'

[mia-autodev]

/approve eec0675