[mia-autodev]

Closes #47: For JaneliaSciComp/mia-workflow-data#226: dataset empiar-11399-dinoflagellate-fib-sem has no snippet that reads it.

Implemented by snippets: Claude Code running in a sandboxed job on the cluster. The agent's own
summary is the first comment.

• Merge: approve this PR, or comment /approve <head sha> (needed when running under a personal token; a
  trusted worker posts it when the output of this head is approved).
  snippets merges once an approval matches the current head and checks are not failing.
• Request changes: submit a review with Request changes, or comment /revise <what to change>.
  Up to 3 revision rounds.
• Stop: close this PR.

[mia-autodev]

snippets: round 1 pushed 773d5d81d827 to snippets/issue-47 · $2.09

I added the snippet tiff-series-http with its example binding for empiar-11399-dinoflagellate-fib-sem, committed as 773d5d8 (not pushed). Smoke on the real data passes; a trial harness verify of four real crops failed only on "crop.json has no licence", because the issue's Record block has no license field.

What changed

One new directory, snippets/tiff-series-http/, and nothing else:

• snippet.py: reads a numbered series of single-page grey TIFFs at plain URLs, one file per z plane, as zyx.
  • The binding gives the first file's URL and params.last, the number in the last file's name. The snippet counts up the last digit run of the file name with the same zero padding.
  • Without last, the URLs are the planes in binding order.
  • open_array reads only the first file's header. Uncompressed contiguous pages are read by row range (access: chunk); anything else is decoded whole.
• snippet.yaml: declares the one param last; requires is numpy, tifffile, imagecodecs.
• examples/empiar-11399-dinoflagellate-fib-sem.yaml: raw is amst_crop_0101.tif with last: 2037, giving 1937×2718×2718 uint8. This is one URL instead of 1,937, which is what made the match agent's draft too large.
• README.md: usage, how last is chosen, and known limits.

No labels are bound. The EMPIAR segmentation is surface meshes, and the voxel masks are N5 at 16 nm on s3.embl.de, which the record says is not allowlisted.

Limit for the reviewer

Only the first file is in the binding's urls, and the harness derives provenance from urls. So crop.json validators and harness stage's prefetch and derived_from cover only that file; the other slices are read by range through io. Full staging provenance needs the harness to expand the series, or a binding that lists every URL (the snippet accepts that too). I put the first real file in urls rather than a URL template because harness verify rejects a crop with no source validators.

How I tested

• Smoke on real data with MIA_HARNESS_ALLOWED_HOSTS=ftp.ebi.ac.uk. I used a cut-down record.yaml (the raw array entry, voxel size and formats), not the full Record block:

  ok: true   shapes.raw: [[1937, 2718, 2718]]   labels: []
  warnings: []   errors: []   bytes_fetched: 1048576
  

• Crop and verify on real data: harness crop --n 4 wrote 4 crops (z = 90, 599, 1029, …). harness verify then failed with the licence error above and no other errors in the output I inspected. The first crop attempt died on a dropped connection from EBI (RemoteProtocolError); the retry succeeded.

• CI checker: ci/check_pr.py --author 'mia-autodev[bot]' --base HEAD exits 0.

• Repo tests: harness/tests and ci pass (243 passed), run with the pinned harness environment's Python because pixi is not on PATH here.

• Snippet behaviour: scratch pytest cases over the harness's local range server pass (5 passed). They cover uncompressed, zlib, big-endian and multi-strip files, several region shapes, explicit URL lists and the error cases. They are not committed, since the PR may only add the snippet directory.

[mia-autodev]

Crops at 773d5d81d827909e6c8f3be0e700c7fd36bb0e6f: passed, 16 of 16 verified (at least 8 needed); the crop job fetched 62914560 bytes.

16 crops

raw	overlay	caption
		i226-e35700127793-b0-000 · z=451 c=None · (2008, 1638)
		i226-e35700127793-b0-001 · z=1211 c=None · (1210, 2316)
		i226-e35700127793-b0-002 · z=1096 c=None · (677, 2131)
		i226-e35700127793-b0-003 · z=302 c=None · (1525, 337)
		i226-e35700127793-b0-004 · z=500 c=None · (522, 526)
		i226-e35700127793-b0-005 · z=1125 c=None · (180, 2340)
		i226-e35700127793-b0-006 · z=1681 c=None · (1847, 825)
		i226-e35700127793-b0-007 · z=1083 c=None · (1775, 2119)
		i226-e35700127793-b0-008 · z=1931 c=None · (2366, 1202)
		i226-e35700127793-b0-009 · z=1584 c=None · (832, 988)
		i226-e35700127793-b0-010 · z=125 c=None · (1386, 1683)
		i226-e35700127793-b0-011 · z=403 c=None · (2314, 1060)
		i226-e35700127793-b0-012 · z=1022 c=None · (1004, 325)
		i226-e35700127793-b0-013 · z=689 c=None · (2183, 1906)
		i226-e35700127793-b0-014 · z=1472 c=None · (483, 752)
		i226-e35700127793-b0-015 · z=769 c=None · (189, 431)

Binding (sha256 3e0f5b6407abb0abfdf16bf7857792fc45ba03197defc396c0e78be414c07d09):

record: empiar-11399-dinoflagellate-fib-sem
record_commit: bd965a656de8f35269c064a77912455b3d4058e7
snippets_commit: 773d5d81d827909e6c8f3be0e700c7fd36bb0e6f
raw:
- snippet: tiff-series-http
  urls:
  - https://ftp.ebi.ac.uk/empiar/world_availability/11399/data/FIBSEM/amst_crop_0101.tif
  params:
    last: 2037
labels: []
reasons:
- 'tiff-series-http with one url and last 2037: the raw is 1,937 single-page TIFFs
  FIBSEM/amst_crop_0101.tif ... amst_crop_2037.tif, one z slice each (record path_pattern
  FIBSEM/amst_crop_[0-9][0-9][0-9][0-9].tif, alignment_notes ''z = file index'', listing_complete:
  true); 2037 - 101 + 1 = 1,937 files, so the numbering has no gaps and the url counts
  up to the whole stack as zyx 1937x2718x2718 uint8 (the shape of the MoBIE raw N5
  in the record notes) without listing 1,937 urls, which made the match agent''s draft
  too large'
- 'no voxel size from the files: the TIFFs carry no calibration (record notes); 8
  nm isotropic is the record''s imaging.voxel_size_nm'
- 'no labels: the EMPIAR labels are surface meshes (EMDB-SFF mesh_list in the .hff,
  STL files in individual_segments.tar.gz), not voxel arrays; the 14 binary organelle
  masks are N5 stores at 16 nm on s3.embl.de, a format this snippet does not read,
  on a host the record says is not allowlisted, and at half the raw''s sampling (969x1359x1359),
  so they cannot pair shape-for-shape with this raw'

[mia-autodev]

/approve 773d5d8