Repository navigation
P1: Implement distributed fleet worker layer #935
Description
Summary
Implement the distributed fleet-worker layer for Loom. The current Loom implementation has an internal worker/arbiter consensus abstraction, but that is not the distributed fleet worker system required by the target FlossWare architecture.
This is an implementation requirement, not merely documentation or a future concept.
Target architecture
agent-setup
-> model-router
-> Loom
-> Worker 1
-> Worker 2
-> Worker N
|
v
Arbiter
|
v
Result
Workers must be independently runnable across the FlossWare fleet and communicate through explicit contracts. Loom should coordinate work without absorbing machine/provider-specific setup logic.
Required capabilities
- Worker lifecycle: start, stop, health, graceful shutdown.
- Worker registration with stable worker identity.
- Capability advertisement so the scheduler can select suitable workers.
- Task submission and leasing/claim semantics.
- Heartbeats and liveness detection.
- Explicit task states and result reporting.
- Failure handling, retry/requeue semantics, and lease expiry.
- Idempotency so retries do not duplicate completed work.
- Worker isolation from the Loom coordinator process.
- Correlation/run/task identifiers sufficient for tracing and provenance.
- Secure transport/authentication appropriate to the deployment model.
- Clear separation between orchestration contracts and worker implementation/runtime.
- Tests covering registration, scheduling/lease races, heartbeat expiry, retry, duplicate delivery, worker failure, graceful shutdown, and result persistence.
Integration requirements
- Integrate with
model-routerfor model/provider selection rather than allowing workers to invent provider routing. - Integrate with
agent-setupfor environment/identity configuration rather than duplicating setup logic. - Preserve the canonical chunk contract and existing Loom persistence boundaries.
- Keep the worker protocol provider- and machine-neutral.
- Avoid making a specific queue, database, container runtime, or model provider a hard architectural dependency unless explicitly defined behind a contract.
- Document the local-development worker path and multi-machine deployment path.
- Add a deterministic local dogfood mode that can exercise multiple workers without requiring the production fleet.
- Extend live dogfood qualification so the distributed worker path is actually exercised before this P1 is considered complete.
Non-goals
- Do not reimplement
chunking. - Do not reimplement scraping.
- Do not move model discovery/routing into Loom or workers.
- Do not treat the existing in-process
ConsensusEnginefan-out as completion of this issue.
Acceptance criteria
- Worker protocol/contracts are defined and documented.
- At least two independently executing workers can register with Loom and execute separate leased tasks.
- Worker heartbeat/liveness and lease expiry work deterministically.
- Worker failure results in safe retry/requeue without duplicate successful completion.
- Results are correlated to the originating run/task and persisted through existing boundaries.
- Workers use the model-router/agent-setup integration points instead of ad-hoc configuration.
- Local multi-worker dogfood passes without production infrastructure.
- Live dogfood exercises the distributed worker path.
- Unit/integration tests cover failure and concurrency cases.
- Documentation explicitly distinguishes internal consensus workers from distributed fleet workers.
- No regression to the canonical chunk contract or existing Loom qualification gate.
Priority
P1 architectural implementation. This blocks claiming the target distributed worker/arbiter architecture is implemented.
Related architecture context
The canonical chunk P0 was completed in Loom PR #927. The next implementation phases include model-router integration, agent-setup control-plane binding, distributed worker execution, and prompt-security hardening. This issue tracks the distributed worker implementation so it cannot get lost while those pieces are being implemented independently.
Activity
sfloess commentedon Oct 18, 2026
[Astra] The generic protocol foundation is now merged in FlossWare/loom#46: ADR-0012 and the Distributed Worker Protocol define stable registration, finite leases, fencing generations, heartbeat/expiry, retry/requeue, idempotency, authenticated operations, and result correlation, with conformance fixtures validated in CI. This does not complete #935. The next implementation slice is the Python reference runtime and deterministic multi-worker dogfood, followed by the HTTP/process boundary and live fleet qualification. Keeping this issue open until those acceptance criteria pass.
sfloess commentedon Oct 18, 2026
[Astra] The first Python runtime slice is now in draft PR #76: FlossWare/loom-ai-python#76. It implements the coordinator state machine for capability-based assignment, leases/fencing, heartbeat expiry, retries, and duplicate-result immutability with deterministic tests. This is intentionally not called complete: remote worker agents, durable fleet state, authenticated transport, and multi-process/live qualification remain next.
sfloess commentedon Oct 18, 2026
[Astra] Follow-on PR #77 is now merged. It adds an authenticated HTTP binding for registration, heartbeat/drain, task submission/claim, lease renewal, result reporting, and observation, plus an independently runnable Worker agent that renews leases while executing. CI passed on Python 3.11, 3.12, and 3.13; the security audit passed. #935 remains open because the coordinator is still in-memory and the HTTP agent is not yet wired through an Orchestrator proxy; durable fleet state, cancellation propagation, true multi-process/live fleet qualification, and full REST-to-persisted-result dogfood remain.
sfloess commentedon Oct 18, 2026
[Astra] Follow-up hardening merged in
loom-ai-python#78(f97ac583). Fleet submissions now deduplicate by idempotency key even when the caller omitstask_id, and reject key reuse when the execution, full Intent, required capabilities, retry policy, deadline, or explicit task identity differs. Regression tests cover generated IDs and conflicting payloads. Python 3.11/3.12/3.13 quality gates and the security audit passed. #935 remains open for durable coordinator state, Orchestrator integration, cancellation propagation, and real multi-process/live dogfood qualification.sfloess commentedon Oct 18, 2026
[Astra] Started the durable fleet-state slice with
loom-ai-python#80: a versioned snapshot-store port plus atomic filesystem writes and corruption/concurrency tests. This is intentionally only the storage primitive, not restart recovery. Next is wiring snapshots intoFleetCoordinator, restoring task/result/history state, invalidating pre-restart leases, and testing retry/deadline behavior. #935 remains open until that integration and the live multi-process qualification are complete.sfloess commentedon Oct 18, 2026
[Astra] The fleet snapshot storage foundation is now merged as
loom-ai-python#80(atomic versioned JSON snapshots; explicitly single-coordinator, not a leadership protocol). I opened draftloom-ai-python#81to wire snapshots intoFleetCoordinator: persist domain state, restore terminal/queued tasks and attempt history, mark restored workers unhealthy, and invalidate active leases on restart. Python matrix/security CI is running. Issue #935 remains open until restart semantics, live multi-process qualification, and the real Orchestrator/REST execution path are verified.sfloess commentedon Oct 18, 2026
[Astra] Durable coordinator recovery is still under review in loom-ai-python#81. The latest revision fixes the Ruff formatting failures from earlier CI attempts and adds an explicit test that a snapshot write failure fail-stops subsequent coordinator mutations. The current head is
9233f04a29913d936e3a6976b9139a715a6723a9; its Python matrix/security checks have just queued. I am not treating the earlier failed revisions as validation, and #935 remains open pending green CI, live multi-process qualification, and the real Orchestrator/REST execution path.sfloess commentedon Oct 18, 2026
[Astra] The durable coordinator recovery slice is now merged in loom-ai-python#81, merge commit
f731ee6fe1b6d53a693fae77b0dceb7242f8e606. The final head passed Python 3.11/3.12/3.13 quality gates, SonarCloud, security audit, and GitGuardian. It includes restart recovery, lease-generation fencing, terminal result/history restoration, deadline handling, and a fail-stop persistence failure test. This is still only the durable state foundation: I am continuing with the missing connection from distributed worker protocol into the Orchestrator/REST execution path; multi-process and live dogfood remain acceptance gates.sfloess commentedon Oct 18, 2026
[Astra] I found that
loom-ai-pythonalready has a dedicatedFleetHTTPServer/FleetWorkerAgenttransport and independent-agent tests. I opened then closed draft PR #82 without merging because it duplicated that existing implementation instead of addressing the actual gap. The remaining work is to connect the existing fleet protocol to Orchestrator/REST, not create another HTTP server.sfloess commentedon Oct 18, 2026
[Astra] The Orchestrator bridge is now merged in loom-ai-python#83, merge commit
277a3bb0bf3ece39b00d6718fcac49283b507e4c. The final head passed Python 3.11/3.12/3.13 quality gates, SonarCloud, security audit, and GitGuardian. The CLI can now route REST Intent submissions throughFleetDispatchWorkerto independently running agents using the existing authenticated fleet protocol. Cancellation/deadline/timeout termination clears leases, and the worker HTTP service no longer accepts task submissions outside the Orchestrator. Issue #935 remains open: live multi-host qualification, deployment security/TLS, restart dogfood, and failure/requeue behavior still need end-to-end qualification. The filesystem coordinator is single-leader, not a distributed leadership protocol.sfloess commentedon Oct 19, 2026
[Astra] The standalone fleet Worker process runner is now merged in loom-ai-python#84, squash commit
12b62b057c96e197680810def8defbcbf1d55d6b. CI passed on Python 3.11, 3.12, and 3.13, plus SonarCloud, security audit, and GitGuardian. The process-level integration test now exercises a separately running Worker registering, claiming, executing, and returning a result through REST → Orchestrator → FleetDispatchWorker, with execution state persisted and observable. This qualifies separate processes on one host, not multi-host deployment or multi-leader coordinator safety. Those remain open acceptance gates, along with cancellation/failure/requeue qualification.sfloess commentedon Oct 19, 2026
[Astra] loom-ai-python#86 is merged (
875e2a0ffe9aa5acba5d163583f1f95055c6659f). The process-boundary REST integration now exercises both successful and failed remote Worker outcomes, asserting terminal fleet state and preservation of the failure category. CI passed on Python 3.11/3.12/3.13 plus SonarCloud, security audit, and GitGuardian. This improves failure observability but does not qualify multi-host transport, multi-leader safety, or the full live fleet acceptance set.