Skip to content

Turn snapshots fail with "git commit-tree failed: fatal: <sha> is not a valid object" since 2026-10-12 (0.10.0 after rollback from 0.10.1); old user turns are re-delivered #6978

Open

Description

@SergiuSSoluS

Hello,

I would like to report a defect in CodeWhale 0.10.0. We run it on Windows 11 in daily work, and the instance
in question holds a long-running session (about 99 MB) that we would very much like to keep. The problem has
been with us since 12 October 2026.

Environment

  • CodeWhale 0.10.0 (npm package installed globally). Binary: %APPDATA%\npm\node_modules\codewhale\bin\downloads\codewhale.exe,
    100 331 520 bytes, last written 2026-10-12 15:48.
  • Current version in npm: 0.10.1.
  • Windows 11 IoT Enterprise LTSC 10.0.26100, git 2.55.0.windows.5.
  • Data dir %USERPROFILE%\.codewhale; workspace C:\Users\Public\CodeWhale.
  • Session journal sessions\fd452c83-….json: 98 989 231 bytes (it was ~72 MB on 2026-10-12 and grows).

Symptom 1 — every turn snapshot fails

WARN snapshot: snapshot 'pre-turn:2: <first words of a user message>' failed:
     git commit-tree failed: fatal: 6a55c90c6ada7206170f178fd20032f813a3b3d9 is not a valid object

Symptom 2 — old user turns are re-delivered as new input

Because snapshots never complete, past turns are never marked done and are fed back. The order is stable and
reproducible: in the most recent run the app re-delivered turns 2, 3 and 4 — the same three messages, in the
same order, as in every previous run (logs tui-2026-10-14-20628, -14004, -23528). The user sees his own
old messages answered again.

Symptom 3 — shell operations frequently fail

WARN codewhale_tui::tools::shell: shell work-graph registration skipped;
     running without a bound operation shell_id=shell_<hex>
     error=To-do state is busy; operation was not registered

WARN codewhale_tui::tools::shell: failed to reconcile shell lifecycle shell_id=shell_<hex>
     error=operation binding shell:shell_<hex> is not registered

273 such lines in the logs; on the user's side roughly every 3rd–4th shell call fails. Possibly the same root
cause as the snapshot failure.

Symptom 4 — at startup the store looks empty

WARN snapshot: boot prune failed: git log failed: fatal: bad object HEAD

When it started

  • first occurrence: 2026-10-12T16:00:16Z — snapshot prune failed: commit-tree failed: fatal: c6d19349dd4298bf49945c1e64e4e7b35c4b334c is not a valid object;
  • 458 lines containing “is not a valid object” in total, over four runs;
  • only two lost objects in total: 6a55c90c6ada7206170f178fd20032f813a3b3d9 and c6d19349dd4298bf49945c1e64e4e7b35c4b334c.

What correlates with the date

On the same day the user updated to 0.10.1 and then rolled back to 0.10.0. The package directory is
stamped 2026-10-12 15:48 (the rollback), and the first snapshot failure is at 19:00 local time.
Hypothesis: a snapshot store was begun by 0.10.1, and after the rollback 0.10.0 cannot complete that chain.

What we tried

  • Reinstalled the same version (npm install -g codewhale@0.10.0) — no effect. The store is not part of
    the program files; boot prune failed: bad object HEAD still appears and snapshots still fail with the same SHA.

Where we looked for the store (found nothing)

Workspace C:\Users\Public\CodeWhale (not a git repo; only .codewhale\state with subagent state);
%USERPROFILE%\.codewhale (no work-graph, objects, refs, .git — only sessions\.work-graph-import-archive);
%LOCALAPPDATA% and %APPDATA% (depth 4); %TEMP%; the app package dir (only bin, scripts, test,
package.json, README.md); the app's state.db is empty and stale (2026-10-04).

Questions

  1. Where are the snapshot store and the work-graph kept, and how can a corrupted chain be repaired on the
    user's side?
  2. How can the turn queue / parent-commit pointer be reset so that snapshots work again and old turns stop
    being re-delivered without discarding the session? The session is 99 MB and losing it would be costly.
  3. Is the To-do state is busy shell failure the same root cause as the snapshot failure? Both began in the
    same period.

Related: cannot move to 0.10.1 with a long session

On 2026-10-12 we also tried 0.10.1: it installs cleanly, but resuming our session fails:

error: selected saved history could not be captured
caused by: session exceeds full-history transport bound; source retained

The session file was ~72 MB then and is 98 989 231 bytes now, so this bound will trigger even more readily.
We cannot test 0.10.1 — and therefore cannot check whether it fixes the snapshot store — while it stands.
Is there a way to raise or bypass it?

Materials available on request

Full log excerpts (458 snapshot lines, 273 work-graph lines), the environment dump, and our extended Russian
report. Happy to attach or translate anything you need.

Thank you for the tool, and for taking the time to look at this. I will gladly provide any further logs,
dumps or reproductions you need.

Best regards,
Sergej Malevich

Activity

  1. added
    needs-triageNew external report awaiting maintainer triage; repro, logs and version output help
    on Oct 14, 2026
  2. Hmbown commented on Oct 15, 2026

    @Hmbown
    Collaborator

    🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

    Thank you @SergiuSSoluS for an exceptionally careful report — the exact first-failure timestamp, the two lost object ids and the per-symptom log counts made this traceable. Answers to your three questions, from reading the v0.10.0 and current main source (no build of your environment was possible, so the replay cause below is the most likely one, not a proven one):

    1. Where the snapshot store lives, and repairing it. The per-workspace side repo is %USERPROFILE%\.codewhale\snapshots\<project_hash>\<worktree_hash>\.git — it is only the /restore / undo history, not the session. Its HEAD names a commit that no longer exists, and 0.10.0 has no repair for that. With Codewhale closed, renaming that snapshots directory (or just the one <hash>\<hash> folder for C:\Users\Public\CodeWhale) stops symptoms 1 and 4; you lose only earlier undo points, and the 99 MB session file is untouched. 0.10.1 and later repair this automatically (falling back to the newest surviving reflog commit, else starting a fresh history) and rebuild a stale index.

    2. The re-delivered turns are most likely not caused by the snapshot failure. In 0.10.0 the pre-turn snapshot runs off-thread and its result is discarded; nothing about turn completion reads it. Also, the pre-turn:2 label is a per-launch counter, not the session's turn number. What does match "the same three messages, in the same order, every run" is the session's parked offline queue: at launch Codewhale restores %USERPROFILE%\.codewhale\sessions\checkpoints\<session_id>.offline_queue.json (you should see "Restored 3 queued message(s) from previous session — ↑ to edit, Ctrl+X to discard"), then sends them one by one after your first turn finishes — i.e. as turns 2, 3, 4. That file should be cleared as the queue drains; on your machine that clear apparently never lands. Safe workaround that keeps the session: with Codewhale closed, move just that .offline_queue.json file (not the .lock, not the session .json) to a backup folder; or press Ctrl+X at launch while the "Restored…" notice shows. If you can, please share (a) the contents of that file and whether a leftover checkpoints\offline_queue.json exists, and (b) any log lines containing persistence write failed with offline-queue or clear-offline-queue — those will tell us why the clear fails on Windows. Current main still has the same restore/clear path, so this stays open until we know.

    3. To-do state is busy is a separate shell bookkeeping defect. The matching end-to-end fix is 12fab9f7cb, shipped in 0.10.1: when registration is skipped, the actual spawned command now stays unbound instead of publishing to a nonexistent operation and being killed. Our earlier reference, 983e5cc1e8, was already in 0.10.0 and covered only the guard. The current Windows guard fixture passes; the actual-command fixture is Unix-only, so this is not a reproduction of your Windows session.

    Moving to 0.10.1 with a long session. The session exceeds full-history transport bound refusal is tracked in #6955. The 0.10.2 candidate first archives off-branch journal entries into a sidecar and retries (8131364, 4dfe852), which is often most of a large file; but if the active conversation alone is still over 8 MiB, resume is still refused, and that limit has not been lifted yet. 0.10.2 has not shipped.

    — Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers. Shell-fix attribution corrected by GPT-6 (AI agent), 2026-10-15.

  3. Hmbown commented on Oct 16, 2026

    @Hmbown
    Collaborator

    🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

    @SergiuSSoluS, a follow-up on the re-sent turns, with two corrections to my earlier reply.

    Corrections

    • In 0.10.0 the "Restored N queued message(s)" notice is never shown when a session is resumed, because the "Resumed session: …" status hides it. That is why you never saw it.
    • Ctrl+X does not discard the queue. To discard it inside Codewhale, run /queue clear before sending anything. With Codewhale closed, moving sessions\checkpoints\<session id>.offline_queue.json to a backup folder works. Leave the .lock file and the session .json where they are.

    What we found. We traced the v0.10.0 source further. The three messages come from that parked queue file. Each launch restores the file and sends one message after each turn you finish, which gives turns 2, 3 and 4. As the queue drains, the app asks to save two messages, then one, then to delete the file. If any of those writes had reached disk, the next launch would re-send two, one or none. You get the same three every time, so writes to that one file are most likely failing on your machine. Your session file keeps growing through the same writer, which suggests the failures are specific to this file: for example, a permissions rule on the file or folder, or another program holding it open. In 0.10.0 such a failure is only logged, as persistence write failed what=offline-queue or what=clear-offline-queue, and the log shows only an error category.

    On main (not released yet) in b6ce70b12e:

    • the restored-queue warning is shown on resume too, says the messages will be sent after your next message, and names /queue clear;
    • a failed queue write is logged with the session id and the full OS error, such as os error 32.

    Messages restored from an earlier launch are still sent automatically after your next message.

    What would settle the cause. Any of these would help:

    1. dir /a /t:w "%USERPROFILE%\.codewhale\sessions\checkpoints", run before a launch and again after quitting. We want to know whether the .offline_queue.json file's last-write time changes.
    2. attrib and icacls on that file and its folder. If you have Sysinternals, also handle.exe offline_queue.
    3. Any log lines containing persistence write failed, persistence request dropped, reported write failures during shutdown or panic. Also any file in %USERPROFILE%\.codewhale\crashes\ that mentions persistence-actor.
    4. How you start Codewhale (--continue, a session id, or auto-resume) and how you quit it (/exit, Ctrl+C, or closing the terminal tab).

    A quick check that needs no build: launch, run /queue clear, wait about five seconds, then /exit. If the queue file is still there, the delete is failing on disk, and the log line above will say why.

    One possibly related report: #6986 says pasted multi-line text in Windows Terminal is sent without Enter. In 0.10.0, pasting while a turn runs could queue one message per line, which would be an easy way to end up with exactly three parked messages. #6986's fix shipped in 0.10.1.

    — Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers.

  4. Hmbown commented on Oct 16, 2026

    @Hmbown
    Collaborator

    🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

    @SergiuSSoluS, an update on the re-sent turns.

    59bfc13 (on main) changes what happens to queued messages restored from an earlier launch: they are no longer sent automatically. They wait in the queue until you run /queue send <n> for the ones you want, or /queue clear to discard them. Messages you queue during the current session still send in order as before. The restore notice now says this and is also written to the transcript. Before, a resume's "Resumed session: …" status hid it, which is why you never saw it.

    So even if the queue file on your machine again fails to clear, an old message can no longer be answered again unless you send it yourself.

    How it was checked: a real TUI run with a parked queue of two messages, then codewhale --continue and a new message.

    • 0.10.2 sent the new message and then both old ones, with no action from the user.
    • This change sent only the new message. /queue send 1 sent the first old one, and the second stayed queued.

    Limits:

    • This is not in 0.10.2, which is frozen. It goes into the release after it.
    • It has not been run on Windows or against your 99 MB session.
    • It does not fix the underlying write that fails to clear the file on your machine. It only makes that failure harmless. If you can share the persistence write failed lines with offline-queue/clear-offline-queue that I asked about earlier, we can still look at the write itself.

    — Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers.

  5. Hmbown commented on Oct 17, 2026

    @Hmbown
    Collaborator

    🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

    One more piece is on main now (8f7ee0c, not yet in a release). The startup snapshot prune repairs a broken side-repo HEAD before it prunes. Before this change, if HEAD named a missing commit, the first launch logged boot prune failed: bad object HEAD, and the HEAD was only repaired later, at the first turn snapshot. The repair now runs at boot, using the same lock-safe repair_broken_head that turn snapshots use.

    — Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers.

  6. removed
    needs-triageNew external report awaiting maintainer triage; repro, logs and version output help
    on Oct 17, 2026
  7. SergiuSSoluS commented on Oct 17, 2026

    @SergiuSSoluS
    Author

    Thanks for the pointer to 8f7ee0c. Reporting from an install where the breakage is still live, because I suspect the boot-time HEAD repair may not cover this case.

    Environment: codewhale 0.10.0 (1be1a703b975), Windows. I did not move to 0.10.1: on this machine 0.10.1 fails to start with session exceeds full-history transport bound; source retained (journal around 99 MB).

    Still reproducing as of 2026-10-17 21:21 UTC, unchanged since 2026-10-12:

    • at startup: WARN snapshot: boot prune failed: git log failed: fatal: bad object HEAD;
    • then every snapshot fails: git commit-tree failed: fatal: 6a55c90c6ada7206170f178fd20032f813a3b3d9 is not a valid object — 62 failures in the current run alone, covering pre-turn, tool and post-turn snapshots.

    Two observations that may matter for the fix:

    1. The failing SHA is byte-identical across six days and several sessions.
    2. That SHA appears nowhere in the current session record (about 2.4 MB of JSON), but appears 53 times in an older session record created before the breakage started.

    That looks less like a broken HEAD inside the side repo and more like a stale parent commit persisted in stored state (per-session or lineage metadata). If the boot prune repairs only the repository HEAD, installs in this state will likely keep failing every snapshot after the fix ships.

    Questions:

    • Does 8f7ee0c (or a follow-up) also clear a stored snapshot parent that no longer exists in the side repo, or is HEAD repair the whole fix?
    • Is there a supported way to drop that stale pointer on an affected install, short of starting a fresh session?

    The same runs show the related work-graph symptom: shell work-graph registration skipped ... error=To-do state is busy; operation was not registered, with a retry of the same command succeeding.

  8. Hmbown commented on Oct 17, 2026

    @Hmbown
    Collaborator

    🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

    Thank you @SergiuSSoluS. Checking whether the SHA was stable across sessions was a good test of the stored-state theory. Short answers first, then the evidence.

    1. Is HEAD repair the whole fix? Yes, for the snapshot failures. Each snapshot gets its parent from only one place: the snapshot side repo's HEAD.

    • In 0.10.0 that is git rev-parse --verify HEAD in crates/tui/src/snapshot/repo.rs (around line 465 at the v0.10.0 tag).
    • On main it is the same lookup, run after repair_broken_head.
    • No session, lineage or checkpoint record is read as a snapshot parent.

    Your two observations fit that:

    • The SHA has been identical for six days because HEAD only moves when a snapshot succeeds. Every commit-tree -p 6a55c90c… fails, so HEAD stays on the missing commit forever. Your boot line, boot prune failed: git log failed: fatal: bad object HEAD, is that same HEAD.
    • The 53 copies in the older session record are almost certainly text that session recorded while the commit still existed, or the failure warnings themselves. I couldn't check without the file. No code path reads them back as a parent.

    2. Dropping the stale pointer on 0.10.0 without starting a fresh session. Do this with Codewhale closed. It only touches the undo side repo, never sessions\.

    1. Find the side repo that names the missing commit:
      findstr /s /m 6a55c90c6ada7206170f178fd20032f813a3b3d9 "%USERPROFILE%\.codewhale\snapshots\*"
      
      The hits sit under one snapshots\<project_hash>\<worktree_hash>\.git\ folder, as HEAD, refs\heads\…, packed-refs or logs\….
    2. Then either:
      • rename that <project_hash>\<worktree_hash> folder (for example, add .broken); or
      • keep the folder and drop only the ref: git --git-dir "<that folder>\.git" update-ref -d HEAD.

    The next launch starts a new undo history. You lose only the restore points from before 12 October, which already can't be reached. Your 99 MB session file is not touched. 0.10.1 and later do the same repair automatically, and they first try the newest surviving commit in the reflog.

    3. Why 0.10.1 refuses the session. session exceeds full-history transport bound; source retained is #6955, not this issue.

    4. To-do state is busy; operation was not registered is fixed in 12fab9f. That fix is in the 0.10.2 candidate, but not in 0.10.0 or 0.10.1.

    — Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers.

  9. Hmbown commented on Oct 19, 2026

    @Hmbown
    Collaborator

    🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

    @SergiuSSoluS, two corrections to our earlier replies. Both are about which version carries which fix.

    1. To-do state is busy; operation was not registered. My last reply said the fix, 12fab9f, is not in 0.10.1. That was wrong: it is in the released v0.10.1 (git tag --contains 12fab9f7cb gives v0.10.1). From 0.10.1 on, the "registration skipped" warning can still be logged, but the command keeps running instead of failing with operation binding shell:… is not registered. Your logs come from 0.10.0, which does not have this fix. It is the same fix as Оболочка отказывает, когда занято состояние списка задач #6987 and bug(shell): a busy To-do/Plan lock becomes a fatal spawn failure — "Shell execution failed: operation binding shell:<id> is not registered", while the guard written for exactly this case promises the command still runs #6435.
    2. Held queued messages and the boot-time HEAD repair. We said these would miss 0.10.2. 0.10.2 is cut from current main, so the 0.10.2 candidate now carries both:
      • 59bfc13: restored messages wait for /queue send <n> or /queue clear instead of being sent again;
      • 8f7ee0c: the boot prune repairs a HEAD that names a missing commit.
        It also includes the journal-archive step for over-bound sessions (8131364). 0.10.2 is still not released.

    Two things have not changed:

    — Written by Claude Opus 5.5 (AI agent) on behalf of the maintainers.

  10. SergiuSSoluS commented on Oct 20, 2026

    @SergiuSSoluS
    Author

    Confirming from the affected install, with the lines you asked for.

    Unchanged: codewhale 0.10.0 (1be1a703b975), Windows. Affected session fd452c83-a2d4-4749-9f57-1ee5d035d164; its .json is 111 MB.

    Queue file. sessions\checkpoints\fd452c83-….offline_queue.json is 793 bytes, exactly three messages, and its last write is 2026-10-14T14:04:43 — it has never been touched since it was created. icacls gives my account full control, there is no read-only attribute, and the file reads fine. Next to it sits a 0-byte ….offline_queue.lock, and nine more .offline_queue.lock files from other sessions in the same folder, all 0 bytes.

    No offline-queue write failures are logged at all. Zero matches for offline-queue or clear-offline-queue across all 13 files in logs\. What does appear, for that same session id, is:

    2026-10-13T21:13:12Z WARN persistence: persistence write failed what=checkpoint:fd452c83-… error_kind=PermissionDenied
    2026-10-13T22:13:11Z WARN persistence: persistence write failed what=checkpoint:fd452c83-… error_kind=PermissionDenied
    2026-10-15T21:13:16Z WARN persistence: persistence write failed what=checkpoint:fd452c83-… error_kind=PermissionDenied
    2026-10-15T21:13:30Z WARN persistence: persistence write failed what=session:fd452c83-…    error_kind=PermissionDenied
    2026-10-15T21:19:52Z WARN persistence: persistence write failed what=session:fd452c83-…    error_kind=PermissionDenied
    2026-10-15T22:13:26Z / 22:13:31Z / 22:13:52Z — same shape
    

    Eight lines, 13–15 Oct, PermissionDenied, none of them about the queue. So the writes that fail are the session and checkpoint files of the very session that owns the parked queue.

    Today's log, still on 0.10.0, 21 occurrences, in pairs:

    WARN …shell: shell work-graph registration skipped; running without a bound operation shell_id=shell_… error=To-do state is busy; operation was not registered
    WARN …shell: failed to reconcile shell lifecycle shell_id=shell_… error=operation binding shell:shell_… is not registered
    

    Snapshots. The side repo no longer exists here — I moved %USERPROFILE%\.codewhale\snapshots aside as you suggested — so today's log has no bad object HEAD and no commit-tree failed. That workaround works.

    Two questions:

    1. On Windows, does error_kind=PermissionDenied for what=session/what=checkpoint map to OS error 32 (sharing violation) or 5 (access denied)? ACLs are full control for the account, so a second holder looks more likely — and the same holder could explain why the queue file never gets rewritten.
    2. Since no offline-queue failure is logged at all, could the clear path be skipped rather than failing, with the file re-created on each launch?

    I can run /queue clear on the affected session and report the file state before and after, if that is the check you want.

  11. Hmbown commented on Oct 20, 2026

    @Hmbown
    Collaborator

    🤖 This reply is from Claude Opus 5.5, an AI agent working on behalf of the Codewhale maintainers. It is posted through @Hmbown's account, but Hunter did not write it.

    Thank you @SergiuSSoluS for the ACL, lock-file and log checks, and for confirming the snapshots workaround. Answers are from the 0.10.0 source, not a Windows run.

    1. Error 5 or no OS error, not 32. For file operations, Rust maps only ERROR_ACCESS_DENIED (5) to PermissionDenied; 32 logs as Uncategorized. Error 5 fits a second holder: replacing a file another program holds open without delete sharing fails with 5. Seven of eight at :13 past the hour suggests something scheduled. But 0.10.0 also logs PermissionDenied, with no OS error, when it refuses a save because the session's runtime binding on disk changed, for example after another Codewhale process saved it. The unreleased 0.10.2 candidate logs the full error (b6ce70b12e).

    2. Most likely skipped. A failed write would log what=offline-queue. 0.10.0 re-saves a restored queue at every launch; your unchanged write time means even that never landed, with no failure logged. It sends nothing, silently, when the window's queue lease does not match its session id. The file is not re-created.

    Yes, please: copy the file aside, run /queue clear, wait ten seconds, /exit, then check the file. Does anything touch that session on a schedule?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions