Skip to content

Desktop notification children still become zombies after the observer-hook fix (#935) #1187

Description

@elee1766

Desktop notification children still become zombies after the observer-hook fix (#935)

Summary

The observer-hook fix discussed in #935 does not cover the desktop notification helpers. They still discard std::process::Child immediately after spawning notify-send (Linux) or osascript (macOS).

Rust does not wait for a child when its std::process::Child handle is dropped. Once the notification command exits, it remains a zombie until its parent waits for it or exits. In a long-lived jcode process this accumulates across notifications.

Environment and observed impact

  • Linux x86_64, initially observed while using jcode v0.81.6.
  • The same notification code is still present in upstream master 983b532702d2b9cb7766182d566d39081ad1abcb, inspected on September 6, 2026.
  • Live inspection found dozens of jcode-owned zombies, approximately 40 of them notify-send. Other child-lifecycle paths also contributed, so the overall count should not be attributed entirely to notifications.
  • After applying local child-reaping fixes and reloading the actual daemon and clients, repeated /proc scans found zero jcode-owned zombies. This is a bounded observation, not proof that every possible child-lifecycle leak has been eliminated.
  • Zombie processes consume process-table entries, not active CPU. Accumulation can eventually contribute to process/PID limits. Resource exhaustion was not reproduced in this investigation.

Confirmed code paths

  1. send_desktop_notification_rich, app-core notifications.rs, lines 701–719
  2. send_desktop_notification, setup-hints cli_launch_hints.rs

Both use the following pattern:

let _ = std::process::Command::new("notify-send")
    // arguments and stdio setup
    .spawn();

The macOS osascript branches use the same ownership pattern. Linux was tested. The macOS consequence is inferred from the code and Unix child semantics, not from a macOS runtime test.

Reproduce through the production notification helper

For deterministic regression coverage, use a fake notify-send instead of relying on a desktop session or notification service:

  1. In a temporary directory, create an executable named notify-send:

    #!/bin/sh
    printf '%s\n' "$$" >> "$JCODE_NOTIFY_PID_FILE"
    exit 0
  2. Add a Linux-only test in crates/jcode-app-core/src/notifications.rs that calls send_desktop_notification_rich("reap-test", None, "test", None) five times.

  3. Launch the test executable through Cargo with that temporary directory prepended to PATH, an empty PID file referenced by JCODE_NOTIFY_PID_FILE, and --test-threads=1. Set these variables in the parent shell, rather than modifying process-wide environment variables from a multithreaded Rust test.

  4. Keep the test process alive. First wait with a deadline until all five child PIDs have been recorded. Then poll /proc/<pid>/stat for those exact children.

  5. Before the fix: exited children remain with state Z and the test process as PPID. Fail if any of the recorded children still exists after a bounded wait.

  6. After the fix: all five /proc/<pid> entries disappear while the parent test process remains alive.

Run the equivalent check for the setup-hints helper too.

Important: do not assert only that the initial zombie count is zero, or stop polling as soon as the count is zero. A child may not have exited yet, producing a false pass. Record exact PIDs, wait for the children to be launched, and require their disappearance. Avoid a process-wide waitpid(-1, ...) sweep in the test, which would hide the bug by reaping the leaked children itself.

A local notification regression using five real notifications failed with five zombies when the reaper was removed and passed with the reaper restored. The fake-command/PID approach above is the recommended more deterministic upstream test design, not an assertion that such a test is already in the fork.

Measured deterministic reproduction

On September 6, 2026, an isolated Linux harness compiled the unchanged notification function bodies extracted from upstream 983b53270 and the fork fixes revision 21922c06c. It supplied a fake notify-send, recorded every child's PID, and kept each parent alive during observation:

Source Helper Children launched Persistent zombies Child PIDs disappeared
Upstream app-core rich notification 5 5 0
Upstream setup-hints notification 5 5 0
Fixed app-core rich notification 5 0 5
Fixed setup-hints notification 5 0 5

Upstream children were still Z after a three-second observation window, with the test parent as PPID. Fixed children disappeared before the deadline while the parent remained alive. The harness reaped its intentional upstream zombies only after recording the result and exiting their parent. All recorded child PIDs were gone at cleanup.

Scope: this is a synthetic source-extraction harness, not a full jcode integration test or a macOS test. It demonstrates both notification ownership defects and the before/after behavior without a notification daemon or a live jcode session.

Real-crate public API integration check

Also ran a Cargo integration test against the actual jcode-app-core crate at feature revision a39df8e3f, rather than copied function bodies. Result: 2 passed, 0 failed, with one ignored helper test explicitly invoked as the subprocess entry point by the two parent tests.

The test exercised both exported functions, send_desktop_notification_rich and send_desktop_notification, through the real crate dependency graph:

  • All five subprocesses received the expected --app-name=jcode, title, and body arguments.
  • Both APIs returned while the notifier processes were deliberately held open behind a release-file gate. This verifies nonblocking behavior without relying on a tight timing threshold.
  • After release, all five recorded child PIDs disappeared while their parent test process was still alive.
  • With notify-send absent from the child process's PATH, both public APIs returned without failure.

This crosses the real app-core/platform/subprocess boundary using a controlled executable fixture. It does not claim that a desktop notification was visually displayed, that the setup-hints CLI workflow was integration-tested, or that macOS was tested.

The initial invocation lacked the scratch-directory environment variable and failed in test setup, before launching any notification children. Supplying that variable explicitly produced the successful result above.

Suggested narrow fix

Transfer each successfully spawned std::process::Child to the existing jcode_base::platform::reap_detached helper, or the appropriate equivalent for the setup-hints dependency boundary. Waiting should be off the notification caller's critical path.

The relevant initial patch is available for reference in gfx-labs/jcode commit e74371b7c. That commit also touches other child-lifecycle paths. Please isolate the notification change and its regression tests rather than assuming the whole commit is required.

Related lifecycle paths to investigate separately

Our broader local patch series is on gfx-labs/jcode:fixes. It also contains unrelated test-lock fixes, so it is not a minimal patch for this report.

  • McpClient::drop calls start_kill() without explicitly awaiting termination. Related existing report: Bug: MCP tools registered but fail to execute with "Failed to send request" #1036. Tokio child cleanup differs from std::process::Child, so this requires its own reproducer and should not be treated as identical to the confirmed notification bug.
  • Exec-based daemon/client reloads preserve the parent PID but discard userspace cleanup state. Child shutdown and reaping must complete before exec. Shared and per-session MCP children, power inhibitors, and cancelled background commands warrant coverage.
  • Awaiting an aborted task's JoinHandle establishes that task teardown finished, but does not by itself prove that every Tokio child has been reaped. Cancellation/reload tests should observe exact child PIDs disappearing rather than infer success from task status alone.
  • A startup sweep can recover already-exited children left by earlier versions, but it is not a substitute for correct ownership and waiting. A broad reaper must not steal exit statuses from active child owners.

Acceptance criteria

  • Repeated desktop notifications do not leave persistent zombie children while the parent remains running.
  • Both notification helpers are covered.
  • A deterministic PID-based regression fails on affected upstream code and passes with the fix.
  • Notifications remain nonblocking for the calling thread.
  • Validation uses the newly built code. For end-to-end checks, confirm the long-lived daemon/client binary revision or use an isolated test daemon, rather than accidentally exercising the old installed server.

Standalone reproduction script

Requirements: Linux with /proc, Python 3, Rust, Git, and a local clone of this repository. Save the script below as repro_notifications.py outside tracked source files. The upstream and fixed revisions must be available locally. If needed, fetch the fixes branch without checking it out or merging it:

git fetch https://github.com/gfx-labs/jcode.git fixes
export JCODE_SCRATCH_DIR="$(mktemp -d)"
python3 /path/to/repro_notifications.py

The script prints four result records and the location of results.json. It does not modify repository files or restart jcode. Its temporary binaries and source remain in the scratch directory for inspection.

Reproducer source
#!/usr/bin/env python3
"""Isolated synthetic PID regression using unchanged extracted production helpers."""
import ctypes
import json
import os
from pathlib import Path
import subprocess
import tempfile
import time

BASE = '983b532702d2b9cb7766182d566d39081ad1abcb'
FIXED = '21922c06c811dae0cedb8675575134b0ff24640f'
# Adopt and reap only this harness's descendants when their test parent exits.
libc = ctypes.CDLL(None, use_errno=True)
if libc.prctl(36, 1, 0, 0, 0) != 0:  # PR_SET_CHILD_SUBREAPER
    raise OSError(ctypes.get_errno(), 'prctl subreaper failed')
root = Path(tempfile.mkdtemp(prefix='notification-repro-', dir=os.environ['JCODE_SCRATCH_DIR']))
fixture = root / 'bin'
fixture.mkdir()
notifier = fixture / 'notify-send'
notifier.write_text('#!/bin/sh\nprintf \'%s\\n\' "$$" >> "$JCODE_NOTIFY_PID_FILE"\nexit 0\n')
notifier.chmod(0o700)

def source(rev, path):
    return subprocess.check_output(['git', 'show', f'{rev}:{path}'], text=True)

def function(text, signature):
    start = text.index(signature)
    end = text.index('\n}', start) + 2
    return text[start:end]

def state(pid):
    try:
        text = Path(f'/proc/{pid}/stat').read_text()
        return text[text.rfind(')') + 2:].split()[:2]
    except FileNotFoundError:
        return None

results = []
for label, rev in [('upstream', BASE), ('fixed', FIXED)]:
    for helper in ['app_core', 'setup_hints']:
        name = f'{label}_{helper}'
        if helper == 'app_core':
            code = function(source(rev, 'crates/jcode-app-core/src/notifications.rs'), 'pub fn send_desktop_notification_rich(')
            platform = function(source(rev, 'crates/jcode-base/src/platform.rs'), 'pub fn reap_detached(')
            code = 'extern crate self as jcode_base;\npub mod platform {\n' + platform + '\n}\n' + code
            call = 'send_desktop_notification_rich("reap-test", None, "test", None);'
        else:
            text = source(rev, 'crates/jcode-setup-hints/src/cli_launch_hints.rs')
            code = function(text, 'fn send_desktop_notification(')
            if label == 'fixed':
                code = function(text, 'fn reap_detached(') + '\n' + code
            call = 'send_desktop_notification("reap-test", "test");'
        code += '\nfn main() { for _ in 0..5 { ' + call + ' } let mut s = String::new(); std::io::stdin().read_line(&mut s).unwrap(); }\n'
        src = root / f'{name}.rs'
        src.write_text(code)
        binary = root / name
        subprocess.run(['rustc', '--edition=2024', str(src), '-o', str(binary)], check=True)
        pidfile = root / f'{name}.pids'
        pidfile.write_text('')
        env = dict(os.environ, PATH=str(fixture) + os.pathsep + os.environ['PATH'], JCODE_NOTIFY_PID_FILE=str(pidfile))
        child = subprocess.Popen([str(binary)], stdin=subprocess.PIPE, text=True, env=env)
        pids = []
        try:
            deadline = time.monotonic() + 5
            while time.monotonic() < deadline:
                pids = [int(p) for p in pidfile.read_text().splitlines()]
                if len(pids) == 5:
                    break
                time.sleep(.02)
            assert len(pids) == 5, (name, pids)
            deadline = time.monotonic() + 3
            while time.monotonic() < deadline:
                states = [state(pid) for pid in pids]
                if label == 'fixed' and all(s is None for s in states):
                    break
                time.sleep(.02)
            states = [state(pid) for pid in pids]
            assert child.poll() is None, 'Parent exited before observation'
            if label == 'upstream':
                assert all(s == ['Z', str(child.pid)] for s in states), states
            else:
                assert all(s is None for s in states), states
            result = dict(case=name, children=5, zombies=sum(s is not None and s[0] == 'Z' for s in states), disappeared=sum(s is None for s in states), parent_alive=True)
            results.append(result)
            print(json.dumps(result), flush=True)
        finally:
            child.communicate('\n', timeout=5)
            # The upstream case intentionally creates zombies. Reap them here,
            # after the observation and test-parent exit, never during the test.
            for pid in pids:
                try:
                    os.waitpid(pid, 0)
                except ChildProcessError:
                    pass
            assert all(state(pid) is None for pid in pids), 'Reproducer left a child behind'

output = dict(upstream=BASE, fixed=FIXED, scope='synthetic harness, unchanged extracted functions, Linux only', results=results, cleanup='all recorded child PIDs gone')
(root / 'results.json').write_text(json.dumps(output, indent=2) + '\n')
print(f'ARTIFACT={root}/results.json')

Cargo integration test source

Save the following as crates/jcode-app-core/tests/notification_issue_acceptance.rs in a test checkout. Then run:

export JCODE_SCRATCH_DIR="$(mktemp -d)"
cargo test --profile selfdev -p jcode-app-core --test notification_issue_acceptance -- --nocapture

The notifier fixture and outputs are created under that scratch directory. No process-wide environment changes are made by the multithreaded Rust test process.

Real-crate integration test
#![cfg(target_os = "linux")]

use std::io::Write;
use std::path::PathBuf;
use std::process::{Child, Command, Stdio};
use std::time::{Duration, Instant};

#[test]
#[ignore = "subprocess entry point for notification API acceptance tests"]
fn notification_api_child() {
    let dir = PathBuf::from(std::env::var_os("JCODE_NOTIFICATION_ACCEPTANCE_DIR").unwrap());
    for _ in 0..3 {
        jcode_app_core::notifications::send_desktop_notification_rich(
            "acceptance title", None, "acceptance body", None,
        );
    }
    for _ in 0..2 {
        jcode_app_core::notifications::send_desktop_notification("acceptance title", "acceptance body");
    }
    std::fs::write(dir.join("ready"), "returned").unwrap();
    let mut line = String::new();
    std::io::stdin().read_line(&mut line).unwrap();
}

struct Probe { child: Child, dir: PathBuf }
impl Drop for Probe {
    fn drop(&mut self) {
        let _ = std::fs::write(self.dir.join("release"), "release");
        if let Some(mut stdin) = self.child.stdin.take() { let _ = stdin.write_all(b"done\n"); }
        let _ = self.child.wait();
    }
}

fn wait_for(mut condition: impl FnMut() -> bool, description: &str) {
    let deadline = Instant::now() + Duration::from_secs(10);
    while !condition() {
        assert!(Instant::now() < deadline, "timed out: {description}");
        std::thread::sleep(Duration::from_millis(20));
    }
}

fn probe(mode: &str) -> Probe {
    use std::os::unix::fs::PermissionsExt;
    let root = PathBuf::from(std::env::var_os("JCODE_SCRATCH_DIR").unwrap());
    let dir = root.join(format!("notification-api-{}-{mode}", std::process::id()));
    std::fs::create_dir(&dir).unwrap();
    let bin = dir.join("bin");
    std::fs::create_dir(&bin).unwrap();
    if mode == "present" {
        let script = bin.join("notify-send");
        std::fs::write(&script, "#!/bin/sh\nprintf '%s|%s|%s|%s\\n' \"$$\" \"$1\" \"$2\" \"$3\" >> \"$JCODE_NOTIFICATION_ACCEPTANCE_DIR/pids\"\nwhile [ ! -e \"$JCODE_NOTIFICATION_ACCEPTANCE_DIR/release\" ]; do /usr/bin/sleep 0.02; done\n").unwrap();
        std::fs::set_permissions(script, std::fs::Permissions::from_mode(0o700)).unwrap();
    }
    let child = Command::new(std::env::current_exe().unwrap())
        .args(["--exact", "notification_api_child", "--ignored", "--nocapture"])
        .env("PATH", &bin)
        .env("JCODE_NOTIFICATION_ACCEPTANCE_DIR", &dir)
        .stdin(Stdio::piped()).stdout(Stdio::null()).stderr(Stdio::inherit())
        .spawn().unwrap();
    Probe { child, dir }
}

#[test]
fn public_notification_apis_are_nonblocking_and_reap_children() {
    let mut p = probe("present");
    wait_for(|| p.dir.join("ready").exists(), "both public notification APIs return while notifier is gated");
    wait_for(|| std::fs::read_to_string(p.dir.join("pids")).unwrap_or_default().lines().count() == 5, "five real subprocess launches");
    let records = std::fs::read_to_string(p.dir.join("pids")).unwrap();
    let pids: Vec<u32> = records.lines().map(|line| {
        let (pid, args) = line.split_once('|').unwrap();
        assert_eq!(args, "--app-name=jcode|acceptance title|acceptance body");
        pid.parse().unwrap()
    }).collect();
    assert!(pids.iter().all(|pid| PathBuf::from(format!("/proc/{pid}")).exists()));
    assert!(p.child.try_wait().unwrap().is_none());
    std::fs::write(p.dir.join("release"), "release").unwrap();
    wait_for(|| pids.iter().all(|pid| !PathBuf::from(format!("/proc/{pid}")).exists()), "all five children are reaped while their parent remains alive");
    assert!(p.child.try_wait().unwrap().is_none());
    println!("PUBLIC_API_ACCEPTANCE: 5 launches, arguments correct, nonblocking return, 5 child PIDs gone, parent alive");
}

#[test]
fn public_notification_apis_tolerate_missing_notifier() {
    let mut p = probe("missing");
    wait_for(|| p.dir.join("ready").exists(), "public API returns without notify-send installed");
    assert!(!p.dir.join("pids").exists());
    assert!(p.child.try_wait().unwrap().is_none());
    println!("PUBLIC_API_ACCEPTANCE: missing notify-send is nonfatal for both public notification APIs");
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    autonomous: clearHands-off: unambiguous bug, obvious fix, no decisions. Don't even look - an agent can fully solve.bugSomething isn't workingtriage: reproducibleClear repro + clear fix path

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions