Skip to content

Commit 8115b1d

Browse files
mishushakovclaude
andauthored
Add template debugging tooling and surface Jupyter logs (#288)
* Add template debugging tooling and surface Jupyter logs Send Jupyter's stdout to the systemd journal instead of /dev/null so startup errors (e.g. failed session creation) are visible via `journalctl -u jupyter`. Add a `make debug-template` workflow (build_debug.py + debug_logs.py) that builds the template via the real systemd start path with a timeout ready-gate, then spawns a sandbox and dumps the jupyter and code-interpreter service journals — for diagnosing a server that fails its readiness check. make_template() gains an optional `ready` override to support this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix wait_for_timeout units in build_debug (ms, not s) wait_for_timeout takes milliseconds (min 1000ms), so 60 collapsed to a 1s ready-gate. Use 60_000 for the intended 60s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Harden debug_logs against command/sandbox timeouts Address Cursor Bugbot review on PR #288: - Bump sandbox TTL 180s -> 600s so the full diagnostic sequence (sleep + per-command 60s budgets) can't outlive the sandbox. - Wrap each command in try/except so one slow or failing command no longer aborts the loop and skips the remaining journals/probes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop redundant sleep in debug_logs The start command runs at build time and the resulting state is snapshotted, so a resumed sandbox already has the services running (and their journals populated). No need to wait after create. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Type the make_template ready parameter Annotate `ready` as `ReadyCmd | None` (the type returned by wait_for_url/wait_for_timeout and accepted by set_start_cmd). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add --max-time to debug_logs health probes A port that accepts TCP but never sends an HTTP response (the half-broken state this tool diagnoses) would otherwise hang curl until the 60s command timeout. --max-time 3 makes each probe fail fast. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Route Jupyter stdout to journal only in debug builds Keep production at StandardOutput=null. make_template(debug=True) now applies a systemd drop-in (jupyter-debug.conf) that flips Jupyter's stdout to the journal, and build_debug.py opts in. Production template behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop redundant systemctl daemon-reload from template build systemd lazily loads the freshly-copied units (and their drop-ins) on the first `systemctl start` at end of build, so the explicit daemon-reload was a no-op. Verified the prod build still reaches a healthy /health gate without it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Drop changeset; debug tooling needs no template release The published template artifact is unchanged (jupyter.service matches main, daemon-reload removal yields an identical image, the journal drop-in only ships in debug builds). Remaining changes are the build script, dev-only scripts, and docs — no version bump warranted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent cf9e095 commit 8115b1d

6 files changed

Lines changed: 116 additions & 13 deletions

File tree

‎Makefile‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,6 @@ start-template-server:
33

44
kill-template-server:
55
docker kill $(shell docker ps --filter expose=49999 --format {{.ID}})
6+
7+
debug-template:
8+
cd template && python build_debug.py && python debug_logs.py

‎template/README.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,35 @@ sbx = Sandbox.create(template="code-interpreter-custom")
5555
execution = sbx.run_code("print('Hello, World!')")
5656
print(execution.logs.stdout)
5757
```
58+
59+
## Debugging a server that won't start
60+
61+
The template runs Jupyter and the code-interpreter server as **systemd**
62+
services (`systemd/jupyter.service`, `systemd/code-interpreter.service`). This is
63+
the path CI and production use — note it is *different* from `make
64+
start-template-server`, which runs the Docker `start-up.sh` path. The two can
65+
diverge, so a server that boots fine under Docker may still fail under systemd.
66+
67+
When a build fails its readiness check (`Waiting for template to be ready ...
68+
timed out`), the real cause is in the service journals. To see them:
69+
70+
```
71+
make debug-template
72+
```
73+
74+
This builds a debug template (gated on a fixed timeout instead of `/health`, so
75+
it finalizes even while the server is crash-looping), spawns a sandbox, and
76+
prints `systemctl status` + the full `journalctl` for both services. It needs
77+
`template/.env` with your `E2B_API_KEY` and the deps from `requirements-dev.txt`.
78+
79+
The debug build also applies a systemd drop-in that routes Jupyter's stdout to
80+
the journal (`make_template(debug=True)`). Production builds keep
81+
`StandardOutput=null`, so Jupyter's request/error logs are only captured in the
82+
debug template.
83+
84+
Inside a running sandbox you can also inspect things directly:
85+
86+
```
87+
journalctl -u jupyter -u code-interpreter
88+
systemctl status code-interpreter
89+
```

‎template/build_debug.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
import os
2+
3+
from dotenv import load_dotenv
4+
from e2b import Template, default_build_logger, wait_for_timeout
5+
from template import make_template
6+
7+
load_dotenv()
8+
9+
alias = os.getenv("E2B_DEBUG_TEMPLATE", "code-interpreter-debug")
10+
11+
Template.build(
12+
make_template(
13+
kernels=["python", "javascript"],
14+
ready=wait_for_timeout(60_000),
15+
debug=True,
16+
),
17+
alias=alias,
18+
cpu_count=2,
19+
memory_mb=2048,
20+
on_build_logs=default_build_logger(min_level="debug"),
21+
)
22+
23+
print(f"Built debug template: {alias}")

‎template/debug_logs.py‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
import os
2+
3+
from dotenv import load_dotenv
4+
from e2b import Sandbox
5+
6+
load_dotenv()
7+
8+
alias = os.getenv("E2B_DEBUG_TEMPLATE", "code-interpreter-debug")
9+
10+
sbx = Sandbox.create(template=alias, timeout=600)
11+
print(f"sandbox: {sbx.sandbox_id}")
12+
13+
CMDS = [
14+
"systemctl --no-pager status jupyter || true",
15+
"systemctl --no-pager status code-interpreter || true",
16+
"journalctl --no-pager -u jupyter || true",
17+
"journalctl --no-pager -u code-interpreter || true",
18+
"curl -s --max-time 3 -o /dev/null -w 'jupyter :8888 -> %{http_code}\\n' http://localhost:8888/api/status || true",
19+
"curl -s --max-time 3 -o /dev/null -w 'server :49999 -> %{http_code}\\n' http://localhost:49999/health || true",
20+
]
21+
22+
try:
23+
for cmd in CMDS:
24+
print(f"\n===== $ {cmd} =====")
25+
try:
26+
result = sbx.commands.run(f"sudo bash -lc {cmd!r}", timeout=60)
27+
if result.stdout:
28+
print(result.stdout)
29+
if result.stderr:
30+
print("[stderr]", result.stderr)
31+
except Exception as e:
32+
# Keep going so one slow/failed command doesn't skip the rest.
33+
print(f"[command failed] {e}")
34+
finally:
35+
sbx.kill()
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Debug-only drop-in: route Jupyter's stdout to the journal (the base unit
2+
# sends it to /dev/null) so ServerApp request/error logs are visible via
3+
# `journalctl -u jupyter`. Applied only by `make_template(debug=True)`.
4+
[Service]
5+
StandardOutput=journal

‎template/template.py‎

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
1-
from e2b import Template, wait_for_url
1+
from e2b import ReadyCmd, Template, wait_for_url
22

33

44
def make_template(
55
kernels: list[str] = ["python", "r", "javascript", "bash", "java"],
66
is_docker: bool = False,
7+
ready: ReadyCmd | None = None,
8+
debug: bool = False,
79
):
810
enabled_kernels = set(["python", "javascript"] + kernels)
911
# Start with base template
@@ -104,16 +106,18 @@ def make_template(
104106
)
105107

106108
if not is_docker:
107-
template = (
108-
template.copy(
109-
"systemd/jupyter.service", "/etc/systemd/system/jupyter.service"
110-
)
111-
.copy(
112-
"systemd/code-interpreter.service",
113-
"/etc/systemd/system/code-interpreter.service",
114-
)
115-
.run_cmd("systemctl daemon-reload")
109+
template = template.copy(
110+
"systemd/jupyter.service", "/etc/systemd/system/jupyter.service"
111+
).copy(
112+
"systemd/code-interpreter.service",
113+
"/etc/systemd/system/code-interpreter.service",
116114
)
115+
if debug:
116+
# Drop-in that routes Jupyter's stdout to the journal for debugging.
117+
template = template.copy(
118+
"systemd/jupyter-debug.conf",
119+
"/etc/systemd/system/jupyter.service.d/debug.conf",
120+
)
117121
else:
118122
template = template.copy("start-up.sh", ".jupyter/start-up.sh").run_cmd(
119123
"chmod +x .jupyter/start-up.sh"
@@ -136,6 +140,7 @@ def make_template(
136140
else:
137141
start_cmd = "sudo systemctl start jupyter"
138142

139-
return template.set_start_cmd(
140-
start_cmd, wait_for_url("http://localhost:49999/health")
141-
)
143+
if ready is None:
144+
ready = wait_for_url("http://localhost:49999/health")
145+
146+
return template.set_start_cmd(start_cmd, ready)

0 commit comments

Comments
 (0)