[big-cir]

What is this PR for?

After restarting an interpreter from the setting page there is no way to ask whether it is reachable. The badge on that page is the dependency download status (InterpreterSetting.Status), which the code that starts and stops processes never touches, so it stays green whether the process is gone, stuck, or was never started. The only remaining way to find out is to run a paragraph, and for Spark that means the check itself takes cluster resources.

This adds a probe that a user asks for explicitly:

POST /api/interpreter/setting/{settingId}/healthcheck

{
  "settingId": "2CH2VMY7B",
  "settingName": "spark",
  "groups": [
    { "groupId": "2CH2VMY7B-shared_process", "healthy": true,  "reason": "OK",
      "alive": true, "running": true,  "probeTookMs": 3 },
    { "groupId": "2CH2VMY7B-user2",         "healthy": false, "reason": "NOT_REACHABLE",
      "alive": true, "running": false, "probeTookMs": 1004 },
    { "groupId": "2CH2VMY7B-user3",         "healthy": false, "reason": "PROBE_TIMEOUT",
      "probeTookMs": 3000 }
  ]
}

Review points:

• The deadline has to come from the caller. isRunning() takes no timeout, and its cost is not uniform: local and yarn read memory, docker opens a socket with a hardcoded one second timeout, and the k8s launcher asks the kube-apiserver through a client built with library defaults (10s per request, up to 10 retries with backoff). The thrift path is worse, since the socket is created without a timeout and the call is retried three times. So the probes run on another thread, and the request stops waiting after a budget shared by the whole setting. That budget is a 3 second constant. The number itself is arbitrary, so please suggest a more meaningful one and I will change it.
• Bounding the response does not reclaim the probe. Cancelling a future only interrupts, and neither a blocking socket connect nor an HTTP client call ends on an interrupt, so a probe thread can stay busy after its result was given up on. The pool is therefore limited to 4 daemon threads. Daemon matters because normal shutdown does not call System.exit, so a stuck non daemon thread would delay it.
• It never starts an interpreter. Restart only closes the processes and the next use creates them again, so a setting with nothing running is reported with the reason NOT_RUNNING rather than as a failure. The frontend sub-tasks need that distinction to render something other than a red badge.
• One setting can own many processes. An isolated setting has a group per user or note, so the response carries an entry per group. Probes are submitted together and collected against the one deadline, so an interpreter that does not answer costs the request its budget once instead of delaying the groups that would have answered right away.
• Two places where this differs from the ticket, both deliberate. The ticket names isAlive, but the interface documents isRunning as "the interpreter can communicate with server" while isAlive only says a process exists, so healthy follows isRunning and both values are reported. The ticket also names two outcomes, healthy and unhealthy, which the response keeps as the healthy flag; the reason next to it is what lets a caller tell a setting that was never started from one that timed out. Happy to reduce it if you prefer the literal reading.
• Known limit. For the local and yarn launchers, isRunning() reads in process state, so an interpreter that is up but no longer answering is still reported healthy there. Only docker, k8s, and existing process check from outside. A thrift level ping would answer that properly, but it needs a schema change and has no bound of its own, so it is left out.

Out of scope, on purpose: GET /api/session calls isRunning() per session on a timer and has the same cost problem, but fixing it is a separate change.

What type of PR is it?

Feature

Todos

• - Add InterpreterHealthCheck, the response model, built through factory methods so healthy cannot disagree with reason
• - Add InterpreterHealthChecker, which probes the groups of a setting under one deadline
• - Add the POST /api/interpreter/setting/{settingId}/healthcheck endpoint with the note based authorization that restart already uses
• - Add unit and REST tests
• - Document the endpoint in docs/usage/rest_api/interpreter.md

What is the Jira issue?

• ZEPPELIN-6576

How should this be tested?

Included tests, all passing on current master:

export JAVA_HOME=<your JDK 11 home>
./mvnw package -pl zeppelin-server --am \
  -Dtest=InterpreterHealthCheckerTest,InterpreterRestApiTest -DfailIfNoTests=false

• InterpreterHealthCheckerTest, 7 tests. Covers the deadline, a slow group not hiding a fast one, group order, not probing a launching group, not starting an interpreter to answer, and a single remote call in the healthy case. (7 passed, 0 failed)
• InterpreterRestApiTest, 2 added. A health check of an unused setting returns NOT_RUNNING with no groups and leaves the setting without one, and an unknown setting returns 404. (13 passed, 0 failed)

License check: ./mvnw clean org.apache.rat:apache-rat-plugin:check -Prat -pl zeppelin-server reports 0 unapproved.

Manually, on a running server:

curl -u <admin> -X POST http://localhost:8080/api/interpreter/setting/<settingId>/healthcheck

Right after a restart this answers NOT_RUNNING without starting anything. Run a paragraph first and the same call reports the group as healthy.

Screenshots (if appropriate)

N/A, REST only.

Questions:

• Does the license files need to update? No. The four new files carry the ASF header and the RAT check reports no unapproved files.
• Is there breaking changes for older versions? No. The endpoint is new and nothing existing changes behavior.
• Does this needs documentation? Yes, and it is included in docs/usage/rest_api/interpreter.md.

[hyunw9]

Following up on the blocking-call concern raised in #5403: the four-thread limit bounds active probes, but the LinkedBlockingQueue is unbounded.
If all four probes ignore interruption, subsequent checks queue and time out without running. Cancelling their futures does not remove those tasks from this executor's queue.
when i reproduced this with the checker implementation, three subsequent requests left three cancelled tasks queued, and the healthy process was never probed.
Could we bound pending work and remove cancelled queued tasks?

[big-cir]

Thanks for reproducing this, you're right.

• The queue is now bounded (64 entries). A probe that doesn't fit is rejected right away with a new reason, PROBE_UNAVAILABLE.
• Cancelled probes are purged from the queue before the request returns.
• A probe that never got a thread is also reported as PROBE_UNAVAILABLE, without probeTookMs. Only a probe that started and ran past the deadline is PROBE_TIMEOUT.

The executor is shared across requests, so while all four threads are held, a healthy interpreter still can't be probed and gets PROBE_UNAVAILABLE. What the change guarantees is that pending work stays bounded and the reason tells this apart from a probe that timed out. Limiting how long a thread is held would need timeouts on the launcher calls themselves (e.g. the Kubernetes API request), so I'd leave that for a follow-up.

[hyunw9]

Thanks for documenting the local/YARN limitation in the PR description. The same distinction appears relevant to Kubernetes: isRunning() checks the Pod phase and started, rather than a response from the interpreter.
This means healthy=true / reason=OK can be returned without establishing interpreter reachability, while the API documentation describes the result as whether the interpreter answered.
Could we document the launcher-specific guarantees in the API docs and avoid presenting OK as confirmed reachability?

[big-cir]

Agreed, and it applies beyond Kubernetes: no launcher's check gets a reply from the interpreter service.

• The docs now define healthy / OK as the launcher reporting the interpreter as running, and list per launcher what is actually checked.
• Groups that were checked now carry checkMethod (e.g. KUBERNETES_POD_PHASE, TCP_CONNECT), so a caller can see what OK is based on.

I kept the healthy / OK names to match the ticket, but the docs no longer present OK as confirmed reachability.