Analysis: generic OJP messaging protocol over the JDBC driver (server↔server, server→client)

[Copilot]

OJP needs a generic messaging substrate to support RAFT-style consensus, cache-invalidation broadcasts, and server-restart notices, without opening any new direct connection between OJP servers — all transport must reuse the existing ojp-jdbc-driver. This adds a design-only analysis under documents/analysis/ covering the protocol, delivery modes, and trade-offs.

• Full analysis (OJP_MESSAGING_PROTOCOL_ANALYSIS.md)
  • Envelope/proto sketch: topic + opaque payload + delivery mode (FIRE_AND_FORGET / GUARANTEED)
  • Options considered with pros/cons: piggybacking on StatementService SQL calls, ad-hoc ConnectionDetails fields, a new MessagingService over the driver (recommended), external brokers, direct server links (disallowed)
  • Server-to-server topology: each server acts as a driver-backed client of its peers (mesh), never opening a direct socket
  • Server-to-client push via a client-initiated Subscribe stream, surfaced through SQLWarning
  • Mapping of the three example use cases to topics/delivery modes, plus ordering guarantees (per-producer/topic FIFO only)
  • Concerns/open questions: inter-server auth story (none exists today), mesh scaling limits, guaranteed-delivery durability across publisher crashes, backpressure
  • Suggested phased rollout
• Executive summary (OJP_MESSAGING_PROTOCOL_SUMMARY.md) — condensed version for quick review
• Index update (documents/analysis/README.md) — links the new documents as the latest analysis

Example of the proposed contract shape:

service MessagingService {
  rpc Publish (PublishRequest) returns (PublishAck);
  rpc Subscribe (SubscribeRequest) returns (stream Envelope);
}

message Envelope {
  string topic = 2;
  bytes payload = 3;
  DeliveryMode delivery_mode = 6;
}

No source code changes are included — this is purely a design analysis for future implementation.

[rrobetti]

@copilot evaluate how these ideas would work with serverless architectures where the clients might be all off at some point. We might need a OJP server to server option for those cases?

[Copilot]

@copilot evaluate how these ideas would work with serverless architectures where the clients might be all off at some point. We might need a OJP server to server option for those cases?

Good catch — added §6.1 to the full analysis to address this directly.

Short answer: the server-to-server mesh already proposed in §5.3 is that server-to-server option, and it already works with zero connected clients. Each OJP server embeds the JDBC driver as a client library and connects directly to its peers' MessagingService, driven by server startup/config (peer list), not by application client activity — no client session is involved in that path at all. So RAFT/cache-invalidation keep working through serverless periods where application clients scale to zero, as long as the OJP server processes themselves stay up.

That last clause is the one open question I flagged (§8.8): if there's a deployment model where the OJP servers themselves scale to zero between requests (not just client apps), this mesh design wouldn't cover that case and would need a different, likely externally-coordinated approach. Let me know if that's a real scenario you're targeting.

Commit: 3d7c2b1 (see repo for actual short hash).

[rrobetti]

@copilot explain this better "each OJP server also holds a
small internal pool of driver client instances pointed at its peers "

looks messy, how will you have an embeded jdbc client just do do communication in OJP server? This cannot be enabled by default, by default I want OJP servers to not need to connect to other ojp servers only if enabled they would.

[Copilot]

@copilot explain this better "each OJP server also holds a
small internal pool of driver client instances pointed at its peers "
...

Rewrote that section (64ae18e) — "internal pool of driver client instances" was a poor description, fixed to be concrete:

• It's not the JDBC driver's public API (no DriverManager.getConnection, no java.sql.Connection). It reuses the driver's lower-level gRPC plumbing — GrpcChannelFactory, retry/circuit-breaker logic — which already lives in ojp-grpc-commons and is already shared by both the driver and the server. A new small MessagingServiceGrpcClient wraps a MessagingServiceGrpc stub with that same plumbing, living in ojp-grpc-commons so ojp-server depends on it symmetrically, same as it already depends on other shared classes.
• "Pool" was wrong wording too: it's one long-lived channel + one subscribe stream per configured peer, not a pool.

On the default-off requirement: added a new ojp.server.mesh.enabled setting, defaulting to false, gating the entire feature — mirrors the existing ojp.server.slowQuerySegregation.enabled pattern. When disabled, no peer list is read and no outbound connections happen at all. Peers are configured via a new, dedicated ojp.server.mesh.peers setting rather than reusing serverEndpoints (which is client-populated and empty exactly when clients are all off, which would defeat the point).

[rrobetti]

@copilot add mermaid diagrams to the analysis explaining how the flow of messages will work in both modes. Mesh ON and Mesh OFF.

[Copilot]

@copilot add mermaid diagrams to the analysis explaining how the flow of messages will work in both modes. Mesh ON and Mesh OFF.

Added in ac3b508: new §5.5 in the full analysis with a topology comparison diagram (Mesh OFF vs Mesh ON) plus full sequence diagrams for each mode, covering cache invalidation, RAFT vote messages, and the server-restart notice. Condensed versions of the topology diagrams are also in the executive summary, linking to §5.5 for the detailed flows. All diagrams were validated with mermaid-cli.

[rrobetti]

@copilot "Mesh OFF is only sufficient for the pure server-to-its-own-clients use case (server.lifecycle) or for single-server deployments. This should be documented plainly for operators so nobody assumes cache invalidation is cluster-wide by default when it isn't."

This is not the initial intent, if mesh is OFF it should use the clients as a mean to communicate with other OJP servers, it must be possible to exchange server to server communication even with mesh off, being recommended for situations where opening a connection between ojp servers is difficult or not possible. Also having the limitation where if the clients are not guaranteed to be alive, for example serverless where all clients might be off at a given point then mesh ON is recommended as per is the only way to guarantee that the messages are flowing. Do a comparison of both approaches with pros and cons.

But is it a problem if all clients are off if the OJP servers don't communicate? if no one is using OJP servers what is the down point of them not communicating ? Is it really a problem?

Review the whole analysis.

[Copilot]

@copilot "Mesh OFF is only sufficient for the pure server-to-its-own-clients use case (server.lifecycle) or for single-server deployments. This should be documented plainly for operators so nobody ass...

Reworked in b57ab1a — §5.3 now describes two server-to-server topologies instead of framing "mesh off" as no communication at all:

• Client-relay (default, always on, no config): a multinode-connected client (existing jdbc:ojp[host1:port1,host2:port2]_url support) already holds sessions to more than one server and now forwards cluster_scope=true envelopes it receives from one server onto the others it's connected to — de-duped by message_id, bounded by a new max_relay_hops field. This is exactly "use the clients as a means to communicate," requires zero new connections, and is the recommended choice when a direct server-to-server link is hard or disallowed.
• Direct mesh (opt-in, unchanged mechanics): kept as the guaranteed alternative, now framed as required for RAFT and recommended when client presence can't be assumed (serverless).

Added a full pros/cons comparison in §5.3.3 (and a condensed version in the executive summary), and rewrote the §5.5 diagrams so "Mesh OFF" shows an actual cross-server relay flow instead of "no communication."

On whether idle-period non-communication is actually a problem: mostly no. If zero clients are connected there's no query traffic and nothing depending on cluster state at that instant. The real risk is at the reconnect boundary — if the first request back needs a decision that depends on cluster convergence. For idempotent cases (cache invalidation, RAFT election finishing a beat late) that's a bounded cold-start cost, likely acceptable. It's a real problem only if the protected state can be actively wrong (not just stale) the moment a client reconnects, e.g. two servers both briefly acting as leader — that's the case direct mesh exists for. Full reasoning with confidence levels is in the rewritten §6.1.

Also updated §6 mapping, §8 concerns (relay coverage is probabilistic and invisible on failure; the driver taking on relay duty is a scope-creep concern I flagged with a suggested topic allowlist so RAFT can never leak onto client-relay), §9 phasing, and §10 summary to match, and reviewed the full document for consistency.

[rrobetti]

@copilot address my comments:
1 -
this is no longer correct

"Hard constraint: the transport must reuse the existing ojp-jdbc-driver (and the gRPC channel/session machinery it already implements). OJP servers are not allowed to open a new, separate connection directly to each other (no raw sockets, no second gRPC server-to-server link, no new listening port purely for inter-server chat). Whatever moves bytes between two OJP processes must travel through the same code path a normal JDBC application already uses."

Also this is no longer like that
3.2 Non-functional
Must not require clients or servers to open any new network endpoint.

2-
Create or modify a mermaid diagram showing the types of connections to implement the recommended solution, both for mesh ON and OFF, what GRPC channels are needed, what is the proto interface used is it a stream? does it add new connections? where? when a OJP server needs to broadcast a message to all OJP servers how does it work? when a OJP server needs to send a message to a specific OJP node how it works? when a OJP server needs to send a broadcast message to all clients how it works?

3-
5.1 Envelope you showed only the messages, I want to see the service as well, how it looks like, and then explain how will the pub sub work.

4-
Please expand this part, may need its own section
"as a hard product rule, not a soft recommendation. Confidence: high (85%) on the trust argument, medium (65%) on exactly how bad the coverage risk is in practice, since that depends on real deployment client-connectivity patterns I don't have data on."

5-
what are the concrete risks of using the client relay for RAFT? It would be beneficial to have it obviously, so we need a strong and clear rational to not implement it, it is less secure, less reliable is not enough. RAFT protocol can survive message loss for example so what is the issue exaclty? be honest, do not try to please, use reliable materials.

6 -
on the Mesh OFF — sequence flow (default behavior, client-relay) and Mesh ON — sequence flow (opt-in) you describe examples where clients are connected to a single OJP server, that is not the standard, the standar is that each client is connected to all OJP servers (except under a network partition or other issue), also it is not clear in you ecample if each OJP server receiving a broadcast message also broadcast it further. This cascading broadcast might be expensive and not useful.

7 - in the mesh OFF it is not clear to me if when boradcasting a message to other OJP servers the publisher OJP server sends it to all its clients or to a single client?

Review all the documents in the light of these comments, you need to review multiple times that fact that item 1 "Hard connstraint..." is in my comments means that previously you did not update the full document properly with the new understandings, do not repeat that mistake.

[rrobetti]

@copilot I don't any consern with network cloghing due to fun out approach for client relay approach? I understand the need to fun out to all clients consensus messages bu do we need it for cache invalidations for example, coule we not send to a single client instead? lets say the user configures caching badly and there are a lot of invalidations, besides the side effect of the invalidations themselves if we fan out to all clients we would have a network issue, maybe we are better off sending to a single client, that is why we need a mechanism to chose if it is a fan out to all clients or how many clients we want the message sent to .

[rrobetti]

@copilot in the consensus analysis you need to give reasons on why other Byantine algorithms were considered and why they were dropped.
Among the reasons should be: We do not fully trust the clients, but the clients are the applications and should be secure, is not a level of untrust as if every node was a unknown player in the internet, it is application nodes secured by the same operator, based on that RAFT + encryption came as a reasonable solution. Also Byzantine algorithms would need 4 nodes minimum, increasing small deployment costs and they are not as mature in Java as RAFT. Write it better, add other reasons.

[Copilot]

@copilot in the consensus analysis you need to give reasons on why other Byantine algorithms were considered and why they were dropped.
Among the reasons should be: We do not fully trust the clients, ...

Added in 1f3ba09: new §2.1 "Why not a Byzantine-fault-tolerant algorithm" in OJP_CONSENSUS_ALGORITHM_ANALYSIS.md, consolidating the rejection reasoning into one place, better-written, with your requested points plus a couple more:

1. Trust is narrow, not wide — and encryption closes exactly that gap. Clients aren't anonymous internet hosts; they're the operator's own application instances on the operator's own isolated network. That's a narrow gap (trusted-for-queries vs. trusted-for-consensus), not a wide one (trusted vs. unknown adversary). A single shared AEAD key closes the narrow gap directly — RAFT + encryption is proportionate to the actual risk.
2. Hard node-count cost. BFT needs N = 3f+1, so tolerating even one faulty node needs 4 nodes minimum, vs. 3 for RAFT (2f+1). For small clusters that's a real cost, not a rounding error.
3. Java maturity gap. Apache Ratis is production-grade; three of the four BFT candidates (PBFT, HotStuff, Tendermint) have no mainstream Java implementation at all — OJP would be implementing a paper, not adopting a library.
4. Added, beyond your list: (a) BFT's extra message phases (pre-prepare/prepare/commit, view changes) are more protocol surface to get wrong than RAFT's single round-trip; (b) BFT wouldn't even fix client-relay's actual residual risk — §5.2 shows that risk is a compromised driver build or shared network path, not a malicious validator, and BFT assumes Byzantine behavior from a fixed validator set, which a relaying client was never part of.

Also finished the relay_fanout prose in the messaging analysis (§5.3.1) with a concrete before/after example (800 → 20 relay calls) — that was a loose end from an earlier round, now tied off in the same commit.

[rrobetti]

@copilot
"server-to-server authentication is a separate, currently-unsolved need (§9, item 3)."

We did solve it, did we not? this will be mTLS

[Copilot]

@copilot
"server-to-server authentication is a separate, currently-unsolved need (§9, item 3)."
...

Right, that line was stale — fixed in b1c4813. The approach was already decided (mTLS, per §9 item 3); only the certificate provisioning/rotation process is still open, not the auth mechanism itself. Updated §3.2's non-functional line and the mesh comparison table to say mTLS directly instead of the vaguer "credential story" phrasing.

[rrobetti]

@copilot I don't understand this max_relay_hops, its is always one hop at the moment, goes to client and if needs to go to a server, client sends to a server. Which hops are you talking about here? Explain it better, give a simple example.

[Copilot]

Fixed in 0d780d2. A "hop" = one client re-publishing the same envelope to one other server it's connected to — not a chain of clients. In the standard topology (every client sees every server) one hop already reaches everyone, so nothing more happens. It only matters in a partial-connectivity topology: added a worked example in §5.3.1 with client X bridging servers A/B and client Y bridging B/C — X's hop to B uses up the default budget of 1, so Y (receiving that already-decremented copy on B) won't relay it on to C unless max_relay_hops is raised to 2.

[rrobetti]

@copilot after the table above we need to explain why each mode was chosen for each use case.

[Copilot]

Added in 0d780d2, right after the table in §5.2: separate rationale for consensus (fire-and-forget by default since RAFT already tolerates loss; switches to GUARANTEED only over encrypted client-relay to cover a slow/disconnected relaying client), cache invalidation (fire-and-forget — a missed one is just a stale entry until the next write/TTL), and server.lifecycle (guaranteed — no self-healing fallback if a client misses the notice, and it's a rare, one-off message so the extra cost is worth it).

[rrobetti]

@copilot I want you to analyse an alternative approach where the communication is via a single bi-directional stream and make the message bear the info about what it is and where it is going. Reason about both options, pros and cons. You must include how this might affect performance as well. Still give a suggestion on what to use.

[Copilot]

Added new §5.1.1 in 0d780d2. Compares the current 3-RPC design against a single rpc Communicate(stream Envelope) returns (stream Envelope) where every message carries its own kind (PUBLISH/ACK, with SUBSCRIBE as the first message) and routing info.

Bidi-stream pros: avoids per-call HTTP/2 stream setup for high-frequency traffic, fewer concurrent streams per peer, free per-connection ordering. Cons: head-of-line blocking on the shared stream (a slow Publish can delay an Ack behind it), routing/validation logic has to move from gRPC method dispatch into hand-rolled kind checks, worse per-method observability, more server-side state per connection.

On performance: gRPC already multiplexes all RPCs over one shared HTTP/2 connection per channel, so the three-RPC design isn't opening new TCP connections per call — only cheap new streams on an already-open connection. The bidi stream's real saving (skipping that per-call stream setup) is a small fraction of consensus's 50–150ms timing budget, while its head-of-line-blocking risk can hurt latency in the other direction.

Suggestion: keep the 3-RPC design as the primary approach. Flagged the direct mesh's server-to-server channel specifically as worth revisiting with a bidi stream later if profiling ever shows the per-call overhead is a real cost there — that link is already permanent, homogeneous, and latency-sensitive, unlike the general client-facing path.

[rrobetti]

@copilot "provisioning/rotation still to be designed" remove this kind of note, provisioning and rotation is the operator's responsibility, OJP might suggest some standards. Check all docs, remove these comments. Add a single text somewhere with indications on how to do them.

[Copilot]

Removed that framing from all 3 spots (§3.2, the mesh comparison table, §9 item 3) in 0d780d2. Added a single consolidated note under §9 item 3: provisioning and rotating mTLS certs and the shared AEAD key is the operator's responsibility, same as a database password — suggested practice is reusing whatever CA/cert tooling and secret manager they already run, with a dual-accept grace period for rotation instead of a hard cutover. Also updated the matching "needs its own design" line in the consensus analysis's open questions to point at this note instead of leaving it open.

[rrobetti]

@copilot remove this question "question for the team: is that a real deployment target?"

[Copilot]

Removed in 0d780d2.

[rrobetti]

@copilot remove question "What sizes are actually expected in production?", state that small clusters are expected in production, with vertical scalling.

[Copilot]

Removed in 0d780d2 — §9 item 1 now states small, vertically-scaled clusters are the expected production shape, with gossip-based fan-out noted only as a future option for larger clusters, not a near-term need.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud