Analysis + ADR-010 + Phases 1 & 2: hybrid JMX / OpenTelemetry driver throttling metrics

[Copilot]

Summary

Lands the throttling-metrics analysis, an ADR adopting the hybrid (JMX core + opt-in OpenTelemetry adapter) approach for the JDBC driver, and both phases of that approach: Phase 1 (JMX in ojp-jdbc-driver, no new runtime deps) and Phase 2 (the separately published ojp-jdbc-driver-otel-metrics adapter module).

Server-side ojp.throttle.* metrics (gRPC interceptor + SlotManager) remain analysis-only in this PR and will be implemented in follow-up PRs per the rollout in §7 of the analysis document.

What this PR adds

Documentation

• documents/analysis/THROTTLING_METRICS_ANALYSIS.md — full analysis covering:
  • The three independent throttling layers OJP has today (ConcurrencyThrottleInterceptor, SlotManager, ClientThrottleManager) and the observability gap in each
  • The operator-facing questions good metrics should answer
  • A concrete ojp.throttle.* OpenTelemetry meter scope, following the existing ojp.sql / ojp.*.pool convention (interface + OTel impl + NoOp + factory)
  • Server-side metrics for the gRPC gate (inflight, limit, utilization, rejected/accepted counters, hold time)
  • Server-side metrics for SlotManager per datasource × lane (slots total/active/available/utilization, queue depth/max, acquired/rejected counters with path and reason, wait time histogram, borrowed gauge, observedPeak, enabled)
  • Driver-side metrics (in-flight, proactive/reactive/effective limits, rejected/acquired counters, server-overload events, AIMD limit-change counter)
  • Cardinality discipline (allowed/disallowed labels)
  • Suggested rollout order (SlotManager → gRPC interceptor → driver)
  • Appendix §9 — deep dive on JMX vs OpenTelemetry for driver-side metrics: baseline facts (driver has zero OTel deps today; server has 21), per-option benefits and downsides (classpath-conflict and silent no-op risks for OTel-in-driver, MBean lifecycle and scalar-only constraints for JMX), a recommended hybrid approach (JMX in driver core + opt-in ojp-jdbc-driver-otel-metrics adapter jar, mirroring HikariCP's pattern), a comparison table, recommendation with ~80% confidence, and conditions that would change the recommendation.
• documents/analysis/README.md — link to the new analysis under "Latest Analysis".
• documents/ADRs/adr-010-hybrid-jmx-otel-driver-metrics.md — records the decision to adopt the hybrid approach (Option C), with scope, configuration (ojp.jdbc.metrics system property), neglected options, accepted trade-offs, and a two-phase rollout (both phases now marked as implemented).

Phase 1 implementation (ojp-jdbc-driver, no new runtime deps)

New package org.openjproxy.jdbc.metrics:

• ClientThrottleMetrics — small sink interface with LimitChangeDirection enum (mirrors the server's SqlStatementMetrics pattern).
• NoOpClientThrottleMetrics — singleton.
• ClientThrottleMetricsMXBean + JmxClientThrottleMetrics — registers one MBean per connHash under org.openjproxy:type=ClientThrottle,connHash=<hash>. Best-effort registration (warns and degrades gracefully on SecurityManager denial / duplicate name), idempotent close().
• ClientThrottleStateProvider — read-only snapshot interface so the MBean doesn't import ClientThrottleManager.
• ClientThrottleMetricsProvider — ServiceLoader SPI used by the factory to discover non-built-in bindings (e.g. the Phase 2 OTel adapter).
• ClientThrottleMetricsFactory — selects the implementation from the ojp.jdbc.metrics system property. Built-ins (jmx default, none) are hard-wired; other values (e.g. otel) are resolved via ServiceLoader. Falls back to JMX with a single WARN when the requested provider is not on the classpath.

Wiring:

• ClientThrottleManager gains setMetrics(...) and records on acquire / reject / server-overload / AIMD limit-change.
• ClientThrottleManager.tryAcquire(...) now increments the inFlight counter on the unlimited path (effectiveLimit == Integer.MAX_VALUE) too, keeping it symmetric with release() so the JMX InFlight attribute and the OTel ojp.client.throttle.inflight gauge reflect concurrent driver load from the first request — not only once throttling kicks in.
• Connection.createThrottleManager(...) creates one metrics instance per connHash; the throttle mode is snapshotted to avoid the state provider pinning the first Connection.

MBean attributes exposed: ConnHash, Mode, InFlight, ProactiveLimit, ReactiveLimit, EffectiveLimit, AcquiredTotal, RejectedTotal, ServerOverloadEventsTotal, LimitIncreaseTotal, LimitDecreaseTotal.

Phase 2 implementation — ojp-jdbc-driver-otel-metrics adapter module

New Maven module registered in the parent pom.xml. The driver core gains zero new runtime dependencies; only this opt-in module depends on OpenTelemetry.

• opentelemetry-api is declared with provided scope so the host application controls the OTel version (avoids the classpath-conflict / version-skew risk called out in Appendix §9).
• OpenTelemetryClientThrottleMetrics publishes meter scope ojp.client.throttle with attribute ojp.connection.hash:
  • Observable gauges: ojp.client.throttle.inflight, ojp.client.throttle.limit.proactive, ojp.client.throttle.limit.reactive, ojp.client.throttle.limit.effective. The driver's Integer.MAX_VALUE "no limit" sentinel is saturated to 0 so dashboards don't show a 2-billion bar before SessionInfo arrives.
  • Monotonic counters: ojp.client.throttle.acquired.total, ojp.client.throttle.rejected.total, ojp.client.throttle.server.overload.total, and ojp.client.throttle.limit.changes.total tagged by direction=increase|decrease.
  • Idempotent close() (guarded by a volatile boolean) unregisters the observable callbacks.
  • When the no-arg constructor resolves GlobalOpenTelemetry.get() and gets back the no-op instance (no SDK registered — exactly the risk flagged in Appendix §9), the adapter logs a single WARN explaining how to register an SDK (OpenTelemetry Java agent, opentelemetry-sdk-extension-autoconfigure, or a manual GlobalOpenTelemetry.set(sdk)). Without this surfacing, -Dojp.jdbc.metrics=otel would silently drop every counter and gauge.
• OpenTelemetryClientThrottleMetricsProvider registered via META-INF/services/org.openjproxy.jdbc.metrics.ClientThrottleMetricsProvider; uses GlobalOpenTelemetry.get() by default; returns NoOpClientThrottleMetrics on NoClassDefFoundError / unexpected failure so metrics never break JDBC functionality.

Usage: drop the adapter jar on the application classpath and start the JVM with -Dojp.jdbc.metrics=otel. Without the jar, jmx (Phase 1) remains the default and nothing changes.

Tests

• JmxClientThrottleMetricsTest — MBean registration, attribute reads, unregistration, duplicate-name tolerance.
• ClientThrottleMetricsFactoryTest — default, none, otel-fallback, and null-guard behaviour.
• ClientThrottleManagerMetricsIntegrationTest — end-to-end manager↔metrics counter recording, plus shouldTrackInFlightEvenWhenLimitIsUnlimited regression test for the unlimited-path inFlight symmetry fix.
• OpenTelemetryClientThrottleMetricsTest — counter/gauge values, direction tagging, Integer.MAX_VALUE saturation, double-close, null-arg validation (uses opentelemetry-sdk-testing + InMemoryMetricReader).
• OpenTelemetryClientThrottleMetricsProviderTest — ServiceLoader discovery and provider name.
• FactorySelectsOtelWhenAdapterPresentTest — end-to-end factory selection of the OTel implementation when the adapter is on the classpath.
• Full ojp-jdbc-driver unit suite passes; new adapter module: 9 tests, 0 failures; mvn checkstyle:check passes for both modules; CodeQL: 0 alerts; GitHub Advisory DB: no advisories for OpenTelemetry 1.62.0 (API + SDK + SDK-testing).

Open questions for reviewers

• datasource attribute identity (for the server-side metrics in the analysis). Use existing connHash initially, or invest in human-friendly names from the start?
• grpc.method cardinality. ~10 RPC methods today; OK to use as a label on ojp.throttle.grpc.*?
• Overlap with existing ojp.sql.slow.executions.total. Lane attribution is proposed only on slot metrics; SQL metrics keep their existing shape — confirm this split.

Deferred to follow-up PRs

• Server-side ojp.throttle.* implementation (gRPC interceptor + SlotManager metrics) per §7 of the analysis.

[rrobetti]

@copilot go deepewton the benefits, implications and potential issues betwen choosing JMX vs Open Telemetry in the OJP JDBC layer metrics.

[Copilot]

@copilot go deepewton the benefits, implications and potential issues betwen choosing JMX vs Open Telemetry in the OJP JDBC layer metrics.

Added Appendix §9 to THROTTLING_METRICS_ANALYSIS.md in 6636770 with a deep dive on the driver-side transport choice:

• Baseline facts I verified first: driver has zero OTel deps today (grep opentelemetry ojp-jdbc-driver/pom.xml → 0; server → 21), driver minimum is Java 11, driver lives in the host's classpath, and metric volume is tiny.
• Option A — JMX: zero new deps, universal consumer support (JConsole, jmx_exporter, every APM agent), matches HikariCP precedent. Downsides: scalars only, needs jmx_exporter YAML, MBean lifecycle is a small footgun.
• Option B — OTel API in the driver: native dimensional model, histograms, trace exemplars. Real downsides: classpath-conflict / version-skew risk (~70% this bites at least once), silent no-op trap when host has API but no SDK, shading defeats the integration purpose, harder to remove.
• Option C (recommended) — Hybrid: JMX in driver core + a separately-published ojp-jdbc-driver-otel-metrics adapter jar. Mirrors HikariCP's path (JMX core + opt-in Micrometer/Dropwizard). Same ClientThrottleMetrics interface either way, so the architectural work is done once. Reversible, opt-in for OTel, no cost for users who don't want it.
• Plus a comparison table, things that would change the recommendation (driver tracing plans, exemplar-linked debugging, raising driver baseline to Java 17), and confidence (~80%).

[rrobetti]

@copilot lets go with Option C, create the ADR and start implementing it.

[Copilot]

@copilot lets go with Option C, create the ADR and start implementing it.

Done — ADR + Phase 1 landed.

ADR-010 (documents/ADRs/adr-010-hybrid-jmx-otel-driver-metrics.md, 320261b): hybrid JMX core + opt-in ojp-jdbc-driver-otel-metrics adapter, with a single ojp.jdbc.metrics system property (jmx default, none, otel reserved for Phase 2 → warns + falls back until the adapter ships).

Phase 1 implementation in ojp-jdbc-driver (320261b, refined in 91acb30) — zero new runtime deps:

• org.openjproxy.jdbc.metrics.ClientThrottleMetrics interface + LimitChangeDirection enum
• NoOpClientThrottleMetrics (singleton)
• ClientThrottleMetricsMXBean + JmxClientThrottleMetrics — registers one MBean per connHash under org.openjproxy:type=ClientThrottle,connHash=<hash>. Best-effort registration (warns and degrades on SecurityManager/duplicate name), idempotent close().
• ClientThrottleStateProvider — read-only snapshot interface so the MBean doesn't import ClientThrottleManager.
• ClientThrottleMetricsFactory — selects implementation from the system property.
• Wired into ClientThrottleManager (acquire / reject / server-overload / AIMD limit-change hooks) and Connection.createThrottleManager (one metrics instance per connHash; mode is snapshotted to avoid the state-provider holding a back-reference to the first Connection).

MBean attributes exposed: ConnHash, Mode, InFlight, ProactiveLimit, ReactiveLimit, EffectiveLimit, AcquiredTotal, RejectedTotal, ServerOverloadEventsTotal, LimitIncreaseTotal, LimitDecreaseTotal.

Validation: new tests JmxClientThrottleMetricsTest, ClientThrottleMetricsFactoryTest, ClientThrottleManagerMetricsIntegrationTest — all pass. Full driver unit suite: 262 ran, 0 failures. mvn checkstyle:check passes. CodeQL: 0 alerts.

Deferred to a follow-up PR per ADR-010: the separate ojp-jdbc-driver-otel-metrics Maven module (Phase 2). The ClientThrottleMetrics interface is exactly the seam that module will plug into, so the architectural work is already done.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• checkstyle.org
  • Triggering command: /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/tools/linux64/java/bin/java /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/tools/linux64/java/bin/java -jar /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/xml/tools/xml-extractor.jar --fileList=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/working/files-to-index8922119038882122708.list --sourceArchiveDir=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/src --outputDir=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/trap/java (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[rrobetti]

@copilot Have you implemented all phases already? if not continue.

[Copilot]

@copilot Have you implemented all phases already? if not continue.

No — Phase 1 was done, Phase 2 was deferred. Now landed in 2f747bf.

Phase 2: ojp-jdbc-driver-otel-metrics adapter module — both phases of ADR-010 are now implemented.

What changed:

• New SPI in driver core: ClientThrottleMetricsProvider (driver core gains zero new runtime deps). ClientThrottleMetricsFactory now does a ServiceLoader lookup for any non-built-in binding (e.g. otel); built-ins (jmx, none) stay hard-wired. Still falls back to JMX with a single WARN when no matching provider is on the classpath.
• New module ojp-jdbc-driver-otel-metrics registered in the parent pom.xml. opentelemetry-api is provided so the host app controls the OTel version.
• OpenTelemetryClientThrottleMetrics publishes meter scope ojp.client.throttle with attribute ojp.connection.hash:
  • 4 observable gauges: inflight, limit.proactive, limit.reactive, limit.effective (the driver's Integer.MAX_VALUE "no limit" sentinel is saturated to 0 for dashboards)
  • 4 monotonic counters: acquired.total, rejected.total, server.overload.total, and limit.changes.total tagged by direction=increase|decrease
  • Idempotent close() unregisters the observable callbacks
• OpenTelemetryClientThrottleMetricsProvider registered via META-INF/services/...ClientThrottleMetricsProvider; uses GlobalOpenTelemetry.get(); returns NoOp on NoClassDefFoundError / unexpected failure so metrics never break JDBC.
• Tests (9, all passing) with opentelemetry-sdk-testing + InMemoryMetricReader: counter/gauge values, direction tagging, Integer.MAX_VALUE saturation, double-close, null-arg validation, ServiceLoader discovery, and end-to-end factory selection of OTel when the adapter is on the classpath.

Validation: mvn checkstyle:check passes for both modules; OTel 1.62.0 (API + SDK + SDK-testing) clean in GitHub Advisory DB; Code Review: 0 comments; CodeQL: 0 alerts.

How to use: add the adapter jar to the application classpath and start the JVM with -Dojp.jdbc.metrics=otel. Without the jar, jmx (Phase 1) remains the default and nothing changes.

ADR-010 updated to mark both phases as implemented. The server-side ojp.throttle.* work (gRPC interceptor + SlotManager) called out in §7 of the analysis is the next deliverable but is out of scope for this PR.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• checkstyle.org
  • Triggering command: /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/tools/linux64/java/bin/java /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/tools/linux64/java/bin/java -jar /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/xml/tools/xml-extractor.jar --fileList=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/working/files-to-index16717784854329409518.list --sourceArchiveDir=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/src --outputDir=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/trap/java (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[rrobetti]

@copilot Double check all the metrics are wired correctly to be exported.

[Copilot]

@copilot Double check all the metrics are wired correctly to be exported.

Did the audit end-to-end (Connection.createThrottleManager → setMetrics → ClientThrottleManager record sites → JMX MBean attrs + OTel counters/gauges). All counter and gauge wiring is correct on both sinks. Found two real export bugs and fixed both in a4e1911:

1. OTel silent no-op trap (Appendix §9 risk). If -Dojp.jdbc.metrics=otel is set but no SDK is registered (no agent / no autoconfigure / no manual GlobalOpenTelemetry.set), GlobalOpenTelemetry.get() returns OpenTelemetry.noop() and every counter and gauge silently drops. OpenTelemetryClientThrottleMetrics now detects this once and logs a single WARN telling the operator how to register an SDK.

2. inflight gauge stuck at 0 before first SessionInfo. ClientThrottleManager.tryAcquire() skipped incrementing inFlight on the unlimited path (effectiveLimit == Integer.MAX_VALUE), but release() always decremented (clamped to 0). The JMX InFlight attribute and the OTel ojp.client.throttle.inflight gauge therefore always read 0 until throttling kicked in. Now inFlight is incremented on the unlimited path too, so the gauge reflects concurrent driver load from the first request and acquire/release stay symmetric.

Validation: full driver suite passes (1028 tests), adapter suite passes (9 tests), checkstyle clean on both modules, CodeQL 0 alerts. Added regression test shouldTrackInFlightEvenWhenLimitIsUnlimited.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• checkstyle.org
  • Triggering command: /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/tools/linux64/java/bin/java /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/tools/linux64/java/bin/java -jar /opt/hostedtoolcache/CodeQL/2.25.4/x64/codeql/xml/tools/xml-extractor.jar --fileList=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/working/files-to-index4169832812229896248.list --sourceArchiveDir=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/src --outputDir=/tmp/codeql-scratch-0fe3065af824c7c2/dbs/java/trap/java (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud