Skip to content

Report a truncated TLS session when the transport output is cancelled - #3740

Merged
He-Pin merged 1 commit into
apache:mainfrom
He-Pin:fix/tls-transport-cancel-truncation
Nov 1, 2026
Merged

He-Pin merged 1 commit into
apache:mainfrom
He-Pin:fix/tls-transport-cancel-truncation

Conversation

@He-Pin

@He-Pin He-Pin commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Motivation

Cancelling a TLS operator's ciphertext output can make received input beyond one inbound pump round unreachable while completing the plaintext output silently. The single-record immediate-loss scenario originally described in #3739 does not reproduce; the related abandonment of remaining input does.

Modification

  • Handle transport-output cancellation explicitly in both TLS engines. Cancel plaintext input immediately, including when the final inbound round is waiting for plaintext demand.
  • Report SessionTruncated for an established session whose peer close_notify has not been processed. Preserve pending plaintext, track established sessions across renegotiation, and skip depleted input without calling closeInbound() solely to detect truncation.
  • Preserve an existing wait for the peer's close_notify once the local close_notify has been pushed. End that wait if both outputs are cancelled or cancellation discards a still-buffered close_notify. A partial record does not cause cancellation to wait for future ciphertext.
  • Share cancellation predicates between engines, preserve legacy-actor error publication order, and make actor termination safe against pump reentrancy.
  • Cover both engines and TLS 1.2/1.3, all closing modes, empty/buffered/fragmented input, renegotiation, graceful close, session validity, failed unwrap and downstream cancellation. Document the behavior and its 2.0.0 upgrade impact.

Result

Direct TLS BidiFlow consumers can observe an inbound session abandoned by cancellation. At most one final inbound round is processed; incomplete records and input beyond that round may still be discarded. The terminal signal obeys backpressure, so consumers must consume or cancel the plaintext output; completionTimeout can bound the wait.

The Scala and Java Tcp.bindWithTls and Tcp.outgoingConnectionWithTls APIs expose ByteString streams and filter out SessionTruncated. This PR documents that limitation without changing those wrappers to fail on the signal.

Tests

  • JDK 17 sbt "stream-tests/Test/testOnly *TlsTransportOutputCancelSpec *TlsGraphStageDeferredCloseSpec": 119 passed, 0 failed, including 84 transport-output cancellation tests and 35 deferred-close tests.
  • The new scripted deferred-initial-close regression fails on 2e8364f8 and passes with the pending-close guard. It freezes the initial output timer to reproduce a buffered close being discarded on cancellation. The four ordinary JDK no-demand close cases already pass on 2e8364f8; they provide boundary coverage, not evidence of that failure in normal JDK shutdown.
  • Before the pending-close guard, JDK 17 sbt "stream-tests/Test/testOnly *Tls*": 379 passed, 0 failed.
  • Earlier directional verification: all eight immediate-upstream-cancellation and graceful-close cases fail before their respective fixes, then pass afterward. Depleted-input and renegotiation regressions were also verified against the corresponding unfixed code.
  • sbt +mimaReportBinaryIssues: passed on Scala 2.13.18 and 3.3.9 before the pending-close guard, which changes no binary shape. The latest full PR validation remains delegated to remote CI.
  • sbt headerCreateAll +headerCheckAll checkCodeStyle, scalafmt diff check and git diff --check: passed.
  • sortImports was blocked by the existing NoSuchMethodError: scala.meta.package$.Input() in stream-tests and multi-node-testkit; unrelated partial edits were removed.
  • An earlier JDK 25 run was interrupted after ScalaTest SocketReporter failed with Broken pipe; final local validation uses JDK 17.
  • docs/paradox and validatePullRequest are delegated to remote CI as requested; an earlier local docs/paradox attempt was interrupted.

References

Refs #3739

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues
What changed in this PR

This PR makes transport-output cancellation observable to TLS consumers by reporting inbound truncation (SessionTruncated) when buffered inbound records become unreachable due to teardown triggered by cancelling the ciphertext output.

Changes:

  • Emit SessionTruncated on the transport-output-cancel teardown path in both TlsGraphStage and the legacy TLSActor implementation.
  • Add a dedicated TLSActor phase to defer the truncation report until plaintext output demand (and proactively cancel read-side input to avoid buffering overrun).
  • Add a new spec covering multiple buffering/demand/cancellation permutations and document the contract in the stream IO guide.
File Description
stream/​src/​main/​scala/​org/​apache/​pekko/​stream/​impl/​io/​TlsGraphStage.scala Signals inbound truncation on TransportOut cancellation before completing.
stream/​src/​main/​scala/​org/​apache/​pekko/​stream/​impl/​io/​TLSActor.scala Introduces a truncation-signalling phase for the legacy actor and cancels inputs to prevent buffer overrun while waiting for demand.
stream-tests/​src/​test/​scala/​org/​apache/​pekko/​stream/​io/​TlsTransportOutputCancelSpec.scala Adds comprehensive regression tests for truncation semantics under transport-output cancellation.
docs/​src/​main/​paradox/​stream/​stream-io.md Documents the new observable contract when transport output is cancelled.

Review effort: Lite

@He-Pin
He-Pin marked this pull request as draft October 30, 2026 19:26
@pjfanning

Copy link
Copy Markdown
Member

@He-Pin is this aimed at 2.0.0? Nothing else blocks a 2.0.0 release at the moment. If we choose to hold this to 2.1.0, then I can create the RC1 for 2.0.0.

@He-Pin
He-Pin requested a review from pjfanning October 31, 2026 07:21
@He-Pin He-Pin added this to the 2.0.0-M5 milestone Oct 31, 2026
@He-Pin
He-Pin force-pushed the fix/tls-transport-cancel-truncation branch from d3a5ce6 to 8cd70ec Compare October 31, 2026 07:21
@He-Pin
He-Pin marked this pull request as ready for review October 31, 2026 07:23

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this. A few concerns are noted inline. The main ones:

  • Silent drops remain in two paths: renegotiation in flight, and a pending plaintext element (both engines for the first).
  • Behaviour change: SessionTruncated is now emitted on any transport-output cancellation of an established session, even with nothing buffered.
  • closeInbound() before close_notify may invalidate the SSLSession and break resumption.
  • Teardown now waits on plaintext demand, so the actor or stage can stay alive if downstream never pulls.

@He-Pin
He-Pin force-pushed the fix/tls-transport-cancel-truncation branch from 8cd70ec to a01263f Compare October 31, 2026 11:22
@He-Pin
He-Pin requested review from pjfanning and a lite review from Copilot October 31, 2026 11:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Changes are still needed because both TLS cancellation paths can treat a depleted transport input as ready, invoke closeInbound, and invalidate the session instead of emitting SessionTruncated while preserving it.

2 open issues
3 resolved since last review

Review effort: Lite

@He-Pin
He-Pin force-pushed the fix/tls-transport-cancel-truncation branch from a01263f to 5ed4a10 Compare October 31, 2026 11:59
@He-Pin
He-Pin requested a lite review from Copilot October 31, 2026 12:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change spans intricate TLS-engine interactions across cancellation, backpressure, renegotiation, and provider-specific state, so maintainer confirmation of the teardown contract is warranted despite extensive tests.

No open issues 🎉

2 resolved since last review

Review effort: Lite

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Some review findings are inline. The one most worth addressing is the plaintext input left open after the transport output is cancelled (TlsGraphStage onCancel). The partial-record and AwaitingClose cases are edge cases worth weighing.

One point outside the diff: Tcp's tlsWrapping (stream/src/main/scala/org/apache/pekko/stream/scaladsl/Tcp.scala, around line 99) uses collect { case sb: SessionBytes => sb.bytes }, which drops SessionTruncated. So users of Tcp().outgoingConnectionWithTls / bindWithTls still get a normal completion when the session is truncated, and the release-note advice to handle SessionTruncated doesn't reach them. That could be a follow-up (fail the ByteString flow on SessionTruncated), but the docs/release note should probably say that the signal is only visible at the TLS BidiFlow level.

@He-Pin
He-Pin force-pushed the fix/tls-transport-cancel-truncation branch from 5ed4a10 to 2e8364f Compare October 31, 2026 15:13
@He-Pin
He-Pin requested review from pjfanning and a lite review from Copilot October 31, 2026 15:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change adds interacting SSLEngine teardown state machines with deferred close_notify handling, while reentrant cancellation and provider-specific close or renegotiation behavior still require maintainer validation.

No open issues 🎉

Review effort: Lite

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates. I re-reviewed 2e8364f8. My earlier points are handled: the plaintext input is now cancelled on transport-output cancellation, AwaitingClose is kept, the failure is no longer re-run (isCompleted = stopped plus the idempotent pumpFinished), the predicates are shared in TlsEngineHelpers, the phase constant is reordered, and the Tcp filtering is documented. Each one has a test. I'm fine with documenting the partial-record behaviour instead of waiting for more ciphertext.

There is one new issue in the graph stage (inline).

Motivation:
Cancelling a ciphertext output can make received TLS data unreachable.
Completing the plaintext output silently hides the unclean shutdown.

Modification:
- Handle transport-output cancellation in both TLS engines, cancel the
  plaintext input immediately, and allow at most one final inbound round.
- Report SessionTruncated for established sessions without a peer
  close_notify. Preserve pending plaintext and session validity, including
  during renegotiation, and skip depleted transport input.
- Preserve an existing wait for the peer's close_notify after the local
  close_notify has been pushed. End that wait if both outputs are cancelled
  or cancellation discards a buffered close_notify.
- Share cancellation predicates, preserve actor error publication order,
  and make actor failure cleanup safe against pump reentrancy.
- Add directional tests and document the 2.0.0 behavior and the TCP
  ByteString APIs' filtering of SessionTruncated.

Result:
Direct TLS BidiFlow consumers can observe an inbound session abandoned by
cancellation. The final signal obeys backpressure; incomplete input and
data beyond the final inbound round can still be discarded.

Tests:
- JDK 17: sbt "stream-tests/Test/testOnly *TlsTransportOutputCancelSpec
  *TlsGraphStageDeferredCloseSpec": 119 passed, 0 failed.
- The deferred-initial-close regression fails on 2e8364f and passes with
  the fix. Four ordinary no-demand close cases already pass on 2e8364f.
- Before the pending-close guard: JDK 17
  sbt "stream-tests/Test/testOnly *Tls*": 379 passed, 0 failed.
- Eight upstream-cancellation and close-handshake regressions across both
  engines and TLS 1.2/1.3 were verified to fail before their fixes, as were
  the depleted-input and renegotiation regressions.
- sbt +mimaReportBinaryIssues: passed on Scala 2.13.18 and 3.3.9 before
  the pending-close guard; the guard changes no binary shape.
- sbt headerCreateAll +headerCheckAll checkCodeStyle: passed.
- scalafmt diff check and git diff --check: passed.
- sortImports was blocked by the existing scala.meta.package$.Input()
  NoSuchMethodError; unrelated partial edits were removed.
- An earlier JDK 25 run was interrupted after a ScalaTest SocketReporter
  Broken pipe error; final local validation uses JDK 17.
- docs/paradox and validatePullRequest are delegated to remote CI;
  an earlier local docs/paradox attempt was interrupted.

References:
Refs apache#3739
@He-Pin
He-Pin force-pushed the fix/tls-transport-cancel-truncation branch from 2e8364f to 8bd6245 Compare November 1, 2026 05:52
@He-Pin
He-Pin requested review from pjfanning and a lite review from Copilot November 1, 2026 05:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The intertwined TLS shutdown, renegotiation, cancellation, and provider-specific SSLEngine changes require remote multi-JDK validation and maintainer review beyond the provided local evidence.

No open issues 🎉

Review effort: Lite

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 8bd62459. The pending-close_notify case is fixed: onCancel records hadPending before clearing the slot, so it only keeps the AwaitingClose wait when our close_notify was actually pushed. The new tests cover both engines with no ciphertext demand, plus the deferred initial-output path, and the docs describe the behaviour. I have no further findings.

@pjfanning pjfanning left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@He-Pin
He-Pin merged commit ce3c32d into apache:main Nov 1, 2026
10 checks passed
@He-Pin
He-Pin deleted the fix/tls-transport-cancel-truncation branch November 1, 2026 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t:stream Pekko Streams

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants