http-netty: avoid "Maximum active streams violated" after cancels on busy h2 connections

[bryce-anderson]

Motivation

An HTTP/2 client can fail requests with "Maximum active streams violated for
this endpoint" when a connection is backed up, for example because the peer
reads slowly. The load balancer frees a stream's concurrency slot as soon as
the stream channel closes. Netty keeps counting the stream as active until its
RST_STREAM, or its final DATA frame, is written, which can take as long as the
socket stays full. During that time the load balancer keeps choosing the
connection, Netty refuses every new stream on it, and the internal retry
tries the same connection again.

Modifications

A stream's concurrency slot is now released when Netty closes the stream, so
the load balancer's count can no longer fall below Netty's.

• A stream that closes before sending HEADERS still releases its slot when the
  channel closes, because Netty never counted it.
• ServiceTalk keeps its own count instead of reading Netty's. Netty creates a
  stream only when HEADERS are written, so it can't see requests that the load
  balancer has admitted but not yet sent.

Result

On a backed-up HTTP/2 connection, a cancelled or timed-out request keeps its
slot until Netty closes the stream. New requests go to another connection
instead of failing.