rust(feat): mcp rate-limiting

[Brandon-Shippy]

Description

Implements gRPC status handling for backoff and retries in the mcp.

We construct each Service with a Retry Policy that outlines how to handle gRPC status codes. In here we implement a retry loop that takes in a closure. With this new structure we wrap our gRPC calls within the retry loop so we can handle the gRPC status codes when they come in.

When the retry layer gives up, we give the agent a soft-signal to not retry this tool without the user providing more context or the general shape of the request is different. We do handle the different tonic statuses differently.

Following the status codes laid out here: https://google.aip.dev/194

UNAVAILABLE = Retry with backoff, then soft signal on exhaustion
RESOURCE_EXHAUSTED, DEADLINE_EXCEEDED, INTERNAL, ABORTED, ALREADY_EXISTS, PERMISSION_DENIED, UNAUTHENTICATED, CANCELLED = Immediate error return with soft signal

Errors without set soft-guidance are given a hard rmcp error code back:

INVALID_ARGUMENT and OUT_OF_RANGE = INVALID_PARAMS
NOT_FOUND = RESOURCE_NOT_FOUND
UNIMPLEMENTED = METHOD_NOT_FOUND
FAILED_PRECONDITION, DATA_LOSS, UNKNOWN = INTERNAL_ERROR

What this does not limit:

• Number of tools being used at once
• Total request count over a session

Verification

• Mock tests
• Unit tests
• Pointed the mcp to an unreachable address and did manual testing to get errors back