Skip to content

Queue payloads through an injectable codec, with an igbinary option #290

Description

@levivannoort

Follow-up to #283 / #285, on the queue side.

packages/queue has no codec seam: Broker\Redis and Broker\Nats hand-roll json_encode/json_decode, so the ~2-4x igbinary decode win measured for the cache (#283) is unavailable to queues. On a queue the decode is paid once per message per delivery, so a redelivery pays it again.

The Redis broker pays it twice on every receive: Connection\Redis::rightPopArray() decodes the popped JSON (Connection/Redis.php:83) and setArray() re-encodes the identical array into the claim key (Broker/Redis.php:95). Dropping to rightPop() + one decode + set() removes an encode per message on its own, whichever codec is configured.

Shape

Mirror cache 5.1.0 (#286), with one queue-specific addition:

  • Utopia\Queue\Codec — encode(mixed): string / decode(string): mixed, both throwing.
  • Codec\Json — today's behaviour.
  • Codec\Igbinary — copy of Cache\Codec\Igbinary, including its two traps: reject '', and rethrow igbinary's warning-plus-null through a set_error_handler.
  • Codec\Compat — decodes either encoding by sniffing the leading bytes, encodes through an injected delegate.

Compat is the part that does not exist in cache. The cache can treat an undecodable payload as a miss and repopulate (Envelope.php:74-87); a queue cannot, because that is a dropped job. Dual-read is what makes the cutover reversible: deploy reading both and writing JSON, then flip writes, with in-flight messages and the failed/dead lists still readable.

Injected as a trailing constructor argument defaulting to new Json(), so every existing caller is unchanged.

Also in scope

A decode failure must be claimed and dead-lettered, never dropped and never retried forever. Today it throws out of receive(). This is also a prerequisite for batched consume (#290): inside a batch an unguarded throw at Broker/Nats.php:638 strands the other N-1 fetched messages for a full ackWait.

ext-igbinary goes under suggest in packages/queue/composer.json — that is both the optional-dependency declaration and what installs it in CI (.github/workflows/tests.yml:104 unions require/require-dev/suggest into the setup-php extension list).

Rollout

  1. queue ships with Compat(writes: Json). Reads both, writes what it writes today. Rollback is free.
  2. appwrite/docker-base ships the extension (feat: add the igbinary extension appwrite/docker-base#108).
  3. Flip to Compat(writes: Igbinary). Keep Compat on the read side permanently — there is no deadline on draining a dead-letter list.

Tracked on appwrite-labs/cloud#5815.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions