You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Queue payloads through an injectable codec, with an igbinary option #290
packages/queue has no codec seam: Broker\Redis and Broker\Nats hand-roll json_encode/json_decode, so the ~2-4x igbinary decode win measured for the cache (#283) is unavailable to queues. On a queue the decode is paid once per message per delivery, so a redelivery pays it again.
The Redis broker pays it twice on every receive: Connection\Redis::rightPopArray() decodes the popped JSON (Connection/Redis.php:83) and setArray() re-encodes the identical array into the claim key (Broker/Redis.php:95). Dropping to rightPop() + one decode + set() removes an encode per message on its own, whichever codec is configured.
Shape
Mirror cache 5.1.0 (#286), with one queue-specific addition:
Utopia\Queue\Codec — encode(mixed): string / decode(string): mixed, both throwing.
Codec\Json — today's behaviour.
Codec\Igbinary — copy of Cache\Codec\Igbinary, including its two traps: reject '', and rethrow igbinary's warning-plus-null through a set_error_handler.
Codec\Compat — decodes either encoding by sniffing the leading bytes, encodes through an injected delegate.
Compat is the part that does not exist in cache. The cache can treat an undecodable payload as a miss and repopulate (Envelope.php:74-87); a queue cannot, because that is a dropped job. Dual-read is what makes the cutover reversible: deploy reading both and writing JSON, then flip writes, with in-flight messages and the failed/dead lists still readable.
Injected as a trailing constructor argument defaulting to new Json(), so every existing caller is unchanged.
Also in scope
A decode failure must be claimed and dead-lettered, never dropped and never retried forever. Today it throws out of receive(). This is also a prerequisite for batched consume (#290): inside a batch an unguarded throw at Broker/Nats.php:638 strands the other N-1 fetched messages for a full ackWait.
ext-igbinary goes under suggest in packages/queue/composer.json — that is both the optional-dependency declaration and what installs it in CI (.github/workflows/tests.yml:104 unions require/require-dev/suggest into the setup-php extension list).
Rollout
queue ships with Compat(writes: Json). Reads both, writes what it writes today. Rollback is free.
Follow-up to #283 / #285, on the queue side.
packages/queuehas no codec seam:Broker\RedisandBroker\Natshand-rolljson_encode/json_decode, so the ~2-4x igbinary decode win measured for the cache (#283) is unavailable to queues. On a queue the decode is paid once per message per delivery, so a redelivery pays it again.The Redis broker pays it twice on every receive:
Connection\Redis::rightPopArray()decodes the popped JSON (Connection/Redis.php:83) andsetArray()re-encodes the identical array into the claim key (Broker/Redis.php:95). Dropping torightPop()+ one decode +set()removes an encode per message on its own, whichever codec is configured.Shape
Mirror cache 5.1.0 (#286), with one queue-specific addition:
Utopia\Queue\Codec—encode(mixed): string/decode(string): mixed, both throwing.Codec\Json— today's behaviour.Codec\Igbinary— copy ofCache\Codec\Igbinary, including its two traps: reject'', and rethrow igbinary's warning-plus-null through aset_error_handler.Codec\Compat— decodes either encoding by sniffing the leading bytes, encodes through an injected delegate.Compatis the part that does not exist in cache. The cache can treat an undecodable payload as a miss and repopulate (Envelope.php:74-87); a queue cannot, because that is a dropped job. Dual-read is what makes the cutover reversible: deploy reading both and writing JSON, then flip writes, with in-flight messages and thefailed/deadlists still readable.Injected as a trailing constructor argument defaulting to
new Json(), so every existing caller is unchanged.Also in scope
A decode failure must be claimed and dead-lettered, never dropped and never retried forever. Today it throws out of
receive(). This is also a prerequisite for batched consume (#290): inside a batch an unguarded throw atBroker/Nats.php:638strands the other N-1 fetched messages for a fullackWait.ext-igbinarygoes undersuggestinpackages/queue/composer.json— that is both the optional-dependency declaration and what installs it in CI (.github/workflows/tests.yml:104unionsrequire/require-dev/suggestinto thesetup-phpextension list).Rollout
Compat(writes: Json). Reads both, writes what it writes today. Rollback is free.appwrite/docker-baseships the extension (feat: add the igbinary extension appwrite/docker-base#108).Compat(writes: Igbinary). KeepCompaton the read side permanently — there is no deadline on draining a dead-letter list.Tracked on appwrite-labs/cloud#5815.