From 1ef2b5ae1ffd8cda3d78e7658d89a2f2f1f7d1f4 Mon Sep 17 00:00:00 2001 From: Aviv Sabadra Date: Wed, 4 Nov 2026 00:02:34 +0200 Subject: [PATCH] Install the snappy codec so the Kafka consumer can read compressed batches kafka-python does not negotiate compression. It has to decompress whatever codec the producer chose, and the codec libraries are optional extras it does not pull in. The image installed no snappy support, so against a producer using snappy every fetch failed with: kafka.errors.UnsupportedCodecError: Libraries for snappy compression codec not found This is worse than a single failed message. UnsupportedCodecError is raised by the consumer iterator while fetching, not by consume(), so it is outside the per-message error handling in kafka_backend(). The exception escapes the loop, the process exits, and on restart the consumer rejoins and re-fetches the same uncommitted batch, failing again. Observed in a stage deployment as a crash loop that advanced the consumer group through 13 generations in three hours without committing an offset. Only snappy is added here, which is what the producer in question uses. lz4 and zstd remain unavailable and would fail the same way. Co-Authored-By: Claude Opus 5 --- docker/install-dependencies.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docker/install-dependencies.sh b/docker/install-dependencies.sh index 33ad9e9..670b75d 100755 --- a/docker/install-dependencies.sh +++ b/docker/install-dependencies.sh @@ -11,6 +11,12 @@ dependencies=( python3-flask python3-gunicorn python3-kafka + # kafka-python negotiates no compression of its own: it must be able to + # decompress whatever codec the producer chose. Without the codec library + # the batch raises UnsupportedCodecError from the consumer iterator, which + # is outside the per-message error handling in kafka_backend(), so the + # process exits and restarts onto the same batch forever. + python3-snappy python3-koji python3-qpid-proton python3-pip