Repository navigation
Add Kafka messaging backend - #125
Merged
Merged
Conversation
Add a 'kafka' messaging backend alongside the existing rhmsg (UMB) and fedora-messaging backends, per the UMB-to-Kafka migration guide. The Messaging Bridge keeps the UMB and Kafka topics in sync, so the producer and consumer can be switched to Kafka independently while other services remain on UMB. The UMB (rhmsg) client is retained for now and can be removed once the migration is confirmed. - messaging.py: add _kafka_publish and a shared kafka_client_config helper; register the 'kafka' backend. - consumer.py: add KafkaMessage wrapper and kafka_backend using kafka-python with manual offset commits (at-least-once); wire into run(). - conf/config.py: add kafka_* config (bootstrap servers, topic/group, offset reset, topic prefix, send timeout, and SASL/SSL auth options). - requirements*: add kafka-python==2.0.2. - docker/install-dependencies.sh: install python3-kafka. - tests: cover the Kafka producer, consumer, and message wrapper. - README: document the three messaging backends and Kafka config. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Read the Kafka SASL username/password from MTS_KAFKA_USERNAME and MTS_KAFKA_PASSWORD so the secret does not have to live in the config file, and clarify that SCRAM-SHA-512 over SASL_SSL connects on port 9096. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
|
@azhuzhu @lcarva @ralphbean do you have merge perms? can you review please? |
The VirtualTopic.* namespace is a UMB addressing convention and has no meaning on Kafka. Align the Kafka topic declarations with the names registered in it-platform/kafka!911 (producer) and !768 (consumer): kafka_topic_prefix eng.mts kafka_consumer_topic eng.mbs.module.state.change The rhmsg settings keep VirtualTopic.* since that backend still targets UMB. Stage uses a stage.* prefix on both, set in the deployment overlay rather than here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The consume loop only caught JSONDecodeError. Any other exception escaped
the loop, closed the consumer and ended the process with the offset
uncommitted, so the restart was re-delivered the same record and failed
again -- a crash loop that makes no further progress.
This is reachable with a well-formed JSON body: the NSVC is built with
'{name}:{stream}:{version}:{context}'.format(**mbs_msg) outside of the
try block in consume(), so a message missing one of those fields raises
KeyError. A connection error while fetching the rules file escapes the
same way, turning a transient fault into a stuck consumer.
Log the failure and commit the offset instead, which is what consume()
already does for a failed tagging attempt.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
asabadra
force-pushed
the
umb-to-kafka
branch
from
November 1, 2026 15:26
9fd2091 to
97fd2e8
Compare
Both jobs failed at the "Set up Python 3.9" step, before tox ever ran: The version '3.9' with architecture 'x64' was not found for Ubuntu 26.04. ubuntu-latest began migrating to Ubuntu 26.04 on 2026-10-19, and the actions/python-versions manifest has no 3.9 build for 26.04, though it does for 3.10 and newer. Pinning the runner restores a combination where 3.9 is available. Python 3.9 has been EOL since 2025-10, so moving the test matrix off it is the durable fix; this only unblocks CI in the meantime. Also bump actions/setup-python v4 to v6. That is unrelated hygiene, not part of the fix above: v4 targets the deprecated node16 runtime, but deprecated runtimes are force-upgraded with a warning rather than failed, so it was never what broke these jobs. This is pre-existing breakage on master rather than anything to do with the Kafka backend, but CI could not report on this branch until it was fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
asabadra
force-pushed
the
umb-to-kafka
branch
from
November 1, 2026 15:33
97fd2e8 to
b90391d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds kafka as a third messaging backend alongside fedora-messaging and rhmsg. MTS can now consume MBS module state change events from and publish tagging events to Kafka. It plugs into the existing messaging_backend dispatch, so existing deployments are unaffected — they stay on their current backend by default.
Changes
Tests
5 unit tests covering the consume path, offset commit/close, invalid bodies, KafkaMessage, and publish (config pass-through and dry-run). All fake the kafka module, so no broker is needed.