Skip to content

fix(logging): give the journal the real severity of each line - #473

Closed
ChuckBuilds wants to merge 1 commit into
mainfrom
fix/journald-log-priority
Closed

fix(logging): give the journal the real severity of each line#473
ChuckBuilds wants to merge 1 commit into
mainfrom
fix/journald-log-priority

Conversation

@ChuckBuilds

Copy link
Copy Markdown
Owner

Found while surveying warnings on a live rig: journalctl -p warning -u ledmatrix returned "-- No entries --" for the last four hours. That seemed wrong, and it was.

Every line reaches the journal as informational

Measured over 24 hours on the rig:

lines containing - ERROR - 55
lines containing - WARNING - 13
journald PRIORITY recorded 6, for every one of them

The app writes to stdout and journald has nothing else to go on, so journalctl -p err -u ledmatrix returns nothing while errors are being logged.

Triage then falls back to grepping message text — slower, and unreliable in a way that bit me during this audit: a search for oom matched the radar logging zoom=9 twenty-four times, and for a few minutes it looked like the OOM killer had been firing. It hadn't.

The fix

systemd reads a leading <N> on each stdout line and takes it as the priority (sd-daemon(3)), so a formatter that prefixes one costs no dependency:

CRITICAL <2>   ERROR <3>   WARNING <4>   INFO <6>   DEBUG <7>

Two details that matter:

  • Every line of a multi-line record is tagged, not just the first. The journal splits them, and an untagged continuation reverts to the default — which would file the body of a traceback as informational while its first line was an error. That's precisely the record you most want to find.
  • Only when JOURNAL_STREAM is set, which systemd sets for services whose output it captures. From a terminal, in the emulator, or under pytest the prefixes would be literal noise. The file handler keeps the plain formatter for the same reason.

Verification

11 new tests, 39 passing across the logging suites.

Mutation-checked:

mutation result
prefix unconditionally outside-systemd test fails
prefix only the first line multi-line test fails
map ERROR to 6 level mapping fails

Related

This also makes the earlier work easier to act on: with real priorities, the SD-wear and log-volume reduction in #468 can be judged by journalctl -p warning rather than by eye.

🤖 Generated with Claude Code

https://claude.ai/code/session_01STMbQE4YctTacQXfbYqKuW

Everything this process writes to stdout reaches the journal as PRIORITY=6,
whatever the Python level was, because journald has nothing else to go on.
Measured on a live rig over 24 hours:

    lines containing " - ERROR - "      55
    lines containing " - WARNING - "    13
    journald PRIORITY recorded          6, for every one of them

So `journalctl -p err -u ledmatrix` returns nothing while errors are being
logged, and `-p warning` likewise. Triage falls back to grepping message text,
which is slower and unreliable: during this audit a search for "oom" matched
the radar logging "zoom=9" twenty-four times and briefly looked like the OOM
killer had been firing.

systemd reads a leading "<N>" on each stdout line and takes it as the priority
(sd-daemon(3)), so a formatter that prefixes one costs no dependency. Every
line of a multi-line record is tagged, not just the first -- the journal splits
them, and an untagged continuation reverts to the default, which would leave
the body of a traceback filed as informational while its first line was an
error.

Applied only when JOURNAL_STREAM is set, which systemd sets for services whose
output it captures. Run from a terminal, in the emulator or under pytest the
prefixes would be literal noise, and the file handler keeps the plain
formatter for the same reason.

Mutation-checked three ways: prefixing unconditionally fails the
outside-systemd test, prefixing only the first line fails the multi-line test,
and mapping ERROR to 6 fails the level mapping. 39 tests pass across the
logging suites.
@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@ChuckBuilds, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 30 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 99667768-ec1b-4c4e-9f05-5a1203bff520

📥 Commits

Reviewing files that changed from the base of the PR and between cf0a551 and 780fca6.

📒 Files selected for processing (2)
  • src/logging_config.py
  • test/test_journald_log_priority.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 5 complexity · 0 duplication

Metric Results
Complexity 5
Duplication 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@ChuckBuilds

Copy link
Copy Markdown
Owner Author

Superseded by #475, which carries this commit unchanged (cherry-picked with -x) alongside the SD-write and log-volume work. Consolidated because CodeRabbit is rate-limiting across the queue. No work lost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant