🔊(back) log as JSON with django-structlog by kernicPanel · Pull Request #289 · suitenumerique/drive-migrator

kernicPanel · 2026-10-27T12:18:58Z

Purpose

A failed migration has to be matched with the Drive logs of the same
request, and the logs of concurrent migrations can't be told apart.

Proposal

Write every log as a JSON line with its context, using django-structlog.

JSON logs (event, level, logger, timestamp), extra fields included
request_id taken from the X-Request-ID header (generated otherwise),
with user_id, on every log of a request
request_started, request_finished and request_failed replace
request.summary and django.request, with duration_ms and the API
error_name
Celery tasks keep the context of the request that sent them, without
their arguments
workspace_id on every log of a migration
Every outgoing call sends an X-Request-ID, kept by its retries, and
each attempt is logged with it, its status and its duration
Request id appended to the gunicorn access log
Readable key-value lines in development instead of JSON

Existing log calls are unchanged: their message goes to the event field.
Log queries and dashboards parsing the text format need to be updated.

martigan · 2026-11-02T12:35:02Z

-    LOGGING = values.DictValue(
-        {
-            "version": 1,
-            "disable_existing_loggers": False,
-            "handlers": {
-                "console": {
-                    "class": "logging.StreamHandler",
-                },
-            },
-            "loggers": {
-                "main": {
-                    "handlers": ["console"],
-                    "level": "DEBUG",
-                },
-            },
-        }
-    )


martigan · 2026-11-02T12:36:24Z

    "django-parler==2.3",
    "django-redis==5.4.0",
    "django-storages[s3]==1.14.2",
+    "django-structlog==10.1.0",


why not using v11?

v11 requires Celery 5.5+ (we're on 5.3.6) and stops trusting X-Request-ID by default, which this PR relies on. We'll upgrade separately.

martigan · 2026-11-02T12:43:33Z

+def test_logging_request_user(console_logs):
+    """A request should be logged with the id of its user."""
+    user = factories.UserFactory()
+    client = APIClient()


I think you can have a client in test parameters with pytest-django

https://pytest-django.readthedocs.io/en/stable/helpers.html#client-django-test-client

Done, thanks.

martigan · 2026-11-02T12:51:54Z

 _QUERY_IN_TEXT_RE = re.compile(r"\?\S*")

+# Logs the calls of every service, whose retries go to the logger of the service
+request_logger = structlog.get_logger(__name__)


seems logger are not specifically named across the project codebase, logger = structlog.get_logger(__name__) is more consistent

Done. The module logger is now logger; the session's parameter became retry_logger to avoid shadowing it.

martigan · 2026-11-02T12:55:07Z

+        started_at = time.monotonic()
+        try:
+            response = super().request(method, url, *args, **kwargs)
+        except Exception as error:


too broad exception, not a good practice, try to narrow (ie requests.exceptions.HTTPError, requests.exceptions.Timeout)

This only logs the attempt and re-raises right after, so nothing is swallowed. Keeping it broad means every failed attempt gets its log line.

martigan · 2026-11-02T12:56:51Z

 from sentry_sdk.integrations.django import DjangoIntegration
 from sentry_sdk.integrations.logging import ignore_logger

+# pylint: disable=too-many-lines


maybe we should split the setting file into chunks

Agreed it's getting long, but it's unrelated to logging. Better in its own PR.

A failed migration has to be matched with the Drive logs of the same request. Each log becomes a JSON object with the request id, taken from the X-Request-ID header, the user, the logger and the level. The Celery tasks keep the context of the request that sent them, but not their arguments, which can hold user data. The request logs of django-structlog replace request.summary and django.request, with the duration and the API error name. The access log gets the request id too. The tests now use the deployed logging.

Read the console lines as production writes them, to pin the fields a request or a task is traced by: request id, user, duration, API error name, traceback, and the request id carried over to the Celery tasks.

Most logs of a migration did not tell its workspace, so the logs of the migrations running at the same time could not be told apart. The workspace id is now bound to the context of the migration task.

A failed call to Drive or Resana could not be found in their logs. Every call now sends an X-Request-ID header, which Drive takes as the request id of its logs, and each attempt is logged with it, its status or error, and its duration. The attempts of a call keep the same id, including the retry loops outside the session: item creation, upload-ended and the Resana download. The S3 upload is logged without its path, which holds the file name.

The JSON lines of the deployed logs are hard to read in a terminal. The development configuration renders them as colored key-value lines, with the tracebacks in clear.

The client fixture of pytest-django already does what these tests need. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LzBgHnj8LE8FhuX6gT72hz

Name the module logger like the other modules, and the service one apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LzBgHnj8LE8FhuX6gT72hz

kernicPanel force-pushed the structlog branch from fa20348 to b16cb21 Compare October 27, 2026 12:41

kernicPanel requested a review from jmaupetit October 27, 2026 17:19

jmaupetit assigned kernicPanel Oct 28, 2026

jmaupetit added the enhancement New feature or request label Oct 28, 2026

martigan reviewed Nov 2, 2026

View reviewed changes

kernicPanel added 5 commits November 5, 2026 14:56

✅(back) assert the JSON logs of the requests and tasks

a3c521f

Read the console lines as production writes them, to pin the fields a request or a task is traced by: request id, user, duration, API error name, traceback, and the request id carried over to the Celery tasks.

🔊(back) log the workspace of a migration with each of its logs

0f63f67

Most logs of a migration did not tell its workspace, so the logs of the migrations running at the same time could not be told apart. The workspace id is now bound to the context of the migration task.

🔊(back) write readable log lines in development

2c4d5c0

The JSON lines of the deployed logs are hard to read in a terminal. The development configuration renders them as colored key-value lines, with the tracebacks in clear.

kernicPanel force-pushed the structlog branch from 726c0e7 to 2c4d5c0 Compare November 5, 2026 13:58

kernicPanel and others added 2 commits November 5, 2026 15:29

fixup! ✅(back) assert the JSON logs of the requests and tasks

8c81ba3

The client fixture of pytest-django already does what these tests need. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LzBgHnj8LE8FhuX6gT72hz

fixup! 🔊(back) log every outgoing call with the request id it sends

3289bff

Name the module logger like the other modules, and the service one apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LzBgHnj8LE8FhuX6gT72hz

kernicPanel requested a review from martigan November 5, 2026 15:25

martigan approved these changes Nov 9, 2026

View reviewed changes

Navigation Menu

Repository navigation

🔊(back) log as JSON with django-structlog - #289

🔊(back) log as JSON with django-structlog#289
kernicPanel wants to merge 7 commits into
mainfrom
structlog

kernicPanel commented Oct 27, 2026 •

edited

Loading

Uh oh!

martigan Nov 2, 2026

Uh oh!

martigan Nov 2, 2026

Uh oh!

kernicPanel Nov 5, 2026

Uh oh!

martigan Nov 2, 2026

Uh oh!

kernicPanel Nov 5, 2026

Uh oh!

martigan Nov 2, 2026

Uh oh!

kernicPanel Nov 5, 2026

Uh oh!

martigan Nov 2, 2026

Uh oh!

kernicPanel Nov 5, 2026

Uh oh!

martigan Nov 2, 2026

Uh oh!

kernicPanel Nov 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kernicPanel commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Proposal

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kernicPanel commented Oct 27, 2026 •

edited

Loading