Skip to content

improve efficiency of dim_ga4__sessions_daily - #392

Draft
dgitis wants to merge 3 commits into
mainfrom
session-window
Draft

dgitis wants to merge 3 commits into
mainfrom
session-window

Conversation

@dgitis

@dgitis dgitis commented Feb 12, 2026

Copy link
Copy Markdown
Collaborator

Description & motivation

Replaced FIRST_VALUE() OVER (window) + DISTINCT with qualify row_number() = 1.

This matches how dim_ga4__sessions works and is more efficient. The previous approach was causing memory issues with one client while this approach works for them.

Why the old approach was expensive:

  • The window function computed the "first value" for every row in the partition, attaching a copy of the result to each row
  • All those rows (with duplicated first-values) were materialized in memory as an intermediate result
  • Then DISTINCT made a second pass to collapse them back down

So for a session with 50 events, BigQuery materialized 50 identical rows just to throw away 49.

The qualify approach: BigQuery sorts each partition, assigns row numbers, and filters to keep only row 1 — no intermediate blowup, no second dedup pass.

Checklist

  • [ y ] I have verified that these changes work locally
  • [ n/a ] I have updated the README.md (if applicable)
  • [ n/a ] I have added tests & descriptions to my models (and macros if applicable)
  • [ y ] I have run dbt test and python -m pytest . to validate existing tests

@dgitis
dgitis requested a review from mtcarlone February 12, 2026 00:07
@michaelf108

Copy link
Copy Markdown

Hi, I noticed the same window + DISTINCT pattern is in stg_ga4__client_key_first_last_events too, just doubled up: it uses FIRST_VALUE and LAST_VALUE for the first and last event, then DISTINCT, then joins back to stg_ga4__events twice to pull the geo/device/source columns onto each side.

I've made a version that moves it over to the same qualify row_number() = 1 pattern you used in #392. Two CTEs (first event, last event) that keep the whole row, joined on client_key, so both self-joins and the DISTINCT drop out, same memory optimisation you described there. Same columns, grain, and one row per client_key, and it passes the existing .py test in the repo.

Happy to open a PR if that's useful. If not, no problem, happy to wait until #392 lands and align with that!

@mtcarlone mtcarlone left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if this a real PR, but lgtm

@mtcarlone

Copy link
Copy Markdown
Collaborator

If this is live, then let's remove the draft status and I can release this on Sept. 8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants