Conversation
|
Hi, I noticed the same window + DISTINCT pattern is in stg_ga4__client_key_first_last_events too, just doubled up: it uses FIRST_VALUE and LAST_VALUE for the first and last event, then DISTINCT, then joins back to stg_ga4__events twice to pull the geo/device/source columns onto each side. I've made a version that moves it over to the same qualify row_number() = 1 pattern you used in #392. Two CTEs (first event, last event) that keep the whole row, joined on client_key, so both self-joins and the DISTINCT drop out, same memory optimisation you described there. Same columns, grain, and one row per client_key, and it passes the existing .py test in the repo. Happy to open a PR if that's useful. If not, no problem, happy to wait until #392 lands and align with that! |
mtcarlone
left a comment
There was a problem hiding this comment.
not sure if this a real PR, but lgtm
|
If this is live, then let's remove the draft status and I can release this on Sept. 8 |
Description & motivation
Replaced
FIRST_VALUE() OVER (window)+DISTINCTwithqualify row_number() = 1.This matches how dim_ga4__sessions works and is more efficient. The previous approach was causing memory issues with one client while this approach works for them.
Why the old approach was expensive:
So for a session with 50 events, BigQuery materialized 50 identical rows just to throw away 49.
The qualify approach: BigQuery sorts each partition, assigns row numbers, and filters to keep only row 1 — no intermediate blowup, no second dedup pass.
Checklist
dbt testandpython -m pytest .to validate existing tests