Backfill clones

[dgitis]

Description & motivation

This PR adds a utility in the analyses folder that generates clone commands that can be copied and pasted into BigQuery and run there.

The way that dbt implements clone commands makes them run very slowly. It regularly takes 5-7 minutes for them to run in dbt when they take less than one minute when run directly in BigQuery.

This creates a problem with running full refreshes as they base_ga4__events model, which clones partitions before running in a pre-hook, will time out once you have more than a few days worth of data.

As a result, we recommended not refreshing the base model in a multi-site implementation.

This now begs the question of what to do when you need to refresh the data regardless of what we recommend and it is this situation that this PR addresses.

The new analysis file lets people generate the code to refresh one property at a time and between certain dates. It also batches and delays refreshes so that it handles BigQuery processing limits for you and users don't need to manually manage cloning limits.

In order to facilitate this, I've refactored the combine_property_data macro so that it supports using the same code when run as pre-hook from base_ga4__events and when run in the clone_backfill analysis. The original macro combined logic that wouldn't work with the analysis that needs to work on specific datasets and on a configurable date range.

Additionally, once the clone has been run, we need to be able to full refresh the base_ga4__events model without triggering the clone pre-hook. This PR adds a variable that lets you manually disable the clone operation pre-hook so that you can generate the clones that you need using the analysis and then run dbt run --select base_ga4__events+ --full-refresh --vars '{clone_disabled: true}' in order to update the project with new or fixed clones.

Checklist

• [y] I have verified that these changes work locally
• [y] I have updated the README.md (if applicable)
• I have added tests & descriptions to my models (and macros if applicable)
• [y] I have run dbt test and python -m pytest . to validate existing tests

[dgitis]

@mtcarlone, I haven't added tests to this yet. I wanted to check with you how you think we should handle tests here.

I suspect the multi-site functionality is used only on a limited number of sites. If we add a test that runs by default that tests the cloning, we end up testing something that people aren't using.

I think we definitely want to test this in CI/CD (specifically the macros and cloning in the pre-hook), but I'm not sure that we want this to run in other situations.

What are your preferences with respect to testing?

We weren't testing the multi-site features at all before so pretty much anything is an upgrade.

[mtcarlone]

@dgitis I think you're spot on with respect to testing. Multi-site seems like a nice to have if the lift isn't great, but if the use cases are limited it can be something implemented later on / if there becomes a larger need to tighten up that functionality.

In addition to the macros/cloning operations in general, one suggestion would be to also test the output of the compiled sql. A "simple" (in quotes, because it might be more complex than I think) way would be to stage an expected output in the repo, and test the output of the compiled sql against that (1:1 match, noting that whitespace might be a little funny at times). In a slightly different manner, we can use the sqlglot package to test that the compiled sql parses, meaning we can assume that the compiled sql is valid.

Just a few thoughts, but otherwise I'm in agreement with you! Lmk if there's anything I can clarify or if there's anything else I can answer/help with!

[dgitis]

I added code to skip the sqlglot tests when sqlglot is not present. This is only really a problem right now because the test runner is running off of the workflows defined in the main branch and I'm adding sqlglot in this branch.

The sqlglot tests pass in my local environment.

Otherwise, this is good to merge now.

[dgitis]

I have now tested this on a reasonably large set of clones (400+).

This new version fixes an issue with the batch operation that wasn't working and it adds support for calling the clones via dbt run-operation.

I've tested both of these new features.

@mtcarlone, please review this when you have time.

[mtcarlone]

lgtm!

[mtcarlone]

This will go out September 8, end of day.