Repository navigation
Conversation
3 tasks done
This was referenced Sep 19, 2026
This was referenced Sep 20, 2026
Author
|
@aoelvp94 I saw Your branch has fixes this PR needs whichever way the client question goes: case folding of quoted identifiers in the relation cache, the missing-table probe before |
fpiped
force-pushed
the
feat/athena-adapter-methods
branch
from
September 20, 2026 13:59
2de6773 to
eebac86
Compare
fpiped
force-pushed
the
feat/athena-adapter-methods
branch
from
September 25, 2026 07:51
eebac86 to
1636ccf
Compare
This was referenced Sep 25, 2026
Open
aoelvp94
pushed a commit
to aoelvp94/dbt
that referenced
this pull request
Sep 27, 2026
As dbt-athena does after the DDL; adapter.add_lf_tags_to_database is implemented in dbt-labs#16376. (cherry picked from commit 8282da8)
fpiped
force-pushed
the
feat/athena-adapter-methods
branch
from
September 27, 2026 21:34
32cf21e to
eabbb99
Compare
aoelvp94
added a commit
to aoelvp94/dbt
that referenced
this pull request
Sep 28, 2026
The dispatch claimed "the Glue API serves only the Data Catalog". That is wrong: S3 Tables is a Glue catalog too, addressed as `CatalogId = <account>:s3tablescatalog/<bucket>`, and so are the catalogs registered in Athena, which dbt-athena resolves in `_get_data_catalog`. The behaviour is unchanged and still correct — those catalogs fall back to their own `information_schema`, which reads the right metadata, just with one billed query per lookup instead of a free API call. What was wrong was the reason given for it. Addressing a non-default catalog needs the account id, which needs an STS or Athena client this adapter does not have, and adding another AWS SDK to the engine for a transport that dbt-labs#16376 replaces is not worth it: that PR resolves the CatalogId through the ADBC driver and reads Glue for every Glue catalog. Reported by @fpiped on dbt-labs#16252. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aoelvp94
pushed a commit
to aoelvp94/dbt
that referenced
this pull request
Sep 28, 2026
As dbt-athena does after the DDL; adapter.add_lf_tags_to_database is implemented in dbt-labs#16376.
aoelvp94
pushed a commit
to aoelvp94/dbt
that referenced
this pull request
Oct 5, 2026
As dbt-athena does after the DDL; adapter.add_lf_tags_to_database is implemented in dbt-labs#16376.
aoelvp94
added a commit
to aoelvp94/dbt
that referenced
this pull request
Oct 5, 2026
A catalog name reaches the adapter bare or already quoted depending on the call path, and the Glue dispatch compared it as-is. A quoted `"awsdatacatalog"` fails the comparison, so every relation and column lookup falls back to `information_schema` — correct results, but one billed query per lookup instead of a free Glue call, and nothing anywhere says so. Rendering is worse: a quoted name re-quoted gives `"""catalog"""`, a syntax error. Both sites now strip the quotes, in `catalog_is_glue` / `list_relations_sql` and in `athena__get_columns_in_relation`. Found by @David-Banquet on dbt-labs#16252, in dbt-labs#16376's Glue path: `AthenaOps::data_catalog` compares the quoted name and `athena.get_data_catalog` then fails on every run. Ours is the quiet version of the same mistake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fpiped
force-pushed
the
feat/athena-adapter-methods
branch
from
October 10, 2026 22:08
eabbb99 to
36d38c0
Compare
![codescene-delta-analysis[bot]](https://avatars.githubusercontent.com/in/53921?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
4 tasks done
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…cros The dbt-athena macro package calls 21 `adapter.*` methods that exist only on the Python `AthenaAdapter`; without them every materialization fails at the first call. This ports them: the pure helpers (`generate_s3_location`, partition formatting, `murmur3_hash`, `TableType`), the Glue / S3 / Athena / STS backed ones, and the two statement runners (`run_query_with_partitions_limit_catching`, `run_operation_with_potential_multiple_runs`). The AWS calls go through the ADBC driver, as the BigQuery adapter does for `copy_table`: a statement with `athena.operation` / `athena.operation.payload` runs the named API call with the connection's credentials and answers with the API response as one JSON cell (dbt-labs/athena, `operations.go`). The composition — catalog ids, which versions to expire, the `swap_table` update and partition move, the docs diff — follows `impl.py` here. Lake Formation stays out: disabled configs are a no-op, enabled ones raise NotSupported. Also what the macros need around them: `relation.render_hive()` / `render_pure()`, `Relation.create(..., s3_path_table_part=...)`, `quote_seed_column`'s `quote_character`, an Athena arm in `build_catalog_relation`, dbt-athena's query comment rules (line comment, none ahead of ALTER / DROP / OPTIMIZE / VACUUM / MSCK), and the time-machine registry entries.
athena__persist_docs passes `for_columns and config.persist_column_docs() and model.columns`, which in Jinja is the columns mapping itself, not a bool; dbt-athena reads it as truthiness.
The driver puts the query statistics in the result schema metadata (`ATHENA:query_id`, `ATHENA:Statistics:DataScannedInBytes`, `ATHENA:UpdateCount`, dbt-labs/athena). The Athena response now reads them the way `AthenaConnectionManager.get_response` builds it: code `OK`, message `OK <rows>`, the rows being Athena's UpdateCount for a CTAS / INSERT / MERGE / DELETE and the fetched rows otherwise, and `data_scanned_in_bytes`. The query id reaches `query_id` and the query log. `run_query_with_partitions_limit_catching` reads the bytes from the same key instead of BigQuery's `bytes_processed`.
… work group check - `generate_s3_location` falls back to `schema_table_unique`, the `AthenaCredentials.s3_data_naming` default, when neither the model nor the profile names a strategy. - `is_work_group_output_location_enforced` answers false without calling GetWorkGroup when the profile names no work group or sets `skip_workgroup_check`.
…n's report
The dbt-athena table materialization runs its CTAS through
`run_query_with_partitions_limit_catching` and ends with
`SELECT '{"rowcount":99,"data_scanned_in_bytes":1552}'` as the `main`
statement. `AthenaConnectionManager.process_query_stats` takes the response
from that JSON object; the Athena response now does the same, so a table
model reports the CTAS rows and bytes instead of the reporting SELECT's.
`submit_python_job` follows dbt-athena's `AthenaPythonJobHelper`: the compiled model runs as a calculation in a Spark session of the profile's `spark_work_group`, through the driver's `athena.start_session`, `athena.get_session_status`, `athena.start_calculation_execution`, `athena.get_calculation_execution` and `athena.stop_calculation_execution` operations (dbt-labs/athena#19). - Model config as `AthenaSparkSessionConfig`: `timeout` (43200 s), `polling_interval` (the profile's `poll_interval`, else 1 s), `engine_config` over the defaults (1 / 2 / 1 DPU), Spark properties for `table_type` iceberg / hudi / delta_lake and the `spark_encryption`, `spark_cross_account_catalog`, `spark_requester_pays` flags; the same validation errors. - Sessions are shared by the process: a model takes a free session with the same engine configuration or starts one (10 minutes idle timeout). A session that ended is dropped and the calculation starts in a new one. - A failed or cancelled calculation reports its id, session, reason and stderr S3 path; a timeout or Ctrl-C stops the calculation.
`add_lf_tags`, `apply_lf_grants` and `add_lf_tags_to_database` follow dbt-athena's `LfTagsManager` and `LfPermissions` through the driver's `lakeformation.*` operations: - table and column LF-Tags become the `lf_tags_config` ones: stale column and table tags are removed, `inherited_tags` are left alone; - the table's data cell filters are created, updated or deleted to match `lf_grants.data_cell_filters`, and each filter's SELECT grantees become its `principals`; - the profile's `lf_tags_database` tags the schema. The configs parse as the pydantic models do, and a failure the API reports per tag or per grant fails the call.
…el relation create_schema passes a relation without an identifier.
As dbt-athena's `GlueCatalogIntegration` and `S3TablesCatalogIntegration`: - catalogs v2 accepts an `athena` block on `glue` catalogs (`catalog_database`, `external_volume`, `file_format`) and a new `s3_tables` type, Athena only, whose `catalog_database` must be `s3tablescatalog/<table-bucket>` and which forbids `external_volume` (S3 Tables manages the storage); - `build_catalog_relation` builds the Iceberg relation from the block, so `generate_database_name` routes a `catalog_name` model to the catalog's database; models without `catalog_name` keep the Glue default; - Athena model configs are read off `config`, since Athena nodes carry no adapter attribute; - `adapter.is_s3_tables_database(database)` for the dbt-athena macros.
…g for Athena stops a calculation on a best-effort basis: on the model timeout the calculation went CANCELING and then COMPLETED, writing its table after dbt had reported the failure, and on Ctrl-C the stop call itself was refused as the run was cancelled. An abandoned calculation is now stopped and its session terminated (athena.terminate_session) and dropped from the pool, through driver calls that go out after cancellation.
dbt-athena answers list_relations_without_caching, get_columns_in_relation, list_schemas and check_schema_exists with Glue API calls for every Glue catalog (awsdatacatalog, S3 Tables, registered Glue catalogs) and uses SQL only for data catalogs outside Glue. The same reads now go through the driver's glue.get_tables / get_table / get_databases / get_database: no Athena query per lookup, and the Hive types the macros expect. Outside Glue, columns and schemas keep the macros' SQL; relations are not listed.
Columns now come from Glue in Hive types. AthenaColumn turns them into the Trino spelling queries use: is_string covers string and varchar, data_type maps string to varchar, binary to varbinary, array<t> to array(t) and an Iceberg timestamp to timestamp(6); string_size is 0 without a length. Without it, a query cast such as dbt_utils.union_relations renders cast(null as string) and Athena rejects it.
On Ctrl-C the stop and the terminate went out in that order, and the process could exit before the second call: the calculation, stopped only on a best-effort basis, ran to completion and wrote its table. Terminating the session ends the calculation, so it goes first; the stop follows, and the failures of both are logged instead of dropped.
filter_missing_schemas passes the quoted catalog name, so data_catalog missed awsdatacatalog and called athena.get_data_catalog with the quotes, which fails on every run.
dbt-labs/athena renames them after review, to match its field metadata key ATHENA:type: ATHENA:query_execution_id, ATHENA:data_scanned_in_bytes and ATHENA:update_count. The update count of a SELECT is 0; the fetched rows still count for it.
main reports an unknown row count as unset instead of 0 (known_rows_affected). Athena knows the rows a query returns and, through its update count, the rows a CTAS, INSERT, MERGE or DELETE wrote; DDL reports none, so its count is now unknown. A reporting SELECT whose stats cannot be read leaves the count unset instead of its own row, and run_query_with_partitions_limit_catching reports dbt-athena's -1 for an unknown count.
fpiped
force-pushed
the
feat/athena-adapter-methods
branch
from
October 14, 2026 07:42
60872b2 to
d8ae091
Compare
There was a problem hiding this comment.
Gates Failed
(3 new files with code health below 9.00)
(5 files with Low Cohesion, Bumpy Road Ahead, Deep, Nested Complexity, Brain Method)
(15 files with Code Duplication, Complex Method, Large Method, Excess Number of Function Arguments, Complex Conditional, String Heavy Function Arguments, Large Assertion Blocks, Lines of Code in a Single File)
Our agent can fix these. Install it.
Gates Passed
Reason for failure
| New code is healthy | Violations | Code Health Impact | |
|---|---|---|---|
| driver_ops.rs | 7 rules | 6.21 | Suppress |
| dispatch.rs | 7 rules | 6.79 | Suppress |
| mod.rs | 3 rules | 8.03 | Suppress |
| Enforce critical code health rules | Violations | Code Health Impact | |
|---|---|---|---|
| driver_ops.rs | 3 critical rules | 6.21 | Suppress |
| dispatch.rs | 3 critical rules | 6.79 | Suppress |
| lakeformation.rs | 1 critical rule | 9.11 | Suppress |
| python.rs | 1 critical rule | 9.24 | Suppress |
| types.rs | 1 critical rule | 5.23 → 4.73 | Suppress |
| Enforce advisory code health rules | Violations | Code Health Impact | |
|---|---|---|---|
| driver_ops.rs | 4 advisory rules | 6.21 | Suppress |
| dispatch.rs | 4 advisory rules | 6.79 | Suppress |
| mod.rs | 3 advisory rules | 8.03 | Suppress |
| lakeformation.rs | 2 advisory rules | 9.11 | Suppress |
| python.rs | 2 advisory rules | 9.24 | Suppress |
| types.rs | 3 advisory rules | 5.23 → 4.73 | Suppress |
| metadata.rs | 1 advisory rule | 9.69 | Suppress |
| semantic.rs | 1 advisory rule | 9.34 → 9.22 | Suppress |
| catalog_relation.rs | 3 advisory rules | 3.08 → 2.98 | Suppress |
| mod.rs | 1 advisory rule | 3.32 → 3.28 | Suppress |
| relation_object.rs | 1 advisory rule | 6.33 → 6.30 | Suppress |
| dbt_catalogs.rs | 1 advisory rule | 3.56 → 3.53 | Suppress |
| relation_impl.rs | 1 advisory rule | 4.70 → 4.69 | Suppress |
| adapter_engine.rs | 1 advisory rule | 7.53 → 7.52 | Suppress |
| adapter_impl.rs | 1 advisory rule | 1.80 → 1.79 | Suppress |
Quality Gate Profile: Clean Code Collective
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
Comment on lines
+344
to
+373
| fn table_prefix_defaults_to_staging_dir_tables() { | ||
| assert_eq!( | ||
| s3_table_prefix("s3://b/stage/", None, None, None, false), | ||
| "s3://b/stage/tables" | ||
| ); | ||
| assert_eq!( | ||
| s3_table_prefix("s3://b/stage", None, Some("s3://d/data"), None, false), | ||
| "s3://d/data" | ||
| ); | ||
| assert_eq!( | ||
| s3_table_prefix( | ||
| "s3://b/stage", | ||
| Some("s3://t/tmp"), | ||
| Some("s3://d/data"), | ||
| None, | ||
| true | ||
| ), | ||
| "s3://t/tmp" | ||
| ); | ||
| assert_eq!( | ||
| s3_table_prefix( | ||
| "s3://b/stage", | ||
| Some("s3://t/tmp"), | ||
| Some("s3://d/data"), | ||
| None, | ||
| false | ||
| ), | ||
| "s3://d/data" | ||
| ); | ||
| } |
There was a problem hiding this comment.
❌ New issue: Large Assertion Blocks
The test suite contains 9 assertion blocks with at least 4 assertions, threshold = 4
Comment on lines
+4496
to
+4525
| // Athena: dbt-athena's `AthenaAdapter` methods, see `adapter/athena`. | ||
| "is_list" => self.athena_is_list(args), | ||
| "is_s3_tables_database" => self.athena_is_s3_tables_database(args), | ||
| "format_value_for_partition" => self.athena_format_value_for_partition(args), | ||
| "format_one_partition_key" => self.athena_format_one_partition_key(args), | ||
| "format_partition_keys" => self.athena_format_partition_keys(args), | ||
| "murmur3_hash" => self.athena_murmur3_hash(args), | ||
| "generate_s3_location" => self.athena_generate_s3_location(args), | ||
| "get_glue_table_type" => self.athena_get_glue_table_type(state, args), | ||
| "clean_up_table" => self.athena_clean_up_table(state, args), | ||
| "delete_from_glue_catalog" => self.athena_delete_from_glue_catalog(state, args), | ||
| "drop_glue_database" => self.athena_drop_glue_database(state, args), | ||
| "expire_glue_table_versions" => self.athena_expire_glue_table_versions(state, args), | ||
| "swap_table" => self.athena_swap_table(state, args), | ||
| "clean_up_partitions" => self.athena_clean_up_partitions(state, args), | ||
| "delete_from_s3" => self.athena_delete_from_s3(state, args), | ||
| "is_work_group_output_location_enforced" => { | ||
| self.athena_is_work_group_output_location_enforced(state, args) | ||
| } | ||
| "upload_seed_to_s3" => self.athena_upload_seed_to_s3(state, args), | ||
| "run_query_with_partitions_limit_catching" => { | ||
| self.athena_run_query_with_partitions_limit_catching(state, args) | ||
| } | ||
| "run_operation_with_potential_multiple_runs" => { | ||
| self.athena_run_operation_with_potential_multiple_runs(state, args) | ||
| } | ||
| "persist_docs_to_glue" => self.athena_persist_docs_to_glue(state, args), | ||
| "add_lf_tags" => self.athena_add_lf_tags(state, args), | ||
| "apply_lf_grants" => self.athena_apply_lf_grants(state, args), | ||
| "add_lf_tags_to_database" => self.athena_add_lf_tags_to_database(state, args), |
There was a problem hiding this comment.
❌ Getting worse: Complex Method
Adapter.call_method_impl already has high cyclomatic complexity, and now it increases in Lines of Code from 463 to 492
Comment on lines
+170
to
+171
| // Athena: `AthenaRelation.s3_path_table_part` | ||
| let s3_path_table_part = iter.next_kwarg::<Option<String>>("s3_path_table_part")?; |
There was a problem hiding this comment.
❌ Getting worse: Large Method
RelationStatic.create increases from 99 to 104 lines of code, threshold = 70
Comment on lines
+430
to
+434
| Some("s3_path_table_part") => Some( | ||
| s3_path_table_part(self.relation.as_ref()) | ||
| .map(Value::from) | ||
| .unwrap_or_else(none_value), | ||
| ), |
There was a problem hiding this comment.
❌ Getting worse: Large Method
RelationObject.get_value increases from 73 to 78 lines of code, threshold = 70
| static COLLATION_RE: Lazy<Regex> = | ||
| Lazy::new(|| Regex::new(r"(?i)\s+COLLATE\s+'([^']+)'").expect("A valid regex")); | ||
|
|
||
| /// `AthenaColumn.array_inner_type`: the element type of `array<t>` or `array(t)`. |
There was a problem hiding this comment.
❌ New issue: Lines of Code in a Single File
This module has 1017 lines of code, improve code health by reducing it to 1000
| numeric_precision: None, | ||
| numeric_scale: None, | ||
| collation: None, | ||
| _iceberg: false, |
There was a problem hiding this comment.
❌ Getting worse: Large Method
Column.try_from_snowflake_raw_data_type increases from 94 to 96 lines of code, threshold = 70
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves #16252 (partial)
Problem
The dbt-athena macro package (#16369) calls 21
adapter.*methods that exist only on the PythonAthenaAdapter(impl.py). None of them existed in the Rust adapter, so every materialization failed: debug builds panicked intime_machine/semantic.rs(Unknown adapter method 'upload_seed_to_s3'), release builds hitUnknown method on adapter object.view.sqlcallsexpire_glue_table_versionsunconditionally, so even a plain view could not run.Solution
crates/dbt-adapter/src/adapter/athena/:mod.rs: the pure methods, ported fromimpl.pyandutils.py:generate_s3_location(all fiveS3DataNaminglayouts),format_value_for_partition,format_one_partition_key,format_partition_keys,murmur3_hash(MurmurHash3 x86_32 asmmh3.hash, checked against the Iceberg reference vectors),is_list,TableType/get_table_type, thepersist_docsstring helpers.driver_ops.rs: the Glue / S3 / Athena / STS calls go through the ADBC driver, as the BigQuery adapter does forcopy_table: a statement withathena.operation/athena.operation.payloadruns the named AWS API call with the connection's credentials and returns the API response as one JSON cell (feat(go): run Glue, S3 and STS operations behind statement options athena#19). The payload and the response are the AWS API JSON shapes; the composition followsimpl.pyhere:CatalogIdresolution as_get_data_catalog(awsdatacatalogands3tablescatalog/*from the STS account id, anything else throughGetDataCatalog), which versionsexpire_glue_table_versionsdeletes, theswap_tableupdate and partition move, the expression batching ofclean_up_partitions, the docs diff ofpersist_docs_to_glue.dispatch.rs: theadapter.<method>entry points, plusrun_query_with_partitions_limit_catchingandrun_operation_with_potential_multiple_runsover the adapter'sexecute.lakeformation.py):add_lf_tagsmakes the table and column LF-Tags thelf_tags_configones (stale tags removed,inherited_tagsleft alone),apply_lf_grantsmakes the table's data cell filters and their SELECT grantees thelf_grantsones,add_lf_tags_to_databasetags the schema with the profile'slf_tags_database; through the driver'slakeformation.*operations. A disabled config is a no-op.generate_s3_locationfalls back toschema_table_unique(theAthenaCredentials.s3_data_namingdefault);is_work_group_output_location_enforcedanswers false without an API call when the profile names no work group or setsskip_workgroup_check.Around it, what the macros need from the rest of the adapter:
relation.render_hive()/relation.render_pure()andRelation.create(..., s3_path_table_part=...)(AthenaRelation).quote_seed_column(column, quote_config, quote_character): dbt-athena's third argument.build_catalog_relationfor Athena withoutcatalogs.yml: Glue, table format fromtable_type/table_format. Withcatalogs.ymlv2, as dbt-athena'sGlueCatalogIntegration/S3TablesCatalogIntegration: anathenablock ongluecatalogs (catalog_database,external_volume,file_format) and a news3_tablestype, Athena only (catalog_database: s3tablescatalog/<table-bucket>required,external_volumeforbidden);external_volumeandfile_formatreach the macros on the catalog relation, andadapter.is_s3_tables_databaseis exposed.Query comments follow
_AthenaQueryComment.add:-- /* ... */line comment, none ahead ofALTER/DROP/OPTIMIZE/VACUUM/MSCK(Athena rejects a block comment there; verified against the API).The 21 names registered in the time-machine
SemanticCategorytable.Python models (
submit_python_job) followAthenaPythonJobHelper: the model runs as a calculation in a Spark session of the profile'sspark_work_group, through the driver'sathena.*_session/athena.*_calculation_executionoperations. Model config asAthenaSparkSessionConfig(timeout,polling_interval,engine_config, the table format andspark_*Spark properties, the same validation errors). Sessions are shared within the process: a model takes a free session with the same engine configuration or starts one, and a session that ended is replaced. A failed calculation reports its id, session, reason and stderr S3 path; a timeout or Ctrl-C terminates its session and then stops it: Athena stops a calculation on a best-effort basis (a stopped one has been seen to run to completion and write its table), while a terminated session ends it.The adapter response follows
AthenaConnectionManager.get_response: codeOK, messageOK <rows>,data_scanned_in_bytes, and the query id. The driver reports the statistics in the result schema metadata (ATHENA:query_execution_id,ATHENA:data_scanned_in_bytes,ATHENA:update_count, feat(go): report query statistics in the result schema metadata athena#22); the rows are Athena's UpdateCount for a CTAS / INSERT / MERGE / DELETE, the fetched rows otherwise, and the table materialization's reportingSELECT '{"rowcount":..,"data_scanned_in_bytes":..}'is read asprocess_query_statsdoes. A driver without the metadata leaves the bytes at 0 and the rows at the fetched count.Metadata reads follow dbt-athena:
list_relations_without_caching,get_relation,get_columns_in_relation,list_schemasandcheck_schema_existsread Glue for every Glue catalog, S3 Tables and Glue catalogs registered in Athena included (_get_data_catalog), through the driver'sglue.get_tables/get_table/get_databases/get_database: no Athena query per lookup. Columns keep Glue's Hive types (current Iceberg columns only, partition keys last), andColumnfollowsAthenaColumnfor them:stringis a string,data_typerendersvarchar,varbinary,array(...)and, on Iceberg,timestamp(6). Outside Glue, columns and schemas keep the SQL macros.No new dependencies. Requires a driver with the operations (dbt-labs/athena#19); an older driver rejects the
athena.operationoption and the first materialization fails with that message. #16369's S3 Tables macros calladapter.is_s3_tables_database, which this PR adds, so this PR lands before #16369 or with it.Testing
Unit tests for the pure helpers, the dispatch arms, the relation additions, the catalog relation arm and the comment rule.
End to end against a real Athena workgroup with the whole stack (#16365, #16274, #16367, #16368, #16369, #16374, #16370, #16375) on the jaffle_shop project plus Iceberg, incremental (append / insert_overwrite with partitions / Iceberg merge) and snapshot (check strategy) models:
seed,run(views, tables, repeated runs),test,build(28/28),snapshot,docs generate. Checked in Glue / S3, not only exit codes: seed CSVs uploaded undertables/<name>/<uuid>/and removed afterwards, no temp Glue tables left, view versions expire to the newest four, dropped tables leave no S3 data, OPTIMIZE / VACUUM ran as bare statements, the Iceberg table sits undertables/ice_orders/<uuid>withtable_type=ICEBERG.Since then the same stack runs a 121-check matrix on the work group (last run 2026-10-13, all passed), each check read back from Glue / S3 / Athena:
hatables (swap_table), 150-partition batching, Iceberg bucket partitions, microbatch, Iceberg merge,on_schema_change: append_new_columns,persist_docsto Glue, view version expiry, snapshots (check and timestamp), unit tests, source freshness,compile --write-catalog, Athena Lambda UDFs in tables, Python models on a Spark work group (Spark and pandas DataFrames, Hive and Iceberg, incremental append), Lake Formation (schema tags on a schema the run creates, table and column tags and a data cell filter with its grantee, then a second config they converge to),external_locationand the defaultschema_table_uniquelayout on a work group without enforced output,skip_workgroup_check, therun_results.jsonadapter response (OK 99, bytes scanned, query id), Ctrl-C leaving the running queryCANCELLED, a Python model's timeout or Ctrl-C ending its calculation and session, and concurrent Iceberg writes rerun onICEBERG_COMMIT_ERROR. It needs Part 7 (#16452:Column.quoted_hive, case folding of quoted identifiers) and the driver's query statistics (dbt-labs/athena#20).catalogs.ymlv2 is exercised in a separate project withuse_catalogs_v2on a real table bucket: a Glue Iceberg model lands under the catalog'sexternal_volume, S3 Tables table and incremental models are written intos3tablescatalog/<bucket>, rebuilt through Glue and appended to. That run needs three changes to other parts, proposed as aoelvp94#1 and aoelvp94#2: #16369's macros brought to dbt-adapters62a5495cf(S3 Tables), andlist_relations(#16374) andathena__get_columns_in_relation(#16369) reading"<catalog>".information_schema, since Athena'sinformation_schemacovers only the current catalog.athena__create_schemain #16369 does not calladd_lf_tags_to_databaseyet; aoelvp94#1 restores the call as dbt-athena has it.The metadata reads were measured on it: no
information_schemaquery outside the catalog query ofcompile --write-catalog(145 before), column types identical to dbt-core 1.11 + dbt-athena 1.11 on 25 columns of a Hive and an Iceberg table, anon_schema_changethat adds a struct column passes (frominformation_schematypes theALTERrenderedrow(...)and Athena refused it), and on a 250-model project the two runs take 72 s and 92 s (75 s and 103 s before). The bench is public: fpiped/dbt2-athena-lab.Findings on the other parts that the run needed as local fixes are reported on their PRs.
Checklist