You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@dbeatty10, seems like it was placed here as a thread and left for good without any intentions.
When we met at coalesce I was told you are planning for release early 2026, can you please shed some light?
Reacted by Juan Ramos , Arnau Villoro, Thiago Barreto, huynv3-vmo, pratikchavan3 and David Banquet
I'm interested in contributing the Athena Fusion adapter.
I currently use dbt extensively with Athena/Iceberg and would be happy to work on this.
Is there already ongoing/internal work on the Athena ADBC driver/connectivity layer?
If so, could we split #13822 into implementation parts similar to the SQL Server/ClickHouse adapter work, so I can pick one or several pieces?
I'm interested in contributing the Athena Fusion adapter. I currently use dbt extensively with Athena/Iceberg and would be happy to work on this. Is there already ongoing/internal work on the Athena ADBC driver/connectivity layer? If so, could we split #13822 into implementation parts similar to the SQL Server/ClickHouse adapter work, so I can pick one or several pieces?
Hey — I've had an Athena adapter running on dbt v2 for a couple of weeks and it's now far enough along to be worth poking at. It runs a real project end to end on a test environment: parse, compile, show, run (tables, views, incremental/merge), seeds, tests, docs. The headline is parse time — about 3 seconds against roughly 25 on dbt-core, which matters because parsing turns out to be a large share of the time our dbt pods actually spend.
Sharing a step-by-step here so anyone can reproduce it.
Trying the Athena adapter on dbt v2
1. Get the code
git clone -b spike/athena-profile-parse https://github.com/aoelvp94/dbt.git
cd dbt
The toolchain is pinned by rust-toolchain.toml, so rustup will fetch it. Expect around two hours cold. The result is target/release/dbt.
Two things that cost me time: on a 24 GB machine -j 4 fills swap and the compiler gets killed, so use -j 2; and on a Mac, closing the lid suspends the build no matter what caffeinate you wrapped it in.
3. Build the Athena driver
Connectivity is a separate ADBC driver, written in Go:
git clone https://github.com/dbt-labs/athena.git
cd athena/go/pkg && make # needs Go 1.26+
Put the resulting shared library in a lib/ directory next to the dbt binary — it's located by climbing up from the executable, so that placement matters.
4. Configure
export DBT_ALLOW_EXPERIMENTAL_ADAPTERS=1 # dbt v2 won't load a non-built-in adapter without itexport AWS_PROFILE=<your-profile>export AWS_DEFAULT_REGION=<your-region>
aws sso login --profile <your-profile>
Point it at any dbt project with an athena profile. If your profiles.yml templates values from env vars — staging dir, data dirs, workgroup — export those too. A handful of dbt-athena profile keys aren't supported yet and fail at connect time with a clear message: endpoint_url, poll_interval, num_boto3_retries, num_iceberg_retries, spark_work_group, skip_workgroup_check, lf_tags_database, and the assume_role_* family.
5. Read-only first
dbt parse
dbt ls --resource-type model
dbt compile --select <some_model>
dbt show --inline 'select 1 as n'
dbt debug
None of these write anything. parse is the one to watch — roughly 5× faster than dbt-core on a few hundred models.
6. Then a real build
dbt run --select <some_model>
This writes for real, to wherever your profile's target points, so send it somewhere disposable first. How you do that depends on the project — in mine the schema name is templated from an environment variable, so one variable redirects every model into a parallel set of throwaway schemas; if yours doesn't do something like that, point the target at a scratch schema instead. Either way, check where it's aimed before you start. Cleanup is just dropping the tables it made.
Working today: table, view, incremental with merge, seeds, tests, docs generate, and compile --write-catalog. insert_overwrite is implemented but lightly exercised, and dbt retry hasn't been tried at all.
If you want to read Rust, the two interesting areas are crates/dbt-adapter/src/metadata/athena/, where relation lookups and the docs catalog go through the Glue API rather than information_schema, and the dbt-athena macro package under crates/dbt-loader.
@aoelvp94 thanks for the mention. For the Glue and S3 calls I would keep them in the ADBC driver rather than add the AWS SDK to the engine: that is Part 6 (#16376), with the driver side in dbt-labs/athena#18 and #19. I am also in touch with an engineer on the Athena team and have asked for their view.
@aoelvp94 thanks for the mention. For the Glue and S3 calls I would keep them in the ADBC driver rather than add the AWS SDK to the engine: that is Part 6 (#16376), with the driver side in dbt-labs/athena#18 and #19. I am also in touch with an engineer on the Athena team and have asked for their view.
Ciao caro, I am OOO this week and next one but I can give a try on our progress so lmk if you need something from my side I can test code later :) Piacere
@aoelvp94 grazie, no rush! Meanwhile I opened three small PRs against your branches with what the end-to-end runs found: aoelvp94#1 (Part 4), aoelvp94#2 (Part 5) and aoelvp94#3 (Part 3). Have a look when you are back.
fpiped's stack built all 11 models on both loads and passed 23 of 24 tests, with Lake Formation, persist_docs, Hive insert_overwrite and a work group with enforced configuration. Parse took 0.6 s, against 1.8 s for dbt-core.
Activity
thiagocbarreto commentedon Jan 12, 2026
Hi! Any updates on this? My team just onboarded Fusion but we need the Athena adapter to finish a data migration process.
Please let me know if any help is needed to speed this up, as I'm game to help!
ttarom commentedon Jan 27, 2026
@dbeatty10, seems like it was placed here as a thread and left for good without any intentions.
When we met at coalesce I was told you are planning for release early 2026, can you please shed some light?
jjuanramos commentedon Jan 27, 2026
we are waiting on this, too 👀
ee07dazn commentedon Feb 10, 2026
any update on this guys ? we are awaiting this for long time! any rough ETA would be helpful please 🙏
xenogon commentedon Mar 27, 2026
bump
ttarom commentedon Apr 1, 2026
@dbeatty10 People are anxious, please shed some light, when can we expect this release to be live?
olexanderos commentedon Apr 1, 2026
If the blocker for you is lack of testers my team is ready to help 😃
David-Banquet commentedon Sep 24, 2026
I'm interested in contributing the Athena Fusion adapter.
I currently use dbt extensively with Athena/Iceberg and would be happy to work on this.
Is there already ongoing/internal work on the Athena ADBC driver/connectivity layer?
If so, could we split #13822 into implementation parts similar to the SQL Server/ClickHouse adapter work, so I can pick one or several pieces?
aoelvp94 commentedon Sep 24, 2026
Hey — I've had an Athena adapter running on dbt v2 for a couple of weeks and it's now far enough along to be worth poking at. It runs a real project end to end on a test environment: parse, compile, show, run (tables, views, incremental/merge), seeds, tests, docs. The headline is parse time — about 3 seconds against roughly 25 on dbt-core, which matters because parsing turns out to be a large share of the time our dbt pods actually spend.
Sharing a step-by-step here so anyone can reproduce it.
Trying the Athena adapter on dbt v2
1. Get the code
git clone -b spike/athena-profile-parse https://github.com/aoelvp94/dbt.git cd dbt2. Build the engine
brew install protobuf # prost-build needs protoc cargo build --release -p dbt-sa-cliThe toolchain is pinned by
rust-toolchain.toml, so rustup will fetch it. Expect around two hours cold. The result istarget/release/dbt.Two things that cost me time: on a 24 GB machine
-j 4fills swap and the compiler gets killed, so use-j 2; and on a Mac, closing the lid suspends the build no matter whatcaffeinateyou wrapped it in.3. Build the Athena driver
Connectivity is a separate ADBC driver, written in Go:
Put the resulting shared library in a
lib/directory next to thedbtbinary — it's located by climbing up from the executable, so that placement matters.4. Configure
Point it at any dbt project with an
athenaprofile. If yourprofiles.ymltemplates values from env vars — staging dir, data dirs, workgroup — export those too. A handful of dbt-athena profile keys aren't supported yet and fail at connect time with a clear message:endpoint_url,poll_interval,num_boto3_retries,num_iceberg_retries,spark_work_group,skip_workgroup_check,lf_tags_database, and theassume_role_*family.5. Read-only first
None of these write anything.
parseis the one to watch — roughly 5× faster than dbt-core on a few hundred models.6. Then a real build
This writes for real, to wherever your profile's target points, so send it somewhere disposable first. How you do that depends on the project — in mine the schema name is templated from an environment variable, so one variable redirects every model into a parallel set of throwaway schemas; if yours doesn't do something like that, point the target at a scratch schema instead. Either way, check where it's aimed before you start. Cleanup is just dropping the tables it made.
Working today:
table,view, incremental withmerge, seeds, tests,docs generate, andcompile --write-catalog.insert_overwriteis implemented but lightly exercised, anddbt retryhasn't been tried at all.The code is at https://github.com/aoelvp94/dbt/tree/spike/athena-profile-parse. I've split it into nine pieces that each compile on their own against
main, and seven are open upstream as PRs (#16365, #16367, #16368, #16369, #16370, #16374, #16375). @fpiped reviewed three of them and those findings are fixed.If you want to read Rust, the two interesting areas are
crates/dbt-adapter/src/metadata/athena/, where relation lookups and the docs catalog go through the Glue API rather thaninformation_schema, and the dbt-athena macro package undercrates/dbt-loader.fpiped commentedon Sep 25, 2026
@aoelvp94 thanks for the mention. For the Glue and S3 calls I would keep them in the ADBC driver rather than add the AWS SDK to the engine: that is Part 6 (#16376), with the driver side in dbt-labs/athena#18 and #19. I am also in touch with an engineer on the Athena team and have asked for their view.
aoelvp94 commentedon Sep 25, 2026
Ciao caro, I am OOO this week and next one but I can give a try on our progress so lmk if you need something from my side I can test code later :) Piacere
fpiped commentedon Sep 25, 2026
@aoelvp94 grazie, no rush! Meanwhile I opened three small PRs against your branches with what the end-to-end runs found: aoelvp94#1 (Part 4), aoelvp94#2 (Part 5) and aoelvp94#3 (Part 3). Have a look when you are back.
David-Banquet commentedon Sep 25, 2026
@fpiped thanks for the links. A short recap for people following here; the details are on #16252.
main) and ran them on the same Iceberg and Hive test project as dbt-core 1.11.15 + dbt-athena 1.11.0. The project is at https://github.com/David-Banquet/dbt-athena-bench and the comparison at [Feature] Athena adapter for dbt Core 2.0 #16252 (comment).column_typeskeys written in upper case are ignored when v2 lower-cases the column names. Issue [v2 Bug] Seedcolumn_typeskeys are matched case-sensitively against lower-cased column names #16459, fix in fix(seed): match column_types keys to seed column names case-insensitively #16460.list_relationscompile error I hit while assembling the stack is the rebase fpiped already flagged on feat(athena): Part 5 — AthenaMetadataAdapter and athena_get_relation #16374.