perf(dashboard): batch dataset lookups during dashboard export - #43017
perf(dashboard): batch dataset lookups during dashboard export#43017ColtenOuO wants to merge 1 commit into
Conversation
ExportDashboardsCommand issued a separate DatasetDAO.find_by_id query (and, for _export, a separate ExportDatasetsCommand run) per native filter / chart customization target, even when multiple targets referenced the same dataset — a common case since one dataset is often targeted by several filters. Batch the lookups with find_by_ids and deduplicate before exporting, so each referenced dataset is queried and exported exactly once regardless of how many targets reference it.
dosubot
Bot
added
dashboard:export
Code Review Agent Run #93725dActionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #43017 +/- ##
=======================================
Coverage 66.42% 66.42%
=======================================
Files 2861 2861
Lines 161614 161615 +1
Branches 37223 37222 -1
=======================================
+ Hits 107349 107355 +6
+ Misses 52223 52217 -6
- Partials 2042 2043 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 participant
SUMMARY
ExportDashboardsCommand._file_contentand_exportissued a separateDatasetDAO.find_by_idquery per native-filter target and per chart-customization target, even when multiple targets referenced the same dataset — a common case, since one dataset is often targeted by several filters on the same dashboard._exportcompounded this by re-running a fullExportDatasetsCommand([id]).run()(a recursiveexport_to_dictplus the dataset's database export) for every occurrence of the same dataset id, with the duplicate output only discarded later by filename dedup in the outer export loop.This change collects all referenced dataset ids up front, resolves them in a single batched
DatasetDAO.find_by_ids()call, and exports each unique dataset exactly once via a singleExportDatasetsCommand(unique_ids).run()call. Behavior is unchanged — same UUID substitution, same "dataset not found" fallback/warning, same file output — just without the redundant DB round-trips and re-serialization work.Benchmarked against a real SQLite session (
DatasetDAO.find_by_id× N vs. a singlefind_by_ids): ~2.4x faster at 3 unique referenced datasets, ~6.2x at 10, ~9.4x at 20. This is against local SQLite with no network latency — on a production Postgres/MySQL backend the per-query round-trip cost is higher, so the absolute savings would be larger.TESTING INSTRUCTIONS
pytest tests/unit_tests/commands/dashboard/export_test.py -q— 18 tests pass, including two new regression tests (test_file_content_batches_dataset_lookup_across_targets,test_export_batches_dataset_export_across_targets) that assertDatasetDAO.find_by_idis never called andDatasetDAO.find_by_ids/ExportDatasetsCommandare each called exactly once, even when several targets reference overlapping dataset ids. Verified these fail against the pre-fix implementation (3 calls tofind_by_id) and pass against the fix.ruff check/ruff format --checkon both changed files pass.ADDITIONAL INFORMATION