Refactor and improvement of AODBcRewriter#2345
Open
sawenzel wants to merge 1 commit intoAliceO2Group:masterfrom
Open
Refactor and improvement of AODBcRewriter#2345sawenzel wants to merge 1 commit intoAliceO2Group:masterfrom
sawenzel wants to merge 1 commit intoAliceO2Group:masterfrom
Conversation
Replaces the patched version with a full refactor that fixes a critical correctness bug introduced by commit 95cc50b ("Fix concatenation for McCollisions indexed Trees"). Problem ------- Stage 2 was added to reorder every table carrying fIndexMcCollisions (including O2mcparticle) so that rows are grouped by their new MC-collision parent. The fIndexMcCollisions column was correctly remapped, but two other index columns inside O2mcparticle were left untouched: • fIndexArray_Mothers — VLA of Int_t pointing to mother particles within the same O2mcparticle table • fIndexSlice_Daughters — fixed [2] array of Int_t giving the [first, last] daughter slice in the same table After reordering, these columns still held old row positions. The pointed-to rows now belonged to different MC collisions, producing structurally invalid output verified by 267,605 cross-collision violations in the example AO2D. O2Physics detected this at analysis time as: [FATAL] MC particle N with PDG P has daughter with index M > MC particle table size (+ offset) Additionally, label tables (O2mctracklabel, O2mcfwdtracklabel, O2mcmfttracklabel, O2mccalolabel) carry fIndexMcParticles / fIndexArrayMcParticles pointing into O2mcparticle. After O2mcparticle is reordered those pointers became stale; they were also not remapped. A latent memory-safety bug was also present: fIndexSlice_Daughters is stored as a fixed-size branch with leaf title fIndexSlice_Daughters[2] (8 bytes), but the generic branch I/O code allocated only 4 bytes for it (treating it as a plain scalar), causing a silent out-of-bounds write on every GetEntry call. Changes ------- 1. BranchDesc gains nElems (= leaf->GetLen()), so the I/O buffer is sized nElems * elemSize for fixed-size arrays as well as scalars. This fixes the fIndexSlice_Daughters buffer underallocation. 2. A new ExtraRemap struct and an extraRemaps parameter on rewriteTable() allow any number of additional integer columns to be remapped in-place within the same write pass. Each extra remap applies remapIdx() to every Int_t value in the branch (scalar, fixed array, or VLA) after GetEntry and before Fill, using the caller-supplied PermMap. 3. In stage2_MCCollIndexedTables, when processing O2mcparticle: - The self-permutation (oldRow → newRow) is computed from rowOrder before calling rewriteTable — this is exact because rowOrder contains unique source rows in output order. - fIndexArray_Mothers and fIndexSlice_Daughters are passed as ExtraRemaps using this self-permutation. The stable sort by new MC-collision position preserves the original within-collision particle order, so daughter slices remain contiguous and remapping [first, last] via the same permutation is correct. 4. processPasteJoinTables now locates the MC-particle permutation in allPerms and builds ExtraRemaps for fIndexMcParticles and fIndexArrayMcParticles in any table that carries those branches. Tables that need only index remapping (no row reordering) are processed with an identity rowOrder via rewriteTable instead of CloneTree. 5. A new AODBcRewriterValidate(fname) function (callable as a standalone ROOT macro) checks: - BC table is strictly monotonic in fGlobalBC. - All fIndexSlice_Daughters values are in [0, nMcParticle) and point to particles sharing the same fIndexMcCollisions as the parent row. - All fIndexArray_Mothers values satisfy the same constraints. Returns true/false and prints [FAIL] lines for each failing DF. Verification ------------ Validated on example_AOD/AO2D_pre.root (merged, pre-rewrite): AODBcRewriterValidate reports PASSED — confirms input was clean. Old rewriter output (AO2D_after.root): AODBcRewriterValidate reports FAILED — 267,605 cross-collision violations in DF_3594457012001 and DF_3594457012003. New rewriter output: AODBcRewriterValidate reports PASSED (3 DFs checked) — zero violations. Co-Authored-By: Claude Opus 4.7 <[email protected]>
|
REQUEST FOR PRODUCTION RELEASES: This will add The following labels are available |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the recently patched version of AODBcRewriter with a full refactor, fixing a correctness bug introduced by commit 95cc50b ("Fix concatenation for McCollisions indexed Trees").
Problem
Stage 2 was added to reorder every table carrying fIndexMcCollisions (including O2mcparticle) so that rows are grouped by their new MC-collision parent. The fIndexMcCollisions column was correctly remapped, but two other index columns inside O2mcparticle were left untouched:
• fIndexArray_Mothers — VLA of Int_t pointing to mother particles
within the same O2mcparticle table
• fIndexSlice_Daughters — fixed [2] array of Int_t giving the
[first, last] daughter slice in the same table
After reordering, these columns still held old row positions. The pointed-to rows now belonged to different MC collisions, producing structurally invalid output verified by 267,605 cross-collision violations in the example AO2D. O2Physics detected this at analysis time as:
[FATAL] MC particle N with PDG P has daughter with index M >
MC particle table size (+ offset)
Additionally, label tables (O2mctracklabel, O2mcfwdtracklabel, O2mcmfttracklabel, O2mccalolabel) carry fIndexMcParticles / fIndexArrayMcParticles pointing into O2mcparticle. After O2mcparticle is reordered those pointers became stale; they were also not remapped.
A latent memory-safety bug was also present: fIndexSlice_Daughters is stored as a fixed-size branch with leaf title fIndexSlice_Daughters[2] (8 bytes), but the generic branch I/O code allocated only 4 bytes for it (treating it as a plain scalar), causing a silent out-of-bounds write on every GetEntry call.
Changes
BranchDesc gains nElems (= leaf->GetLen()), so the I/O buffer is sized nElems * elemSize for fixed-size arrays as well as scalars. This fixes the fIndexSlice_Daughters buffer underallocation.
A new ExtraRemap struct and an extraRemaps parameter on rewriteTable() allow any number of additional integer columns to be remapped in-place within the same write pass. Each extra remap applies remapIdx() to every Int_t value in the branch (scalar, fixed array, or VLA) after GetEntry and before Fill, using the caller-supplied PermMap.
In stage2_MCCollIndexedTables, when processing O2mcparticle:
processPasteJoinTables now locates the MC-particle permutation in allPerms and builds ExtraRemaps for fIndexMcParticles and fIndexArrayMcParticles in any table that carries those branches. Tables that need only index remapping (no row reordering) are processed with an identity rowOrder via rewriteTable instead of CloneTree.
A new AODBcRewriterValidate(fname) function (callable as a standalone ROOT macro) checks: