Fix parent-bucket scoping in StreamStringTermsAggregator - #21447
Fix parent-bucket scoping in StreamStringTermsAggregator#21447harshavamsi wants to merge 3 commits into
Conversation
9a664ce to
366cce4
Compare
PR Reviewer Guide 🔍(Review updated until commit 4f9f35b)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 4f9f35b Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 30d2ace
Suggestions up to commit cc87077
Suggestions up to commit 6844f93Suggestions up to commit c637106
Suggestions up to commit 772f18a
|
|
❌ Gradle check result for 366cce4: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
366cce4 to
b57e4ec
Compare
|
Persistent review updated to latest commit b57e4ec |
|
❌ Gradle check result for b57e4ec: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit b794d80 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #21447 +/- ##
============================================
+ Coverage 71.58% 71.64% +0.06%
- Complexity 77353 77386 +33
============================================
Files 6170 6170
Lines 359700 359718 +18
Branches 52459 52458 -1
============================================
+ Hits 257493 257725 +232
+ Misses 81808 81579 -229
- Partials 20399 20414 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Persistent review updated to latest commit f5c9b3f |
|
❌ Gradle check result for f5c9b3f: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 772f18a |
772f18a to
c637106
Compare
|
Persistent review updated to latest commit c637106 |
|
@rishabhmaurya Added an IT for this case. |
|
{"run-benchmark-test":"id_17"} |
|
Persistent review updated to latest commit 6844f93 |
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/8609/ . Final results will be published once the job is completed. |
|
The benchmark job https://build.ci.opensearch.org/job/benchmark-pull-request/8609/ failed. |
|
❌ Gradle check result for 6844f93: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
{"run-benchmark-test":"id_17"} |
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/8621/ . Final results will be published once the job is completed. |
|
The benchmark job https://build.ci.opensearch.org/job/benchmark-pull-request/8621/ failed. |
|
@harshavamsi do you mind taking a look at the failure |
|
|
||
| // Pair up each candidate as (bucketOrd, segmentOrdinal). We keep them in two | ||
| // parallel arrays sized bucketsForOwner. | ||
| long[] candidateBucketOrds = new long[Math.toIntExact(bucketsForOwner)]; |
There was a problem hiding this comment.
we should not be creating a regular java long array of size with upper bound upto max ordinals, which seems to be the case here. Please make use of bigarrays or it will result into hard OOMs rather than CBE.
There was a problem hiding this comment.
It is fine to create java array of size of topN requested which is effectiveSegmentSize or segmentSize which is usually much smaller
There was a problem hiding this comment.
You can see how in previous logic we made use of reusableIndices bigArray until we knew the effective segment size and then we lazily used java int array of that size to perform quick select.
There was a problem hiding this comment.
thanks, this is indeed a valid concern. Using BigArrays now.
|
@harshavamsi added some comments, I highly suggest running some stress tests leading to CBEs for this part and validating its working as expected. |
6844f93 to
cc87077
Compare
|
Persistent review updated to latest commit cc87077 |
| for (int i = 0; i < effectiveSegmentSize; i++) { | ||
| indices[i] = i; | ||
| } | ||
| Arrays.sort(indices, (a, b) -> Long.compare(candidateSegmentOrds[a], candidateSegmentOrds[b])); |
There was a problem hiding this comment.
is it possible in this new logic to retrieve in the ordinal order itself and avoid sorting here
There was a problem hiding this comment.
We get each ordinal in insertion order and not segment ordinal order, which means we cannot avoid sorting since downstream reduce expects keys to be sorted
There was a problem hiding this comment.
but we can retrieve in ordinal order right? by running a loop to max ordinal and then checking which ones made to topN instead of sorting?
There was a problem hiding this comment.
So sorting here would be O(NlogN) where N is the topN value which is usually small. But if we want to loop through the entire set of unique values, those could be very high per batch. So it would be O(k) where K could be a much higher value. wdyt?
There was a problem hiding this comment.
since we already do O(K) throughout the logic, another O(K) still keeps complexity linear in my opinion and would avoid log factor. I'm not too adamant on it, so if you think sort here is simplifying the code, go ahead
|
❌ Gradle check result for cc87077: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 30d2ace |
When StreamStringTermsAggregator ran as a sub-aggregator under another terms aggregator, every parent bucket received the same inner bucket list. The collector wrote into a flat docCounts array keyed on segment ordinal, ignoring the owningBucketOrd passed into collect(), and selectTopBuckets walked that same global array once per parent ordinal. Route collection through LongKeyedBucketOrds keyed on (owningBucketOrd, segmentOrdinal) so each parent bucket maintains its own doc-count state, and teach selectTopBuckets to enumerate buckets per owner via ordsEnum(owningBucketOrd). Thread CardinalityUpperBound into createStreamStringTermsAggregator so the factory can pick between FromSingle and FromMany implementations. Add a deterministic reproducer in StreamStringTermsAggregatorTests that fails on the pre-fix code by asserting two parent buckets produce distinct inner results. Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>
Add a streaming search integration test that verifies nested string terms buckets remain scoped to their parent bucket. Assert through profiling that both aggregation levels use StreamStringTermsAggregator. Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>
Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>
30d2ace to
4f9f35b
Compare
|
Persistent review updated to latest commit 4f9f35b |
|
❌ Gradle check result for 4f9f35b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
When StreamStringTermsAggregator ran as a sub-aggregator under another terms aggregator, every parent bucket received the same inner bucket list. The collector wrote into a flat docCounts array keyed on segment ordinal, ignoring the owningBucketOrd passed into collect(), and selectTopBuckets walked that same global array once per parent ordinal.
Route collection through LongKeyedBucketOrds keyed on (owningBucketOrd, segmentOrdinal) so each parent bucket maintains its own doc-count state, and teach selectTopBuckets to enumerate buckets per owner via ordsEnum(owningBucketOrd). Thread CardinalityUpperBound into createStreamStringTermsAggregator so the factory can pick between FromSingle and FromMany implementations.
Add a deterministic reproducer in StreamStringTermsAggregatorTests that fails on the pre-fix code by asserting two parent buckets produce distinct inner results.
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.