[core] Fix LookupMergeFunction to use sequence.field for picking high level records#7221
Open
Liulietong wants to merge 1 commit intoapache:masterfrom
Open
[core] Fix LookupMergeFunction to use sequence.field for picking high level records#7221Liulietong wants to merge 1 commit intoapache:masterfrom
Liulietong wants to merge 1 commit intoapache:masterfrom
Conversation
… level records When sequence.field is configured, LookupMergeFunction.pickHighLevel() should select the record with the highest sequence value instead of the lowest level number. This ensures correct behavior when out-of-order data arrives. Previously, pickHighLevel() only compared level numbers, which could lead to incorrect results when: - L1 has sequence=7 (older) - L2 has sequence=8 (newer) - L0 has sequence=6 (oldest, out-of-order arrival) The old logic would pick L1 (level 1 < level 2), but the correct behavior should pick L2 (sequence 8 > 7). This fix: 1. Adds sequenceComparator field to LookupMergeFunction 2. Modifies pickHighLevel() to use sequence comparator when available 3. Modifies getResult() to sort records by sequence before adding to merge function 4. Only sets sequenceComparator when user-defined sequence field is configured, preserving original behavior when sequence.field is not set 5. Adds test cases to verify the fix, backward compatibility, and descending sort order Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix #7220
When
sequence.fieldis configured,LookupMergeFunction.pickHighLevel()should select the record with the highest sequence value instead of the lowest level number.Problem
Previously,
pickHighLevel()only compared level numbers:This could lead to incorrect results when out-of-order data arrives:
The old logic would pick L1 (level 1 < level 2), but the correct behavior should pick L2 (sequence 8 > 7).
Changes
sequenceComparatorfield toLookupMergeFunctionpickHighLevel()to use sequence comparator when availablegetResult()to sort records by sequence before adding to merge functionsequenceComparatorwhen user-defined sequence field is configured, preserving original behavior whensequence.fieldis not setTests
Added 3 test cases:
testSequenceFieldWithMultipleLevels- verifies sequence.field is used correctlytestWithoutSequenceFieldPreservesOriginalBehavior- verifies backward compatibilitytestSequenceFieldWithDescendingSortOrder- verifies descending sort order worksImpact
Only affects
changelog-producer = lookupwithsequence.fieldconfigured. No impact on:sequence.field