[CALCITE-7817] Merge ANTI JOIN should preserve unmatched NULL keys

[bvolpato]

Jira Link

CALCITE-7817

Changes Proposed

Enumerable merge ANTI joins currently stop processing left rows when they reach a NULL join key. Under strict equality, NULL keys do not match and those left rows must survive the ANTI join.

Preserve left NULL keys in all three merge-join advancement paths. Correct the existing ANTI test expectations and add repeated-NULL coverage. The tests also cover empty inputs, NULL keys on the right, and the overload with an additional predicate.

Reproduction

With ascending, NULLS LAST inputs, an equality ANTI join of left [1, 2, NULL] and right [1] returns [2]. The correct result is [2, NULL].

./gradlew :core:test --tests 'org.apache.calcite.runtime.EnumerablesTest'

Validation

• The updated test class fails in four methods against the baseline runtime.
• Compiling the changed runtime and running the complete EnumerablesTest class directly passes all 53 enabled tests; one existing test is disabled.
• Generated EnumerableMergeJoin execution returns [2, NULL] after the fix.
• ./gradlew autostyleApply and git diff --check pass.
• Focused runs used freshly compiled changed classes with cached supporting artifacts. Native Gradle test-class compilation on pristine main was blocked by java.io.IOException: No space left on device; the full Gradle build has not been validated locally.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[mihaibudiu]

Can you find out using git Blame who wrote this test and ask for their review?

[julianhyde]

It was @rubenada in commit 95e40f4, for CALCITE-3834.

[rubenada]

New result looks good.

[rubenada]

LGTM.
Looking at this with fresh eyes, it does look like a bug on the original code of EnumerableMerge AntiJoin

[rubenada]

nit: why not putting these new test cases on the existing above method, on the ANTI join section?

[mihaibudiu]

@rubenada you have LGTM but not approved officially. Still waiting for the nit?

[rubenada]

@rubenada you have LGTM but not approved officially. Still waiting for the nit?

It's not a blocking issue.

[vlsi]

Suggestions, none blocking:

1. Please add an end-to-end test next to EnumerableJoinTest.testMergeJoinAntiWithCompositeKeyAndNullValues. That test covers only the composite key, which already kept NULL keys, so nothing at the SQL level pins this fix. This test fails on the base commit (was "empid=100\nempid=110") and passes with the PR:

   /** Test case for
    * <a href="https://issues.apache.org/jira/browse/CALCITE-7817">[CALCITE-7817]
    * Enumerable merge ANTI join drops left rows with NULL keys</a>. */
   @Test void testMergeJoinAntiWithSingleKeyAndNullValues() {
     tester(false, new HrSchema())
         .withHook(Hook.PLANNER, (Consumer<RelOptPlanner>) planner -> {
           planner.addRule(EnumerableRules.ENUMERABLE_MERGE_JOIN_RULE);
           planner.removeRule(EnumerableRules.ENUMERABLE_JOIN_RULE);
         })
         .withRel(builder -> builder
             .scan("s", "emps").as("e1")
             .sort(builder.field("commission"))
             .scan("s", "emps").as("e2")
             .filter(builder.equals(builder.field("deptno"), builder.literal(20)))
             .sort(builder.field("commission"))
             .antiJoin(
                 builder.equals(builder.field(2, 0, "commission"),
                     builder.field(2, 1, "commission")))
             .project(builder.field("empid"))
             .build())
         .explainHookContains("EnumerableMergeJoin")
         .returnsUnordered("empid=100", "empid=110", "empid=150");
   }

2. The commit subject lacks the [CALCITE-7817] prefix. It would also help to state in the commit or PR description that the existing ANTI expectations in EnumerablesTest (for example [3] → [3, null], and [] → [null] for "Both empty") encoded the bug.

[vlsi]

The two cases exercise different paths: [1, 2, null, null] vs [1] reaches the NULL keys after the right input runs out, and [1, null, null] vs [1, 2] reaches them inside advanceLeft right after a match. A one-line comment above each case would say so; the repetition of NULL itself is not what they test. This fits with @rubenada's suggestion to move them into the ANTI section of the existing method.

[vlsi]

This comment still says "if we reach a null key, we are done", but the next line continues for LEFT and ANTI.

[vlsi]

The @return of advanceLeft still says it returns false when a "null key is found". For LEFT and ANTI a NULL key no longer ends this loop. It was already wrong for LEFT, but the PR edits the neighboring comment, so it could fix this one too.