Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CalcitePPLJoinIT #3369

Conversation

qianheng-aws
Copy link
Contributor

Description

This PR includes 2 changes:

  1. Fix PredicateAnalyzer for in and notIn, it build the DSL query with redundant brackets because of calling wrong API.
  2. For the tables used in this IT, change some fields from text type to keyword type as we don't support text push down for now. Issue [BUG][Calcite Engine] Incorrect Pushdown of Equality/Inequality (=, !=) on text Fields #3334 will fix it.

./gradlew :integ-test:integTest --tests '*Calcite*IT' succeed locally except below 3 tests. They failed because of inconsistent order on the final results while it should not promise that order. Need another PR to fix the test themself.

  • CalcitePPLJoinIT.testJoinTwoColumnsAndDisjointFilters
  • CalcitePPLJoinIT.testJoinWithCondition
  • CalcitePPLJoinIT.testJoinWithTwoJoinConditions

Related Issues

Resolves #3368

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <qianheng@amazon.com>
…t type

Signed-off-by: Heng Qian <qianheng@amazon.com>
@@ -5,7 +5,7 @@
"type": "keyword"
},
"occupation": {
"type": "text"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@qianheng-aws , please create an issue of supporting text type pushdown. I will add it as child of #3309. Note, the new issue may not require product code change since it could be a part of adding external property to Calcite plan. But at least we need to add some tests to verify the text (may include more) type code be pushdown.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a issue to fix text push down, #3334

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this issue as a subtask of that as well, #3370

@LantaoJin
Copy link
Member

Merging to dev branch firstly. cc @penghuo

@LantaoJin LantaoJin merged commit e2d7333 into opensearch-project:feature/calcite-engine Mar 3, 2025
6 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
calcite calcite migration releated
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants