Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Run filter null join key optimization once #3657

Merged
merged 1 commit into from
Jan 9, 2025

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Jan 9, 2025

Only run the filter null join key optimization once before filter pushdown, instead of 3 times together with the other rules, otherwise you end up with something like [not(is_null(key)) & not(is_null(key))] & not(is_null(key)).

@github-actions github-actions bot added the fix label Jan 9, 2025
Copy link

codspeed-hq bot commented Jan 9, 2025

CodSpeed Performance Report

Merging #3657 will degrade performances by 33.32%

Comparing colin/filter-null-join-key-once (f94058b) with main (e0a2f69)

Summary

⚡ 2 improvements
❌ 1 regressions
✅ 24 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main colin/filter-null-join-key-once Change
test_iter_rows_first_row[100 Small Files] 129.4 ms 194.1 ms -33.32%
test_show[100 Small Files] 23.8 ms 15.8 ms +50.64%
test_tpch_sql[1-in-memory-native-8] 197.3 ms 156.1 ms +26.38%

Copy link

codecov bot commented Jan 9, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.08%. Comparing base (426ddd0) to head (f94058b).
Report is 5 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3657      +/-   ##
==========================================
+ Coverage   78.04%   78.08%   +0.04%     
==========================================
  Files         727      728       +1     
  Lines       89762    90092     +330     
==========================================
+ Hits        70052    70352     +300     
- Misses      19710    19740      +30     
Files with missing lines Coverage Δ
...rc/daft-logical-plan/src/optimization/optimizer.rs 94.97% <100.00%> (+0.06%) ⬆️

... and 3 files with indirect coverage changes

@colin-ho colin-ho requested a review from kevinzwang January 9, 2025 01:09
Copy link
Member

@kevinzwang kevinzwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@colin-ho colin-ho merged commit bb85070 into main Jan 9, 2025
42 of 43 checks passed
@colin-ho colin-ho deleted the colin/filter-null-join-key-once branch January 9, 2025 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants