Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SPI for delegating row expression optimizer #24144

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add SPI for delegating row expression optimizer #24144

wants to merge 3 commits into from

Conversation

tdcmeehan
Copy link
Contributor

@tdcmeehan tdcmeehan commented Nov 25, 2024 •
edited
Loading

Description

As part of RFC-0006, we need to support out of process expression evaluation. Add support for pluggable expression optimization and planner support to utilize the new SPI.

Motivation and Context

RFC-0006. See #24126 for larger changes that include the Presto sidecar as described in the RFC.

Impact

No impact by default as the old in-memory evaluation is the default.

Test Plan

Tests have been added.

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

== RELEASE NOTES ==

SPI Changes
* Add ``CoordinatorPlugin#getExpressionOptimizerFactories`` to customize expression evaluation in the Presto coordinator. :pr:`24144`

@tdcmeehan tdcmeehan requested a review from presto-oss November 25, 2024 18:56
@prestodb-ci prestodb-ci added the from:IBM PR from IBM label Nov 25, 2024
@prestodb-ci prestodb-ci requested review from a team, sh-shamsan and pdabre12 and removed request for a team November 25, 2024 18:56
pdabre12
pdabre12 reviewed Nov 25, 2024
View reviewed changes
Copy link
Contributor

@pdabre12 pdabre12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tdcmeehan.
Just some initial comments

presto-main/src/main/java/com/facebook/presto/server/testing/TestingPrestoServer.java Outdated Show resolved Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java Show resolved Hide resolved
@tdcmeehan tdcmeehan force-pushed the expr-spi-init branch 12 times, most recently from 17fd8e1 to 563611e Compare December 2, 2024 17:34
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 2, 2024 •
edited
Loading

Codenotify: Notifying subscribers in CODENOTIFY files for diff 15fd5d8...226a3c3.

No notifications.

@tdcmeehan tdcmeehan mentioned this pull request Dec 10, 2024
Draft
6 tasks
ZacBlanco
ZacBlanco reviewed Dec 18, 2024
View reviewed changes
Copy link
Contributor

@ZacBlanco ZacBlanco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At a high-level, the changes make sense. I had a few questions about the configuration and how we might be able to simplify it to be more straightforward for users

presto-tests/src/test/java/com/facebook/presto/memory/TestMemoryManager.java Outdated Show resolved Hide resolved
presto-tests/src/test/java/com/facebook/presto/memory/TestMemoryManager.java Outdated Show resolved Hide resolved
presto-main/src/main/java/com/facebook/presto/server/testing/TestingPrestoServer.java Outdated Show resolved Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java Outdated Show resolved Hide resolved
...-main/src/main/java/com/facebook/presto/sql/relational/DelegatingRowExpressionOptimizer.java Outdated Show resolved Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/expressions/ExpressionOptimizerManager.java Outdated Show resolved Hide resolved
@tdcmeehan tdcmeehan force-pushed the expr-spi-init branch 3 times, most recently from a7b6402 to 2d609d0 Compare December 19, 2024 15:51
@tdcmeehan tdcmeehan mentioned this pull request Dec 19, 2024
Merged
6 tasks
pdabre12
pdabre12 reviewed Dec 20, 2024
View reviewed changes
Copy link
Contributor

@pdabre12 pdabre12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments,

...main/src/test/java/com/facebook/presto/sql/planner/iterative/rule/TestRemoveMapCastRule.java Outdated Show resolved Hide resolved
...s/src/test/java/com/facebook/presto/tests/expressions/TestDelegatingExpressionOptimizer.java Outdated Show resolved Hide resolved
presto-tests/src/test/java/com/facebook/presto/tests/expressions/TestExpressionOptimizers.java Outdated Show resolved Hide resolved
presto-tests/src/test/java/com/facebook/presto/tests/expressions/TestExpressionOptimizers.java Outdated Show resolved Hide resolved
presto-main/src/main/java/com/facebook/presto/sql/relational/ConnectorRowExpressionService.java Show resolved Hide resolved
@tdcmeehan tdcmeehan force-pushed the expr-spi-init branch 2 times, most recently from 97d0c3b to 8c8d1f4 Compare December 20, 2024 20:57
@tdcmeehan tdcmeehan force-pushed the expr-spi-init branch 3 times, most recently from 3cbc135 to 6e124d2 Compare January 9, 2025 20:53
@tdcmeehan tdcmeehan force-pushed the expr-spi-init branch 6 times, most recently from 9d80d3e to 2cf08d2 Compare January 17, 2025 21:37
@tdcmeehan
Copy link
Contributor Author

tdcmeehan commented Jan 21, 2025 •
edited
Loading

Updates:

  1. Removed DelegatingExpressionOptimizer and all related tests.
  2. Added the ability to configure multiple expression optimizers and switch them based on a session property. Should be useful for fuzzer testing.

Thanks for the comments, please take another @ZacBlanco @rschlussel @aaneja @pdabre12

@tdcmeehan tdcmeehan force-pushed the expr-spi-init branch 2 times, most recently from 616bb09 to 226a3c3 Compare January 22, 2025 21:23
@tdcmeehan
Add ExpressionOptimizerManager
2662b33
@tdcmeehan
Use ExpressionOptimizerProvider
226a3c3 
The runtime should consolidate to the `ExpressionOptimizerProvider` factory
so that it can be customized without significant refactoring.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pdabre12 pdabre12 pdabre12 left review comments

@rschlussel rschlussel rschlussel left review comments

@ZacBlanco ZacBlanco ZacBlanco left review comments

@aaneja aaneja aaneja left review comments

@shrinidhijoshi shrinidhijoshi Awaiting requested review from shrinidhijoshi shrinidhijoshi is a code owner

@jaystarshot jaystarshot Awaiting requested review from jaystarshot jaystarshot is a code owner

@feilong-liu feilong-liu Awaiting requested review from feilong-liu feilong-liu is a code owner

@elharo elharo Awaiting requested review from elharo elharo is a code owner

@ClarenceThreepwood ClarenceThreepwood Awaiting requested review from ClarenceThreepwood ClarenceThreepwood is a code owner

@presto-oss presto-oss Awaiting requested review from presto-oss presto-oss is a code owner automatically assigned from prestodb/committers

@sh-shamsan sh-shamsan Awaiting requested review from sh-shamsan sh-shamsan was automatically assigned from prestodb/ibm-reviewers

@sdruzkin sdruzkin Awaiting requested review from sdruzkin sdruzkin is a code owner

@shangxinli shangxinli Awaiting requested review from shangxinli shangxinli is a code owner

@hantangwangd hantangwangd Awaiting requested review from hantangwangd hantangwangd is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
from:IBM PR from IBM
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@tdcmeehan @aaneja @ZacBlanco @rschlussel @pdabre12 @prestodb-ci