Add a native function namespace manager #23358

pdabre12 · 2024-08-01T19:41:14Z

Description

Adds a native function namespace manager

Motivation and Context

To help resolve : #23000

Impact

Test Plan

Unit and end-to-end tests. More comprehensive end-to-end tests will be written in the future.

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

== RELEASE NOTES ==

General C++ changes
* Add a native function namespace manager :pr:`23358`

Depends on : #23671

linux-foundation-easycla · 2024-08-01T19:41:19Z

The committers listed above are authorized under a signed CLA.

✅ login: pdabre12 / name: Pratik Joseph Dabre (4ef734e, a08fe11, 6dff08e, 4af2e6a, dd80fd7, 062b54e, 2d7cbe0, e9989b4)

steveburnett

LGTM! (docs)

Pull branch, new local doc build, the doc looks good. Thanks!

steveburnett

LGTM! (docs)

Pull updated branch, new local doc build, doc looks good. Thanks!

tdcmeehan

Just some initial comments

presto-common/src/main/java/com/facebook/presto/common/type/TypeSignature.java

...n/java/com/facebook/presto/functionNamespace/AbstractSqlInvokedFunctionNamespaceManager.java

presto-function-namespace-managers-common/pom.xml

presto-grpc-api/pom.xml

presto-jdbc/pom.xml

presto-main/pom.xml

presto-main/src/main/java/com/facebook/presto/metadata/FunctionAndTypeManager.java

tdcmeehan

Refactored qualifyObjectName from static to instance method ✅

tdcmeehan

Add utility functions to resolve intermediate type in aggregate funct… ❓

Thank you for the really extensive tests for this utility. I think the tests should be broken down to the various different types, because the tests are rather large.

tdcmeehan

[native] Fix bug when handling multiple params and no params aggregat… ❓

Can you help me understand when this bug manifests? Is there any way to add a unit test for it?

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.h

aditi-pandit · 2024-12-06T12:52:51Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.h

 #include "presto_cpp/main/types/TypeParser.h"
 #include "presto_cpp/presto_protocol/core/presto_protocol_core.h"
 #include "velox/core/Expressions.h"

+namespace {
+inline std::string formatDefaultNamespacePrefix(


Its bit odd this function has name formatDefaultNamespacePrefix and then takes the default namespace prefix as a parameter. Maybe you should call this addDefaultNamespace and pick up the default namespace in the function without passing it as a parameter everywhere.

Yes, calling this addDefaultNamespace makes sense.
I am confused about the picking up of the default namespace from the functions part, the functions passed down to the formatDefaultNamespacePrefix function won't have any namespace attached.

For eg:
presto.default.$operator.$add should map to {namespace-prefix}.add , in this case native.default.add

presto/presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

Line 41 in 40161c0

{"presto.default.$operator$add",

The call to the formatDefaultNamespacePrefix() function will take in the default namespace prefix and the function name, add the default namespace prefix to the function name and return the string.

tdcmeehan

A few nits

.../main/java/com/facebook/presto/sidecar/functionNamespace/NativeFunctionNamespaceManager.java

presto-spi/src/main/java/com/facebook/presto/spi/function/FunctionMetadataManager.java

presto-spi/src/main/java/com/facebook/presto/spi/function/FunctionNamespaceManagerContext.java

.../main/java/com/facebook/presto/sidecar/functionNamespace/NativeFunctionNamespaceManager.java

tdcmeehan

Almost there

presto-main/src/main/java/com/facebook/presto/metadata/FunctionAndTypeManager.java

presto-main/src/main/java/com/facebook/presto/metadata/StaticFunctionNamespaceStore.java

presto-main/src/main/java/com/facebook/presto/sql/planner/PlanOptimizers.java

presto-main/src/main/java/com/facebook/presto/sql/planner/optimizations/AddExchanges.java

presto-main/src/main/java/com/facebook/presto/sql/relational/SqlToRowExpressionTranslator.java

presto-main/src/test/java/com/facebook/presto/sql/TestExpressionInterpreter.java

.../main/java/com/facebook/presto/sidecar/functionNamespace/NativeFunctionNamespaceManager.java

pdabre12 · 2024-12-17T22:26:05Z

@rschlussel Can you please have another look? Thanks.

tdcmeehan

Only one minor thing.

@rschlussel would you like to have a look as well?

.../main/java/com/facebook/presto/spi/function/SqlInvokedAggregationFunctionImplementation.java

...s/src/main/java/com/facebook/presto/functionNamespace/json/JsonBasedUdfFunctionMetadata.java

presto-main/src/main/java/com/facebook/presto/metadata/StaticFunctionNamespaceStore.java

rschlussel · 2024-12-19T16:36:00Z

presto-main/src/main/java/com/facebook/presto/metadata/FunctionAndTypeManager.java

@@ -129,6 +131,7 @@ public class FunctionAndTypeManager
    private final LoadingCache<FunctionResolutionCacheKey, FunctionHandle> functionCache;
    private final CacheStatsMBean cacheStatsMBean;
    private final boolean nativeExecution;
+    private CatalogSchemaName currentDefaultNamespace = DEFAULT_NAMESPACE;


this class isn't really thread safe anymore because this field is mutable, and there are no guards on how it is read/written. It might be better to set the default namespace in FunctionsConfig instead of in the config for any particular function namespace. Then we also wouldn't need to check that there is only one.
Downside is that we will only fail later (at runtime?) if the default namespace doesn't exist (never gets loaded).

also, let's change this name to defaultNamespace, as this isn't something that we expect to be updated. and change the static variable DEFAULT_NAMESPACE to something that's clearer now, e.g. JAVA_BUILTIN_NAMESPACE.

Last comment about default_namespace, but have you checked all the things that are using DEFAULT_NAMESPACE right now? Is it still correct for them to do that?

Added a config in FunctionsConfig, the default value is going to be JAVA_BUILTIN_NAMESPACE so we always have a default namespace configured.

Yes, I went over usages of DEFAULT_NAMESPACE (now JAVA_BUILTIN_NAMESPACE), they are used in the right places.

presto-main/src/main/java/com/facebook/presto/metadata/FunctionAndTypeManager.java

presto-spi/src/main/java/com/facebook/presto/spi/function/FunctionMetadataManager.java

rschlussel · 2024-12-19T16:47:38Z

presto-main/src/main/java/com/facebook/presto/metadata/FunctionAndTypeManager.java

@@ -698,7 +728,8 @@ private FunctionHandle resolveFunctionInternal(Optional<TransactionId> transacti

    private FunctionHandle resolveBuiltInFunction(QualifiedObjectName functionName, List<TypeSignatureProvider> parameterTypes)
    {
-        checkArgument(functionName.getCatalogSchemaName().equals(DEFAULT_NAMESPACE), "Expect built-in functions");
+        checkArgument(functionName.getCatalogSchemaName().equals(currentDefaultNamespace) ||


why do we use both DEFAULT_NAMESPACE and currentDefaultNamespace here?

There might be some functions an eg: key_sampling_percent function

presto/presto-main/src/main/java/com/facebook/presto/sql/planner/optimizations/KeyBasedSampler.java

Lines 151 to 155 in a391cec

sampledArg = call(

functionAndTypeManager,

QualifiedObjectName.valueOf(JAVA_BUILTIN_NAMESPACE, getKeyBasedSamplingFunction(session)),

DOUBLE,

ImmutableList.of(arg));

, which would always be under the JAVA_BUILTIN_NAMESPACE.

presto-main/src/main/java/com/facebook/presto/metadata/FunctionAndTypeManager.java

pdabre12 · 2024-12-21T02:49:28Z

I will get back to this soon.

…te functions

…oint

…ation tests

pdabre12 · 2025-01-22T16:55:41Z

@rschlussel
Thank you for the review.
Addressed your comments. Can you please have another look?

pdabre12 force-pushed the native-function-namespace-manager branch from 49d3b9d to 5ed5a18 Compare August 13, 2024 23:37

pramodsatya mentioned this pull request Aug 15, 2024

[native] Retrieve Json function metadata for prestissimo functions #22332

Merged

pdabre12 force-pushed the native-function-namespace-manager branch 2 times, most recently from bcb09bc to ce7ca4b Compare August 17, 2024 00:06

steveburnett previously approved these changes Sep 10, 2024

View reviewed changes

pdabre12 dismissed steveburnett’s stale review via 4d00991 September 12, 2024 18:28

pdabre12 force-pushed the native-function-namespace-manager branch from 4d00991 to c3ed29f Compare September 13, 2024 23:03

pdabre12 requested a review from steveburnett September 13, 2024 23:05

pdabre12 changed the title ~~[WIP] Native function namespace manager~~ [WIP] Add a native function namespace manager Sep 13, 2024

steveburnett previously approved these changes Sep 25, 2024

View reviewed changes

pdabre12 dismissed steveburnett’s stale review via 3fde055 September 27, 2024 00:48

pdabre12 force-pushed the native-function-namespace-manager branch 2 times, most recently from 3fde055 to ebfa361 Compare September 27, 2024 05:55

pdabre12 changed the title ~~[WIP] Add a native function namespace manager~~ Add a native function namespace manager Sep 27, 2024

pdabre12 force-pushed the native-function-namespace-manager branch from f203d25 to 88e1152 Compare November 1, 2024 01:09

pdabre12 marked this pull request as ready for review November 1, 2024 16:40

pdabre12 requested review from elharo, a team, shrinidhijoshi, jaystarshot, feilong-liu and ClarenceThreepwood as code owners November 1, 2024 16:40

pdabre12 requested a review from presto-oss November 1, 2024 16:40

tdcmeehan self-assigned this Nov 11, 2024

tdcmeehan requested changes Nov 11, 2024

View reviewed changes

tdcmeehan reviewed Nov 12, 2024

View reviewed changes

tdcmeehan requested changes Nov 13, 2024

View reviewed changes

tdcmeehan reviewed Nov 13, 2024

View reviewed changes

pdabre12 force-pushed the native-function-namespace-manager branch from d5ccae3 to b229336 Compare December 6, 2024 00:44

aditi-pandit reviewed Dec 6, 2024

View reviewed changes

pdabre12 force-pushed the native-function-namespace-manager branch from b229336 to c748230 Compare December 9, 2024 18:33

tdcmeehan reviewed Dec 9, 2024

View reviewed changes

pdabre12 force-pushed the native-function-namespace-manager branch 3 times, most recently from f2c832d to d8a89d3 Compare December 10, 2024 21:29

pdabre12 requested review from aditi-pandit, rschlussel and tdcmeehan December 10, 2024 21:31

pdabre12 force-pushed the native-function-namespace-manager branch from d8a89d3 to f73aca4 Compare December 16, 2024 21:59

tdcmeehan requested changes Dec 17, 2024

View reviewed changes

pdabre12 force-pushed the native-function-namespace-manager branch from f73aca4 to 5567990 Compare December 17, 2024 21:16

pdabre12 requested a review from tdcmeehan December 17, 2024 21:17

tdcmeehan requested changes Dec 18, 2024

View reviewed changes

.../main/java/com/facebook/presto/spi/function/SqlInvokedAggregationFunctionImplementation.java Outdated Show resolved Hide resolved

rschlussel reviewed Dec 19, 2024

View reviewed changes

pdabre12 added 8 commits January 22, 2025 08:44

[native] Add presto.default-namespace Prestissimo config property

4ef734e

Add utility functions to resolve intermediate type in generic aggrega…

a08fe11

…te functions

[native] Add additional function metadata to function signatures endp…

e9989b4

…oint

Add a new module presto-function-namespace-managers-common

dd80fd7

Change built-in namespace to JAVA_BUILTIN_NAMESPACE

6dff08e

[native] Introduce native function namespace SPI

062b54e

[native] Integrate SPI with sidecar and add e2e native function valid…

4af2e6a

…ation tests

Add OpenAPI documentation for /v1/functions

2d7cbe0

pdabre12 force-pushed the native-function-namespace-manager branch from 5567990 to 2d7cbe0 Compare January 22, 2025 16:47

pdabre12 requested review from hantangwangd and ZacBlanco as code owners January 22, 2025 16:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a native function namespace manager #23358

Add a native function namespace manager #23358

pdabre12 commented Aug 1, 2024 •

edited

Loading

linux-foundation-easycla bot commented Aug 1, 2024 •

edited

Loading

steveburnett left a comment

steveburnett left a comment

tdcmeehan left a comment

tdcmeehan left a comment

tdcmeehan left a comment

tdcmeehan left a comment

aditi-pandit Dec 6, 2024

pdabre12 Dec 10, 2024 •

edited

Loading

tdcmeehan left a comment

tdcmeehan left a comment

pdabre12 commented Dec 17, 2024

tdcmeehan left a comment

rschlussel Dec 19, 2024

rschlussel Dec 19, 2024

rschlussel Dec 19, 2024

pdabre12 Jan 22, 2025

pdabre12 Jan 22, 2025

rschlussel Dec 19, 2024 •

edited

Loading

pdabre12 Jan 22, 2025

pdabre12 commented Dec 21, 2024

pdabre12 commented Jan 22, 2025

	sampledArg = call(
	functionAndTypeManager,
	QualifiedObjectName.valueOf(JAVA_BUILTIN_NAMESPACE, getKeyBasedSamplingFunction(session)),
	DOUBLE,
	ImmutableList.of(arg));

Add a native function namespace manager #23358

Are you sure you want to change the base?

Add a native function namespace manager #23358

Conversation

pdabre12 commented Aug 1, 2024 • edited Loading

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

linux-foundation-easycla bot commented Aug 1, 2024 • edited Loading

steveburnett left a comment

Choose a reason for hiding this comment

steveburnett left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

aditi-pandit Dec 6, 2024

Choose a reason for hiding this comment

pdabre12 Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

tdcmeehan left a comment

Choose a reason for hiding this comment

pdabre12 commented Dec 17, 2024

tdcmeehan left a comment

Choose a reason for hiding this comment

rschlussel Dec 19, 2024

Choose a reason for hiding this comment

rschlussel Dec 19, 2024

Choose a reason for hiding this comment

rschlussel Dec 19, 2024

Choose a reason for hiding this comment

pdabre12 Jan 22, 2025

Choose a reason for hiding this comment

pdabre12 Jan 22, 2025

Choose a reason for hiding this comment

rschlussel Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

pdabre12 Jan 22, 2025

Choose a reason for hiding this comment

pdabre12 commented Dec 21, 2024

pdabre12 commented Jan 22, 2025

pdabre12 commented Aug 1, 2024 •

edited

Loading

linux-foundation-easycla bot commented Aug 1, 2024 •

edited

Loading

pdabre12 Dec 10, 2024 •

edited

Loading

rschlussel Dec 19, 2024 •

edited

Loading