PG_FUNCTION_INFO_V2

[yrashk]

Hey @pgedc/members,

PG_FUNCTION_INFO_V1 did quite well for Postgres over the years. However, I keep thinking about cases where it may be insufficient and wanted to run some ideas by you to spark a conversation about it – and what kind of proposal we can make to the core.

Some of the issues

I tried to note the concerns I had most often below.

Unsafe type signatures

The safety of our extensions often relies on being defensive about the types of arguments the functions receive. A Postgres user with sufficient privileges can create a function with any parameters/return signature calling our native function. Unless that function inspects FunctionCallInfo deeply (which comes at a non-zero cost), all bets are off in this case, and a crash or other malfunction can occur.

Load-time symbol resolution

Sometimes, it would be meaningful to select a different function implementation based on some load-time parameters – either based on the host system or on something coming from the database. In the case of more optimal generated code specific to CPUs, function multiversioning can alleviate this (subject to limitations of multiversioning, but for all other cases, there's no straightforward way to pick the right function when we load it.

Symbol concatenation

PG_FUNCTION_INFO_V2 defines pg_finfo_FUNCTION_NAME symbol. Some languages make it much harder to construct names like this. This also makes it much harder to list all exported names (we need to find symbols!)

This is less critical than the ones above, but it is still a concern.

Sketch of the proposal

What if we had a single function that would return a "list" of available functions?

typedef struct {
  char *function_name; // name
  void *ptr; // actual function pointer
  int n_args; // number of arguments
  Oid return_type; // return type
  char volatile; // volatility
  // and whatever else we want, `strict`, etc.
  Oid arguments[FLEXIBLE_ARRAY_MEMBERS]; // array of `n_args` arguments
} Pg_FunctionInfo;

/* the above records would be unique per (name * arguments * return_type) so that we can have different signatures available */

typedef struct {
  int api_version; // 2?
  int total_entries; // total number of entries
  Pg_FunctionInfo entries[FLEXIBLE_ARRAY_MEMBER];
} Pg_FunctionInfoList;


Pg_FunctionInfoList * __pg_function_info() {
    // returns a pointer to the list structure, filling in the details as needed.   
}

(Please ignore arbitrary naming and imperfect data structure – the point to make here is conceptual)

The most difficult part design-wise here would be identifying the types involved. This would imply that the __pg_function_info would have to resolve the types by name when using non-standard types. There is a non-zero risk that those types can change OIDs during the lifecycle, but we can address this. [1]

With the above, we can solve all three concerns I noted before. It should also work well with the _V1 approach. Postgres would have to try to load __pg_function_info first; if not available, fall back to PG_FUNCTION_INFO_V1.

What do you think?

---

[1] We can perhaps mandate that such types should be declared in a different part of Pg_FunctionInfoList (or however it would be called) as exports (or even imports from other extensions?), and then the extension should receive a callback upon such a change. Definitely a more involved change but not infeasible.

[yrashk]

To track some of the sketching: pgedc/postgres#2

[danolivo]

I think this issue is connected to another one - code versioning.
My two recent extensions don't have any UI at all, employing only hooks. This is intentional behaviour because the application generates databases and manipulates objects according to some internal logic, and the library must not interfere with the state of the database—only with the core behaviour.
After a couple of versions, my clients raised a question: how to define which code version of the library is used at the instance or specific backend.
Attaching the version to the file name doesn't seem like a reliable approach. I am going to discover this issue and raise a discussion in the hackers' mailing list, but may someone already have an opinion on this subject? Don't hesitate to explain it here!

[danolivo]

1. • glad to now!
2. • Yes, you understand it correctly. Maybe it is a problem with my English, but I see such requirements occasionally: a proprietary application storing data inside a Postgres instance wants monopoly access/control and doesn't trust any extra objects inside the database except a few extensions and objects generated by the application itself. That's the main reason.

[danolivo]

I generally like the OMNI_MODULE_INFO structure, but what is the purpose for identity and name fields? Do we really need it?
Looking into the history of PG_MODULE_MAGIC (https://www.postgresql.org/message-id/flat/20060507211705.GB3808%40svana.org), I see that they weren't disagree with the idea of module info - they just didn't predict the future that time.
But the only one detail: Is it safe to use dlsym for the structs/functions with the same name? Is it possible, that after loading function of the next module overwrites function with the same name from previously loaded module? I vaguely recall something about flat function call table, but don't remember precisely is it a problem or not.

[yrashk]

@danolivo the purpose of the name is to specify the name of the module (covering the edge case of having more than one .so file in an extension – technically possible) and identity is an identifier of the module that never changes, surviving module renames.

But the only one detail: Is it safe to use dlsym for the structs/functions with the same name? Is it possible, that after loading function of the next module overwrites function with the same name from previously loaded module? I vaguely recall something about flat function call table, but don't remember precisely is it a problem or not.

Are you referring to a case where the filename of the shared object file is not changing but the underlying file does?

[danolivo]

@yrashk

identifier of the module that never changes, surviving module renames

That was my point: it seems strange a little, what a purpose here? Differ forks?
Anyway, as I see, the (name, version) data is enough for now to identify changes and maintain separate shmem segments for different versions, if needed.

shared object file is not changing but the underlying file does?

Exported function of a module will be replaced in flat namespace by the code of another function from another module with the same name, but loaded later.
User can get version of an extension in any moment, looking up pg_extension. But how it can get version of just a module after the PG_init() call ?
It seems to me that PG_init() should return module_info data and save it into the DynamicFileList on load.

[danolivo]

@yrashk ,
It seems you have some use cases in the pocket that caused current implementation of the module info in the omni extensions. I started a discussion:

https://github.com/orgs/pgedc/discussions/45

Can you explain there your experience on module management issues and cases that triggered your current design?