feat: allow spread operators in to-many relationships #3640

laurenceisla · 2024-07-05T00:56:21Z

laurenceisla · 2024-07-06T00:13:57Z

My approach right now is to generate this query for a to-many request:

curl 'localhost:3000/clients?select=name,...projects(name,id)'

SELECT "test"."clients"."name",
       "clients_projects_1"."name",
       "clients_projects_1"."id"
FROM "test"."clients"
LEFT JOIN LATERAL (
  SELECT json_agg("projects_1"."name") AS "name",
         json_agg("projects_1"."id") AS "id"
  FROM "test"."projects" AS "projects_1"
  WHERE "projects_1"."client_id" = "test"."clients"."id"
) AS "clients_projects_1" ON TRUE

Right now this gives the expected result. But aggregates are not working correctly, because they are designed to be selected in the top query with a GROUP BY. A solution would be to not do the json_agg() inside the sub-query and do it in the top one and treat it as another aggregate (with GROUP BY). Like this:

SELECT "test"."clients"."name",
      json_agg("clients_projects_1"."name") AS "name",
      json_agg("clients_projects_1"."id") AS "id"
FROM "test"."clients"
LEFT JOIN LATERAL (
  SELECT "projects_1"."name",
         "projects_1"."id"
  FROM "test"."projects" AS "projects_1"
  WHERE "projects_1"."client_id" = "test"."clients"."id"
) AS "clients_projects_1" ON TRUE
GROUP BY "test"."clients"."name"

Not sure which one is better/easier right now... I'm thinking the latter.

src/PostgREST/Error.hs

wolfgangwalther · 2024-07-06T09:29:21Z

Right now this gives the expected result. But aggregates are not working correctly, because they are designed to be selected in the top query with a GROUP BY. A solution would be to not do the json_agg() inside the sub-query and do it in the top one and treat it as another aggregate (with GROUP BY).

Having the json_agg in the outer query would make the query cleaner, imho.

laurenceisla · 2024-07-17T01:19:59Z

Some caveats I encountered:

Repeated values and order

Do we want to keep repeated values in the results? For example (not the best use case, just to illustrate):

curl 'localhost:3000/project?select=name,...tasks(tasks:name,due_dates:due_date)'

[
  {
    "name": "project 1",
    "tasks": ["task 1", "task 2", "task 3", "task 4"],
    "due_dates": [null, "2024-08-08", "2024-08-08", null]
  }
]

Here we're repeating null and "2024-08-08", so maybe we don't want to do this and just return [null, "2024-08-08"] (perhaps also remove null values?). Doing this will not guarantee the same dimensions for the same aggregated relationship, and definitely not the order of the results (which wasn't guaranteed before either). Doing a DISTINCT inside the json_agg() is a possible solution (the next caveat has an example query).

Nested To-Many Spreads

I have a doubt on what to expect with nested to-many spreads. For example, on a non-nested to-many spread like this one:

curl 'localhost:3000/entities?select=name,...child_entities(children:name)'

We would expect:

[
  {"name": "entity 1", "children": ["child entity 1", "child entity 2"]},
  {"name": "entity 2", "children": ["child entity 3"]},
  "..."
]

But what if we nest another to-many spread embedding with a new column to aggregate:

curl 'localhost:3000/entities?select=name,...child_entities(children:name,...grandchild_entities(grandchildren:name))'

I understand that we're hoisting all the aggregates to the top level, and not grouping by the intermediate columns (entities.name), because they should also be aggregated. I'm assuming that the result should be the same as above but also with the aggregated grandchild_entities.name.

[
  {"name": "entity 1", "children": ["child entity 1", "child entity 2"], "grandchildren": ["grandchild entity 1", "grandchild entity 2", "..."]},
  {"name": "entity 2", "children": ["child entity 3"], "grandchildren": []},
  "..."
]

This cannot be achieved by a simple GROUP BY, because duplicated values will be returned by entities.name (which, perhaps, is not what we want). A solution would also be to use DISTINCT. The query would look like this:

SELECT "api"."entities"."name",
       json_agg(DISTINCT "entities_child_entities_1"."children") AS "children",
       json_agg(DISTINCT "entities_child_entities_1"."grandchildren") AS "grandchildren"
FROM "api"."entities"
LEFT JOIN LATERAL (
  SELECT "child_entities_1"."name" AS "children",
         "child_entities_grandchild_entities_2"."grandchildren" AS "grandchildren"
  FROM "api"."child_entities" AS "child_entities_1"
  LEFT JOIN LATERAL (
    SELECT "grandchild_entities_2"."name" AS "grandchildren"
    FROM "api"."grandchild_entities" AS "grandchild_entities_2"
    WHERE "grandchild_entities_2"."parent_id" = "child_entities_1"."id"
  ) AS "child_entities_grandchild_entities_2" ON TRUE
  WHERE "child_entities_1"."parent_id" = "api"."entities"."id"
) AS "entities_child_entities_1" ON TRUE
GROUP BY "api"."entities"."name";

If there is no sensible interpretation of the query, another option is to prohibit these intermediate columns altogether (aggregates like sum, avg, etc. should still be possible).

src/PostgREST/Plan.hs

laurenceisla · 2024-09-07T01:54:20Z

OK, this is what I got implemented so far. For example, using the tables in our spec test:

Factories <-02M-> processes <-M2M-> supervisors

curl 'localhost:3000/factories?select=name,...processes(processes:name,...supervisors(supervisors:name))'

[
 {
  "name": "Factory C",
  "processes": ["Process C1", "Process C2", "Process XX"],
  "supervisors": ["Peter", "Peter", null]
 },
 {
  "name": "Factory B",
  "process": ["Process B1", "Process B1", "Process B2", "Process B2"],
  "supervisors": ["Peter", "Sarah", "Mary", "John"]
 },
 {
  "name": "Factory A",
  "process": ["Process A1", "Process A2"],
  "supervisors": ["Mary", "John"]
 },
 {
  "name": "Factory D",
  "process": [null],
  "supervisors": [null]
 }
]⏎

[
  {
  	"name":"Factory C",
  	"processes":["Process C1", "Process C2", "Process XX"],
  	"supervisors":[{"name": "Peter"}, {"name": "Peter"}, null]},
  {
  	"name":"Factory B",
  	"processes":["Process B1", "Process B1", "Process B2", "Process B2"],
  	"supervisors":[{"name": "Peter"}, {"name": "Sarah"}, {"name": "Mary"}, {"name": "John"}]},
  {
  	"name":"Factory A",
  	"processes":["Process A1", "Process A2"],
  	"supervisors":[{"name": "Mary"}, {"name": "John"}]},
  {
  	"name":"Factory D",
  	"processes":[null],
	"supervisors":[null]
  }
]

As I mentioned in previous comments, some values will repeat, since we're grouping by the factory "name" without doing a DISTINCT or NOT NULL. The next step would be to implement the .. operator as mentioned here: #3640 (comment), it shouldn't be too complicated.

There's a problem when the embeddings have no values, as seen in the "Factory D" example, which has no processes and no supervisors. This is the same issue as this SO question. One solution is to do a COALESCE(NULLIF(..., '[null]'), '[]'), but this does not take into consideration valid null values (the row exists but the column value is null). The best solution is in one of the answers (filtering by the PK of the relationship), but it doesn't seem like a trivial task.

laurenceisla

This feature should be ready for review now.

~~I'm leaving the .. for DISTINCT and NOT NULL for another PR to keep it cleaner.~~ Edit: Nvm. I figured that it should be OK to include that feature here too, although in different commits.

Here are some comments on the changes done:

src/PostgREST/Query/SqlFragment.hs

src/PostgREST/ApiRequest/Types.hs

src/PostgREST/Plan/ReadPlan.hs

docs/references/api/resource_embedding.rst

steve-chavez

Looks great now! Awesome work! 💯

jdgamble555 · 2024-11-05T02:47:06Z

This is awesome guys, I was going to ask about aggregations, but just works!

Sweet!

wolfgangwalther

Terrific work with the test cases, very extensive. My head explodes, though.

Because we just discussed commit message / prefixes in another PR - what's your opinion on docs/feat commits? Should they be split like in this PR or do they belong together, i.e. was the idea to squash this?

I think they should go into the same feat: commit. A feature without docs is not a feature.

wolfgangwalther · 2024-11-05T18:21:31Z

docs/references/api/resource_embedding.rst

+It is expected to get ``null`` values in the resulting array.
+You can exclude them with :ref:`stripped_nulls`.


I haven't looked at the code, but I assume this with a special case handling this for the aggregation.

I don't think it's a good idea. This will lead to inconsistent results, because: Assume you have a regular aggregation, some spread embedding aggregation, a regular array and a json array - all with some null values in them. Some of them will be stripped, but others won't.

json(b)_strip_nulls only strips nulls in objects for a reason, I don't think we should change that.

We had the discussion about it here: #3640 (comment)

This will lead to inconsistent results [...] Some of them will be stripped, but others won't.

From the convo above, I was also on the fence about it, but I figured that adding "this only works on to-many spreads" to the docs would clarify some things (I forgot to do that btw). Still, I agree, I think the inconsistency you mention is enough to look for an alternative (maybe another parameter in the header?).

Still, I agree, I think the inconsistency you mention is enough to look for an alternative (maybe another parameter in the header?).

I think filtering NULLs should be very explicit.

I don't remember seeing any tests with filters in the tests (but I didn't look again now).

Is something like this supposed to work?

get "/factories?select=factory:name,...processes(process:name)&processes.process=not.is.null"

And also any other filter on the embedding?

And also any other filter on the embedding?

Yes, filters on the spread embed resource work, there's a couple of tests with them.

Is something like this supposed to work?

Yes, it will work on a single embed resource and won't include the null values. But deeply nested resources could include nulls inside the array when the value is null or when no embedded row is returned. This is a problem with the current implementation.

Hmm... with the fixed implementation (non-flattened arrays) this may not be a problem anymore, since it should return empty arrays instead of null... but I'm not entirely sure, I need to check the new design of the resulting queries to verify.

Hmm... with the fixed implementation (non-flattened arrays) this may not be a problem anymore, since it should return empty arrays instead of null...

Yes, AFAICT this is correct. Since the array_agg is done in the same sub-query selection, it would return null on a failed JOIN. But it returns [null] when the JOIN is successful and the value is null (which is what we want). So just the explicit filter should be needed here, not the header.

test/spec/Feature/Query/SpreadQueriesSpec.hs

laurenceisla · 2024-11-05T22:52:23Z

what's your opinion on docs/feat commits? Should they be split like in this PR or do they belong together, i.e. was the idea to squash this?

I think they should go into the same feat: commit. A feature without docs is not a feature.

Makes sense, yes. I'll squash them to avoid problems when merging.

wolfgangwalther · 2024-11-06T08:14:18Z

docs/references/api/resource_embedding.rst

+     }
+   ]
+
+The order of the values inside the resulting array is unspecified.


Imho, this is unsatisfactory, it would basically make the feature unusable for me. This is because we snapshot test all our api responses and if we can't generate predictable output, then we can't use the feature. So ordering is very important.

Would something like this be hard to do?

get "/factories?select=factory:name,...processes(name)&processes.order=name"

(I hope I got the syntax right, this should be the regular "sort the embedded response" syntax, right?)

For the spread, this could then move the ORDER BY into the aggregate function call. This would only allow to specify a single ORDER BY for multiple spread aggregates - which I consider a good thing, because this would ensure the array items still match between arrays.

Would something like this be hard to do?

For the spread, this could then move the ORDER BY into the aggregate function call. This would only allow to specify a single ORDER BY for multiple spread aggregates - which I consider a good thing, because this would ensure the array items still match between arrays.

I don't think it'd be hard to do. Yes, the syntax is OK, internally it would need to treat every order done inside a to_many spread as an array_agg order for every column, instead of a subquery one.

Right now, the order as you mentioned in your example works: it orders the subquery and the aggregated columns will be sorted. But it's not guaranteed to behave the same way for other more complex cases, as mentioned in this convo. So yes, I'll implement the order by in the aggregate here.

laurenceisla

Some advances I'm making that are ready for reviewing:

Non-flattend arrays as specified here feat: allow spread operators in to-many relationships #3640 (comment)
ORDER BY inside the aggregate (with some caveats mentioned below)

The aggregates on the whole relationship are not yet implemented, e.g. ...to_many(count()).sum().

laurenceisla · 2024-11-23T18:48:46Z

docs/references/api/resource_embedding.rst

+.. code-block:: json
+
+   [
+     {
+       "first_name": "Quentin",
+       "film_titles": [
+         "Reservoir Dogs",
+         "Pulp Fiction"
+       ],
+       "film_years": [
+         1992,
+         1994
+       ]
+     }
+   ]
+
+Note that the field must be selected in the spread relationship for the order to work.


Since the json_agg(col) aggregate is done outside of the subquery selection (to avoid cases like json_agg(sum(col))), we cannot order the json_agg by columns that are not selected in that subquery. Here's the generated query for this example:

Query

WITH pgrst_source AS -- Subquery for the current example (SELECT "public"."directors"."first_name", "directors_films_1"."film_titles", "directors_films_1"."film_years" FROM "public"."directors" LEFT JOIN LATERAL (SELECT json_agg("directors_films_1")::jsonb AS "directors_films_1", COALESCE( json_agg("directors_films_1"."film_titles" ORDER BY "directors_films_1"."film_years") ,'[]' )::jsonb AS "film_titles", COALESCE( json_agg("directors_films_1"."film_years" ORDER BY "directors_films_1"."film_years") ,'[]' )::jsonb AS "film_years" FROM (SELECT "films_1"."title" AS "film_titles", "films_1"."year" AS "film_years" FROM "public"."films" AS "films_1" WHERE "films_1"."director_id" = "public"."directors"."id") AS "directors_films_1") AS "directors_films_1" ON TRUE WHERE "public"."directors"."first_name" LIKE $1) -- SELECT NULL::bigint AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, coalesce(json_agg(_postgrest_t), '[]') AS body, nullif(current_setting('response.headers', TRUE), '') AS response_headers, nullif(current_setting('response.status', TRUE), '') AS response_status, '' AS response_inserted FROM (SELECT * FROM pgrst_source) _postgrest_t

Maybe selecting all the columns in the non-aggregated subquery could be an alternative? (computed columns still won't work, I think).

Just noticed there's also an issue when using aliases in the columns. In the example, order=film_years (the alias) works, but order=year does not. This needs to be fixed.

Any fundamental reason this can't become SELECT jsonb_agg(... ORDER BY ...) FROM public.films WHERE .., i.e. without the subquery in FROM?

Edit: Ah, this, I think:

(to avoid cases like json_agg(col.sum()))

Not sure whether I understand that part, yet.

(to avoid cases like json_agg(col.sum()))

Not sure whether I understand that part, yet.

No, I don't. I'm especially confused by the mixed syntax of SQL and PostgREST-request here. Why exactly did you decide to use the subquery?

Edit: Ah, this, I think:

Yes. For example, ...films(years.max()), would try to do this:

SELECT json_agg(max(years)) FROM public.films WHERE ...

Which returns ERROR: calls to aggregate functions cannot be nested.

Edit: Fix syntax 🤦

For example, ...films(max(years)), would try to do this:

This time the syntax was mixed again, but the other way around :D

So, I guess you mean: ...films(years.max()).

Ok, I see that now, yes. It makes sense to treat the spread as another query layer, so I guess the requirement to have the columns selected for ordering is OK.

I'm not liking the drawbacks of having to include the order inside the select.

Having the order in films(name)&order=year works, but just by adding ... like ...films(name)&order=year, then it breaks.

We lose top-level ordering, like:

films(name,technical_specs(runtime))&films.order=technical_specs(runtime)

In general, having an order exception for spreads is bad UX.

The solution I'm working right now is to include the order in the subquery, and then use it for the ordering, e.g. ...films(title)&order=year:

select json_agg(x.title order by year) from ( select title, year from films ) x;

It needs to take into consideration cases where the order could already be selected and be careful not to collide the aliases, so it's not so straight forward there. If there's any drawback for this approach, let me know.

The solution I'm working right now is to include the order in the subquery, and then use it for the ordering

This is now complete, I added it in a separate commit for better reviewing.

laurenceisla · 2024-11-23T19:03:07Z

src/PostgREST/Query/QueryBuilder.hs

+      Spread{rsSpreadSel, rsAggAlias} ->
+        if relSpread == Just ToManySpread then
+          let
+            selection = selectJsonArray <> (if null rsSpreadSel then mempty else ", ") <> intercalateSnippet ", " (pgFmtSpreadSelectItem True rsAggAlias order <$> rsSpreadSel)


Still need to "SELECT json_agg(<subquery_alias>) AS "<subquery_alias>" to use it for not.is.null or !inner conditions. Can be seen how it's added in the previous comment's example.

wolfgangwalther · 2024-11-23T19:30:57Z

The aggregates on the whole relationship are not yet implemented, e.g. ...to_many(count()).sum().

Does it make sense to leave this out for this PR? This seems already complex enough :)

laurenceisla · 2024-11-23T20:29:54Z

Does it make sense to leave this out for this PR? This seems already complex enough :)

I wanted to include it since it would solve what's mentioned in the original issue #3041 (under Spread on Count). But yes, I would consider it a separate feature, we could leave it for another PR and don't let this one close the issue completely.

wolfgangwalther · 2024-12-04T18:21:30Z

Does it make sense to leave this out for this PR? This seems already complex enough :)

I wanted to include it since it would solve what's mentioned in the original issue #3041 (under Spread on Count). But yes, I would consider it a separate feature, we could leave it for another PR and don't let this one close the issue completely.

I looked at the issue again and I think we need to take the following into account:

get "/processes?select=process:name,...supervisors(count())

and

get "/processes?select=process:name,...supervisors(supervisor:name,count())

We mostly discussed the second case, in which I argued that I expect an array of counts as a return, matching the array of supervisors.

But we didn't really discuss the first case, which seems to be the case in the issue. I think the first case should not return a single item array, but indeed the overall count.

The basic idea would be: We use array aggregation for x2m embeddings. But once we aggregate inside this embedding without any GROUP BY columns, then we don't have an x2m embedding anymore, but an x2o. The "relation" we are embedding is guaranteed to return only one row.

Taking this into account I wonder whether we actually need the more complex syntax ...to_many(x, count()).sum() or so syntax. I don't think so.

laurenceisla · 2024-12-05T15:18:09Z

The basic idea would be: We use array aggregation for x2m embeddings. But once we aggregate inside this embedding without any GROUP BY columns, then we don't have an x2m embedding anymore, but an x2o. The "relation" we are embedding is guaranteed to return only one row.

Yes, that's a nice approach, I agree. There wouldn't be a need for the more complex syntax anymore. I haven't checked yet but there may be some caveats with nested spreads and this implementation. I'll let you know if I find something along the way.

steve-chavez · 2024-12-09T21:35:18Z

The order of the values inside the resulting array is unspecified.
ORDER BY inside the aggregate (with some caveats mentioned below)

Thinking about the ORDER issue, isn't the whole problem that multiple columns inside the spread are supported? /clients?select=name,...projects(name,id)

The motivation comes from the comment on #3041 (comment). But I think the main use case is just forming an array of one column and running aggregates on them, for this we wouldn't need to worry about ORDER.

Perhaps we could leave multiple columns for later?

laurenceisla · 2024-12-31T22:12:11Z

Perhaps we could leave multiple columns for later?

The ordering is now complete and ready for review. If everything's OK there would be no need to leave for later.

wolfgangwalther

I only looked at the docs and the AggregateFunctionsSpec.hs file. Still have a few questions, let's discuss those first, before I continue with the other test file.

Maybe not everything we discussed is implemented or maybe I missed something else, not sure.

wolfgangwalther · 2025-01-01T14:31:41Z

test/spec/Feature/Query/AggregateFunctionsSpec.hs

+        it "supports aggregates inside nested to-one spread relationships" $ do
+          get "/supervisors?select=name,...processes(...process_costs(cost.sum()))&order=name" `shouldRespondWith`
            [json|[
-              {"supervisor": 1, "supervisor_count": 2},
-              {"supervisor": 2, "supervisor_count": 2},
-              {"supervisor": 3, "supervisor_count": 3},
-              {"supervisor": 4, "supervisor_count": 1}]|]
+              {"name":"Jane","sum":[null]},
+              {"name":"John","sum":[270.00]},
+              {"name":"Mary","sum":[220.00]},
+              {"name":"Peter","sum":[290.00]},
+              {"name":"Sarah","sum":[180.00]}]|]
            { matchHeaders = [matchContentTypeJson] }


The [null] for Jane here appears odd to me. Jane doesn't supervise any processes, right? This means there are no rows joined - I would expect null (without array!) here, I think.

Or maybe an empty array.

But [null] indicates to me: "Jane has processes to supervise, but none of them have any cost associated.".

Yes, I agree that it should show an empty array if there's no processes. But this seems to be a caveat with to-one spreads.

For example, a normal embedding returns an empty array as expected:

curl 'localhost:3000/supervisors?select=name,processes(process_costs(cost.sum()))&name=eq.Jane' [{"name":"Jane","processes":[]}]

But the to-one spread does not:

curl 'localhost:3000/supervisors?select=name,processes(...process_costs(cost.sum()))&name=eq.Jane' [{"name":"Jane","processes":[{"sum": null}]}]

The to-many spread is based on the data returned before, so it kinda makes sense for it to return [null]:

curl 'localhost:3000/supervisors?select=name,...processes(...process_costs(cost.sum()))&name=eq.Jane' [{"name":"Jane","sum":[null]}]

If no to-one spread is done, then the to-many spread returns an empty array as expected:

curl 'localhost:3000/supervisors?select=name,...processes(process_costs(cost.sum()))&name=eq.Jane' [{"name":"Jane","process_costs":[]}]

I think the issue is with to-one spreads, it needs to be fixed at that level (not in this PR). For example, if we add a column to GROUP BY in the to-one spread, then we get the expected empty array:

curl 'localhost:3000/supervisors?select=name,processes(...process_costs(process_id,cost.sum()))&name=eq.Jane' [{"name":"Jane","processes":[]}]

wolfgangwalther · 2025-01-01T14:41:54Z

test/spec/Feature/Query/AggregateFunctionsSpec.hs

+          get "/operators?select=name,...processes(id,...factories(...factory_buildings(size.sum())))&order=name" `shouldRespondWith`
            [json|[
-              {"factory": 1, "process_costs_count": 2},
-              {"factory": 2, "process_costs_count": 4},
-              {"factory": 3, "process_costs_count": 1}]|]
+              {"name":"Alfred","id":[6, 7],"sum":[[240], [240]]},
+              {"name":"Anne","id":[1, 2, 4],"sum":[[350], [350], [170]]},
+              {"name":"Jeff","id":[2, 3, 4, 6],"sum":[[350], [170], [170], [240]]},
+              {"name":"Liz","id":[],"sum":[]},
+              {"name":"Louis","id":[1, 2],"sum":[[350], [350]]}]|]
            { matchHeaders = [matchContentTypeJson] }


The empty arrays for Liz seem to be correct for me, because Liz does not operate any processes.

wolfgangwalther · 2025-01-01T14:51:29Z

test/spec/Feature/Query/AggregateFunctionsSpec.hs

+              get "/factories?select=factory:name,...processes(processes_count:count())&order=name" `shouldRespondWith`
+                [json|[
+                  {"factory":"Factory A","processes_count":[2]},
+                  {"factory":"Factory B","processes_count":[2]},
+                  {"factory":"Factory C","processes_count":[4]},
+                  {"factory":"Factory D","processes_count":[0]}]|]
+                { matchHeaders = [matchContentTypeJson] }
+            it "works alongside other columns in the embedded resource" $ do
+              get "/factories?select=name,...processes(category_id,count())&order=name" `shouldRespondWith`
+                [json|[
+                  {"name":"Factory A","category_id":[1, 2],"count":[1, 1]},
+                  {"name":"Factory B","category_id":[1],"count":[2]},
+                  {"name":"Factory C","category_id":[2],"count":[4]},
+                  {"name":"Factory D","category_id":[],"count":[]}]|]
+                { matchHeaders = [matchContentTypeJson] }


Note the asymmetry here between the count. I think the first example should also return [] instead of [0], even if possibly slightly unintuitive.

The general idea: I can't spread something that doesn't exist. Since there is no process for factory D, I can't spread anything. The "count" conceptually never happened, so I can't return a 0 for it.

Interesting, this may be related with the issue mentioned above, the aggregates may be the actual problem... probably.

Again, using the example of the non-spread embed:

curl 'localhost:3000/factories?select=factory:name,processes(processes_count:count())&id=eq.4' [{"factory":"Factory D","processes":[{"processes_count": 0}]}]

The to-many spread takes the above response into consideration, so it returns the [0]. Now, if we add a column to GROUP BY, then we get an expected empty array.

curl 'localhost:3000/factories?select=factory:name,processes(category_id,processes_count:count())&id=eq.4' [{"factory":"Factory D","processes":[]}]

So, the inconsistency seems to be at the aggregate level and it should be fixed there.

wolfgangwalther · 2025-01-01T14:55:10Z

test/spec/Feature/Query/AggregateFunctionsSpec.hs

+            it "works by itself in the embedded resource" $ do
+              get "/supervisors?select=name,...processes(count())&order=name" `shouldRespondWith`
+                [json|[
+                  {"name":"Jane","count":[0]},
+                  {"name":"John","count":[2]},
+                  {"name":"Mary","count":[2]},
+                  {"name":"Peter","count":[3]},
+                  {"name":"Sarah","count":[1]}]|]
+                { matchHeaders = [matchContentTypeJson] }


Could have probably commented it on any of the aggregates before, too - shouldn't this return a count without array?

(Maybe that even resolves the concerns I had in the comments before about NULL and 0.)

The idea was "if an embedding only has aggregates in it, it's not considered to-many anymore, but to-one", right? So this should just return count: 0, count: 2 etc.

Done, the new tests are passing now.

wolfgangwalther · 2025-01-01T14:58:27Z

test/spec/Feature/Query/AggregateFunctionsSpec.hs

+    it "prevents the use of aggregates on to-many spread embeds" $
+      get "/factories?select=...processes(id.count())" `shouldRespondWith`
+        [json|{
+          "hint":null,
+          "details":null,
+          "code":"PGRST123",
+          "message":"Use of aggregate functions is not allowed"
+        }|]
+        { matchStatus = 400
+        , matchHeaders = [matchContentTypeJson] }


Further up we have examples of this:

get "/factories?select=name,...processes(count())&order=name"

I don't understand what the difference is, so that the case here is forbidden.

Ah, this is for when db-aggregates-enabled is not set to true. The to-one spread used to bypass this restriction before it was fixed, I added the to-many test for completion.

wolfgangwalther · 2025-01-01T15:02:07Z

Ah, I guess I missed the fact that you only changed the ordering stuff, not anything else :D

laurenceisla · 2025-01-02T15:07:57Z

Ah, I guess I missed the fact that you only changed the ordering stuff, not anything else :D

Yup 😄, the "do not wrap into arrays if there's no GROUP BY columns" is still a WIP.

…pped in an array (treated as a to-one spread)

laurenceisla · 2025-01-02T19:20:23Z

test/spec/Feature/Query/AggregateFunctionsSpec.hs

+              get "/supervisors?select=supervisor:id,...processes(processes:name,...operators(operators_count:count()))&order=id" `shouldRespondWith`
+                [json|[
+                  {"supervisor":1,"processes":["Process A1", "Process B2"],"operators_count":[2, 2]},
+                  {"supervisor":2,"processes":["Process A2", "Process B2"],"operators_count":[3, 2]},
+                  {"supervisor":3,"processes":["Process B1", "Process C1", "Process C2"],"operators_count":[1, 0, 2]},
+                  {"supervisor":4,"processes":["Process B1"],"operators_count":[1]},
+                  {"supervisor":5,"processes":[],"operators_count":[]}]|]
+                { matchHeaders = [matchContentTypeJson] }
+              get "/supervisors?select=supervisor:id,...processes(processes:name,...operators(operators_count:count()))&processes.order=name.desc&order=id" `shouldRespondWith`
+                [json|[
+                  {"supervisor":1,"processes":["Process B2", "Process A1"],"operators_count":[2, 2]},
+                  {"supervisor":2,"processes":["Process B2", "Process A2"],"operators_count":[2, 3]},
+                  {"supervisor":3,"processes":["Process C2", "Process C1", "Process B1"],"operators_count":[2, 0, 1]},
+                  {"supervisor":4,"processes":["Process B1"],"operators_count":[1]},
+                  {"supervisor":5,"processes":[],"operators_count":[]}]|]
+                { matchHeaders = [matchContentTypeJson] }


This should be expected. Processes and operators have a to-many spread. Since there's only count() in the operators side then it doesn't wrap it in an extra array. In other words, I think it's expected that operators_count is [2,2] instead of [[2],[2]]. If the processes weren't spread, then the results can be seen more clearly:

curl 'localhost:3000/supervisors?select=supervisor:id,processes(processes:name,...operators(operators_count:count()))&order=id'

[{"supervisor":1,"processes":[{"processes": "Process A1", "operators_count": 2}, {"processes": "Process B2", "operators_count": 2}]}, {"supervisor":2,"processes":[{"processes": "Process A2", "operators_count": 3}, {"processes": "Process B2", "operators_count": 2}]}, {"supervisor":3,"processes":[{"processes": "Process B1", "operators_count": 1}, {"processes": "Process C1", "operators_count": 0}, {"processes": "Process C2", "operators_count": 2}]}, {"supervisor":4,"processes":[{"processes": "Process B1", "operators_count": 1}]}, {"supervisor":5,"processes":[]}]

Also added an order test to check if it was working correctly.

laurenceisla · 2025-01-02T19:22:23Z

Ah, I guess I missed the fact that you only changed the ordering stuff, not anything else :D

Yup 😄, the "do not wrap into arrays if there's no GROUP BY columns" is still a WIP.

NVM, it should be complete now.

laurenceisla mentioned this pull request Jul 5, 2024

Spread operator for many-to-many relationships, aliases, and aggregations #3041

Open

laurenceisla force-pushed the feat-spread-m2m branch from 1a02416 to dd89c12 Compare July 5, 2024 23:58

laurenceisla commented Jul 6, 2024

View reviewed changes

src/PostgREST/Error.hs Outdated Show resolved Hide resolved

laurenceisla force-pushed the feat-spread-m2m branch from dd89c12 to dce8597 Compare July 10, 2024 01:07

laurenceisla force-pushed the feat-spread-m2m branch 2 times, most recently from 6507878 to bd93514 Compare July 26, 2024 01:50

laurenceisla commented Jul 26, 2024

View reviewed changes

src/PostgREST/Plan.hs Outdated Show resolved Hide resolved

laurenceisla mentioned this pull request Aug 20, 2024

Fixes when using aggregates inside Spread Embedded Resources #3693

Merged

3 tasks

laurenceisla force-pushed the feat-spread-m2m branch 2 times, most recently from 38abc0e to 6e64707 Compare September 7, 2024 00:49

laurenceisla force-pushed the feat-spread-m2m branch 7 times, most recently from 9002110 to b3e5483 Compare September 18, 2024 23:37

laurenceisla commented Sep 18, 2024

View reviewed changes

src/PostgREST/Query/SqlFragment.hs Show resolved Hide resolved

src/PostgREST/Query/SqlFragment.hs Outdated Show resolved Hide resolved

src/PostgREST/ApiRequest/Types.hs Outdated Show resolved Hide resolved

src/PostgREST/Plan/ReadPlan.hs Outdated Show resolved Hide resolved

laurenceisla changed the title ~~feat: WIP allow spread operators in to-many relationships~~ feat: allow spread operators in to-many relationships Sep 18, 2024

laurenceisla marked this pull request as ready for review September 18, 2024 23:40

laurenceisla force-pushed the feat-spread-m2m branch 4 times, most recently from 67e6419 to 87a13ef Compare September 25, 2024 22:14

laurenceisla force-pushed the feat-spread-m2m branch from 87a13ef to 19466a8 Compare October 3, 2024 17:33

steve-chavez reviewed Oct 3, 2024

View reviewed changes

docs/references/api/resource_embedding.rst Show resolved Hide resolved

steve-chavez approved these changes Nov 4, 2024

View reviewed changes

laurenceisla force-pushed the feat-spread-m2m branch from 7900716 to 02d8308 Compare November 5, 2024 01:10

wolfgangwalther self-requested a review November 5, 2024 07:05

wolfgangwalther reviewed Nov 5, 2024

View reviewed changes

laurenceisla force-pushed the feat-spread-m2m branch from 02d8308 to e969f91 Compare November 6, 2024 00:19

wolfgangwalther reviewed Nov 6, 2024

View reviewed changes

laurenceisla force-pushed the feat-spread-m2m branch 3 times, most recently from dca7c2d to daf47a5 Compare November 23, 2024 18:38

laurenceisla commented Nov 23, 2024

View reviewed changes

laurenceisla force-pushed the feat-spread-m2m branch from daf47a5 to c3ee0e2 Compare December 31, 2024 21:45

laurenceisla added 3 commits December 31, 2024 16:58

feat: allow spreading one-to-many and many-to-many embedded resources

a64cce1

refactor

50926c6

add ordering on spread embeds without the need of selecting them

4600a2b

laurenceisla force-pushed the feat-spread-m2m branch from c3ee0e2 to 4600a2b Compare December 31, 2024 21:58

wolfgangwalther reviewed Jan 1, 2025

View reviewed changes

when a to-many spread has only aggregates selected, then it's not wra…

1ebc7e0

…pped in an array (treated as a to-one spread)

laurenceisla commented Jan 2, 2025

View reviewed changes

add missing tests

a8ecc05

		It is expected to get ``null`` values in the resulting array.
		You can exclude them with :ref:`stripped_nulls`.

feat: allow spread operators in to-many relationships #3640

Are you sure you want to change the base?

feat: allow spread operators in to-many relationships #3640

Conversation

laurenceisla commented Jul 5, 2024 • edited Loading

laurenceisla commented Jul 6, 2024 • edited Loading

wolfgangwalther commented Jul 6, 2024

laurenceisla commented Jul 17, 2024 • edited Loading

Repeated values and order

Nested To-Many Spreads

laurenceisla commented Sep 7, 2024

laurenceisla left a comment • edited Loading

Choose a reason for hiding this comment

steve-chavez left a comment

Choose a reason for hiding this comment

jdgamble555 commented Nov 5, 2024

wolfgangwalther left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfgangwalther Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurenceisla commented Nov 5, 2024

Choose a reason for hiding this comment

laurenceisla Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

laurenceisla left a comment

Choose a reason for hiding this comment

laurenceisla Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

wolfgangwalther Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurenceisla Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurenceisla Nov 23, 2024 • edited Loading

Choose a reason for hiding this comment

wolfgangwalther commented Nov 23, 2024

laurenceisla commented Nov 23, 2024 • edited Loading

wolfgangwalther commented Dec 4, 2024

laurenceisla commented Dec 5, 2024

steve-chavez commented Dec 9, 2024

laurenceisla commented Dec 31, 2024

wolfgangwalther left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurenceisla Jan 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wolfgangwalther commented Jan 1, 2025

laurenceisla commented Jan 2, 2025

Choose a reason for hiding this comment

laurenceisla commented Jan 2, 2025

laurenceisla commented Jul 5, 2024 •

edited

Loading

laurenceisla commented Jul 6, 2024 •

edited

Loading

laurenceisla commented Jul 17, 2024 •

edited

Loading

laurenceisla left a comment •

edited

Loading

wolfgangwalther Nov 6, 2024 •

edited

Loading

laurenceisla Nov 6, 2024 •

edited

Loading

laurenceisla Nov 23, 2024 •

edited

Loading

wolfgangwalther Nov 23, 2024 •

edited

Loading

laurenceisla Nov 23, 2024 •

edited

Loading

laurenceisla Nov 23, 2024 •

edited

Loading

laurenceisla commented Nov 23, 2024 •

edited

Loading

laurenceisla Jan 2, 2025 •

edited

Loading