fix(duckdb): return null typed pyarrow arrays and disable creating tables with all null columns in duckdb

[cpcloud]

Address all-NULL column handling in the DuckDB backend. null pyarrow Arrays are returned (previously int32), and it is now an error on the Ibis side to create all null columns with create_table in the DuckDB backend. Fixes #9669.

[github-actions]

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

[ncclementi]

@cpcloud The flink backend is failing, it doesn't like the null type column. I got it to xfail by adding something similar to what you added in _register_in_memory_table() for other backends, in the execute() for flink, although maybe it could go higher up in the to_pyarrow() method. Looks like for flink we don't have the _register_in_memory_table() method.

In backends/flink/__init__.py :

    def execute(self, expr: ir.Expr, **kwargs: Any) -> Any:
        """Execute an expression."""
        self._register_udfs(expr)

        table_expr = expr.as_table()

        if null_columns := table_expr.op().schema.null_fields:
            raise exc.IbisTypeError(
                f"{self.name} cannot yet reliably handle `null` typed columns; "
                f"got null typed columns: {null_columns}")

        sql = self.compile(table_expr, **kwargs)
        df = self._table_env.sql_query(sql).to_pandas()

[NickCrews]

@cpcloud im motivated to help this one across the finish line. If I fix the flink tests is this good to merge? Any thoughts regarding my incline comments?

[NickCrews]

A few thoughts, but thank you very much for this fix! I would love to help push this across the finish line.

• If you point me to what you think needs to happen with flink, I can help
• if you give a thumbs up to my suggestions, I can implement them

[cpcloud]

@NickCrews Can you give this another review pass and/or approve?

[NickCrews]

@cpcloud I tacked on two commits that I think are improvements, if those pass CI and you are happy, looks great to me to merge. I can then, in a followup PR, tackle adding in the nested null support that I describe in this comment. Thanks!

[NickCrews]

Actually, hold on, we aren't converting null-typed scalars correctly. Pushing up a fixup commit soon.

[NickCrews]

OK, with that I think I'm happy.

[cpcloud]

Moving off the 10.0 milestone.