Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(polars): add bitwise operations (bit_and, bit_or, bit_xor) #10634

Merged
merged 2 commits into from
Dec 31, 2024

Conversation

IndexSeek
Copy link
Member

Description of changes

Add support for [bit_and], [bit_or], and [bit_xor] for the Polars backend. This uses bitwise_and, bitwise_or, and bitwise_xor Polars expression methods.

The is_in tests are failing still:
FAILED ibis/backends/tests/test_aggregation.py::test_reduction_ops[polars-is_in-bit_xor] - AssertionError:
FAILED ibis/backends/tests/test_aggregation.py::test_reduction_ops[polars-is_in-bit_and] - AssertionError:
FAILED ibis/backends/tests/test_aggregation.py::test_reduction_ops[polars-is_in-bit_or] - AssertionError:

I've tried to repro to see why they are failing, but I may be overlooking something. Here is how I repro'd:

In [1]: from ibis.interactive import *
In [2]: import numpy as np
In [3]: import pandas as pd
In [4]: ibis.set_backend("polars")
In [5]: alltypes = ibis.read_parquet("ci/ibis-testing-data/parquet/functional_alltypes.parquet")
In [6]: alltypes = alltypes.filter(_.id < 1550, _.string_col.isin(["1", "7"]))
In [7]: alltypes.agg(tmp=_.bigint_col.bit_or())
Out[7]: 
┏━━━━━━━┓
┃ tmp   ┃
┡━━━━━━━┩
│ int64 │
├───────┤
│    78 │
└───────┘
In [8]: np.bitwise_or.reduce(alltypes.execute().bigint_col).squeeze()
Out[8]: np.int64(78)

Here is what the test is reporting:

E           AssertionError: 
E           Not equal to tolerance rtol=0.001, atol=0
E           
E           Mismatched elements: 1 / 1 (100%)
E           Max absolute difference among violations: 48
E           Max relative difference among violations: 0.61538462
E            ACTUAL: array(126)
E            DESIRED: array(78)

ibis/backends/tests/test_aggregation.py:568: AssertionError
============================================== short test summary info ==============================================
FAILED ibis/backends/tests/test_aggregation.py::test_reduction_ops[polars-is_in-bit_or] - AssertionError: 

I'm wondering maybe if rows aren't being properly filtered or something.

Issues closed

@github-actions github-actions bot added tests Issues or PRs related to tests polars The polars backend labels Dec 30, 2024
@cpcloud
Copy link
Member

cpcloud commented Dec 31, 2024

@IndexSeek Thanks!

The issue was that you didn't thread the where argument through the rule.

I was able to solve the problem by reusing the _reductions mapping, which we use to construct reduction dispatch rules that have this bit of code which handles the filtering:

       if op.where is not None:
           arg = arg.filter(translate(op.where, **kw))

@cpcloud cpcloud added this to the 10.0 milestone Dec 31, 2024
@cpcloud cpcloud added the feature Features or general enhancements label Dec 31, 2024
@cpcloud cpcloud enabled auto-merge (squash) December 31, 2024 11:28
@cpcloud cpcloud merged commit 56a66de into ibis-project:main Dec 31, 2024
89 checks passed
@IndexSeek
Copy link
Member Author

@IndexSeek Thanks!

The issue was that you didn't thread the where argument through the rule.

I was able to solve the problem by reusing the _reductions mapping, which we use to construct reduction dispatch rules that have this bit of code which handles the filtering:

       if op.where is not None:
           arg = arg.filter(translate(op.where, **kw))

You're welcome! I suspected it had something to do with the filter, thanks for sharing that snippet. That will probably be useful in future ops.

Adding it to the _reductions dictionary is a much cleaner approach. ✨

@IndexSeek IndexSeek deleted the polars-bitwise branch December 31, 2024 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Features or general enhancements polars The polars backend tests Issues or PRs related to tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants