Optimization Ideas

[chris-ha458]

Following #55 I've tried some more things.

1. Change the tokenization NON_ALPHA.split(content) in minhash.py etc

• Blingfire . Blingfire is used for very fast tokenization and has customized c++ code. I tried to use blingfire.text_to_words() to replace the tokenization. The original Python re version was just as fast as that.
• Use re2 for a faster regex engine and dropin replace the regex compilation. re2 is generally believed to be faster than python re. Turns out, in this particular case re is much faster than re2

2. My guess is that it will be much more fruitful to write the fingerprinting part in Cython. (from More Pandas #55)

• This is on my list but it is not high on the list (If i have to write compiled code I'd rather choose rust atm).
• I also thought about replacing a chunk of the fingerprinting stage. More specifically, tokenization and ngramification (with or without hashing as well). As see above, piecemeal changes won't show significant improvements and we'd need to combine multiple steps to remove or reduce memory allocations and py/nonpy object movement. But after several previous difficulties, I'm not confident that changing only two steps would be enough, even if written in highly vectorized code or low level code. Trying to incorporate the hash as well, especially in non python code, would terribly affect overall customizability, readability and maintainability of this code base imo.

[ChenghaoMou]

I have tried simhash-cpp earlier for BigScience, and it was fast but also less approachable from the Python ecosystem. So, I would say Cython code is one step towards optimization on the optimization-vs-user-friendliness curve. Or we can wait for the no-GIL version of Python, or use Rust, which seems to have quite some popularity recently in AI communities.

[chris-ha458]

The way this repo uses sets and dicts mean it has a lot to gain from python 3.11 and 3.12 even without the no-GIL optimizations.
If and when no-GIL arrives, I am sure there will have to be a lot of adaptations to take advantage of the change.

My preference between cython and Rust is definitely Rust.
In my experience, when coupled with Python it brings the best of both worlds, easier prototyping with fast and correct code.

As for integrating Rust into a poetry without sweeping changes, is a bit of a challenge.

PyO3/pyo3#953

Some examples where this was done : https://github.com/sdispater/pendulum/blob/master/build.py

If you have experience, knowledge or opinionation regarding how we should approach this, feel free to share.

Since there is interest, I'll try to prepare some PRs.
I fully expect many if not most initial attempts get shot down, since all have different tradeoffs affecting different aspects I might not be cognizant of.
It might be useful to maintain a long standing branch that only gets merged when the benefit and cost (especially regarding maintenance) becomes clear.

Also I am hoping to implement some changes such as double-hashing to minimize object movement and calling between Rust/Python in preparation for such integration and changes.