Performance issues

[bowspider-man]

I have a dataset of about 1 billion. My machine has a 64-core cpu and 128g memory. I found that with this configuration, I can only deduplicate 30 million data at a time. I would like to ask if there are any parameter settings that can allow me to deduplicate more data at a time while keeping my machine configuration unchanged.

[ChenghaoMou]

Which script are you using and how big is your dataset (physically)?

[bowspider-man]

Which script are you using and how big is your dataset (physically)?

I use this script: python -m text_dedup.minhash, the physical size of my dataset is 4.7T

[ChenghaoMou]

I see. Unfortunately, right now, none of the scripts were designed to be used for such a low-memory case directly.

You would need to refactor the script to process the data in partitions and in stages, and it will take longer time to finish. For example, you can read each parquet file iteratively, calculate the hashes, store the results. Then read the hashes to form the graph, maybe even use a database, then find the connected components on top of that.

[github-actions]

Stale issue message