Questions on SimHash Deduplication

[mohamedlekarim]

Hello,

Is there a threshold that controls the level of duplication aggressiveness in the SimHash implementation, similar to the one in the Mishmash approach?
I’m also wondering if we can adjust these parameters:

• NUM_BUCKET
• BIT_DIFF
• F
• NGRAM
• BATCH_SIZE
  Could you please provide an overview of these parameters and suggest optimal values for each?

[ChenghaoMou]

Thanks for the question.

You can try increase bit-diff or decrease ngram size to allow more matches, at the price of being slower and/or more false positives.

Optimal settings should be tuned based on your dataset and experiments. I would suggest using a random subset to see the effect before committing to larger runs. Here is a good resource on how it works: https://github.com/seomoz/simhash-cpp?tab=readme-ov-file#architecture